Unicode Characters in text sent via Dispatch API

[The Glorious Hypetrain]

Hi Folks,

I'm trying to send some special characters in the text of a dispatch request to the dispatch API, namely ★ and ➢.

I've tried a few different methods of python encoding, including .encode('utf-8') and urlencode. The former returned ????? as seen in https://www.nationstates.net/page=dispatch/id=1638607, and urlencode returned a literal string of encoded text as seen in https://www.nationstates.net/page=dispatch/id=1638611 .


Is there a right encoding/method to send unicode characters and have them represented correctly in the text?

[PhilTech]

Sorry if I will misunderstand your query. Based on my understanding, it seems that you simply want to paste the unicode symbols in your dispatch? If so, you simply need to copy and paste the unicode characters in your dispatch. I've tried it here and works just fine. This type of formatting code is supported by NS.

I also included a bunch of them in my dispatches/factbooks myself.

[The Glorious Hypetrain]

Sorry if I will misunderstand your query. Based on my understanding, it seems that you simply want to paste the unicode symbols in your dispatch? If so, you simply need to copy and paste the unicode characters in your dispatch. I've tried it here and works just fine. This type of formatting code is supported by NS.

I also included a bunch of them in my dispatches/factbooks myself.

Hi PhilTech - I'm not trying to paste them into my dispatch, rather have them sent along an API request and display correctly. I'm making a dispatch via the NS api, and in the request I have unicode symbols in the text. My code looks like this (using the dispatch api example here):

import requests

headers = {
'X-Password': 'hunter2',
'User-Agent': 'UserAgent Example',
}

data = {
'nation': 'testlandia',
'c': 'dispatch',
'dispatch': 'add',
'title': 'Test Dispatch',
'text': '★➢➢➢➢★',
'category': '3',
'subcategory': '305',
'mode': 'prepare'
}

response = requests.post('https://www.nationstates.net/cgi-bin/api.cgi', headers=headers, data=data)

This is then followed up with the correct execute command and x-pin header, but I've omitted that here for brevity.

I think something is off with how I'm sending the request, because any combination of symbols results in "?" being displayed once the request is received.

[Trotterdam]

Where's the manual for the library you're using? I suspect it's an issue with that, rather than NationStates.

HTTP requests officially use Latin-1 by default, not UTF-8 unless you explicitly specify that. So I suspect that what your library does is look at the UTF-8 text it's asked to transmit, try to translate it to Latin-1, realize it can't, and so replace it with question marks.

[The Glorious Hypetrain]

I wasn't able to get the encoding change to place nice with the API (different forms of encoding returned 400 bad request, error on my end but wasn't able to deduce where).

I ended up doing a find and replace with the correct html escape sequence, and that was accepted by the API and renders correctly (see https://www.nationstates.net/page=dispatch/id=1638821 for details). I used https://unicode-table.com/ to find the escapes and just replaced them in my application for the time being. Got the idea from this thread viewtopic.php?f=15&t=483646

[Trotterdam]

html escape sequence

...That sounds like it really shouldn't work. API queries are not in HTML or XML, even if the replies are. So if it works, that's a bug, even if it's one that happens to be convenient to you right now.

[The Glorious Hypetrain]

I don't think it's a bug per-se. The text I'm sending isn't be changed by the api and is rendering correctly. If I wanted to represent U+27A2, I just can substitute with ➢ and then it's rendered correctly. The original unicode character was lost on submission (I think there might have been a utf-8 to latin-1 conversion at some point, but even when I specified the charset it wasn't accepting unicode characters)

[Frisbeeteria]

The site has been modified many times since its introduction in 2002, but has not always provided consistent character sets. What works in one place may fail utterly in a different place.

Establishing and maintaining a modern and consistent character set is very much on the To Do list, but it's not trivial to fix. There are other tasks with a higher priority, so don't hold your breath for a quick solution.