Character Encoding and Unicode

Why it matters

A product used in Saudi Arabia or India depends on its text handling: a customer name saved as علي, an invoice PDF with question marks in the address, a search that cannot find a user because her name was typed with a different accent form, a 30-character limit that rejects a 30-letter Arabic name because it was measured in bytes. Each is a defect the user can meet on the first day, often in the field that carries their own name.

For the tester, encoding defects are cheap to find and expensive to miss. They hide behind English test data, they appear only at a seam between systems (browser to API, API to database, database to export), and once garbled text is saved back it is stored as if it were correct, so every later reader repeats the damage. A small dataset and a clear picture of where bytes become characters finds them before release.

Code points, UTF-8 and the other encodings

Unicode defines code points from U+0000 to U+10FFFF, and the text an application handles is a sequence of them. An encoding maps each code point to bytes, which are written in hexadecimal, two digits per byte from 00 to FF. UTF-8, defined in RFC 3629, uses one byte for the 128 ASCII characters, so a file that contains only ASCII characters is already valid UTF-8; two bytes for most accented Latin letters and for Greek, Cyrillic, Hebrew and Arabic; three for Devanagari, Chinese, Japanese and most other scripts in everyday use; and four for most emoji and everything else above U+FFFF.

UTF-16 stores most characters in one 16-bit unit and characters above U+FFFF in two units called a surrogate pair. JavaScript strings, Java and the Windows APIs count in these units, which is why '😀'.length is 2. Older single-byte encodings, such as Windows-1252 for Western European languages and Windows-1256 or ISO 8859-6 for Arabic, hold at most 256 characters each, so they cannot hold, for example, Arabic and Devanagari in the same text; they survive in legacy databases, CSV exports and e-mail systems, and they are where most mojibake starts.

String:  A ع न 😀                     4 code points

Char   Code point   UTF-8 bytes      UTF-16 units
A      U+0041       41               1
ع      U+0639       D8 B9            1
न      U+0928       E0 A4 A8         1
😀     U+1F600      F0 9F 98 80      2 (surrogate pair)

Total: 4 characters, 10 UTF-8 bytes, 5 UTF-16 units

Try it: Hex Encoder Try it: Unicode Encoder

Mojibake: right bytes, wrong decoder

Mojibake is the garbled text that appears when bytes written in one encoding are read as another. The bytes are usually intact; the reader guessed wrong. café stored as UTF-8 is 63 61 66 C3 A9; a component that reads those bytes as Windows-1252 shows café, because in that table C3 is Ã and A9 is ©. The damage spreads when the garbled text is saved back: café is then re-encoded as valid UTF-8, every later reader shows it faithfully, and the original can be recovered only by reversing the wrong conversion record by record, which does not always succeed.

Every layer declares or assumes an encoding: the Content-Type: text/html; charset=utf-8 header and <meta charset="utf-8"> in HTML; JSON, which RFC 8259 requires to be UTF-8 when it is exchanged outside a closed ecosystem, and whose application/json media type defines no charset parameter; URLs, where the UTF-8 bytes are percent-encoded (ع travels as %D8%B9); database columns and connections; CSV files with or without a byte order mark (EF BB BF); e-mail templates and log files. Mojibake lives at the seam where two of them disagree.

The shape of the garbage tells you which family of defect you are looking at:

• é and € in Latin text, Arabic as runs of Ø and Ù (سلام becomes سلام), Devanagari as runs of à¤ and à¥: UTF-8 bytes read as a Western single-byte encoding.
• ? in place of letters: the text was converted into an encoding that has no slot for those characters, and they are lost for good.
• �, the replacement character U+FFFD: the decoder met a byte sequence that is invalid in UTF-8, often Windows-1256 text read as UTF-8 or a string cut in the middle of a multi-byte character.
• Empty boxes, or a vowel sign drawn on a dotted circle: the bytes decode correctly, but the font has no glyph for the character, or the shaping engine (the part of the renderer that joins letters and places marks on them) met a mark with no letter before it, for example after the text was cut between the two. Look at rendering and truncation, not at the encoding.

Try it: Hex Encoder Try it: Text Diff

Normalisation and counting characters

The same visible text can be stored as different code points. é is one code point (U+00E9) or two (e followed by the combining acute accent U+0301); an Arabic letter with a short-vowel mark such as fatha is always two, and أ is one (U+0623) or two (ا plus hamza above U+0654). Unicode normalisation, defined in UAX #15, converts text to one agreed form: NFC composes such sequences and NFD decomposes them. The compatibility forms NFKC and NFKD go further and also fold the ligature fi to fi, the full-width Ａ to A and the Arabic presentation form ﻻ to ل plus ا; that is right for search and wrong for storing a name as typed. Normalisation does not convert one digit system into another: the Arabic-Indic ٣ stays ٣ even under NFKC, a locale matter for the next lesson.

Unless the product normalises at its boundaries, two users can register José twice, a search for a name typed on one keyboard misses the same name typed on another, and a password set in one form fails in the other. Ask which form the product uses and where it applies it; W3C internationalisation guidance recommends NFC for text on the web.

Length has four answers. A user counts grapheme clusters, the user-perceived characters defined in UAX #29; a database column may count bytes; JavaScript's length counts UTF-16 units; Python counts code points, while Go and Rust count bytes. The four agree only for plain ASCII:

String        Graphemes  Code points  UTF-16 units  UTF-8 bytes
Ali               3           3            3             3
علي               3           3            3             6
مُحَمَّد              4           8            8            16
José (NFC)        4           4            4             5
José (NFD)        4           5            5             6
हिंदी              2           5            5            15
👍🏽                1           2            4             8

Try it: String Length Counter Try it: Unicode Dataset Generator

Worked example: testing a name field end to end

Requirement: "Display name, 2 to 30 characters, shown on the profile, in e-mails and in the CSV export of the user list." Encoding is tested as a round trip: enter a fixed dataset, save, reload, read it back through the API, export it, and compare every stop with the original, by bytes rather than by eye.

• Build the dataset once and keep it: Latin with accents, Arabic with and without diacritics, Devanagari with vowel signs, an emoji with a skin tone, and the same name in NFC and NFD. The Unicode dataset generator produces strings of each kind with their counts, and the string length counter gives the four counts for any string you add yourself.
• Enter each string, save, reload the profile and compare the displayed text with the input in a diff tool; then fetch the same user from the API and compare the hex bytes of the JSON value with the hex of the original.
• Probe the limit: علي repeated ten times is 30 letters and 60 bytes. If it is rejected while 30 Latin letters are accepted, the limit counts bytes; if 👍🏽 counts as 4, it counts UTF-16 units. Record which, and ask whether the requirement meant user-perceived characters.
• Search for the NFD José after saving the NFC one, and the reverse; then open the CSV export in a spreadsheet and in a plain text editor and look for Ã, Ø, ? and �.
• Trigger one e-mail and one PDF that print the name, and read the server log, where Arabic often appears as \u0639 escapes: correct, as long as the log viewer decodes them, and the Unicode decoder turns them back into text.

POST /api/v1/profile HTTP/1.1
Host: api.example.com
Content-Type: application/json

{"displayName":"علي"}

GET /api/v1/profile HTTP/1.1
Host: api.example.com

HTTP/1.1 200 OK
Content-Type: application/json

{"displayName":"علي"}         OK: value bytes D8 B9 D9 84 D9 8A
{"displayName":"علي"}      mojibake: read once as Windows-1252, saved again
{"displayName":"???"}         lost: converted to an encoding without Arabic

Try it: Unicode Dataset Generator Try it: String Length Counter Try it: Text Diff Try it: Unicode Decoder

Frequently asked questions

References

• The Unicode StandardUnicode ConsortiumThe universal character set and its encoding forms (UTF-8, UTF-16, UTF-32), normalization and the properties of every code point.
• RFC 3629: UTF-8, a transformation format of ISO 10646IETF · 2003How code points are encoded as one to four bytes in UTF-8, and which byte sequences are invalid.
• UAX #15: Unicode Normalization FormsUnicode ConsortiumThe NFC, NFD, NFKC and NFKD forms — the reason two visually identical strings can compare unequal until normalised.
• UAX #29: Unicode Text SegmentationUnicode ConsortiumHow text is divided into grapheme clusters, words and sentences — why a "character" count can differ from a code-point or byte count.