ASCII vs Unicode

ASCII is the 1960s character set that defined 128 characters: English letters, digits, basic punctuation, and 33 control codes. Unicode is the modern character set that aims to encode every character in every living language — currently about 154,000 characters and growing.

Almost every modern system uses Unicode underneath. ASCII is the historical baseline that fits inside the first 128 Unicode code points: every ASCII file is also a valid UTF-8 file.

Where ASCII ends up

ASCII defined the byte values 0-127 for printable characters. The values 128-255 were later used for various “extended ASCII” encodings (Latin-1, Windows-1252, Shift-JIS, etc.) — each adding accents, currency symbols, or CJK characters in their own way. These caused the classic “mojibake” problem: a file written on one system, opened on another, and displayed as é instead of é.

The web and modern file formats (JSON, XML, YAML, Markdown, modern TXT) all agreed on UTF-8 as the default and largely removed the problem. Old encodings still appear in legacy data: many .csv exports from old Excel installations still use Windows-1252.

UTF-8 vs UTF-16 vs UTF-32

Unicode has several serialization formats:

• UTF-8 — variable width (1-4 bytes per code point), ASCII-compatible (the first 128 are identical bytes to ASCII). The default for the web, all major operating systems, and most file formats.
• UTF-16 — variable width (2 or 4 bytes). Used internally by Windows, Java, and JavaScript strings (historically; modern JS engines may also use UTF-8 internally).
• UTF-32 — fixed width (always 4 bytes per code point). Simple but wasteful for ASCII-heavy data; used internally by some systems that need fast random access to code points.

When a system says “Unicode” without specifying, modern contexts almost always mean UTF-8.

What this means for regex

The regex engine in your code interprets byte sequences based on the locale or the encoding flag. The dot . in regex matches “any character except newline” in ASCII mode but matches “any code point” in Unicode mode. Character classes like \w (word characters) include many more letters in Unicode mode — A-Z and a-z, but also é, ñ, Ω, any CJK ideograph, and many others.

Most modern regex engines default to Unicode mode. If you want ASCII-only behavior, you need explicit flags (like [A-Za-z] instead of \w).

A practical example: emoji

The “😀” character (U+1F600, “grinning face”) is one Unicode code point that takes 4 bytes in UTF-8 (F0 9F 98 80). It’s a single “character” by Unicode’s definition but 4 bytes by UTF-8’s. Functions that operate on bytes will see 4 things; functions that operate on characters will see 1.

This is why “length” in JavaScript counts U+1F600 as 2 (it accounts for UTF-16 surrogate pairs). string.length in JS operates on UTF-16 code units, not code points.

Tooling

For inspecting the bytes and code points of any pasted string — including invisible characters and Unicode escapes — see the Unicode Inspector. It runs in your browser and shows you the breakdown without transmitting anything.

For text that mixes ASCII and Unicode (most real documents), the Case Converter tool handles capitalization rules across scripts.

See also

• Regex Cheat Sheet — when Unicode-aware regex is the actual question
• PDF to Text — for extracting text from PDFs that have mixed encodings
• JSON to CSV — when character encoding decides whether your CSV opens correctly