Singapore H2 Computing Wiki

Character Encoding: ASCII and Unicode

4 min read

Character Encoding: ASCII and Unicode

You can read this note directly if you understand that computers store bit patterns. Character encoding answers this question: how can a computer store text such as letters, digits, punctuation, and symbols using numbers?

Characters Are Not Drawings

A character is the abstract symbol, such as A.

A font controls how that character is drawn on screen or paper.

An encoding controls how that character is stored as data.

For example, A can appear in many fonts, but it still has the same Unicode code point U+0041.

The Basic Idea

A character encoding maps characters to numeric codes.

Caption: A character has a Unicode code point, and an encoding such as UTF-8 stores that code point as one or more bytes before decoding it back to text.

For example, the character A has Unicode code point U+0041, which is denary 65.

In Python:

print(ord("A"))
print(chr(65))

Output:

65
A

ord() gives the numeric code point for a one-character string. chr() converts a code point back to a character.

ASCII

ASCII is an older character encoding standard. It represents a limited set of characters, including:

  • uppercase and lowercase English letters;
  • digits;
  • common punctuation;
  • control characters such as newline.

Common examples:

CharacterDenary code
A65
B66
a97
048
space32

ASCII is useful historically and remains important because many common characters have the same code point in Unicode.

Limitation: ASCII cannot represent most characters used by languages outside basic English, nor many symbols and emoji.

Beginner checkpoint: the character "7" is not the same as the integer 7. The character "7" has ASCII/Unicode code point 55. The integer 7 is a numeric value used in arithmetic.

Unicode

Unicode is a much larger standard. It assigns code points to characters from many writing systems and symbol sets.

Examples:

CharacterUnicode code pointPython value from ord()
AU+004165
éU+00E9233
你U+4F6020320

Unicode solves the problem that different older encodings could assign different meanings to the same byte value.

Encoding Versus Decoding

Encoding turns text into bytes.

Decoding turns bytes back into text.

In Python, bytes are displayed with a leading b, such as b'Hi'. That leading b means the value is bytes, not an ordinary text string.

Example:

message = "Hi"
data = message.encode("utf-8")
print(data)
print(data.decode("utf-8"))

Output:

b'Hi'
Hi

UTF-8 is a common encoding for Unicode text. It uses one byte for many basic ASCII characters, and more bytes for many other characters.

Example:

print("A".encode("utf-8"))
print("你".encode("utf-8"))

The exact byte output is different because 你 needs more bytes in UTF-8 than A.

For a beginner, the key lesson is not to memorise every byte pattern. The key lesson is that the same text must be encoded before storage and decoded with the matching rule when read back.

Why the Encoding Must Match

If text is encoded using one encoding but decoded using the wrong encoding, the result may be wrong or the program may raise an error.

Practical rule:

with open("message.txt", "w", encoding="utf-8") as file:
    file.write("Hi")
 
with open("message.txt", "r", encoding="utf-8") as file:
    text = file.read()

Specifying the encoding makes the program’s assumption explicit.

Beginner Model

Think in layers:

LayerExample
characterA
code pointU+0041
denary value65
byte pattern in UTF-801000001

For many English characters, ASCII and UTF-8 look the same at byte level. That does not mean ASCII and Unicode are the same system. Unicode covers far more characters.

Common Mistakes

  • Saying ASCII and Unicode are fonts.
  • Thinking every character uses exactly one byte.
  • Confusing a character such as "7" with the integer 7.
  • Forgetting that the same bytes must be decoded with the correct encoding.
  • Assuming ASCII can represent all text.

Check Your Understanding

  1. What does ord("A") return?
  2. What does chr(65) return?
  3. Why is Unicode needed if ASCII already exists?
  4. What is the difference between encoding and decoding?

Answers:

  1. 65.
  2. "A".
  3. ASCII represents only a limited set of characters; Unicode represents characters from many writing systems and symbol sets.
  4. Encoding turns characters into bytes; decoding turns bytes back into characters.

Graph View

Backlinks