What are UTF-16 and UTF-32, and how are they different from UTF-8?

In one line

The rule

Character encoding is a method to represent letters, symbols, and emojis as numbers so computers can store and share them.

• UTF-8: Uses 1 to 4 bytes for each character.
• UTF-16: Uses 2 or 4 bytes for each character.
• UTF-32: Uses 4 bytes for every character.

A byte is 8 bits — enough to store one English letter in simple encodings.

Let's understand it simply

Imagine you want to write a letter to a friend, but your friend speaks a different language. You both agree on a secret code for every letter and symbol. Computers do the same. They turn every letter, number, or emoji into numbers using a code called encoding.

Think of UTF-8, UTF-16, and UTF-32 as different notebooks. Some notebooks use small pages (UTF-8), some use medium (UTF-16), and some use big pages (UTF-32). If you only write short words (like English), the small notebook saves paper. But if you write long words or use many symbols (like Hindi, Chinese, emojis), you may need bigger pages.

How it works

1. UTF-8

• Each character is stored using 1 to 4 bytes.
• English letters use 1 byte. Hindi, Chinese, emojis use more (up to 4 bytes).
• It is "backward compatible" with old ASCII (the basic English code).

1. UTF-16

• Each character uses 2 bytes for most common symbols.
• Rare symbols (like some emojis) use 4 bytes.
• It is good for Asian languages, which have many symbols.

1. UTF-32

• Every character always uses 4 bytes, even for simple English letters.
• It is simple but uses more memory.

1. Reading and writing

• The computer reads the bytes and uses the encoding rule to turn them back into letters and symbols.

1. Why different sizes?

• To save space (memory) and make sharing text between computers easier.

The story behind it

Long ago, computers only used English, so they made a simple code called ASCII (7 bits, 128 symbols). But as computers spread, people wanted to write in Hindi, Tamil, Chinese, Arabic, and use emojis. ASCII was too small. So, a group called Unicode Consortium started in the late 1980s to make a big code that can fit every symbol in every language. They created Unicode, and UTF-8, UTF-16, and UTF-32 are ways to store Unicode symbols in computer memory.

One fun fact: UTF-8 was designed by Ken Thompson and Rob Pike in 1992. They wanted a system that worked well for English and also for all other languages.

Where you see it in real life

• WhatsApp messages: When you send messages in English, Hindi, or with emojis, the app uses UTF-8 or UTF-16 to store and send your text.
• Websites: Most websites use UTF-8, so you can read content in any language.
• Windows files: Some Windows programs use UTF-16 to store text, especially for Asian languages.
• Programming: When you write code in Python or Java, strings (text) are stored in one of these formats.
• Mobile phones: Your phone uses these encodings to show your contacts' names, SMS, and app texts.
• Databases: When banks store your name in their system, they use UTF-8 or UTF-16 to handle all languages.

Why it matters

• Universal communication: Lets people write and read in any language on any device.
• Saves memory: UTF-8 saves space for English, which is important for fast websites and apps.
• Handles emojis: Modern chat apps need to store and show emojis, which need more bytes.
• Programming: Knowing the difference helps you avoid bugs when handling text in different languages.
• Exams and jobs: Questions about encoding come in computer science exams and interviews.

Worked example

Let's see how the Hindi letter 'क' (ka) is stored in each encoding:

Given: Character = 'क' (Unicode code point U+0915)
Step 1: Find its code point in hexadecimal: 0915
Step 2: UTF-8 encoding: 3 bytes (E0 A4 95)
Step 3: UTF-16 encoding: 2 bytes (09 15)
Step 4: UTF-32 encoding: 4 bytes (00 00 09 15)

So, 'क' uses 3 bytes in UTF-8, 2 bytes in UTF-16, and 4 bytes in UTF-32.

Common mistakes

• Thinking all encodings use the same size: They don't. UTF-8 is variable (1–4 bytes), UTF-16 is mostly 2 bytes, UTF-32 is always 4 bytes.
• Assuming UTF-8 is only for English: UTF-8 can store any language, but uses more bytes for non-English.
• Mixing up encoding and font: Encoding is how text is stored; font is how it looks on screen.
• Not handling emojis: Emojis often need 4 bytes in UTF-8 and UTF-16, so some programs break if not careful.

Remember it

"UTF-8 is small and flexible, UTF-16 is good for Asian scripts, UTF-32 is big but simple."