What is UTF-16 and how is it different from UTF-8?

In one line

The rule

UTF-8 and UTF-16 are character encoding schemes that represent text as sequences of bytes, but use different methods for encoding characters, especially those outside the basic English alphabet.

• UTF-8: Uses 1 to 4 bytes for each character.
• UTF-16: Uses 2 or 4 bytes for each character.

Let's understand it simply

Imagine you want to send a message in Hindi, English, or even emojis on WhatsApp. Your phone needs to turn every letter, symbol, or emoji into numbers, because computers only understand numbers. This turning process is called encoding.

Think of UTF-8 and UTF-16 as two different ways of packing these letters into boxes (bytes). UTF-8 uses small boxes for English letters and bigger boxes for other languages or emojis. UTF-16 mostly uses medium-sized boxes (2 bytes), but sometimes needs a bigger box (4 bytes) for rare symbols.

How it works

1. UTF-8
2. English letters (A-Z, a-z, 0-9) use 1 byte (8 bits).
3. Letters from other languages (like Hindi, Tamil, Chinese) use 2, 3, or 4 bytes.
4. Emojis and rare symbols use 4 bytes.
5. Each character can be a different size, so the text is flexible and saves space for English.
6. UTF-16
7. Most common characters (including many from Indian scripts) use 2 bytes (16 bits).
8. Some rare symbols and emojis use 4 bytes (two 2-byte units called "surrogates").
9. All characters are at least 2 bytes, so even English letters take more space than in UTF-8.
10. It is faster for some languages but uses more space for English.

The story behind it

In the early days of computers, only English letters were needed, so one byte (8 bits) was enough. But as computers spread worldwide, people wanted to write in Hindi, Chinese, Arabic, and use emojis. The old system, called ASCII, could not handle these. Unicode was created to include every language and symbol. UTF-8 (made by Ken Thompson and Rob Pike in 1992) and UTF-16 (from the Unicode standard in the 1990s) are two ways to store this huge set of characters. UTF-8 became popular on the web because it saves space for English, while UTF-16 is common in Windows and Java.

Where you see it in real life

• Websites: Most websites use UTF-8, so you can read Hindi news, Tamil blogs, or see emojis.
• Windows computers: Many Windows programs use UTF-16, so files like Word documents can store many languages easily.
• Mobile apps: Android and iOS use UTF-8 for messages, but sometimes use UTF-16 inside apps.
• Programming: Java and .NET use UTF-16 for their strings, while Python and JavaScript use UTF-8.
• Databases: MySQL and other databases often store text in UTF-8, so names from any language are saved correctly.
• Text files: Notepad can save files in UTF-8 or UTF-16; you might see the option when saving.

Why it matters

• Universal communication: Lets you send messages in any language, including Indian scripts and emojis.
• Saves space: UTF-8 saves storage for English-heavy texts, making websites faster to load.
• Compatibility: UTF-8 is backward compatible with ASCII, so old programs can still read basic English text.
• Software development: Knowing the difference helps programmers avoid bugs when handling text in different languages.

Worked example

Let's encode the Hindi letter 'अ' (U+0905) in both UTF-8 and UTF-16:

Given: Unicode code point for 'अ' = U+0905
Step 1: UTF-8 encoding
  - U+0905 in binary: 0000 1001 0000 0101
  - UTF-8 for U+0800 to U+FFFF: 1110xxxx 10xxxxxx 10xxxxxx
  - Fill in bits: 1110 0000 1010 0000 1000 0101
  - Hex: E0 A4 85
Step 2: UTF-16 encoding
  - U+0905 fits in 2 bytes: 09 05
  - Hex: 09 05

So, 'अ' uses 3 bytes in UTF-8, but only 2 bytes in UTF-16.

Common mistakes

• Thinking UTF-8 and UTF-16 are languages: They are not languages, just ways to store text.
• Assuming UTF-16 is always better for Indian languages: Sometimes it is, but UTF-8 is more common and flexible.
• Forgetting about file size: UTF-16 files are often bigger for English text, but not always for other languages.
• Mixing up bytes and characters: One character can take more than one byte, especially in UTF-8.

Remember it

UTF-8 is like a stretchy bag—small for English, bigger for other scripts. UTF-16 is like a suitcase—always medium or big, never tiny.