What is UTF-8 and how is it related to Unicode?

In one line

The rule

UTF-8 is a variable-length character encoding for Unicode, using 1 to 4 bytes for each character.

There is no formula, but the key idea is:

• Each Unicode character gets a unique code number (code point).
• UTF-8 turns that code point into a set of 1–4 bytes for storage or transmission.

Let's understand it simply

Imagine you want to write your name on a cricket bat, but you want people from every country to read it. If you only use English letters, some people can't read it. Unicode is like a giant dictionary that gives every letter, symbol, and emoji from every language its own number. But computers store numbers as bytes (like small boxes that can hold up to 256 things). UTF-8 is a clever way to pack these Unicode numbers into these boxes. Some letters fit in one box, some need two, three, or even four. This way, all languages can be written and read on any computer.

How it works

1. Unicode assigns a code point: Every character (like 'A', 'अ', '你', '😊') gets a unique number, called a code point (like U+0041 for 'A').
2. UTF-8 encodes the code point: It checks the number. If it's small (like English letters), it uses 1 byte. If it's bigger (like Hindi or Chinese), it uses 2, 3, or 4 bytes.
3. Each byte has a special pattern: The first bits of each byte show if it's the start or part of a character.
4. No confusion: A computer reading a UTF-8 file can always tell where one character ends and the next begins.
5. Backwards compatible: English text in UTF-8 looks the same as old ASCII files, so old programs can still read it.

The story behind it

Unicode was created in the late 1980s and early 1990s by a group of engineers from different countries. Their problem: computers in Japan, India, Europe, and America all used different codes for letters. This made sharing text (like emails or websites) very hard. UTF-8 was invented by Ken Thompson and Rob Pike in 1992 as a way to store any Unicode text using as little space as possible, and to work with old English-only systems.

Where you see it in real life

• WhatsApp messages: You can send texts in Hindi, Tamil, English, or even emojis. UTF-8 makes sure your friend sees the same thing.
• Websites: Most websites use UTF-8, so you can read news in any language on the same browser.
• Programming: When you write code in Python or Java, UTF-8 lets you use names and comments in your own language.
• File names: You can save files as 'दिवाली.jpg' or 'résumé.pdf' because your computer uses UTF-8.
• SMS and Emails: Messages in different scripts (like Bengali or Arabic) are possible because of Unicode and UTF-8.
• Bank ATMs: Menus in multiple languages use UTF-8 to show the right script.

Why it matters

• Universal communication: People from any country can write and read in their own language on the same device.
• No data loss: Old systems (ASCII) would break or show '???' for unknown letters. UTF-8 avoids this.
• Internet standard: Most websites, apps, and devices use UTF-8, so your data is safe everywhere.
• Programming and jobs: Understanding UTF-8 is important for web development, app making, and data science.

Worked example

Let's encode the letter 'A' and the Hindi letter 'अ' in UTF-8.

Given: 'A' (Unicode U+0041), 'अ' (Unicode U+0905)
Step 1: Find the Unicode code point
        'A' = U+0041 (decimal 65)
        'अ' = U+0905 (decimal 2309)
Step 2: Check the size
        65 fits in 1 byte (0–127)
        2309 needs 3 bytes (128–2047: 2 bytes, 2048–65535: 3 bytes)
Step 3: Encode
        'A' in UTF-8: 01000001 (1 byte: 0x41)
        'अ' in UTF-8: 0xE0 0xA4 0x85 (3 bytes)
Step 4: Result
        'A' uses 1 byte, 'अ' uses 3 bytes in UTF-8

Common mistakes

• Thinking Unicode and UTF-8 are the same: Unicode is the list of all characters; UTF-8 is one way to store them.
• Assuming all characters use 1 byte in UTF-8: Only English letters do. Other languages and emojis use more.
• Believing UTF-8 can't store some scripts: UTF-8 can store every Unicode character, from any language.
• Mixing up UTF-8 and UTF-16: UTF-16 is another way to store Unicode, but it uses 2 or 4 bytes, not 1–4.

Remember it

Unicode is the world's alphabet; UTF-8 is the smart box that packs it for computers.