Take the 2-minute tour ×

I am asking for the count of all the possible valid combinations in Unicode with explanation. I know a char can be encoded as 1,2,3 or 4 bytes. I also don't understand why continuation bytes have restrictions even though starting byte of that char clears how long it should be.

share|improve this question

5 Answers 5

up vote 12 down vote accepted


1,111,998: 17 planes × 65,536 characters per plane - 2048 surrogates - 66 noncharacters

Note that UTF-8 and UTF-32 could theoretically encode much more than 17 planes, but the range is restricted based on the limitations of the UTF-16 encoding.

109,384 code points are actually assigned in Unicode 6.0.


The purpose of this restriction in UTF-8 is to make the encoding self-synchronizing.

For a counterexample, consider the Chinese GB18030 encoding. There, the letter ß is represented as the byte sequence 81 30 89 38, which contains the encoding of the digits 0 and 8. So if you have a string-searching function not designed for this encoding-specific quirk, then a search for the digit 8 will find a false positive within the letter ß.

In UTF-8, this cannot happen, because the non-overlap between lead bytes and trail bytes guarantees that the encoding of a shorter character can never occur within the encoding of a longer character.

share|improve this answer
The "self-synchronizing" article you linked doesn't explain what's self-synchronizing at all –  Pacerier Feb 1 '12 at 9:24

Unicode allows for 17 planes, each of 65,536 possible characters (or 'code points'). This gives a total of 1,114,112 possible characters. At present, only about 10% of this space has been allocated.

The precise details of how these code points are encoded differ with the encoding, but your question makes it sound like you are thinking of UTF-8. The reason for restrictions on the continuation bytes are presumably so it is easy to find the beginning of the next character (as continuation characters are always of the form 10xxxxxx, but the starting byte can never be of this form).

share|improve this answer
According to these "planes" even the last three byte of a 4 byte char could express 64 of them. Am I wrong? –  Ufuk Hacıoğulları May 7 '11 at 22:08
Yes, that is for synchronization, see cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt –  ninjalj May 7 '11 at 22:09
That's outdated I think. It doesn't use 6 bytes anymore –  Ufuk Hacıoğulları May 7 '11 at 22:26
@Andy: That makes sense: the original spec for UTF-8 worked for bigger numbers. The 21-bit limit was a sop to the folks who had locked themselves into 16-bit characters, and thus did UCS-2 beget the abomination known as UTF-16. –  tchrist May 7 '11 at 22:42
@Simon: There are 34 noncharacter code points, anything that when bitwise-addded with 0xFFFE == 0xFFFE, so two such code points per plane. Also, there are 31 noncharacter code points in the range 0x00_FDD0 .. 0x00_FDEF. Plus you should subtract from that the surrogates, which are not legal for open interchange due to the UTF-16 flaw, but must be supported inside your program. –  tchrist May 7 '11 at 22:47

Unicode is not a encoding, is a character mapping. According to Wikipedia Unicode currently contains 109242 distinct characters.

share|improve this answer
That's not true –  Ufuk Hacıoğulları May 7 '11 at 21:36
I'm asking how many it can support in principle –  Ufuk Hacıoğulları May 7 '11 at 22:06
@Ufuk Hacıoğulları: Then accept Simon's answer. –  orlp May 7 '11 at 22:07
@Ufuk: What isn’t true? Unicode is not an encoding form. –  tchrist May 7 '11 at 22:43
Unicode is a character encoding standard. First answer from unicode.org/faq/basic_q.html: “Unicode is the universal character encoding,” so saying that “Unicode is not an encoding” is wrong. (I once made that mistake myself.) –  Philipp May 8 '11 at 9:36

Unicode supports 1,112,114 code points. There are 2048 surrogate code point, giving 1,110,066 scalar values. Of these, there are 66 noncharacters, leading to 1,110,000 possible encoded characters (unless I made a calculation error).

share|improve this answer
Can you look at my answer? Why is there 1,112,114 code points? –  Ufuk Hacıoğulları May 8 '11 at 17:06
This number comes from the number of planes that is addressable using the UTF-16 surrogate system. You have 1024 low surrogates and 1024 high surrogates, giving 1024² non-BMP code points. This plus the 65,536 BMP code points gives exactly 1,114,112. –  Philipp May 8 '11 at 18:40

To give a metaphorically accurate answer, all of them.

Continuation bytes in the UTF-8 encodings allow for resynchronization of the encoded octet stream in the face of "line noise". The encoder, merely need scan forward for a byte that does not have a value between 0x80 and 0xBF to know that the next byte is the start of a new character point.

In theory, the encodings used today allow for expression of characters whose Unicode character number is up to 31 bits in length. In practice, this encoding is actually implemented on services like Twitter, where the maximal length tweet can encode up to 4,340 bits' worth of data. (140 characters [valid and invalid], times 31 bits each.)

share|improve this answer
Actaully, in theory it is not limited to 31 bits, you can go bigger on a 64 bit machine. perl -le 'print ord "\x{1FFF_FFFF_FFFF}"' prints out 35184372088831 on a 64-bit machine, but gives integer overflow on a 32-bit machine. You can use bigger chars like that inside your perl program, but if you try to print them out as utf8, you get a mandatory warning unless you disable such: perl -le 'print "\x{1FFF_FFFF}"' Code point 0x1FFFFFFF is not Unicode, may not be portable at -e line 1. ######. There is a difference between "loose utf8" and "strict UTF-8": the former is not restricted. –  tchrist May 7 '11 at 22:37
The encodings used today don't allow for 31-bit scalar values. UTF-32 would allow for 32-bit values, UTF-8 for even more, but UTF-16 (used internally by Windows, OS X, Java, .NET, Python, and therefore the most popular encoding scheme) allows for just over one million (which should still be enough). –  Philipp May 8 '11 at 9:43
"All of them" isn't quite accurate; there are characters in legacy encodings that aren't in Unicode. For example, the Apple logo in MacRoman, and a couple of the graphics characters in ATASCII. OTOH, there's a Private Use Area, so these characters can be mapped with Unicode; they're just not part of the standard. –  dan04 May 8 '11 at 14:44
@tchrist: Python 3 does use UTF-16; for example, on my system I can say len(chr(0x10000)), giving 2 (code units). OS X's kernel uses UTF-8, correct—but the high-level APIs (Cocoa etc.) use UTF-16. –  Philipp May 8 '11 at 18:44
@Philip: I only use Python 2, whose Unicode support leaves a lot to be desired. I’m a systems guy, so I don’t do end-user chrome-platting: all the syscalls I use on OS X take UTF-8, which the kernel converts into NFC for you. My UTF-16 experiences in Java have been bad: try a regex bracketed charclass match with literal some non-BMP codepoints in their, like [𝒜-𝒵], and you’ll see why I find exposing UTF-16 to be a botch. It’s a mistake to make programmers think in encoding forms instead of in logical characters. –  tchrist May 8 '11 at 19:09

Your Answer


