Florian Ragwitz > perl-5.13.4 > perlunicode


Annotate this POD


Source   Latest Release: perl-5.21.3


perlunicode - Unicode support in Perl


Important Caveats


People who want to learn to use Unicode in Perl, should probably read the Perl Unicode tutorial, perlunitut, before reading this reference document.


Input and Output Layers

Perl knows when a filehandle uses Perl's internal Unicode encodings (UTF-8, or UTF-EBCDIC if in EBCDIC) if the filehandle is opened with the ":utf8" layer. Other encodings can be converted to Perl's encoding on input or from Perl's encoding on output by use of the ":encoding(...)" layer. See open.


Regular Expressions

The regular expression compiler produces polymorphic opcodes. That is, the pattern adapts to the data and automatically switches to the Unicode character scheme when presented with data that is internally encoded in UTF-8, or instead uses a traditional byte scheme when presented with byte data.



BOM-marked scripts and UTF-16 scripts autodetected


use encoding needed to upgrade non-Latin-1 byte strings


See "Byte and Character Semantics" for more details.

Byte and Character Semantics


In future, Perl-level operations will be expected to work with characters rather than bytes.

However, as an interim compatibility measure, Perl aims to provide a safe migration path from byte semantics to character semantics for programs. For operations where Perl can unambiguously decide that the input data are characters, Perl switches to character semantics. For operations where this determination cannot be made without additional information from the user, Perl decides in favor of compatibility and chooses to use byte semantics.

Under byte semantics, when use locale is in effect, Perl uses the semantics associated with the current locale. Absent a use locale, and absent a use feature 'unicode_strings' pragma, Perl currently uses US-ASCII (or Basic Latin in Unicode terminology) byte semantics, meaning that characters whose ordinal numbers are in the range 128 - 255 are undefined except for their ordinal numbers. This means that none have case (upper and lower), nor are any a member of character classes, like [:alpha:] or \w. (But all do belong to the \W class or the Perl regular expression extension [:^alpha:].)


The bytes pragma will always, regardless of platform, force byte semantics in a particular lexical scope. See bytes.

The use feature 'unicode_strings' pragma is intended to always, regardless of platform, force character (Unicode) semantics in a particular lexical scope. In release 5.12, it is partially implemented, applying only to case changes. See "The "Unicode Bug"" below.


Unless explicitly stated, Perl operators use character semantics for Unicode data and byte semantics for non-Unicode data. The decision to use character semantics is made transparently. If input data comes from a Unicode source--for example, if a character encoding layer is added to a filehandle or a literal Unicode string constant appears in a program--character semantics apply. Otherwise, byte semantics are in effect. The bytes pragma should be used to force byte semantics on Unicode data, and the use feature 'unicode_strings' pragma to force Unicode semantics on byte data (though in 5.12 it isn't fully implemented).



Effects of Character Semantics

Character semantics have the following effects:

Unicode Character Properties

Most Unicode character properties are accessible by using regular expressions. They are used (like bracketed character classes) by using the \p{} "matches property" construct and the \P{} negation, "doesn't match property".




This formality is needed when properties are not binary, that is if they can take on more values than just True and False. For example, the Bidi_Class (see "Bidirectional Character Types" below), can take on a number of different values, such as Left, Right, Whitespace, and others. To match these, one needs to specify the property name (Bidi_Class), and the value being matched against (Left, Right, etc.). This is done, as in the examples above, by having the two components separated by an equal sign (or interchangeably, a colon), like \p{Bidi_Class: Left}.


Most Unicode character properties have at least two synonyms (or aliases if you prefer), a short one that is easier to type, and a longer one which is more descriptive and hence it is easier to understand what it means. Thus the "L" and "Letter" above are equivalent and can be used interchangeably. Likewise, "Upper" is a synonym for "Uppercase", and we could have written \p{Uppercase} equivalently as \p{Upper}. Also, there are typically various synonyms for the values the property can be. For binary properties, "True" has 3 synonyms: "T", "Yes", and "Y"; and "False has correspondingly "F", "No", and "N". But be careful. A short form of a value for one property may not mean the same thing as the same short form for another. Thus, for the General_Category property, "L" means "Letter", but for the Bidi_Class property, "L" means "Left". A complete list of properties and synonyms is in perluniprops.

Upper/lower case differences in the property names and values are irrelevant, thus \p{Upper} means the same thing as \p{upper} or even \p{UpPeR}. Similarly, you can add or subtract underscores anywhere in the middle of a word, so that these are also equivalent to \p{U_p_p_e_r}. And white space is irrelevant adjacent to non-word characters, such as the braces and the equals or colon separators so \p{ Upper } and \p{ Upper_case : Y } are equivalent to these as well. In fact, in most cases, white space and even hyphens can be added or deleted anywhere. So even \p{ Up-per case = Yes} is equivalent. All this is called "loose-matching" by Unicode. The few places where stricter matching is employed is in the middle of numbers, and the Perl extension properties that begin or end with an underscore. Stricter matching cares about white space (except adjacent to the non-word characters) and hyphens, and non-interior underscores.



Every Unicode character is assigned a general category, which is the "most usual categorization of a character" (from http://www.unicode.org/reports/tr44).



    Short       Long

    L           Letter
    Lu          Uppercase_Letter
    Ll          Lowercase_Letter
    Lt          Titlecase_Letter
    Lm          Modifier_Letter
    Lo          Other_Letter

    M           Mark
    Mn          Nonspacing_Mark
    Mc          Spacing_Mark
    Me          Enclosing_Mark

    N           Number
    Nd          Decimal_Number (also Digit)
    Nl          Letter_Number
    No          Other_Number

    P           Punctuation (also Punct)
    Pc          Connector_Punctuation
    Pd          Dash_Punctuation
    Ps          Open_Punctuation
    Pe          Close_Punctuation
    Pi          Initial_Punctuation
                (may behave like Ps or Pe depending on usage)
    Pf          Final_Punctuation
                (may behave like Ps or Pe depending on usage)
    Po          Other_Punctuation

    S           Symbol
    Sm          Math_Symbol
    Sc          Currency_Symbol
    Sk          Modifier_Symbol
    So          Other_Symbol

    Z           Separator
    Zs          Space_Separator
    Zl          Line_Separator
    Zp          Paragraph_Separator

    C           Other
    Cc          Control (also Cntrl)
    Cf          Format
    Cs          Surrogate   (not usable)
    Co          Private_Use
    Cn          Unassigned

Single-letter properties match all characters in any of the two-letter sub-properties starting with the same letter. LC and L& are special cases, which are both aliases for the set consisting of everything matched by Ll, Lu, and Lt.

Because Perl hides the need for the user to understand the internal representation of Unicode characters, there is no need to implement the somewhat messy concept of surrogates. Cs is therefore not supported.

Bidirectional Character Types

Because scripts differ in their directionality (Hebrew is written right to left, for example) Unicode supplies these properties in the Bidi_Class class:

    Property    Meaning

    L           Left-to-Right
    LRE         Left-to-Right Embedding
    LRO         Left-to-Right Override
    R           Right-to-Left
    AL          Arabic Letter
    RLE         Right-to-Left Embedding
    RLO         Right-to-Left Override
    PDF         Pop Directional Format
    EN          European Number
    ES          European Separator
    ET          European Terminator
    AN          Arabic Number
    CS          Common Separator
    NSM         Non-Spacing Mark
    BN          Boundary Neutral
    B           Paragraph Separator
    S           Segment Separator
    WS          Whitespace
    ON          Other Neutrals



The world's languages are written in a number of scripts. This sentence (unless you're reading it in translation) is written in Latin, while Russian is written in Cyrllic, and Greek is written in, well, Greek; Japanese mainly in Hiragana or Katakana. There are many more.


A complete list of scripts and their shortcuts is in perluniprops.

Use of "Is" Prefix



In addition to scripts, Unicode also defines blocks of characters. The difference between scripts and blocks is that the concept of scripts is closer to natural languages, while the concept of blocks is more of an artificial grouping based on groups of Unicode characters with consecutive ordinal values. For example, the "Basic Latin" block is all characters whose ordinals are between 0 and 127, inclusive, in other words, the ASCII characters. The "Latin" script contains some letters from this block as well as several more, like "Latin-1 Supplement", "Latin Extended-A", etc., but it does not contain all the characters from those blocks. It does not, for example, contain digits, because digits are shared across many scripts. Digits and similar groups, like punctuation, are in the script called Common. There is also a script called Inherited for characters that modify other characters, and inherit the script value of the controlling character.

For more about scripts versus blocks, see UAX#24 "Unicode Script Property": http://www.unicode.org/reports/tr24

The Script property is likely to be the one you want to use when processing natural language; the Block property may be useful in working with the nuts and bolts of Unicode.



Some people just prefer to always use \p{Block: foo} and \p{Script: bar} instead of the shortcuts, for clarity, and because they can't remember the difference between 'In' and 'Is' anyway (or aren't confident that those who eventually will read their code will know).

A complete list of blocks and their shortcuts is in perluniprops.

Other Properties


Unicode defines all its properties in the compound form, so all single-form properties are Perl extensions. A number of these are just synonyms for the Unicode ones, but some are genunine extensions, including a couple that are in the compound form. And quite a few of these are actually recommended by Unicode (in http://www.unicode.org/reports/tr18).

This section gives some details on all the extensions that aren't synonyms for compound-form Unicode properties (for those, you'll have to refer to the Unicode Standard.












Matches a character that has a non-canonical decomposition.

To understand the use of this rarely used property=value combination, it is necessary to know some basics about decomposition. Consider a character, say H. It could appear with various marks around it, such as an acute accent, or a circumflex, or various hooks, circles, arrows, etc., above, below, to one side and/or the other, etc. There are many possibilities among the world's languages. The number of combinations is astronomical, and if there were a character for each combination, it would soon exhaust Unicode's more than a million possible characters. So Unicode took a different approach: there is a character for the base H, and a character for each of the possible marks, and they can be combined variously to get a final logical character. So a logical character--what appears to be a single character--can be a sequence of more than one individual characters. This is called an "extended grapheme cluster". (Perl furnishes the \X construct to match such sequences.)

But Unicode's intent is to unify the existing character set standards and practices, and a number of pre-existing standards have single characters that mean the same thing as some of these combinations. An example is ISO-8859-1, which has quite a few of these in the Latin-1 range, an example being "LATIN CAPITAL LETTER E WITH ACUTE". Because this character was in this pre-existing standard, Unicode added it to its repertoire. But this character is considered by Unicode to be equivalent to the sequence consisting of first the character "LATIN CAPITAL LETTER E", then the character "COMBINING ACUTE ACCENT".

"LATIN CAPITAL LETTER E WITH ACUTE" is called a "pre-composed" character, and the equivalence with the sequence is called canonical equivalence. All pre-composed characters are said to have a decomposition (into the equivalent sequence) and the decomposition type is also called canonical.

However, many more characters have a different type of decomposition, a "compatible" or "non-canonical" decomposition. The sequences that form these decompositions are not considered canonically equivalent to the pre-composed character. An example, again in the Latin-1 range, is the "SUPERSCRIPT ONE". It is kind of like a regular digit 1, but not exactly; its decomposition into the digit 1 is called a "compatible" decomposition, specifically a "super" decomposition. There are several such compatibility decompositions (see http://www.unicode.org/reports/tr44), including one called "compat" which means some miscellaneous type of decomposition that doesn't fit into the decomposition categories that Unicode has chosen.


Perl has added the Non_Canonical type, for your convenience, to mean any of the compatibility decompositions.




This is the same as \h and \p{Blank}: A character that changes the spacing horizontally.


This is a synonym for \p{Present_In=*}



Mnemonic: Perl's (original) space



Mnemonic: Perl's (original) word.


This matches any alphanumeric character in the ASCII range, namely [A-Za-z0-9].


This matches any alphabetic character in the ASCII range, namely [A-Za-z].


This matches any blank character in the ASCII range, namely [ \t].


This matches any control character in the ASCII range, namely [\x00-\x1F\x7F]


This matches any digit character in the ASCII range, namely [0-9].


This matches any graphical character in the ASCII range, namely [\x21-\x7E].


This matches any lowercase character in the ASCII range, namely [a-z].


This matches any printable character in the ASCII range, namely [\x20-\x7E]. These are the graphical characters plus SPACE.


This matches any punctuation character in the ASCII range, namely [\x21-\x2F\x3A-\x40\x5B-\x60\x7B-\x7E]. These are the graphical characters that aren't word characters. Note that the Posix standard includes in its definition of punctuation, those characters that Unicode calls "symbols."


This matches any space character in the ASCII range, namely [ \f\n\r\t\x0B] (the last being a vertical tab).


This matches any uppercase character in the ASCII range, namely [A-Z].







Another confusion with both these properties is that the definition is not that the code point has been assigned, but that the meaning of the code point has been determined. This is because 66 code points will always be unassigned, and, so the Age for them is the Unicode version the decision to make them so was made in. For example, U+FDD0 is to be permanently unassigned to a character, and the decision to do that was made in version 3.1, so \p{Age=3.1} matches this character and \p{Present_In: 3.1} and up matches as well.




This is the same as \s, including beyond ASCII.

Mnemonic: Space, as modified by Perl. (It doesn't include the vertical tab which both the Posix standard and Unicode consider to be space.)




This is the same as \w, including beyond ASCII.

User-Defined Character Properties


    # assuming property Is_Foreign defined in Lang::
    package main;  # property package name required

    package Lang;  # property package name not required




    sub InKana {
        return <<END;


You could also have used the existing block property names:

    sub InKana {
        return <<'END';


    sub InKana {
        return <<'END';


    sub InNotKana {
        return <<'END';


    sub InFooAndBar {
        return <<'END';

It's important to remember not to use "&" for the first set; that would be intersecting with nothing (resulting in an empty set).

User-Defined Case Mappings

You can also define your own mappings to be used in lc(), lcfirst(), uc(), and ucfirst() (or their string-inlined versions, \L, \l, \U, and \u). The principle is similar to that of user-defined character properties: to define subroutines with names ToLower (for lc() and lcfirst()); ToTitle (for ucfirst()); and ToUpper (for uc()).

The string returned by the subroutines needs to be two hexadecimal numbers separated by two tabulators: the two numbers being, respectively, the source code point and the destination code point. For example:

    sub ToUpper {
        return <<END;

defines a mapping for uc() (and \U) that causes only the character "a" to be mapped to "A"; all other characters will remain unchanged.




    use Config;

    sub ToUpper {
        my $official = do "$Config{privlib}/unicore/To/Upper.pl";
        $utf8::ToSpecUpper{'i'} = 
        return $official;

This takes the official mappings and overrides just one, for "LATIN SMALL LETTER I". The keys to the hash must be in UTF-8 (or on EBCDIC platforms, UTF-EBCDIC), as illustrated by the inverse function.

    sub ToLower {
        my $official = do $lower;
        return $official;

This example is for an ASCII platform, and \xc4\xb0 is the UTF-8 string that represents \N{LATIN CAPITAL LETTER I WITH DOT ABOVE}, U+0130.


The mappings are in effect only for the package they are defined in, and only on scalars that have been marked as having Unicode characters, for example by using utf8::upgrade(). You can get around the latter restriction in the scope of a use subs:

    use subs qw(uc ucfirst lc lcfirst);

    sub uc($) {
        my $string = shift;
        return CORE::uc($string);

    sub lc($) {
        my $string = shift;

        $string =~ 
              s/I (?! [^\p{ccc=0}\p{ccc=Above}]* \x{0307} )/\x{131}/gx;

        # dot_above so the end result will be i.
        $string =~ s/I ([^\p{ccc=0}\p{ccc=Above}]* ) \x{0307}/i$1/gx;
        return CORE::lc($string);

These examples (also for Turkish) make sure the input is in UTF-8, and then call the corresponding official function, which will use the ToUpper() and ToLower() functions you have defined in the package. The lc() example shows how you can add context-dependent casing. (For Turkish, there other required functions: ucfirst, lcfirst, and ToTitle. These are very similar to the ones given above.)

Character Encodings for Input and Output

See Encode.

Unicode Regular Expression Support Level

The following list of Unicode support for regular expressions describes all the features currently supported. The references to "Level N" and the section numbers refer to the Unicode Technical Standard #18, "Unicode Regular Expressions", version 11, in May 2005.

Unicode Encodings


Security Implications of Unicode

Read Unicode Security Considerations. Also, note the following:

Unicode in Perl on EBCDIC



Usually locale settings and Unicode do not affect each other, but there are a couple of exceptions:

When Unicode Does Not Happen

While Perl does have extensive ways to input and output in Unicode, and few other 'entry points' like the @ARGV which can be interpreted as Unicode (UTF-8), there still are many places where Unicode (in some encoding or another) could be given as arguments or received as results, or both, but it is not.


One reason why Perl does not attempt to resolve the role of Unicode in these cases is that the answers are highly dependent on the operating system and the file system(s). For example, whether filenames can be in Unicode, and in exactly what kind of encoding, is not exactly a portable concept. Similarly for the qx and system: how well will the 'command line interface' (and which of them?) handle Unicode?

The "Unicode Bug"

The term, the "Unicode bug" has been applied to an inconsistency with the Unicode characters whose ordinals are in the Latin-1 Supplement block, that is, between 128 and 255. Without a locale specified, unlike all other characters or code points, these characters have very different semantics in byte semantics versus character semantics.


In byte semantics, they are considered to be unassigned characters, meaning that the only semantics they have is their ordinal numbers, and that they are not members of various character classes. None are considered to match \w for example, but all match \W. (On EBCDIC platforms, the behavior may be different from this, depending on the underlying C language library functions.)

The behavior is known to have effects on these areas:


 $ perl -le'
     $s1 = "\xC2";
     $s2 = "\x{2660}";
         print /\w/ || 0;



Work is being done to correct this, but only some of it was complete in time for the 5.12 release. What has been finished is the important part of the case changing component. Due to concerns, and some evidence, that older code might have come to rely on the existing behavior, the new behavior must be explicitly enabled by the feature unicode_strings in the feature pragma, even though no new syntax is involved.

See "lc" in perlfunc for details on how this pragma works in combination with various others for casing. Even though the pragma only affects casing operations in the 5.12 release, it is planned to have it affect all the problematic behaviors in later releases: you can't have one without them all.


Forcing Unicode in Perl (Or Unforcing Unicode in Perl)




Using Unicode in XS





Download the files in the version of Unicode that you want from the Unicode web site http://www.unicode.org). These should replace the existing files in \$Config{privlib}/unicore. (\%Config is available from the Config module.) Follow the instructions in README.perl in that directory to change some of their names, and then run make.

It is even possible to download them to a different directory, and then change utf8_heavy.pl in the directory \$Config{privlib} to point to the new directory, or maybe make a copy of that directory before making the change, and using @INC or the -I run-time flag to switch between versions at will (but because of caching, not in the middle of a process), but all this is beyond the scope of these instructions.


Interaction with Locales

Use of locales with Unicode data may lead to odd results. Currently, Perl attempts to attach 8-bit locale info to characters in the range 0..255, but this technique is demonstrably incorrect for locales that use characters above that range when mapped into Unicode. Perl's Unicode support will also tend to run slower. Use of locales with Unicode is discouraged.

Problems with characters in the Latin-1 Supplement range

See "The "Unicode Bug""

Problems with case-insensitive regular expression matching

There are problems with case-insensitive matches, including those involving character classes (enclosed in [square brackets]), characters whose fold is to multiple characters (such as the single character LATIN SMALL LIGATURE FFL matches case-insensitively with the 3-character string ffl), and characters in the Latin-1 Supplement.

Interaction with Extensions

When Perl exchanges data with an extension, the extension should be able to understand the UTF8 flag and act accordingly. If the extension doesn't know about the flag, it's likely that the extension will return incorrectly-flagged data.




    sub my_escape_html ($) {
        my($what) = shift;
        return unless defined $what;

Sometimes, when the extension does not convert data but just stores and retrieves them, you will be in a position to use the otherwise dangerous Encode::_utf8_on() function. Let's say the popular Foo::Bar extension, written in C, provides a param method that lets you store and retrieve data according to these prototypes:



    sub param {
      if (defined $value) {
      } else {
        return $ret;





Problems on EBCDIC platforms

There are a number of known problems with Perl on EBCDIC platforms. If you want to use Perl there, send email to perlbug@perl.org.



Porting code from perl-5.6.X



perlunitut, perluniintro, perluniprops, Encode, open, utf8, bytes, perlretut, "${^UNICODE}" in perlvar http://www.unicode.org/reports/tr44).

syntax highlighting: