byte characters and "real" characters (utf8)
When we memorize a text string some character encoding type is used to represent our character-set. Using single-byte-encoding such as in ASCII we have one char pr. seven bits which will only allow us to type 2^7 = 128 different chars.

UTF-8 (wikipedia excerpt)
UTF-8 (8-bit UCS/Unicode Transformation Format) is a variable-length character encoding for Unicode. It is able to represent any character in the Unicode standard, yet the initial encoding of byte codes and character assignments for UTF-8 is backwards compatible with ASCII. For these reasons, it is steadily becoming the preferred encoding for e-mail, web pages, and other places where characters are stored or streamed.

Perl and utf8
In Perl the internal encoding is utf8 and we can say utf8::upgrade($single-byte-char-string); # to upgrade a scalar value from ASCII to UTF-8 encoding. This can be very tricky and one must take care to get the encoding right from the beginning. When printing utf8 content to file-handles it can be a good idea to include utf8 encoding as an argument when opening the handle: open FH, ">:utf8", "filename"; will open the handle FH for output with utf8-layer included.

Another thing is support with the URI module and utf8 filenames. If you f.ex want to escape an url you will have to use encode('utf8', $name) to return an octet sequence for the escape to work properly.

Exercise
To state an example we'll execute a couple of one-liners from our virtual console to make it clearer how our encoding tools works out.

#1 perl -e 'use utf8;use Encode; print join ",", map sprintf("%x",ord), split(//,
encode("utf8","☺"));print "\n"'

First we say use utf8 to "allow" the smiley-face character in our code-text. Then we encode to a utf8 byte-string and split each byte for hex maps joined by commas (,). This should give us: e2,98,ba which is the bytes used for the utf8 byte-sequence-level representation of a smiley-face.

Now. To prove that we're not complete fools (only forgot to say use utf8;)

#2 perl -e 'print "\xe2\x98\xba"'

Should print out a smiley-face...

resources
http://en.wikipedia.org/wiki/ASCII
http://en.wikipedia.org/wiki/UTF-8

/GS42K0

# (c) dion.rasmussen@gmail.com