3.1 Perceptions of Characters
3.1.1 Introduction
The word character is used in many contexts, with different
meanings. Human cultures have radically differing writing systems, leading to
radically differing concepts of a character. Such wide variation in end user
experience can, and often does, result in misunderstanding. This variation is
sometimes mistakenly seen as the consequence of imperfect technology.
Instead, it derives from the great flexibility and creativity of the human
mind and the long tradition of writing as an important part of the human
cultural heritage.
The developers of W3C specifications, and the developers of software based
on those specifications, are likely to be more familiar with usages they have
experienced and less familiar with the wide variety of usages in an
international context. Furthermore, within a computing context, characters
are often confused with related concepts, resulting in incomplete or
inappropriate specifications and software.
This section examines some of these contexts, meanings and confusions.
3.1.2 Units of a Writing System, and
Units of Aural Rendering
The glossary entry in [Unicode 3.0] gives:
Character. (1) The smallest component of written language that has
semantic values; refers to the abstract meaning and or shape ...
In some scripts, characters have a close relationship to
phonemes, while in others they are closely related to meanings. Even
when characters (loosely) correspond to phonemes, this relationship may not
be simple, and there is rarely a one-to-one correspondence between character
and phoneme.
Example: Japanese hiragana and katakana are syllabaries, not phonemic
alphabets. A character in these scripts corresponds not to a phoneme, but to
a syllable.
Example: Korean Hangul combines symbols for phonemes into square syllabic
blocks. Depending on the user and the application, either the individual
symbols or the syllabic clusters can be considered to be characters.
Example: Indic scripts use semi-regular or irregular ways to combine
consonants and vowels into clusters. Depending on the user and the
application, either individual consonants or vowels, or the consonant or
consonant-vowel clusters can be perceived as characters.
[S][I]
Specification writers and software developers MUST NOT assume that there is a
one-to-one correspondence between characters and phonemes.
3.1.3 Units of Visual
Rendering
Visual rendering introduces the notion of a glyph. Glyphs are
defined by [ISO/IEC 9541] as "a recognizable abstract
graphic symbol which is independent of a specific design". There is
not a one-to-one correspondence between characters and
glyphs:
- A single character can be represented by multiple glyphs (each glyph is
then part of the representation of that character). These glyphs may be
physically separated from one another.
- A single glyph may represent a sequence of characters (this is the case
with ligatures, among others).
- A character may be rendered with very different glyphs depending on the
context.
- A single glyph may represent different characters (e.g. capital Latin
A, capital Greek A and capital Cyrillic A).
Each glyph can be represented by a number of different glyph images; a set
of glyph images makes up a font. Glyphs can be construed as the
basic units of organization of the visual rendering of text, just as
characters are the basic unit of organization of encoded text.
See Appendix A for examples of the
complexities of character to glyph mapping.
[S][I]
Specification writers and software developers MUST NOT assume a one-to-one
mapping between character codes and units of displayed text.
Some scripts, in particular Arabic and Hebrew, are written from right to
left. Text including characters from these scripts can run in both directions
and is therefore called bidirectional text (see examples A.6 to A.8).
The Unicode Standard requires that characters be
stored and interchanged in logical order. [S] Protocols, data formats and APIs MUST
store, interchange or process text data in logical order. [S] Where protocols or APIs
support selection, they MUST support logical selection (range of character
offsets), and they SHOULD support visual selection (area on display), because
many users prefer visual selection. Visual selection can give rise to
discontiguous logical selections and vice versa (see examples A.6 to A.8).
[S] Specifications of
protocols and APIs that involve selection MUST provide for discontiguous
logical selections, at least to the extent necessary to support scripts
involving character reordering. [S] Such specifications MAY, however, make
actual implementation of discontiguous logical selections optional for
implementations that choose not to support scripts involving character
reordering. This is analogous to HTML4 not requiring the
bidirectional algorithm from those implementations that choose not to support
bidirectional scripts (cf. section 8.2 of [HTML
4.0]).
3.1.4 Units of Input
In keyboard input, it is not always the case that
keystrokes and input characters correspond one-to-one. A limited number of
keys can fit on a keyboard and many writing systems have too many characters
to allow such a correspondence and must rely on more complex input
methods, which transform keystroke sequences into character sequences.
Other languages may make it necessary to input some characters with special
modifier keys. See Appendix A for examples of
non-trivial input.
[S][I]
Specification writers and software developers MUST NOT assume that a single
keystroke results in a single character, nor that a single character can be
input with a single keystroke (even with modifiers), nor that keyboards are
the same all over the world.
3.1.5 Units of Collation
String comparison, as used in sorting and searching for instance, is based
on collation units which are not in general in a one-to-one
relationship to encoded characters. String comparison can aggregate a
character sequence into a single collation unit with its own position in the
sorting order, can separate a single character into multiple collation units,
and can distinguish various aspects of a character (case, presence of
diacritics, etc.) to be sorted separately (multi-level sorting).
Example: In traditional Spanish sorting, the letter sequences "ch" and
"ll" are treated as atomic collation units. Although Spanish sorting, and to
some extent Spanish everyday use, treat "ch" as a single unit, current
digital encodings treat it as two letters, and keyboards do the same (the
user types 'c', then 'h').
Example: In most languages, the letter æ is sorted as two consecutive
collation units: a and e.
Example: The sorting of text written in a bicameral script (i.e. a script
which has distinct upper and lower case letters) is usually required to
ignore case differences in a first pass; case is then used to break ties in a
later pass.
Example: Treatment of accented letters in sorting is dependent on the
script or language in question. The letter "ö" is considered as a modified
"o" in French, as an extra character quite related to "o" in German and as a
letter completely independent from "o" (and sorting after "z") in Swedish.
Note: collation methods may also vary by application (e.g. bilingual
dictionaries are sorted differently than telephone books in Japanese).
[S][I] Even though
software fundamentally works on the level of encoded characters, the
capabilities of software to sort (or otherwise process) text MUST NOT be
restricted to those collation units that have a one-to-one relation to
encoded characters. [I] Software developers MUST NOT merely use a
one-to-one mapping as their string-compare function, as for example in
sorting operations.
3.1.6 Units of Storage
Computer storage and communication rely on units of physical storage and
information interchange, such as bits and bytes (also known as octets, as
nowadays the word bytes is generally considered to mean 8-bit bytes). A
frequent error in specifications and implementations is the equating of
characters with units of physical storage. The mapping between characters and
such units of storage is actually quite complex, and is discussed in the next
section, 3.2 Digital Encoding of Characters.
[S][I]
Specification writers and software developers MUST NOT assume a one-to-one
relationship between characters and units of physical storage.
3.1.7 Summary
The term character is used differently in a variety of contexts
and often leads to confusion when used outside of these contexts. In the
context of the digital representations of text, a character can be defined
informally as a small logical unit of text. Text is then
defined as sequences of characters. While such an informal definition is
sufficient to create or capture a common understanding in many cases, it is
also sufficiently open to create misunderstandings as soon as details start
to matter. In order to write effective specifications, protocol
implementations, and software for end users, it is very important to
understand that these misunderstandings can occur.
[S] Specifications
using the term character MUST specify which of the possible meanings
they intend. [S] Specifications SHOULD avoid the use of
the term character if a more specific term is available.
3.2 Digital Encoding of Characters
To be of any use in computers, in computer communications and in
particular on the World Wide Web, characters must be encoded. In fact,
much of the information processed by computers over the last few decades has
been encoded text, exceptions being images, audio, video and numeric data. To
achieve text encoding, a large variety of encoding schemes have been devised,
which can loosely be defined as mappings between the character sequences that
users manipulate and the sequences of bits that computers manipulate.
Given the complexity of text encoding and the large variety of schemes for
character encoding invented throughout the computer age, a more formal
description of the encoding process is useful. The process of defining a text
encoding can be described as follows (see [UTR #17] for
a more detailed description):
A set of characters to be encoded is
identified. The characters are pragmatically chosen to express text and
to efficiently allow various text processes in one or more target
languages. They may not correspond precisely to what users perceive as
letters and other characters. The set of characters is called a
repertoire.
Each character in the repertoire is then
associated with a (mathematical, abstract) non-negative integer, the
code point (also known as a character number or
code position). The result, a mapping from the repertoire to the
set of non-negative integers, is called a coded character set
(CCS).
To enable use in computers, a suitable base
datatype is identified (such as a byte, a 16-bit unit of storage or
other) and a character encoding form (CEF) is used, which
encodes the abstract integers of a CCS into sequences of the code
units of the base datatype. The encoding form can be extremely
simple (for instance, one which encodes the integers of the CCS into the
natural representation of integers of the chosen datatype of the
computing platform) or arbitrarily complex (a variable number of code
units, where the value of each unit is a non-trivial function of the
encoded integer).
To enable transmission or storage using
byte-oriented devices, a serialization scheme or character
encoding scheme (CES) is next used. A CES is a mapping of the code units
of a CEF into well-defined sequences of bytes, taking into account the
necessary specification of byte-order for multi-byte base datatypes and
including in some cases switching schemes between the code units of
multiple CCSes (an example is ISO 2022). A CES, together with the CCSes
it is used with, is identified by an IANA charset
identifier. Given a sequence of bytes representing text and a
charset identifier, one can in principle unambiguously
recover the sequence of characters of the text.
Note: The term character encoding is somewhat
ambiguous, as it is sometimes used to describe the actual process of encoding
characters and sometimes to denote a particular way to perform that process
(as in this file is in the X character encoding). Context normally
allows the distinction of those uses, once one is aware of the ambiguity.
Note: Unfortunately, there are some important cases of
charset identifiers that denote a range of slight variants of an encoding
scheme, where the differences may be crucial (e.g. the well-known
yen/backslash case) and may vary over time. In those cases, recovery of the
character sequence from a byte sequence is not totally unambiguous. See the
[XML Japanese profile] for examples of
such ambiguous charsets.
In very simple cases, the whole encoding process can be collapsed to a
single step, a trivial one-to-one mapping from characters to bytes; this is
the case, for instance, for US-ASCII and ISO-8859-1.
Text data
is said to be in a Unicode encoding form if it is encoded in
UTF-8, UTF-16 or UTF-32.
3.3 Transcoding
Transcoding is the process of converting text data from one Character Encoding Form to another. Transcoders work only
at the level of character encoding and do not parse the text; consequently,
they do not deal with character escapes such as numeric character references
(see 3.7 Character Escaping) and do not fix up
embedded character encoding information (for instance in an XML declaration
or in an HTML <meta> element).
Note: Transcoding may involve one-to-one, many-to-one, one-to-many or
many-to-many mappings. In addition, the storage order of characters varies
between encodings: some, such as Unicode, prescribe logical ordering while
others use visual ordering; among encodings that have separate diacritics,
some prescribe that they be placed before the base character, some after.
Because of these differences in sequencing characters, transcoding may
involve reordering: thus XYZ may map to yxz.
A normalizing transcoder
is a transcoder that converts from a legacy encoding to a Unicode encoding
form and ensures that the result is in Unicode Normalization
Form C (see 4.2.1 Unicode-normalized
Text). For most legacy encodings, it is possible to construct a
normalizing transcoder; it is not possible to do so if the encoding's
repertoire contains characters not in Unicode.
3.4 Strings
Various specifications use the notion of a string, sometimes
without defining precisely what is meant and sometimes defining it
differently from other specifications. The reason for this variability is
that there are in fact multiple reasonable definitions for a string,
depending on one's intended use of the notion; the term string is
used for all these different notions because these are actually just
different views of the same reality: a piece of text stored inside a
computer. This section provides specific definitions for different notions of
string which may be reused elsewhere.
Byte string: A string
viewed as a sequence of bytes representing characters in a particular
encoding. This corresponds to a CES. As a definition
for a string, this definition is most often useless, except when the textual
nature is unimportant and the string is considered only as a piece of opaque
data with a length in bytes. Specifications in general SHOULD NOT define a
string as a byte string.
Code unit string: A
string viewed as a sequence of code units representing characters in a
particular encoding. This corresponds to a CEF. This
definition is useful in APIs that expose a physical representation of string
data. Example: For the [DOM Level 1], UTF-16 was chosen
based on widespread implementation practice.
Character string: A
string viewed as a sequence of characters, each represented by a code point
in [Unicode]. This is usually what programmers
consider to be a string, although it may not match exactly what most users
perceive as characters. This is the highest layer of abstraction that ensures
interoperability with very low implementation effort. [S] This definition is generally the most
useful and SHOULD be used by most specifications, following the examples of
Production [2] of [XML 1.0], the SGML declaration
of [HTML 4.01], and the character model of [RFC 2070].
Example: Consider the string comprising the characters U+10330 GOTHIC
LETTER AHSA, U+2260 NOT EQUAL TO and U+0030 DIGIT ZERO, encoded in UTF-16 in
big-endian byte order. The rows of the following table show the string viewed
as a character string, code unit string and byte string, respectively:
| Character string |
U+10330 |
U+2260 |
U+0030 |
| Code unit string |
D800 |
DF30 |
2260 |
0030 |
| Byte string |
D8 |
00 |
DF |
30 |
22 |
60 |
00 |
30 |
Note: It is also possible to view a
string as a sequence of graphemes. In this case the string is
divided into text units that correspond to the user's perception of where
character boundaries occur in a visually rendered text. However, there is no
standard rule for the segmentation of text in this way, and the segmentation
will vary from language to language and even from user to user. Examples of
possible approaches can be found in sections 5.12 and 5.15 of [Unicode 3.0].
3.5 Reference Processing Model
Many Internet protocols and data formats, most notably the very important
Web formats HTML, CSS and XML, are based on text. In those formats,
everything is text but the relevant specifications impose a structure on the
text, giving meaning to certain constructs so as to obtain functionality in
addition to that provided by plain text. HTML and XML are markup
languages, defining entities entirely composed of text but with
conventions allowing the separation of this text into markup and
character data. Citing from [XML 1.0], section 2.4:
Text consists of intermingled character data and
markup. [...] All text that is not markup constitutes the character
data of the document.
For the purposes of this section, the important aspect is that everything
is text, that is, a sequence of characters.
Since its early days, the Web has seen the development of a Reference
Processing Model, first described for HTML in [RFC
2070]. This model was later embraced by XML and CSS. It is applicable to
any data format or protocol that is text-based as described above. The
essence of the Reference Processing Model is the use of Unicode as a common
reference. Use of the Reference Processing Model by a specification does not,
however, require that implementations actually use Unicode. The requirement
is only that the implementations behave as if the processing took place as
described by the Model.
[S] A
specification conforms to the Reference Processing Model if all of the
following apply:
- [S] The
specification MUST be defined in terms of Unicode characters, not bytes
or glyphs.
- [S] The
specification SHOULD NOT restrict the range of characters that can be
used. The total Unicode range of code points from 0 to 0x10FFFF inclusive
should be usable; any exceptions should be listed and
justified.
- [S] The
specification MAY allow use of any character encoding which can be
transcoded to Unicode for its text entities. [S] It MAY choose to
disallow or deprecate some encodings and to make others mandatory.
[S] Independent
of the actual encoding, the specified behavior MUST be the same as if the
processing happened as follows:
- [S] the
encoding of any text entity received by the application implementing
the specification MUST be determined and the text entity MUST be
interpreted as a sequence of Unicode characters. [S] This MUST be
equivalent to transcoding the entity to some Unicode encoding form
and receiving it in that encoding form;
- [S] all
processing MUST take place on this sequence of Unicode
characters;
- [S] if text
is output by the application, the sequence of Unicode characters MUST
be encoded using an encoding chosen among those allowed by the
specification.
- [S] If the
specification is such that multiple text entities are involved (such as
an XML document referring to external parsed entities), it MAY choose to
allow these entities to be in different character encodings. [S] In all cases, this
Reference Processing Model MUST be applied to all entities.
[S] All
specifications that involve text MUST specify processing according to the
Reference Processing Model.
Note: All specifications that derive from [XML 1.0]
automatically inherit this Reference Processing Model. XML is entirely
defined in terms of Unicode characters and mandates the UTF-8 and UTF-16
encodings while allowing any other encoding for parsed entities.
Note: When specifications choose to allow encodings other than Unicode
encodings, implementers should be aware that the correspondence between the
characters of a legacy encoding and Unicode characters may in practice depend
on the software used for transcoding. See the [XML Japanese profile] for examples of such
inconsistencies.
3.6 Choice and Identification of Character
Encodings
Because encoded text cannot be interpreted and processed without
knowing the encoding, it is vitally important that the character encoding
scheme (see 3.2 Digital Encoding of Characters) is
known at all times and places where text is exchanged or processed. [S]Specifications MUST
either specify a unique encoding, or provide mechanisms such that the
encoding of text can always be reliably identified. [S]When designing a new
protocol, format or API, specifications SHOULD mandate a unique character
encoding.
Mandating a unique character encoding is simple, efficient, and robust.
There is no need for specifying, producing, transmitting, and interpreting
encoding tags. At the receiver, the encoding will always be understood. There
is also no ambiguity if data is transferred non-electronically and later has
to be converted back to a digital representation. Even when there is a need
for compatibility with existing data, systems, protocols and applications,
multiple encodings can often be dealt with at the boundaries or outside a
protocol, format, or API. The DOM ([DOM Level 1]) is an
example of where this was done. The advantages of choosing a unique encoding
become more important the smaller the pieces of text used are and the closer
to actual processing the specification is.
[S] When a unique
encoding is mandated, the encoding MUST be UTF-8, UTF-16 or UTF-32.
[S] If compatibility
with ASCII is desired, UTF-8 (see [RFC 2279]) is
RECOMMENDED; on the Internet, the IETF Charset Policy [RFC 2277] specifies that "Protocols MUST be able to use
the UTF-8 charset". [S] For APIs, UTF-16 (see [RFC 2781]) or UTF-32 are RECOMMENDED.
Note: The [XML 1.0] specification requires all
conforming XML processors to accept both UTF-16 and UTF-8.
3.6.1 Character Encoding Identification
[S] If the unique
encoding approach is not chosen, then protocols MUST provide character
encoding identification mechanisms. [S] These mechanisms SHOULD be along the
lines of the [MIME] Internet specification. The
MIME charset parameter is defined such that it provides sufficient
information to unambiguously decode the sequence of bytes of the payload into
a sequence of characters. The values are drawn from the IANA charset
registry [IANA].
Note: The term charset derives from "character set", an expression
with a long and tortured history (see [Connolly] for
a discussion). [S]
Specifications SHOULD avoid using the expression "character set", as well as
the term "charset", except when the latter is used to refer to the MIME
charset parameter or its IANA-registered values. The terms
character encoding or character encoding scheme are
recommended. In XML, the XML declaration or the text declaration contains a
pseudo-attribute called encoding which actually identifies the
charset.
The IANA charset registry is the official list of names and aliases
for character encodings on the Internet.
[S] If the unique
encoding approach is not taken, specifications SHOULD mandate the use of the
IANA charset names, and in particular the names labeled in the registry as
MIME preferred names, to designate character encodings in protocols,
data formats and APIs. [S] The "x-" convention for unregistered
names SHOULD NOT be used, having led to abuse in the past (use of "x-" for
character encodings that were widely used, even long after there was an
official registration). [I][C] Content developers and software that
tag textual data MUST use one of the names mandated by the appropriate
specification (e.g. the XML specification when editing XML text) and SHOULD
use the MIME preferred name of an encoding to tag data in that
encoding. [I][C] An IANA-registered charset name
MUST NOT be used to tag textual data in an encoding other than the one
identified in the IANA registration of that name.
[S] If the unique
encoding approach is not chosen, specifications MUST designate at least one
of the UTF-8 and UTF-16 encoding forms of Unicode as admissible encodings and
SHOULD choose at least one of UTF-8 or UTF-16 as mandated encoding forms
(encoding forms that MUST be supported by implementations of the
specification). [S] They MAY define either UTF-8 or UTF-16 as
a default (or both if they define suitable means of distinguishing them), but
they MUST NOT use any other character encoding as a default. [S] Also, they MUST NOT use
unreliable heuristics.
[I]
Receiving software MUST determine the encoding from available
information according to appropriate specifications. [I] When a IANA-registered
charset name is recognized, receiving software MUST interpret the
received data according to the encoding associated with the name in the IANA
registry. [I]
When no charset is provided the receiving software MUST adhere to the default
encoding(s) specified in the specification.
[I] Receiving
software MAY recognize as many encodings (names and aliases) as
appropriate. A field-upgradeable mechanism may be appropriate for this
purpose. Certain encodings are more or less associated with certain languages
(e.g. Shift-JIS with Japanese); trying to support a given language or set of
customers may mean that certain encodings have to be supported. The encodings
that need to be supported may change over time. This document does not give
any advice on which encoding may be appropriate or necessary for the support
of any given language.
[I] Software MUST
completely implement the mechanisms for character encoding identification and
SHOULD implement them in such a way that they are easy to use (for instance
in HTTP servers). [I] On interfaces to other protocols,
software SHOULD support conversion between Unicode encoding forms as well as
any other necessary conversions. [C] Content MUST make use of the offered
facilities by always indicating character encoding; where the facilities
offered for character encoding identification include defaults (e.g. in [XML 1.0]), relying on such defaults is sufficient to
satisfy this identification requirement.
Because of the layered Web architecture (e.g. formats used over
protocols), there may be multiple and at times conflicting information about
character encoding. [S] Specifications MUST define
conflict-resolution mechanisms (e.g. priorities) for these cases;
[I][C] software and
content MUST follow them carefully.
3.6.2 Private Use Code Points
Unicode designates certain ranges of code points for private use: the
Private Use Area (U+E000-F8FF) and planes 15 and 16 (U+F0000-FFFFD and
U+100000-10FFFD). These code points are guaranteed to never be allocated to
standard characters, and are available for use by private agreement between a
producer and a recipient. However, their use is strongly discouraged, since
private agreements do not scale on the Web. Code points from different
private agreements may collide, and a private agreement and therefore the
meaning of the code points can quickly get lost.
[S] Specifications
MUST NOT define any assignments of private use code points. [S] Conformance to a
specification MUST NOT require the use of PUA characters. [S] Specifications SHOULD
NOT provide mechanisms for agreement on private use code points between
parties and MUST NOT require the use of such mechanisms. [S][I] Specifications and
implementations SHOULD be designed in such a way as to not disallow the use
of these code points by private arrangement. As an example, XML does
not disallow the use of private use code points.
[S] Specifications
MAY define markup to allow the transmission of symbols not in Unicode or to
identify specific variants of Unicode characters.
Example: [SVG], in section
10.14, defines an element altGlyph which allows the
identification of specific display variants of Unicode characters.
3.7 Character Escaping
In text-based protocols or formats where characters can be either part of
character data or of markup (see 3.5 Reference
Processing Model), it is often the case that certain characters are
designated as having certain specific protocol/format functions in certain
contexts (e.g. "<" and "&" serve as markup delimiters in HTML and
XML). These syntax-significant characters cannot be used to represent
themselves in text data in the same way as all other characters do. Also,
often formats are represented in an encoding that does not allow to represent
all characters directly.
To express syntax-significant or unrepresentable characters, a technique
called escaping is used. This works by creating an additional
syntactic construct, defining additional characters or defining character
sequences that have special meaning. Escaping a character means expressing it
using such a construct, appropriate to the format or protocol in which the
character appears; expanding an escape (or unescaping) means
replacing it with the character that it represents.
Certain guidelines apply to the way specifications define character
escapes. [S] These
guidelines MUST be followed when designing new W3C protocols and formats and
SHOULD be followed as much as possible when revising existing protocols and
formats.
- [S]
Specifications MUST NOT invent a new escaping mechanism if an appropriate
one already exists.
- [S] The number
of different ways to escape a character SHOULD be minimized (ideally to
one). [A well-known counter-example is that for historical
reasons, both HTML and XML have redundant decimal (&#ddddd;) and
hexadecimal (&#xhhhh;) escapes.]
- [S] Explicit
end delimiters MUST be provided. [S] Escapes such as \uABCD where the end
delimiter is a space or any character other than [01-9A-F] SHOULD be
avoided: it is not clear visually, and it can cause an editor to
insert spurious line-breaks when word-wrapping on spaces. Forms like
SPREAD's &UABCD; [SPREAD] or XML's
&#xhhhh;, where the escape is explicitly terminated by a semicolon,
are much better.
- [S] Escaped
characters SHOULD be acceptable wherever unescaped characters are; this
does not preclude that a syntax-significant character, when escaped,
loses its significance in the syntax. [S] In particular, escaped characters
SHOULD be acceptable in identifiers and comments.
Certain guidelines apply to content developers, as well as to software
that generates content:
- [I][C] Escapes
SHOULD be avoided when the characters to be expressed are representable
in the character encoding of the document.
- [I][C] Since
character set standards usually list character numbers as hexadecimal,
content developers SHOULD use the hexadecimal form of escapes when there
is one.
4 Early Uniform Normalization
4.1 Motivation
As explained at length in Requirements for String Identity Matching
and String Indexing [CharReq], the existence,
in many character encoding schemes, of multiple representations for what
users perceive as the same string makes it necessary to define character data
normalization. Without a precise specification, it is not possible to
determine reliably whether or not two strings are identical. Such a
specification must take into account character encoding, the way
normalization is to be performed and where or when (by sender or recipient)
to perform it.
String identity is central to the correct functioning of much software,
and in particular of large parts of the Web infrastructure (protocols,
formats, etc.). Incorrect string matching can have far reaching consequences,
including the creation of security holes. Consider a contract, encoded in
XML, for buying goods: each item sold is described in an <artículo>
element; unfortunately, "artículo" is subject to different representations in
the character encoding of the contract. Suppose that the contract is viewed
and signed by means of a user agent that looks for <artículo> elements,
extracts them (matching on the element name), presents them to the user and
adds up their prices. If different instances of the <artículo> element
happen to be represented differently in a particular contract, then the buyer
and seller may see (and sign) different contracts if their respective user
agents perform string identity matching differently, which is fairly likely
in the absence of a well-defined specification. The absence of a well-defined
specification also means that there is no way to resolve the ensuing
contractual dispute.
The Unicode Consortium provides four standard normalization forms (see
Unicode Normalization Forms [UTR #15]). For use on the
Web, this document defines W3C Text Normalization by picking the most
appropriate of these (NFC) and additionally addressing the issues of legacy
encodings and of character escapes (which can denormalize text when
unescaped).
Roughly speaking, NFC is defined such that combining character sequences
(a base character followed by one or more combining characters) are replaced,
as far as possible, by canonically equivalent precomposed characters. Text in
a Unicode encoding form is said to be in NFC if it doesn't contain any
combining sequence that could be replaced and if any remaining combining
sequence is in canonical order.
This document also specifies that normalization is to be performed
early (by the sender) as opposed to late (by the recipient).
The reasons for that choice are manifold:
Almost all legacy data as well as data created by current software
is normalized (using NFC).
The number of Web components that generate or transform text is
considerably smaller than the number of components that receive text and
need to perform matching or other processes requiring normalized text.
Current receiving components (browsers, XML parsers, etc.)
implicitly assume early normalization by not performing normalization
themselves. This is a vast legacy.
Web components that generate and process text are in a much better
position to do normalization than other components; in particular, they
may be aware that they deal with a restricted repertoire only.
Not all components of the Web that implement functions such as
string matching can reasonably be expected to do normalization. This, in
particular, applies to very small components and components in the lower
layers of the architecture.
Forward-compatibility issues can be dealt with more easily: less
software needs to be updated, namely only the software that generates
newly introduced characters.
It improves matching in cases where the character encoding is partly
undefined, such as URIs [RFC 2396] in which
non-ASCII bytes have no defined meaning.
It is a prerequisite for comparison of encrypted strings (see [CharReq], section 2.7).
It simplifies definitions and implementations for string indexing
(see [CharReq], section 4.6).
It increases interoperability and predictability when string data is
exposed in an API.
4.2 Definitions for W3C Text
Normalization
4.2.1 Unicode-normalized Text
Text data is, for the purposes of this specification,
Unicode-normalized if it is in a Unicode encoding form and
is in Unicode Normalization Form C (according to revision 18 of [UTR #15]).
4.2.2 Fully Normalized Text
Text data is fully normalized if:
the data is Unicode-normalized and does not contain
any character escapes whose unescaping would cause the data to become no
longer Unicode-normalized; or
the data is in a legacy encoding and, after
transcoding to a Unicode encoding form by a normalizing transcoder, the data
satisfies clause 1 above.
In the remainder of this specification, normalized is used to
mean fully normalized, unless otherwise indicated.
Note: A consequence of this definition is that legacy text (i.e. text in a
legacy encoding) is always normalized unless i) a normalizing transcoder
cannot exist for that encoding (e.g. because the repertoire contains
characters not in Unicode) or ii) the text contains escapes which, once
expanded, result in un-normalized text.
Note: Full normalization is specified against the context of a markup
language (or the absence thereof), which specifies the form of escapes. For
plain text (no escapes) in a Unicode encoding form, full normalization and
Unicode-normalization are equivalent.
4.2.3 Examples
The string "suçon", expressed as the sequence of five characters U+0073
U+0075 U+00E7 U+006F U+006E and encoded in a Unicode encoding form, is both
Unicode-normalized and fully normalized. The same string encoded in a legacy
encoding for which there exists a normalizing transcoder would be fully
normalized but not Unicode-normalized (since not in a Unicode encoding
form).
In an XML or HTML context, the string "sucçon" is also both
fully normalized and, if encoded in a Unicode encoding form,
Unicode-normalized. Expanding "ç" yields "suçon" as above, which
contains no replaceable combining sequence.
The string "suçon", expressed as the sequence of six
characters U+0073 U+0075 U+0063 U+0327 U+006F U+006E (U+0327 is the
COMBINING CEDILLA) and encoded in a Unicode encoding form, is neither
Unicode-normalized (since the combining sequence U+0063 U+0327 is replaceable
by the precomposed U+00E7) nor fully normalized (since in a Unicode encoding
form but not Unicode-normalized).
In an XML or HTML context, the string "suçon" is not fully
normalized, regardless of encoding form, because expanding "̧"
yields the sequence "suc¸on" which is not Unicode-normalized (c¸ is
replaceable by ç). Unicode-normalization, however, is defined only for plain
text, doesn't know that ̧ represents a character in XML or HTML
and considers it just a sequence of characters. Therefore, the string
"suçon" in a Unicode encoding form is Unicode-normalized
since it contains no replaceable combining sequence. (The latter example does
not imply that Unicode-normalization is sufficient to meet the normalization
requirements of the Web; it just illustrates a case where
Unicode-normalization and full normalization differ).
The string "<elem>/ foobar</elem>", where the '/' immediately
after '<elem>' stands for the character U+0338 COMBINING LONG SOLIDUS
OVERLAY, is neither Unicode-normalized nor fully normalized, since the U+0338
/ combines with the '>' (yielding U+226F NOT GREATER-THAN).
Note: From this example, it follows that it is impossible to produce a
normalized XML or HTML document containing the character U+0338 COMBINING
LONG SOLIDUS OVERLAY immediately following an element tag, comment, CDATA
section or processing instruction. It is noteworthy that U+0338 COMBINING
LONG SOLIDUS OVERLAY also combines with '<', yielding U+226E NOT
LESS-THAN. Consequently, U+0338 COMBINING LONG SOLIDUS OVERLAY should remain
excluded from XML identifiers.
4.3 Responsibility for
Normalization
[I] Producers MUST
produce text data in normalized form, unless they are willing to accept the
consequences (loss of integrity and security, high probability of rejection
by recipients) of un-normalized data. For the purpose of W3C
specifications and their implementations, the producer of text data is the
sender of the data in the case of protocols and the tool that produces the
data in the case of formats.
Note: As an optimization, it is perfectly acceptable for a system
to define the producer to be the actual producer (e.g. a small device)
together with a remote component (e.g. a server serving as a kind of proxy)
to which normalization is delegated. In such a case, the communications
channel between the device and proxy server is considered to be
internal to the system, not part of the Web. Only data normalized by
the proxy server is to be exposed to the Web at large.
Note: Normalization is the responsibility of the producer as a whole. This
specification does not assign responsibility for normalization to any
particular component of the producer (for instance a DOM implementation).
Note: Implementers of producer software are encouraged to delegate
normalization to their respective data sources wherever possible. Examples of
data sources are operating systems, libraries, and keyboard drivers.
[I] The recipients
of text data MUST verify the normalization of data they receive and reject
un-normalized data, unless they are willing to accept the consequences (loss
of integrity and security) of un-normalized data. [I] Recipients MUST NOT
normalize the data that they receive. [I] Recipients which transcode text data from
a legacy encoding to a Unicode encoding form MUST use a normalizing transcoder.
Note: The prohibition of normalization by recipients is necessary to avoid
the security issues mentionned in section 4.1.
[I] When a
recipient returns un-normalized text to a sender (e.g. to indicate an error
or fault), that recipient MAY return the text without normalizing
it.
[I] If a software
module functions as both a producer and a recipient of text data (e.g. a
browser/editor), normalization MUST be applied in the producer part but MUST
NOT be applied in the recipient part.
[I] Intermediate
(recipient/producer) components whose role involves modification of text data
MUST ensure that their modifications do not result in denormalization of any
data exposed (sent on the network, saved to disk, returned in an API call,
etc.).
Note: Consequently, an intermediate component, in a system that packages a
payload in some control information, may modify the control
information without having to renormalize the payload.
[I] In addition,
software MUST behave as if normalization took place after
each modification, so that any subsequent matching, indexing or other
normalization-sensitive operations always behave as if they
were dealing with normalized data.
Example: If the "z" is deleted from the (normalized) string "cz¸" (where
"¸" represents a combining cedilla, U+0327), normalization is necessary to
turn the denormalized result "c¸" into the properly normalized "ç". Analogous
cases exist for insertion and concatenation. If the software that deletes the
"z" later uses the string in a normalization-sensitive operation, it needs to
normalize the string before this operation to ensure correctness; otherwise,
normalization may be deferred until the data is exposed.
[I] Intermediate
components whose role does not involve modification of the data (e.g. caching
proxies) MUST NOT reject un-normalized data and MUST NOT perform
normalization.
[S] In
specifications of markup languages, syntax-significant characters MUST be
chosen that do not combine with any other characters in NFC. This is
to avoid problems such as U+0338 COMBINING LONG SOLIDUS OVERLAY combining
with the '<' and '>' delimiters in XML (see the last example in section
4.2.3 Examples above.
5 Compatibility and Formatting
Characters
This specification does not address the suitability of particular
characters for use in markup languages, in particular formatting characters
and compatibility equivalents. For detailed recommendations about the use of
compatibility and formatting characters, see [UXML].
[S] Specifications
SHOULD exclude compatibility characters in the syntactic elements (markup,
delimiters, identifiers) of the formats they define.
6 String Identity Matching
One important operation that depends on early normalization is string
identity matching [CharReq], which is a subset of
the more general problem of string matching. There are various degrees of
specificity for string matching, from approximate matching such as regular
expressions or phonetic matching, to more specific matches such as
case-insensitive or accent-insensitive matching and finally to identity
matching. In the Web environment, where multiple encodings are used to
represent strings, including some encodings which allow multiple
representations for the same thing, identity is defined to
occur if and only if the compared strings contain no user-identifiable
distinctions. This definition allows strings to match when they differ only
in their encodings (including escapes) or when strings in the same encoding
differ only by their use of precomposed and decomposed character sequences,
yet does not make equivalent strings that differ in case or accentuation.
To avoid unnecessary conversions and, more importantly, to ensure
predictability and correctness, it is necessary for all components of the Web
to use the same identity testing mechanism. [S][I] To meet this requirement and support
the above definition of identity, string identity matching MUST be performed
as if the following steps were followed:
[S][I] Early
uniform normalization to fully normalized form, as defined in 4.2.2 Fully Normalized Text
[S][I]
Conversion to a common encoding of UCS, if necessary
[S][I]
Expansion of all escapes
[S][I]
Binary comparison
[S][I] In
accordance with section 4 Early Uniform
Normalization, the first step MUST be performed by the
producers of the strings to be compared. This ensures
1) that the identity matching process can produce correct results using the
next three steps and 2) that a minimum of effort is spent on solving the
problem.
[S][I] Forms of
string matching other than identity SHOULD be based on the above
steps. Taking into account normalization and escapes is necessary so
that, for example, a case-insensitive match of 'suçon' against
'SUÇON' or against 'SUC¸ON' returns TRUE.
Note: The expansion of escapes (step 3 above) is dependent on context,
i.e. on which markup or programming language is considered to apply when the
string matching operation is performed. Consider a search for the string
"suçon" in an XML document containing "suçon" but not "suçon". If
the search is performed in a plain text editor, the context is plain
text (no markup or programming language applies), the ç escape
is not recognized, hence not expanded and the search fails. If the search is
performed in an XML browser, the context is XML, the escape (defined
by XML) is expanded and the search succeeds.
An intermediate case would be an XML editor that purposefully
provides a view of an XML document with entity references left unexpanded. In
that case, a search over that pseudo-XML view will deliberately not
expand entities: in that particular context, entity references are not
considered escapes and need not be expanded.
7 String Indexing
There are many situations where a software process needs to access a
substring or to point within a string and does so by the use of indices, i.e.
numeric "positions" within a string. Where such indices are exchanged between
components of the Web, there is a need for an agreed-upon definition of
string indexing in order to ensure consistent behavior. The requirements for
string indexing are discussed in [CharReq], section 4. The two main
questions that arise are: "What is the unit of counting?" and "Do we start
counting at 0 or 1?".
Depending on the particular requirements of a process, the unit of
counting may correspond to any of the definitions of a string provided in
section 3.4 Strings (which all follow the pattern
"a <foo string> is a sequence of <unit>s"). In
particular:
- [S][I] Character string is RECOMMENDED.
(Example: [XPath])
- [S][I] Code unit string MAY be used if the
choice of character string would severely impact the efficiency of
internal operations. (Example: use of UTF-16 in [DOM Level 1]).
- Counting graphemes will become a
good option where user interaction is the primary concern, once a
suitable definition is widely accepted.
It is noteworthy that there exist other, non-numeric ways of identifying
substrings which have favorable properties. For instance, substrings based on
string matching are quite robust against small edits; substrings based on
document structure (in structured formats such as XML) are even more robust
against edits and even against translation of a document from one human
language to another. [S] Consequently, specifications that need a
way to identify substrings or point within a string SHOULD provide ways other
than string indexing to perform this operation. [I][C] Users of such
specifications (software developers, content developers) SHOULD prefer those
other ways whenever possible.
Experience shows that more general, flexible and robust specifications
result when individual characters are understood and processed as substrings,
identified by a position before and a position after the substring.
Understanding indices as boundary positions between the
counting units also makes it easier to relate the indices resulting from the
different string definitions. [S] Specifications SHOULD use this form of
indexing, regardless of the choice of counting units. [S] APIs in addition SHOULD
NOT specify single character or single encoding-unit arguments.
Example: uppercase('ß') cannot return the proper result
('SS') if the return type of the uppercase function is defined
to be a single character.
The issue of index origin, i.e. whether we count from 0 or 1, actually
arises only after a decision has been made on whether it is the units
themselves that are counted or the positions between the units. [S] With the latter,
starting with an index of 0 for the position at the start of the string is
the RECOMMENDED solution, with the last index then being equal to the number
of counting units in the string.
According to the definition in [RFC 2396], URI
references are restricted to a subset of US-ASCII. This RFC also specifies an
escaping mechanism to encode arbitrary byte values, using the %HH convention.
However, because the RFC does not define the mapping from characters to
bytes, the %HH convention by itself is of limited use. This chapter defines
how to address this issue in W3C specifications in a way consistent with the
model defined in this document and with deployed practice.
[I] W3C
specifications that define protocol or format elements (e.g. HTTP headers,
XML attributes,...) whose role is that they be interpreted as URI references
(or specific subsets of URI references, such as absolute URI references,
URIs,...) MUST allow these protocol or format elements to contain characters
disallowed by the URI syntax. The disallowed characters include all
non-ASCII characters, plus the excluded characters listed in Section 2.4 of
[RFC 2396], except for the number sign (#) and percent
sign (%) characters and the square bracket characters re-allowed in [RFC 2732].
When passing such protocol or format elements to software components that
cannot deal with anything else than characters legal in URI references, a
conversion is needed. [S] W3C specifications MUST define when this
conversion is to be made. [S] The conversion MUST take place as late as
possible, i.e. all characters should be preserved as long as possible.
[S] W3C
specifications MUST specify that conversion to a URI reference is carried out
as follows:
- [S] Each
disallowed character is converted to UTF-8, resulting in one or more
bytes.
- [S] The
resulting bytes are escaped using the URI escaping mechanism (that is,
each byte is converted to %HH, where HH is the byte value expressed
using hexadecimal notation).
- [S] The
original character is replaced by the resulting character
sequence.
Example: The string "http://www.w3.org/People/Dürst/" is not a legal URI
because the character "ü" is not allowed in URIs. The representation of "ü"
in UTF-8 consists of two bytes with the values 0xC3 and 0xBC. The string is
therefore converted to "http://www.w3.org/People/D%C3%BCrst/".
Note: [I-D URI-I18N] recently proposed the term
Internationalized Resource Identifiers (IRI). A future version of
this specification will adopt this term if it proves to be useful.
Request for feedback: There is currently a small but
serious difference between [I-D URI-I18N] and the
provisions above. US-ASCII characters which are not allowed in URIs (such as
space, backslash, and curly brackets) are also not allowed in IRIs, but are
allowed in protocol and format elements as defined above because they are
escaped (as described above) during the conversion to a legal URI. We are
requesting feedback on how to deal with this discrepancy.
Note: The intent of this chapter is not to limit URI references to a
subset of US-ASCII characters forever, but to ensure that W3C technology
correctly and predictably interacts with systems that are based on the
definition of URI references while taking advantage of the capabilities of
W3C technology to handle the UCS repertoire.
Note: The ability to use URI references with encodings other than UTF-8 is
not affected by this chapter. However, in such a case, the %HH-escaped form
has always to be used. As an example, if the HTTP server at www.w3.org used
only ISO-8859-1, the above string, when used in a protocol or format element
defined by a W3C specification, would always have to be given in the escaped
form, "http://www.w3.org/People/D%FCrst/" ('ü' in ISO-8859-1 is 0xFC).
[S] A W3C
specification that defines new syntax for URIs, such as a new kind of
fragment identifier, MUST specify that characters outside the US-ASCII
repertoire are encoded in URIs using UTF-8 and %HH-escaping. This
makes such new syntax fully interoperable with the above provisions.
Example: [XPointer] defines fragment identifiers
for XML documents. An XPointer to an element with the ID résumé
is xpointer(id('résumé')). The resulting fragment identifier is
xpointer(id('r%C3%A9sum%C3%A9')) as the escaped UTF-8 of 'é' is
%C3%A9. A URI reference to that element in a document called
doc.xml is doc.xml#xpointer(id('r%C3%A9sum%C3%A9')). Using
[XLink], and assuming the encoding of the XML document can represent 'é'
directly, this can be written as
xlink:href="doc.xml#xpointer(id('résumé'))". This looks very
convenient and obvious, but it should be noted that it is only possible due
to the fact that the encoding is aligned between representation (in XLink
href) and interpretation (in XPointer).
9 Referencing the Unicode Standard and
ISO/IEC 10646
Specifications often need to make references to the Unicode standard or
International Standard ISO/IEC 10646. Such references must be made with care,
especially when normative. The questions to be considered are:
- Which standard should be referenced?
- How to reference a particular version?
- When to use versioned vs unversioned references?
ISO/IEC 10646 is developed and published jointly by ISO (the International Organisation for
Standardisation) and IEC (the International
Electrotechnical Commission). The Unicode Standard is developed and published
by the Unicode Consortium, an
organization of major computer corporations, software producers, database
vendors, national governments, research institutions, international agencies,
various user groups, and interested individuals. The Unicode Standard is
comparable in standing to W3C Recommendations.
ISO/IEC 10646 and Unicode define exactly the same CCS (same repertoire,
same code points) and encoding forms. They are actively maintained in
synchrony by liaisons and overlapping membership between the respective
technical committees. In addition to the jointly defined CCS and encoding
forms, the Unicode Standard adds normative and informative lists of character
properties, normative character equivalence and normalization specifications,
a normative algorithm for bidirectional text and a large amount of useful
implementation information. In short, Unicode adds semantics to the
characters that ISO/IEC 10646 merely enumerates. Conformance to Unicode
implies conformance to ISO/IEC 10646, see [Unicode
3.0] Appendix C.
[S] Since
specifications in general need both a definition for their characters and the
semantics associated with these characters, specifications SHOULD include a
reference to the Unicode Standard, whether or not they include a reference to
ISO/IEC 10646. By providing a reference to The Unicode Standard
implementers can benefit from the wealth of information provided in the
standard and on the Unicode Consortium Web site.
The fact that both ISO/IEC 10646 and Unicode are evolving (in synchrony)
raises the issue of versioning: should a specification refer to a specific
version of the standard, or should it make a generic reference, so that the
normative reference is to the version current at the time of reading
the specification? In general the answer is both. [S] A generic reference
MUST be made if it is desired that characters allocated after a specification
is published are usable with that specification. [S] A specific reference
MAY be included to ensure that functionality depending on a particular
version is available and will not change over time (an example would be the
set of characters acceptable as Name characters in [XML
1.0], which is an enumerated list that parsers must implement to validate
names).
A generic reference can be formulated in two ways:
- By explicitly including a generic entry in the bibliography
section of a specification and simply referring to that entry in the body
of the specification. Such a generic entry contains text such as "...
as it may from time to time be revised or amended".
- By including a specific entry in the bibliography and adding
text such as "... as it may from time to time be revised or
amended" at the point of reference in the body of the
specification.
It is an editorial matter, best left to each specification, which of these
two formulations is used. Examples of the first formulation can be found in
the bibliography of this specification (see the entries for [ISO/IEC 10646] and [Unicode]).
Examples of the latter, as well as a discussion of the versioning issue with
respect to MIME charset parameters for UCS encodings, can be found in
[RFC 2279] and [RFC 2781].
[S] All
generic references to Unicode MUST refer to [Unicode 3.0] or later. [S] Generic references to ISO/IEC 10646 MUST
be written such that they make allowance for the future publication of
additional parts of the standard. They MUST refer to [ISO/IEC 10646-1:2000] or later, including any
amendments.
Appendix A: Examples of Characters,
Keystrokes and Glyphs
A few examples will help make sense of all this complexity (which is
mostly a reflection of the complexity of human writing systems). Let us start
with a very simple example: a user, equipped with a US-English keyboard,
types "Foo", which the computer encodes as 16-bit values (the UTF-16 encoding
of Unicode) and displays on the screen.
Example A.1: Basic Latin
| Keystrokes |
Shift-f |
o |
o |
| Input characters |
F |
o |
o |
| Encoded characters (byte
values in hex) |
0x0046 |
0x006F |
0x006F |
| Display |
Foo |
The only complexity here is the use of a modifier (Shift) to input the
capital F.
A slightly more complex example is a user typing "çé" on a French-Canadian
keyboard, which the computer again encodes in UTF-16 and displays. We assume
that this particular computer uses a fully composed form of UTF-16.
Example A.2: Latin with diacritics
| Keystrokes |
¸ c |
é |
| Input characters |
ç |
é |
| Encoded characters (byte
values in hex) |
0x00E7 |
0x00E9 |
| Display |
çé |
A few interesting things are happening here: when the user types the
cedilla (¸), nothing happens except for a change of state of the keyboard
driver; the cedilla is a dead key. When the driver gets the c, it
provides a complete ç character to the system, which represents it as a
single 16-bit code unit and displays a ç glyph. The user then presses the
dedicated é key, which results in, again, a character represented by two
bytes. Most systems will display this as one glyph, but it is also possible
to combine two glyphs (the base letter and the accent) to obtain the same
rendering.
On to a Japanese example: our user employs an input method to type "
", which the computer
encodes in UTF-16 and displays.
Example A.3: Japanese
| Keystrokes |
n i h o n g o <space><return> |
| Input characters |
 |
 |
 |
| Encoded characters (byte
values in hex) |
0x65E5 |
0x672C |
0x8A9E |
| Display |
|
The interesting aspect here is input: the user types Latin characters,
which are converted on the fly to kana (not shown here), and then to kanji
when the user requests conversion by pressing <space>; the kanji
characters are finally sent to the application when the user presses
<return>. The user has to type a total of nine keystrokes before the
three characters are produced, which are then encoded and displayed rather
trivially.
An Arabic example will show different phenomena:
Example A.4: Arabic
| Keystrokes |
 |
 |
 |
 |
|
| Input characters |
|
|
|
|
|
|
| Encoded characters (byte
values in hex) |
0x0644 |
0x0627 |
0x0644 |
0x0627 |
0x0639 |
0x0639 |
| Display |
 |
Here the first two keystrokes each produce an input character and an
encoded character, but the pair is displayed as a single glyph (, a lam-alif ligature).
The next keystroke is a lam-alif, which some Arabic keyboards have; it
produces the same two characters which are displayed similarly, but this
second lam-alif is placed to the left of the first one. The last two
keystrokes produce two identical characters which are rendered by two
different glyphs (a medial form followed to its left by a final form). We
thus have 5 keystrokes producing 6 characters and 4 glyphs laid out
right-to-left.
A final example in Tamil, typed with an ISCII
keyboard, will illustrate some additional phenomena:
Example A.5: Tamil
| Keystrokes |
 |
 |
 |
 |
 |
 |
| Input characters |
|
|
|
|
|
|
| Encoded characters (byte
values in hex) |
0x0B9F |
0x0BBE |
0x0B99 |
0x0BCD |
0x0B95 |
0x0BCB |
| Display |
 |
Here input is straightforward, but note that contrary to the preceding
accented Latin example, the diacritic (virama, vowel killer) is entered
after the to which
it applies. Rendering is interesting for the last two characters. The last
one () clearly consists of two
glyphs which surround the glyph of the next to last character ().
A number of operations routinely performed on text can be impacted by the
complexities of the world's writing systems. An example is the operation of
selecting text on screen by a pointing device in a bidirectional (bidi)
context. First, let's have some bidi text, in this case Arabic letters
(written right-to-left) mixed with Arabic-Hindi digits (left-to-right):
Example A.6: Bidirectional text
| In memory |
  <space>    <space>    |
| On screen |
 |
In the presence of bidi text, two possible selection modes must be
considered. The first is logical selection mode, which selects all the
characters logically located between the end-points of the user's
mouse gesture. Here the user selects from between the first and second
letters of the second word to the middle of the number. Logical selection
looks like this:
Example A.7: Example of logical selection
| In memory |
|
| On screen |
 |
It is a consequence of the bidirectionality of the text that a single,
continuous logical selection in memory results in a discontinuous selection
appearing on the screen. This discontinuity, as well as the somewhat
unintuitive behavior of the cursor, makes many users prefer a visual
selection mode, which selects all the characters visually
located between the end-points of the user's mouse gesture. With the same
mouse gesture as before, we now obtain:
Example A.8: Example of visual selection
| In memory |
|
| On screen |
 |
In this mode, popular with users, a single visual selection range results
in two logical ranges, which MUST be accommodated by protocols, APIs and
implementations.
Special thanks go to Ian Jacobs for ample help with editing. Tim
Berners-Lee and James Clark provided important details in the section on
URIs. The W3C I18N WG and IG, as well as others, provided many comments and
suggestions.
References
Normative References
- IANA
- Internet Assigned Numbers Authority, Official
Names for Character Sets. (See ftp://ftp.isi.edu/in-notes/iana/assignments/character-sets.)
- ISO/IEC 10646
- ISO/IEC 10646-1:2000, Information
technology -- Universal Multiple-Octet Coded Character Set (UCS) --
Part 1: Architecture and Basic Multilingual Plane, as, from
time to time, amended, replaced by a new edition or expanded by the
addition of new parts. (See http://www.iso.ch/ for the latest
version.)
- ISO/IEC 10646-1:2000
- ISO/IEC 10646-1:2000, Information
technology -- Universal Multiple-Octet Coded Character Set (UCS) --
Part 1: Architecture and Basic Multilingual Plane. (See http://www.iso.ch/iso/en/CatalogueDetailPage.CatalogueDetail?CSNUMBER=29819.)
- MIME
- Multipurpose
Internet Mail Extensions (MIME). Part One: Format of Internet Message
Bodies, N. Freed, N. Borenstein, RFC 2045, November 1996,
<http://www.ietf.org/rfc/rfc2045.txt>.
Part Two: Media Types, N. Freed, N. Borenstein, RFC 2046,
November 1996. Part Three: Message Header Extensions for
Non-ASCII Text, K. Moore, RFC 2047, November 1996. Part
Four: Registration Procedures, N. Freed, J. Klensin, J. Postel,
RFC 2048, November 1996. Part Five: Conformance Criteria and
Examples, N. Freed, N. Borenstein, RFC 2049, November 1996.
- RFC 2119
- S. Bradner, Key
words for use in RFCs to Indicate Requirement Levels, IETF
RFC 2119. (See http://www.ietf.org/rfc/rfc2119.txt.)
- RFC 2396
- T. Berners-Lee, R. Fielding, L. Masinter, Uniform Resource
Identifiers (URI): Generic Syntax, IETF RFC 2396, August
1998. (See http://www.ietf.org/rfc/rfc2396.txt.)
- RFC 2732
- R. Hinden, B. Carpenter, L. Masinter, Format for Literal
IPv6 Addresses in URL's, IETF RFC 2732, 1999. (See http://www.ietf.org/rfc/rfc2732.txt.)
- UTR #15
- Mark Davis, Martin Dürst, Unicode
Normalization Forms, Unicode Technical Report #15. (See http://www.unicode.org/unicode/reports/tr15/
for the latest revision). Revision
18 (November 1999) is at http://www.unicode.org/unicode/reports/tr15/tr15-18.html.
- Unicode
- The Unicode Consortium, The Unicode Standard -- Version
3.0, ISBN 0-201-61633-5, as updated from time to time by the
publication of new versions. (See http://www.unicode.org/unicode/standard/versions/
for the latest version and additional information on versions of the
standard and of the Unicode Character Database).
- Unicode 3.0
- The Unicode Consortium, The Unicode Standard -- Version
3.0, ISBN 0-201-61633-5. (See http://www.unicode.org/unicode/standard/versions/Unicode3.0.html.)
Other References
- CharReq
- Martin J. Dürst, Requirements for String
Identity and Character Indexing Definitions for the WWW, W3C
Working Draft. (See http://www.w3.org/TR/WD-charreq.)
- Connolly
- D. Connolly, Character
Set Considered Harmful, W3C Note. (See http://www.w3.org/MarkUp/html-spec/charset-harmful.)
- CSS2
- Bert Bos, Håkon Wium Lie, Chris Lilley, Ian Jacobs, Eds., Cascading Style Sheets,
level 2 (CSS2 Specification), W3C Recommendation. (See http://www.w3.org/TR/REC-CSS2.)
- DOM Level 1
- Vidur Apparao et al., Document Object
Model (DOM) Level 1 Specification, W3C Recommendation. (See
http://www.w3.org/TR/REC-DOM-Level-1/.)
- HTML 4.0
- Dave Raggett, Arnaud Le Hors, Ian Jacobs, Eds., HTML 4.0
Specification, W3C Recommendation, 18-Dec-1997 (See http://www.w3.org/TR/REC-html40-971218/.)
- HTML 4.01
- Dave Raggett, Arnaud Le Hors, Ian Jacobs, Eds., HTML 4.01
Specification, W3C Recommendation, 24-Dec-1999. (See http://www.w3.org/TR/html401/.)
- I-D URI-I18N
- Larry Masinter, Martin Dürst, Internationalized
Resource Identifiers (IRI), Internet-Draft, January 2001,
expired. (See http://www.w3.org/International/2001/01/draft-masinter-url-i18n-07.txt.)
- Info URI-I18N
- Internationalization:
URIs and other identifiers. (See http://www.w3.org/International/O-URL-and-ident.)
- ISO/IEC 9541-1
- ISO/IEC 9541-1:1991, Information
technology -- Font information interchange -- Part 1:
Architecture. (See http://www.iso.ch/iso/en/CatalogueDetailPage.CatalogueDetail?CSNUMBER=17277
for the latest version.)
- MathML2
- David Carlisle, Patrick Ion, Robert Miner, Nico Poppelier, Eds., Mathematical Markup Language
(MathML) Version 2.0, W3C Recommendation, 21 February 2001.
(See http://www.w3.org/TR/MathML2/.)
- Nicol
- Gavin Nicol, The
Multilingual World Wide Web, Chapter 2: The WWW As A
Multilingual Application. (See http://www.mind-to-mind.com/i18n/articles/multilingual/multilingual-www.html.)
- RFC 2070
- F. Yergeau, G. Nicol, G. Adams, M. Dürst, Internationalization
of the Hypertext Markup Language, IETF RFC 2070, January
1997. (See http://www.ietf.org/rfc/rfc2070.txt.)
- RFC 2277
- H. Alvestrand, IETF Policy on
Character Sets and Languages, IETF RFC 2277, BCP 18, January
1998. (See http://www.ietf.org/rfc/rfc2277.txt.)
- RFC 2279
- F. Yergeau, UTF-8, a
transformation format of ISO 10646, IETF RFC 2279, January
1998. (See http://www.ietf.org/rfc/rfc2279.txt.)
- RFC 2781
- P. Hoffman, F. Yergeau, UTF-16, an encoding of
ISO 10646, IETF RFC 2781, February 2000. (See http://www.ietf.org/rfc/rfc2781.txt.)
- SPREAD
- SPREAD
- Standardization Project for East Asian Documents Universal Public
Entity Set. (See http://www.ascc.net/xml/resource/entities/index.html)
- SVG
- Jon Ferraiolo, Ed., Scalable Vector Graphics (SVG)
1.0 Specification, W3C Recommendation, 4 September 2001.
(See http://www.w3.org/TR/SVG/.)
- UTR #17
- Ken Whistler, Mark Davis, Character
Encoding Model, Unicode Technical Report #17. (See http://www.unicode.org/unicode/reports/tr17/.)
- UXML
- Martin Dürst and Asmus Freytag, Unicode in XML and other
Markup Languages, Unicode Technical Report #20 and W3C Note.
(See http://www.w3.org/TR/unicode-xml.)
- XLink
- Steve DeRose, Eve Maler, David Orchard, Eds, XML Linking Language (XLink)
Version 1.0, W3C Recommendation, 27 June 2001. (See http://www.w3.org/TR/xlink/.)
- XML 1.0
- Tim Bray, Jean Paoli, C. M. Sperberg-McQueen, Eve Maler, Eds., Extensible Markup Language
(XML) 1.0, W3C Recommendation. (See http://www.w3.org/TR/REC-xml.)
- XML Japanese
profile
- MURATA Makoto Ed., XML Japanese
Profile, W3C Note. (See http://www.w3.org/TR/japanese-xml/.)
- XPath
- James Clark, Steve DeRose, Eds, XML Path Language (XPath)
Version 1.0, W3C Recommendation, 16 November 1999. (See http://www.w3.org/TR/xpath.)
- XPointer
- Steve DeRose, Eve Maler, Ron Daniel Jr., Eds, XML Pointer
Language (XPointer) Version 1.0, W3C Candidate
Recommendation, 11 September 2001. (See http://www.w3.org/TR/2001/CR-xptr-20010911/.)
Change Log (Non-Normative)
Normalization: changed from "recipients MUST NOT normalize" to "recipients
MUST check and reject un-normalized data".
Clarified conformance model, in particular introduced [S][I][C] specifiers
for requirements. Still needs some work.
Made numerous other changes listed in Character Model for
the World Wide Web 1.0 Last Call Comments (Members only).
Fixed countless typos and unclear/ambiguous sentences.
Updated references.
<space> 
