3.2

Lexical Translations

LEXICAL STRUCTURE

3.2   Lexical Translations

A raw Unicode character stream is translated into a sequence of Java tokens, using

the following three lexical translation steps, which are applied in turn:

1. A translation of Unicode escapes ( 3.3) in the raw stream of Unicode charac 

ters to the corresponding Unicode character. A Unicode escape of the form

\uxxxx

, where

xxxx

 is a hexadecimal value, represents the Unicode character

whose encoding is

xxxx

. This translation step allows any Java program to be

expressed using only ASCII characters.

2. A translation of the Unicode stream resulting from step 1 into a stream of

input characters and line terminators ( 3.4).

3. A translation of the stream of input characters and line terminators resulting

from step 2 into a sequence of Java input elements ( 3.5) which, after white

space ( 3.6) and comments ( 3.7) are discarded, comprise the tokens ( 3.5)

that are the terminal symbols of the syntactic grammar ( 2.3) for Java.

Java always uses the longest possible translation at each step, even if the result

does not ultimately make a correct Java program, while another lexical translation

would. Thus the input characters

a  b

 are tokenized ( 3.5) as

a

,

,

b

, which is

not part of any grammatically correct Java program, even though the tokenization

a

,

,

,

b

 could be part of a grammatically correct Java program.

3.3   Unicode Escapes

Java implementations first recognize

Unicode escapes

 in their input, translating

the ASCII characters

\u

 followed by four hexadecimal digits to the Unicode char 

acter with the indicated hexadecimal value, and passing all other characters

unchanged. This translation step results in a sequence of Unicode input charac 

ters:

UnicodeInputCharacter:

UnicodeEscape

RawInputCharacter

UnicodeEscape:

\

UnicodeMarker HexDigit HexDigit HexDigit HexDigit

UnicodeMarker:

u

UnicodeMarker

 u

12