New Transliteration Test Files

The Test_*.html files show the transliteration of characters for given languages. The sample for each language consists of "What Is Unicode" in Thai, followed by other available text. The text is broken apart into sentences for ease of viewing (note: we know of some problems with the sentence rules for Japanese and Chinese). The left column is the original, and the right is the romanization. The program also converts back to the original script. If there is a discrepancy between the source and the reverse transformation, that is indicated by making the background red from that point on.

Note: If you have some more text that you would like added to the sample, just let me know. I am particularly interested in name lists, since they are the typical source.

Standards

The goal is to follow a given standard, such as ISO* or UNGEGN wherever possible. We also need to round-trip, so in some cases, that means adding some additional accent marks to disambiguate characters. And often the source standards are missing some characters, such as characters with combining Hamzas in Arabic. Remember that the goal for these is transliteration (unambiguously representing all the letters in the original), not transcription (representing the best pronunciation).

• Thai: ISO 11940 < http://homepage.mac.com/sirbinks/pdf/Thai.r2.pdf > plus a few items:
  • Accents may be added to the Latin for disambiguation.
  • In the next release, we'd like to do the UNGEGN version < http://www.eki.ee/wgrs/rom1_th.pdf > which is probably more useful (and readable), and follows more closely the Thai standard.
  • Spaces are provided at word-breaks, using the Thai BreakIterator.
  • An inherent vowel (ọ) is added, as in UNGEGN. The dot is for disambiguation.
    • Note: if the inherent vowel positions cannot be algorithmically determined, let me know and I will remove them.
• Arabic: Generally follows UNGEGN < http://www.eki.ee/wgrs/rom1_ar.pdf >
  • Accents may be added to the Latin for disambiguation.
  • Occasionally deviates in the direction of ISO 233 < http://homepage.mac.com/sirbinks/pdf/Arabic.pdf >
    • with underdot instead of cedilla for letter like SAD, since those are explicitly in Unicode for transliteration of Arabic
    • adding extra non-Arabic-language letters, like PEH. Note: not all extended Arabic characters are handled yet.
  • Does not do assimilation of "al", nor hyphenation of it.
    • While it could be done, we need to determine whether a prefix "al" could occur other than as the definite article (since no space is used).
  • This is transliteration. For transcription one would want an engine that added points appropriately to the Hebrew.
• Hebrew: Generally follows UNGEGN < http://www.eki.ee/wgrs/rom1_he.pdf >, with some exceptions:
  • Accents may be added to the Latin for disambiguation.
  • Combinations of dagesh, shin/sin dot that would produce different letters are not yet called out.
  • Note that the final forms are not preserved. Thus, when going from Latin to Hebrew, a character is given final form depending on its position.
    • E.g. מםמם => mmmm => מממם
  • This is transliteration. For transcription one would want an engine that added points appropriately to the Hebrew.
  • See also < http://homepage.mac.com/sirbinks/pdf/Hebrew.r1.pdf > for the ISO version. The Chicago Manual of Style has a clear table of mappings for the vowel marks.
• Han: Uses the CEDICT data plus Unicode Unihan kMandarin values for pinyin. Doesn't roundtrip!
  • Note: the Chinese pronunciation of Han characters varies by context and grammar, though nowhere near as much a Japanese.
    • Ideally we'd have an underlying engine for this. In 2.4 we will have a plug-in interface so that people could add one, such as the IBM engine.
    • The data from CEDICT and Unihan don't list the most frequent choice first, so we will be updating that.
• Greek/UNGEGN: Uses a modern Greek transliteration, based on the UNGEGN rules at < http://www.eki.ee/wgrs/rom1_el.pdf >. This version will not roundtrip ancient Greek.
• Greek: Uses a classic Greek transliteration. This version will not roundtrip modern Greek.

Notes

1. For readability, the files have a few other things besides just the transliteration:
   • The first word of the sentences are titlecased, as are names (where we have a name-list, such as in Thai).
   • The Latin in the original is mapped to the private-use zone before conversion, and then again after conversion. This does have the downside that any rules (such as in Han) that need to know the context (e.g. for inserting spaces or capitalization) will gum up a little bit. This is just an artifact of the test display.
2. I don't think that ISO 11940 is a particularly good way to romanize, but it is at least complete and a standard. So what I am interested in just for now is whether the samples in the file follow it (with the above exceptions).
3. Some of the files also have a set of characters at the end, one character per row, with a following row listing the hex and name.
4. The source rules for all of these is in the following URL. So if you want to know the details of how the characters are handled, that is the place to look.
   • http://source.icu-project.org/repos/icu/icu/trunk/source/data/translit/