Mandarin, Japanese and Korean: three scripts, three different problems

Lector now reads Mandarin, Japanese and Korean. Between them, those three account for almost every learner who studies an East Asian language.

I expected this to be simple. Add the hard one first (Mandarin), then the next two would be a piece of cake. Unfortunately, I was mistaken. Each language broke the reader in a unique way.

The shared problem, and where it stops being shared

The core feature of the product is the reading loop. Given the text on screen, find the word the reader tapped, then find that word in the dictionary. English makes this easy because words are delimited by spaces, and the spelling of the word matches the dictionary.

Mandarin breaks the first step, Korean breaks the second, and Japanese breaks both

Mandarin: find the words

Chinese writes no spaces. 我喜欢读书 is five characters and three words, but a reader who taps 喜 wants 喜欢. So, the reader segments each sentence before you tap anything. The words come from the browser's own Unicode segmenter rather than a bundled model.

The second Mandarin problem is that a character does not tell you how it sounds. So pinyin sits above every word while you read, and each reading retires once you mark that word known. The annotation thins out as you improve, which is the whole point.

The dictionary took two rounds to get right. Entries are keyed on Simplified, and the build took that key from the dump's own Simplified rows. That sounds reasonable, but it's not. A row tagged alternative gives the Simplified spelling of a different character rather than of the headword.

今 lists 当 as an alternative, so 今 claimed the key 当. 8,358 keys were claimed that way, and the honest entries were displaced. 这 answered chī.

A second bug was subtler. A Han character carries several readings, and the build kept whichever it met first, so 的 shipped as di. 的 is the most frequent character in Chinese, which means the reader printed a wrong reading above almost every sentence. Both are fixed, and the build now fails if any of 26 named words comes out with the wrong pronunciation. Coverage was healthy through both bugs, which highlights the importance of that integrity check.

Japanese: the words are not in the dictionary

Japanese writes no spaces either, so I pointed the same segmenter at it and most of the language held up. Nouns, adjectives, katakana and numerals all stay whole. 日本語, 図書館, 毎日 and コーヒー are each one token, and every particle is separate, so a tap on 日本語 does not return 日本語を.

Conjugated verbs snapped.

The segmenter knows where a noun ends and does not model verb morphology, so it cut the kanji stem away from its okurigana. Neither half is a word by itself. No dictionary entry matches 読 or んで, so tapping a verb answered nothing and the furigana above it was blank.

Japanese now runs through a real morphological analyser instead, with the IPADIC dictionary. 読んでいました stays whole. The analyser loads once per session, takes about a quarter of a second, and adds 17 MB to the image.

It also solved a problem I had been about to solve badly. A Japanese reading depends on the sentence around it. 本 is ほん in 本を読む and もと in 本を正す. A dictionary holds one reading per headword, so no dictionary can answer that question, and it holds no headword for 読ん at all. The analyser reads the word in place, so the furigana follows the context rather than the entry.

I tried to avoid the analyser first. I built a ranking that picked a single character's most likely reading from how it behaves in compounds. It was confidently wrong. It chose すい for 水 over みず, and こく for 国 over くに. Both of those are the Sino-Japanese reading, where a learner reading a sentence wants the native one.

I threw it away. It is much easier to ship a plausible wrong answer in a language you cannot read. That is the argument for measuring instead of trusting your eye.

Korean: the words are in the dictionary, under a different spelling

Korean looked like the hard one and was the easy one, for a reason I only found by checking rather than assuming.

Korean writes spaces, so the segmenter has nothing to do. And the Wiktionary dump enumerates Korean conjugation: 먹다 lists 먹어요, 먹습니다, 먹었어요 and 먹었습니다 as its own forms. So the machinery that resolves katte to kat in Afrikaans resolves 먹었어요 to 먹다 with no new code at all. Japanese needed an analyser precisely because its dump does not do this.

What Korean does need is a peel. Korean attaches its grammar to the word with no space, and the dump enumerates none of that:

Note that the last row works differently from the others. A postposition attaches to a finished word, so peeling it leaves a key you can look up as it stands. A verb ending does not: 좋아하 is a stem and not a word, and the dictionary form is 좋아하다, so that peel has to put the 다 back.

Here is what each layer bought, measured against 5,000 real Korean word-with-particles units taken from Tatoeba:

That first row is the number that matters. Without the peel, half of every Korean page is untappable, and it is the half carrying the grammar.

The peel runs last. The exact spelling and the conjugation table both get their turn first, because Korean is full of words that are also particles. 보다 is the verb to see and also than. 나 is I and also or. 도 is a province and also too.

A real word has to win before anything comes off the end of it.

One more Korean detail, and it stays invisible until it is wrong. Hangul syllables carry no break opportunity of their own, so a browser breaks a Korean line anywhere it likes. 도서관에서 can land as 도서관에, with 서 alone on the next line. Korean has real break points at its spaces, so the reader now holds each unit together and breaks between them.

Korean needs no reading above the text, and that is a feature of the script rather than a gap in the pack. Hangul tells you how it sounds. It is the only writing system in wide use that was deliberately designed, and this is what the design bought.

What the three packs contain

Do not read that last row as a ranking. Coverage asks how much of a typical text the dictionary resolves. For Mandarin and Japanese the sample is a frequency list of words. For Korean it cannot be. The standard frequency list for Korean is a list of morphemes. Its most common entries are 이, 는, 을 and 하, and 하 is the stem of 하다 rather than a word anyone taps.

So Korean is scored against whole units of running text instead, particles attached. That is a harder test than the other two sit. All three numbers are honest, and they are not measured the same way.

The empty cells are real. Mandarin ships a cloze bank built from Tatoeba, so practice has sentences waiting on day one. Japanese and Korean do not have one yet, so their practice draws on the sentences in the lessons you read. That is the next job on both.

Where they still fall short

Japanese draws furigana above a conjugated verb and cannot yet define one. The analyser reports that 読んでいました is a form of 読む. The lookup does not use that answer yet, so the reading is right and the tap comes back empty. It is the first thing on the list.

Korean's remaining 6.5% is four separate problems rather than one long list. The biggest is an ending that fuses into the syllable before it. 될 is 되 plus ㄹ written as one character, and no amount of trimming the end of a string reaches 되다. The others are auxiliary compounds (도와줄게), contractions (그걸 for 그것을), and words the writer ran together without a space.

One of those four fails in a worse way than the rest, and it is worth naming. A fused ending sometimes lands on a real word rather than on nothing. 갈 in 밥을 먹으러 갈 때 is 가다 plus ㄹ, and the dictionary holds a different 갈 that means common reed. So the reader answers confidently and wrongly, which is worse for a learner than answering nothing at all. Ranking two valid answers needs a frequency signal that the Korean pack does not carry yet.

Mandarin segments with the browser's segmenter, and the cloze bank was built with jieba. The two disagree on some boundaries, so a tap sometimes selects the wrong word.

What I took away from it

The useful lesson was about order of operations. On Japanese I assumed the problem was the dictionary. I spent a day building a ranking to fix it, and the real problem was the segmentation upstream. On Korean I assumed I needed the same analyser, and I checked the dump first. The check took ten minutes and saved the whole job.

Every one of these packs is a filter over a public Wiktionary dump. Each filter came from counting what sits in the file, and not from what I know about the language. That is just as well, because I do not read any of the three.

For self-hosters, all three dictionaries run on your own instance, so a lookup never leaves the machine. Every pack is pinned by checksum and baked into the image at build time.

To ask for a language, use the form on the home page. For anything else, the Discord is open.

← Back to the blog