Pinyin query: 110 lines of trie code, word frequency boosting, atomic user dictionary persistence.

This is part 4/8 of the TUX IM Dev Log: From Zero to 0.1 series.

Part 4 · Pinyin Mode: trie, Word Frequency, User Dictionary

This post starts the technical deep dive. Pinyin query is the hottest code path in the entire input method — every keypress requires multiple trie lookups. I’ll cover three things: the trie data structure itself, the word frequency learning mechanism, and atomic user dictionary persistence.

trie: 110 Lines Supporting All Pinyin Queries

First, the code (tux_im/input/lexicon/_trie.py):

@dataclass
class _TrieNode:
    children: dict[str, _TrieNode] = field(default_factory=dict)
    entries: list[LexEntry] = field(default_factory=list)
    is_word: bool = False

class Trie:
    __slots__ = ("_root", "_size")

    def __init__(self) -> None:
        self._root = _TrieNode()
        self._size = 0

    def insert(self, code: str, word: str, freq: int = 0) -> None:
        code = code.lower().strip()
        if not code or not word:
            return
        node = self._root
        for ch in code:
            node = node.children.setdefault(ch, _TrieNode())
        if not node.is_word:
            self._size += 1
        node.entries.append(LexEntry(word=word, freq=freq, code=code))
        node.entries.sort(key=lambda e: (-e.freq, e.word))

The entire file is 110 lines, zero IO, zero logging, zero side effects. All 14 trie algorithm tests don’t need to mock the filesystem.

Why pure data structure is critical? Because trie is on the hot path — every keypress queries it. If trie mixed in IO, the entire hot path would be slowed down; mixed in logging, unit tests would be interrupted; mixed in state, tests would be non-repeatable. Pure = fast = testable = trustworthy.

For a buffer like nihao, it’s first segmented into ['ni3', 'hao3'] (via PinyinMode.segment()), then for each syllable, candidates are generated by looking up the trie and taking the top-k results, indexed by segment position.

Word Frequency Learning

TUX IM learns user preferences silently. Every time a user selects a candidate, the frequency of that word is incremented. This data is persisted atomically to prevent corruption.

Atomic User Dictionary Persistence

The user dictionary is saved to disk atomically using a write-to-temp-then-rename strategy, ensuring that a crash during write doesn’t corrupt the dictionary file.