TaatikNet: Sequence-to-Sequence Learning for Hebrew Transliteration | by Morris Alper | Jun, 2023

A easy demonstration of character-level seq2seq studying utilized to a posh process: changing between Hebrew textual content and Latin transliteration

This text describes TaatikNet and tips on how to simply implement seq2seq fashions. For code and documentation, see the TaatikNet GitHub repo. For an interactive demo, see TaatikNet on HF Spaces.

Many duties of curiosity in NLP contain changing between texts in numerous types, languages, or codecs:

• Machine translation (e.g. English to German)
• Textual content summarization and paraphrasing (e.g. lengthy textual content to quick textual content)
• Spelling correction
• Abstractive query answering (enter: context and query, output: textual content of reply)

Such duties are identified collectively as Sequence-to-Sequence (Seq2seq) Studying. In all of those duties, the enter and desired output are strings, which can be of various lengths and that are normally not in one-to-one correspondence with one another.

Suppose you’ve a dataset of paired examples (e.g. lists of sentences and their translations, many examples of misspelled and corrected texts, and many others.). These days, it’s pretty simple to coach a neural community on these so long as there may be sufficient information in order that the mannequin might study to generalize to new inputs. Let’s check out tips on how to practice seq2seq fashions with minimal effort, utilizing PyTorch and the Hugging Face transformers library.

We’ll deal with a very fascinating use case: studying to convert between Hebrew textual content and Latin transliteration. We’ll give an summary of this process under, however the concepts and code introduced listed below are helpful past this explicit case — this tutorial needs to be helpful for anybody who needs to carry out seq2seq studying from a dataset of examples.

In an effort to show seq2seq studying with an fascinating and pretty novel use case, we apply it to transliteration. Usually, transliteration refers to changing between completely different scripts. Whereas English is written with the Latin script (“ABC…”), the world’s languages use many various writing programs, as illustrated under:

What if we wish to use the Latin alphabet to put in writing out a phrase from a language initially written in a distinct script? This problem is illustrated by the numerous methods to put in writing the title of the Jewish vacation of Hanukkah. The present introduction to its Wikipedia article reads:

Hanukkah (/ˈhɑːnəkə/; Hebrew: חֲנֻכָּה‎, Modern: Ḥanukka, Tiberian: Ḥănukkā) is a Jewish festival commemorating the restoration of Jerusalem and subsequent rededication of the Second Temple initially of the Maccabean Revolt towards the Seleucid Empire within the 2nd century BCE.

The Hebrew phrase חֲנֻכָּה‎ could also be transliterated in Latin script as Hanukkah, Chanukah, Chanukkah, Ḥanukka, or considered one of many different variants. In Hebrew in addition to in lots of different writing programs, there are numerous conventions and ambiguities that make transliteration advanced and never a easy one-to-one mapping between characters.

Within the case of Hebrew, it’s largely potential to transliterate textual content with nikkud (vowel indicators) into Latin characters utilizing a posh algorithm, although there are numerous edge instances that make this deceptively advanced. Moreover, trying to transliterate textual content with out vowel indicators or to carry out the reverse mapping (e.g. Chanukah → חֲנֻכָּה) is far more tough since there are various potential legitimate outputs.

Fortunately, with deep studying utilized to current information, we will make nice headway on fixing this downside with solely a minimal quantity of code. Let’s see how we will practice a seq2seq mannequin — TaatikNet — to discover ways to convert between Hebrew textual content and Latin transliteration by itself. We notice that this can be a character-level process because it includes reasoning on the correlations between completely different characters in Hebrew textual content and transliterations. We are going to talk about the importance of this in additional element under.

As an apart, you’ll have heard of UNIKUD, our mannequin for including vowel factors to unvocalized Hebrew textual content. There are some similarities between these duties, however the important thing distinction is that UNIKUD carried out character-level classification, the place for every character we realized whether or not to insert a number of vowel symbols adjoining to it. Against this, in our case the enter and output texts might not precisely correspond in size or order as a result of advanced nature of transliteration, which is why we use seq2seq studying right here (and never simply per-character classification).

As with most machine studying duties, we’re lucky if we will accumulate many examples of inputs and desired outputs of our mannequin, in order that we might practice it utilizing supervised studying.

For a lot of duties relating to phrases and phrases, an awesome useful resource is Wiktionary and its multilingual counterparts — assume Wikipedia meets dictionary. Particularly, the Hebrew Wiktionary (ויקימילון) accommodates entries with structured grammatical info as proven under:

Particularly, this consists of Latin transliteration (agvaniya, the place the daring signifies stress). Together with part titles containing nikkud (vowel characters), this provides us the (freely-licensed) information that we have to practice our mannequin.

In an effort to create a dataset, we scrape this stuff utilizing the Wikimedia REST API (example here). Please notice that unique texts in Wiktionary entries have permissive licenses for spinoff works (CC and GNU licenses, details here) and require share-alike licensing (TaatikNet license here); usually, if you happen to carry out information scraping just remember to are utilizing permissively licensed information, scraping appropriately, and utilizing the right license to your spinoff work.

We carry out varied preprocessing steps on this information, together with:

• Eradicating Wiki markup and metadata
• Changing bolding to signify stress with acute accents (e.g. agvaniya → agvaniyá)
• Unicode NFC normalization to unify identically showing glyphs comparable to בּ (U+05D1 Hebrew Letter Wager + U+05BC Hebrew Level Dagesh or Mapiq) and בּ (U+FB31 Hebrew Letter Wager with Dagesh). You possibly can examine these your self by copy-pasting them into the Show Unicode Character tool. We additionally unify equally showing punctuation marks comparable to Hebrew geresh (׳) and apostrophe (‘).
• Splitting multiple-word expressions into particular person phrases.

After information scraping and preprocessing, we’re left with almost 15k word-transliteration pairs (csv file available here). A number of examples are proven under:

The transliterations are on no account constant or error-free; for instance, stress is inconsistently and sometimes incorrectly marked, and varied spelling conventions are used (e.g. ח might correspond to h, kh, or ch). Reasonably than trying to wash these, we’ll merely feed them on to the mannequin and have it make sense of them by itself.

Now that we’ve got our dataset, let’s get to the “meat” of our mission — coaching a seq2seq mannequin on our information. We name the ultimate mannequin TaatikNet after the Hebrew phrase תעתיק taatik which means “transliteration”. We are going to describe TaatikNet’s coaching on a excessive stage right here, however you’re extremely really useful to peruse the annotated training notebook. The coaching code itself is sort of quick and instructive.

To attain state-of-the-art outcomes on NLP duties, a standard paradigm is to take a pretrained transformer neural community and apply switch studying by persevering with to fine-tune it on a task-specific dataset. For seq2seq duties, essentially the most pure selection of base mannequin is an encoder-decoder (enc-dec) model. Frequent enc-dec fashions comparable to T5 and BART are wonderful for frequent seq2seq duties like textual content summarization, however as a result of they tokenize textual content (break up it into subword tokens, roughly phrases or chunks of phrases) these are much less applicable for our process which requires reasoning on the extent of particular person characters. For that reason, we use the tokenizer-free ByT5 enc-dec mannequin (paper, HF model page), which performs calculations on the extent of particular person bytes (roughly characters, however see Joel Spolsky’s excellent post on Unicode and character sets for a greater understanding of how Unicode glyphs map to bytes).

We first create a PyTorch Dataset object to encapsulate our coaching information. We may merely wrap the information from our dataset csv file with no modifications, however we add some random augmentations to make the mannequin’s coaching process extra fascinating:

def __getitem__(self, idx):
row = self.df.iloc[idx]
out = {}
if np.random.random() < 0.5:
out['input'] = row.phrase if np.random.random() < 0.2 else row.nikkud
out['target'] = row.transliteration
else:
out['input'] = randomly_remove_accent(row.transliteration, 0.5)
out['target'] = row.nikkud
return out

This augmentation teaches TaatikNet to just accept both Hebrew script or Latin script as enter and to calculate the corresponding matching output. We additionally randomly drop vowel indicators or accents to coach the mannequin to be sturdy to their absence. Usually, random augmentation is a pleasant trick when you want to your community to study to deal with varied forms of inputs with out calculating all potential inputs and outputs out of your dataset forward of time.

We load the bottom mannequin with the Hugging Face pipeline API utilizing a single line of code:

pipe = pipeline("text2text-generation", mannequin='google/byt5-small', device_map='auto')

After dealing with information collation and setting hyperparameters (variety of epochs, batch dimension, studying price) we practice our mannequin on our dataset and print out chosen outcomes after every epoch. The coaching loop is normal PyTorch, aside from the consider(…) perform which we outline elsewhere and which prints out the mannequin’s present predictions on varied inputs:

for i in trange(epochs):
pipe.mannequin.practice()
for B in tqdm(dl):
optimizer.zero_grad()
loss = pipe.mannequin(**B).loss
losses.append(loss.merchandise())
loss.backward()
optimizer.step()
consider(i + 1)

Evaluate some outcomes from early epochs and on the finish of coaching:

Epoch 0 earlier than coaching: kokoro => okoroo-oroa-oroa-oroa-oroa-oroa-oroa-oroa-oroa-oroa-oroa-oroa-oroa-oroa-oroa-oroa-oroa-oroa-oroa-o
Epoch 0 earlier than coaching: יִשְׂרָאֵל => אלאלאלאלאלאלאלאלאלאלאלאלאלאלאלאלאלאלאלאלאלאלאלאלא
Epoch 0 earlier than coaching: ajiliti => ajabiliti siti siti siti siti siti siti siti siti siti siti siti siti siti siti siti siti siti sit
Epoch 1: kokoro => מְשִׁית
Epoch 1: יִשְׂרָאֵל => mará
Epoch 1: ajiliti => מְשִׁית
Epoch 2: kokoro => כּוֹקוֹרְבּוֹרוֹר
Epoch 2: יִשְׂרָאֵל => yishishál
Epoch 2: ajiliti => אַדִּיטִי
Epoch 5: kokoro => קוֹקוֹרוֹ
Epoch 5: יִשְׂרָאֵל => yisraél
Epoch 5: ajiliti => אֲגִילִיטִי
Epoch 10 after coaching: kokoro => קוֹקוֹרוֹ
Epoch 10 after coaching: יִשְׂרָאֵל => yisraél
Epoch 10 after coaching: ajiliti => אָגִ'ילִיטִי

Earlier than coaching the mannequin outputs gibberish, as anticipated. Throughout coaching we see that the mannequin first learns tips on how to assemble valid-looking Hebrew and transliterations, however takes longer to study the connection between them. It additionally takes longer to study uncommon gadgets comparable to ג׳ (gimel + geresh) similar to j.

A caveat: We didn’t try and optimize the coaching process; the hyperparameters had been chosen relatively arbitrarily, and we didn’t put aside validation or check units for rigorous analysis. The aim of this was solely to supply a easy instance of seq2seq coaching and a proof of idea of studying transliterations; nonetheless, hyperparameter tuning and rigorous analysis could be a promising route for future work together with the factors talked about within the limitations part under.

A number of examples are proven under, demonstrating conversion between Hebrew textual content (with or with out vowels) and Latin transliteration, in each instructions. You could strive taking part in with TaatikNet your self on the interactive demo on HF Spaces. Notice that it use§qs beam search (5 beams) for decoding and inference is run on every phrase individually.

For the sake of simplicity we carried out TaatikNet as a minimal seq2seq mannequin with out in depth tuning. Nonetheless, in case you are fascinated by enhancing outcomes on conversion between Hebrew textual content and transliteration, there are various promising instructions for future work:

• TaatikNet solely tries to guess the suitable spelling (in Hebrew or Latin transliteration) primarily based on letter or sound correspondences. Nonetheless, you may wish to convert from transliteration to legitimate Hebrew textual content given the context (e.g. zot dugma → זאת דוגמא relatively than the incorrectly spelled *זות דוגמע). Attainable methods to perform this might embody retrieval augmented technology (accessing a dictionary) or coaching on pairs of Hebrew sentences and their Latin transliterations to be able to study contextual cues.
• Unusually fashioned inputs might trigger TaatikNet’s decoding to get caught in a loop, e.g. drapapap → דְּרַפָּפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּאפָּ. This could be dealt with by augmentation throughout coaching, extra various coaching information, or utilizing cycle consistency in coaching or decoding.
• TaatikNet might not deal with some conventions which are fairly uncommon in its coaching information. For instance, it typically doesn’t correctly deal with ז׳ (zayin+geresh) which signifies the uncommon international sound zh. This may point out underfitting or that it will be useful to make use of pattern weights throughout coaching to emphasise tough examples
• The convenience of seq2seq coaching comes at the price of interpretability and robustness — we would wish to know precisely how TaatikNet makes its choices, and be sure that they’re utilized persistently. An fascinating potential extension could be distilling its data right into a set of rule-based situations (e.g. if character X is seen in context Y, then write Z). Maybe latest code-pretrained LLMs might be useful for this.
• We don’t deal with “full spelling” and “defective spelling” (כתיב מלא / חסר), whereby Hebrew phrases are spelled barely in another way when written with or with out vowel indicators. Ideally, the mannequin could be skilled on “full” spellings with out vowels and “faulty” spellings with vowels. See UNIKUD for one strategy to dealing with these spellings in fashions skilled on Hebrew textual content.

For those who strive these or different concepts and discover that they result in an enchancment, I’d be very fascinated by listening to from you, and crediting you right here — be at liberty to achieve out by way of my contact data under this text.

We now have seen that it’s fairly simple to coach a seq2seq mannequin with supervised studying — educating it to generalize from a big set of paired examples. In our case, we used a character-level mannequin (TaatikNet, fine-tuned from the bottom ByT5 mannequin), however almost the identical process and code might be used for a extra normal seq2seq process comparable to machine translation.

I hope you’ve realized as a lot from this tutorial as I did from placing it collectively! Be at liberty to contact me with any questions, feedback, or strategies; my contact info could also be discovered at my web site, linked under.