The question
We started with one question: does correcting Arabic text need a giant model? The best published result came from combining nine models. We wanted to know how much a single small model, running on an ordinary machine, could do.
We chose AraT5 as the base: an encoder–decoder model with 367 million parameters. We trained it on a single GPU with 6 GB of memory.
The data
There is not enough hand-corrected Arabic text. So we built our data from three sources, each teaching the model something different.
Injected errors
450,000 Wikipedia paragraphs into which we inserted spelling and grammar errors and stripped punctuation, automatically. This taught the basics.
Real errors
Reader comments from the QALB collection, corrected by human editors. This taught how people actually make mistakes.
Errors on request
About 67,000 examples written by a large language model with specific kinds of error: grammar, word choice, and fourteen dialects.
When we analysed the QALB corrections, a third were punctuation and a third were hamza and spelling. True grammar errors were under a quarter, but they are the hardest.
35%
Punctuation
34%
Hamza and spelling
22%
Grammar and word choice
The leak
Our first run on the official QALB benchmark gave 80.35. That would have put us beside the best published systems. It was too good to be true.
So we checked. The widely used copy of the data online had been re-split at random, which put official test sentences into its training portion. 83% of the development and test text was inside what we had trained on. The model was not correcting; it was remembering.
We found no published paper that mentions this leak. Anyone who trained on that copy and scored on the official QALB test got an inflated number.
We threw the number away. We removed from every training source any example sharing six consecutive words with the official tests, and started again.
Training from scratch
We went back to the original AraT5, before it had seen any of our data, and trained it for five hours. The first four on a broad mix of sources; the last hour at a low learning rate, weighted toward real text.
Word error rate
The error rate on the development set fell from 34.1 in the original text to 15.8. More important: the new model left correct paragraphs as they were.
5 h
Training on one GPU
22,817
Training steps
15.8
Final error rate
Making it smaller
Then we asked: how much of the model do we really need? The first finding was that about two thirds of its vocabulary is rarely used. Removing it took the model from 367 to 257 million parameters for a loss of only 0.3 points.
Then we removed layers and taught the smaller models to imitate the large one. The result is three models, the smallest of which beats an open model of one billion parameters.
Word error rate on 78 test paragraphs; lower is better.
What did not work
Not every idea worked. We publish the failures because knowing them saves others the trip.
Rules alone
We tried having a morphological analyser detect errors with fixed rules. It caught 48% of the errors, but it also flagged 53% of the correct words. We dropped the idea.
Agreement errors
The full model fixes only 20% of agreement errors in our experiment. A small model that predicts the grammatical change and then generates the word automatically reached 48%. Better, but far from the theoretical ceiling of 86%.
What we are testing now
Our current experiment: an encoder that reuses its own layers instead of carrying twelve different ones. Only four unique layers, with the sentence passed through them repeatedly.
39%
Of the encoder's parameters
7.6 vs 7.1
Error rate on a QALB sample
20.6 vs 15.8
Error rate on our own test
On QALB text it nearly matches the full model. On our own test, where dialect and grammar live, it is still clearly behind. That is what we are working on.
What we learned
Check your data before your model
The biggest jump in our score was an illusion. The most important day of the project was the day we found that out.
Much of the size does no work
We removed 110 million parameters from the model and lost only 0.3 points. What matters is what the model was trained on, not how many parameters it has.
Measure what you break
A checker that fixes ten errors and breaks one correct sentence is a checker nobody will trust.