AI literary translator ranking with the model names hidden

In Search of the Best A.I. Literary Translator – Detailed

My wife reads faster than books get translated into Bulgarian. Some of the ones she wants don’t exist in Bulgarian and probably never will, because the market is small and the translation doesn’t pay for itself.

So I tried machine translation. It came out unreadable: sentences that are grammatically correct and mean nothing, and dialogue punctuated the English way. My wife got to page thirty and gave up.

I started with DeepL, which everyone calls the best. It turned out to be all talk, and later in the test it came last of ten.

The same happened with qwen3.8-max. I only included it because it is supposedly the best model for translation. It came second to last and was the only one that dropped parts of the text. All talk again.

So I decided to run my own study and find out which AI literary translator really does the job best. In Bulgarian specifically, because a model that translates beautifully into Spanish isn’t necessarily good at Bulgarian, and the English-language rankings say nothing about it. Bulgarian has things English doesn’t. Gender follows the word the translator chose, so the same ship is correctly “she” if you called it a galley and “he” if you later called it a ship. The definite article sticks to the end of the word, there is a separate mood for retelling what you didn’t witness, and dialogue opens with a dash. At every one of those points the models differed.

What I wanted to find out

Which model, with what processing around it, can translate a work of fiction into Bulgarian well enough that someone finishes the book. I also wanted to know whether it is possible to measure at all who translates better.

How I tested the AI translators

I took Robert E. Howard’s story “Shadows in the Moonlight” from 1934. Howard died in 1936, so in the EU the text has been free since 2007. The story is 296 paragraphs and 12,079 words. As a yardstick I used an old published human translation: 305 paragraphs and 12,276 words, with every paragraph of the original matched to one in it.

The translators were eight language models: gpt-6-astra, claude-opus-5, gemini-3.1-pro, gemini-3.8-flash, deepseek-v4-pro, grok-4.6, kimi-k3 and qwen3.8-max. Next to them I put DeepL, which is not a language model and serves as the floor. All of them ran through the same translation setup, and every model got byte for byte the same task. In the first round they all translated with no extra steps.

I scored them three ways. First, mechanical checks over the whole text. Then a full read of all 296 paragraphs of every translation against the original, by the same reader, Fable 5.1, with serious, moderate and minor errors defined in advance. And finally a blind comparison: six passages from the ten translations, unlabelled and shuffled, in front of four judges from four different makers, Fable 5.1, Astra Pro, Gemini Pro and DeepSeek. Each of them has a relative among the translators, which is why there were four.

There were a few rules. The judges don’t know who translated what. They score all the translations at once, so they measure on the same scale. Their instructions also explain how gender works in Bulgarian, because without that almost every translation looked wrong. And I checked whether the models had seen the published translation during training.

What came out

Whether a model works depends on who answers the call

Most models are bought through a middleman, and behind one model name a different company can answer each call. For DeepSeek the catalogue lists 16 such companies, for Kimi 20. Behaviour changes with whichever company answers.

Through one of the providers, deepseek-v4-pro cut the translation off at every ceiling I tried: 6,000, 16,000 and 48,000 tokens. Directly through DeepSeek’s own API it translated the whole story without a single break. gemini-3.8-flash through Google AI Studio returned 19 empty responses and stopped at 20% of the book, while through Google Vertex it went all the way. My first thought was censorship, because blood gets spilled in the story. I repeated the same call nine times and the empty response never came back. It was a temporary fault at the provider.

One of the failures was mine: out of habit I set a ceiling of 32,000 tokens on DeepSeek, and the translation stopped exactly there. Since then I pin the provider explicitly and check it with a live call before I start the big book.

Counting can’t tell a good translation from a bad one

The mechanical checks count words, dialogue lines, dashes, quotation marks, very short paragraphs, and missing or repeated paragraphs. One solid thing came out of it: none of the nine dropped a whole paragraph. After that, counting stops helping.

Identical on the numbers, four tiers apart on reading

Six of the models come out practically identical: length between 0.948 and 0.975 of the original and 150 or 151 dialogue lines with a dash. kimi-k3 has excellent numbers and a text full of words that don’t exist in Bulgarian, and in the blind comparison it is among the last. Reading split those same six models into four clearly different levels. Counting only catches the obvious failures.

The full read produced these findings:

Translation Findings, count Severe, count Moderate, count Minor, count
gemini-3.1-pro 15 0 4 10
gpt-6-astra 22 0 6 15
grok-4.6 92 0 24 68
gemini-3.8-flash 57 1 14 42
deepseek-v4-pro 77 1 30 45
human translation 125 2 13 105
claude-opus-5 89 4 30 55
kimi-k3 70 8 53 9
qwen3.8-max 108 15 25 68
DeepL 146 16 79 46

The totals aren’t quite comparable, because each reader decides for itself what to note. The serious column is comparable, because serious was defined the same way for everyone. DeepL is last with 16, and almost all of them are words with two meanings. “Quarter!”, which in a fight is a plea for mercy, becomes “Четвърт!”, the fraction. “Painter”, the rope on a boat, becomes “Художникът”, the artist. In one paragraph the word is wrong, in the next it is right, because every sentence is translated on its own. qwen3.8-max drops text and the reader feels the gap. DeepL writes smooth Bulgarian that says something else, and an editor without the original in front of them won’t catch it. According to the reader, bringing the DeepL translation up to print costs 60 to 70% of a new translation.

Which AI literary translator came out best in the first round

Round one: ten candidates, three judges

One judge gave only an order without points, so the chart shows three. The first three places are the same for every judge: gpt-6-astra, gemini-3.8-flash and deepseek-v4-pro. Further down, opinions differ by up to three places.

I also checked how much each judge goes easy on its relative. Fable 5.1 doesn’t move claude-opus-5 at all, Astra Pro lifts gpt-6-astra by 0.33 places, DeepSeek lifts its own model by one place, and Gemini Pro lifts gemini-3.8-flash by 1.33 places and gives it 10 out of 10 on all six criteria, the only perfect score in the first round. The bias is real but small: if I remove each judge from the scoring of its own family, the top doesn’t move.

Slower doesn’t mean better

gemini-3.8-flash is the fastest, seven and a half minutes for the whole story, and it is second in the ranking. grok-4.6 works 10.5 times longer and comes in sixth or seventh. Waiting longer doesn’t buy a better translation.

What a translation pipeline adds

A plain translation is one pass through the text. A translation pipeline adds extra processing around that pass. I tried one on the four best models from the first round. The other four stopped here, because the same effort on a model four levels down answers nothing new.

Each model’s plain translation and its pipeline versions were scored side by side by one blind judge, so compare the points only within one model. gpt-6-astra goes up from 6 to 8 points, gemini-3.1-pro from 7 to 8. gemini-3.8-flash drops from 8 to 7, deepseek-v4-pro from 7 to 6. For both drops the judges say the same thing: the translation becomes more precise in detail and in gender, but pays for it with literalisms and words that don’t exist. So there is no general answer to whether you should switch a pipeline on. It depends on the model, and the only way to find out is to try both.

If you just add up the findings, the pipeline looks harmful: 90 before, 106 after. The breakdown shows the opposite. Moderate errors fall from 29 to 19, while minor remarks rise from 60 to 87, and that holds for each of the four models on its own. The pipeline trades errors of meaning for style remarks. Four out of four in the same direction happens by chance once in sixteen times, so it is a signal, but not yet proof.

The human translation came in below the best machines

The human translation, criterion by criterion

Across the different scorings the published translation sits between sixth and eighth place out of ten. Its biggest loss is completeness, 4 out of 10, the lowest score after qwen3.8-max, which really does drop text. The translator condenses on purpose. Of 125 departures from the original, 66 are deliberate choices, 35 are errors of meaning and 23 are language or typography. The criterion, though, counts every cut as a loss, even when it works. The human translation has both errors and style, and this scale only sees the errors.

Completeness isn’t the only reason. For Bulgarian the translation gets 6 out of 10, level with claude-opus-5 and below four of the machines: gpt-6-astra gets 9, gemini-3.8-flash gets 8. It keeps using the short definite article where Bulgarian grammar needs the full one. There is wrong verb government too, one case of it twice. About ten technical errors remain in the text, such as a line-break hyphen in the middle of a word, doubled words and Latin letters inside Cyrillic words. On gender it scores 8 out of 10, and the review finds one such slip: once Olivia turns masculine. These most likely come from scanning the text, not from the translation. The articles and the verb government, though, are in the translation itself.

The translation is by Ivan Zlatarski, made around 1991 for a publisher, with no editing or proofreading, as was common at the time. Where and how it appeared in print is still unclear. It is probably in the 2003 Bulgarian collection “Конан: Пътят на славата” (Elf), which credits a different translator. That explains both kinds of errors: the articles come from a fast translation, and the technical errors from scanning the book.

The models hadn’t seen the translation before

The published translation may have ended up in the data the models were trained on. Then they would simply be remembering it. I checked two ways. First, the longest common run of words: between a model and the human translation it averages 5.2 to 5.7 words, and between the models themselves 8.8. Then I gave the models a passage of the human translation and asked them to continue: the continuation resembles the human as much as it resembles their own translation. Nothing stands out. The stronger check, the same test on a book with no published Bulgarian translation, I skipped on purpose.

Longest wording a candidate shares with another text

The second round: the judge sits down to translate

After the first round, Fable 5.1 did all the reading and scoring. For two days it checked other people’s translations without handing in one of its own. So I had it translate the story too, with the same instructions as the other models. Like the four best models, it made a plain translation and versions through the pipeline, three texts in all.

Fable couldn’t judge its own translation, so two outside judges scored it, gemini-3.8-flash and deepseek-v4-pro. And all the translations came back into play. The ten from the first round plus Fable’s three, thirteen texts at once in front of each judge, on the same six passages. It doesn’t work any other way: an eight given without knowing there is a nine in the same room isn’t the same eight.

Round two: thirteen candidates, two outside judges

gpt-6-astra stays first. Fable is right behind it. But I don’t count it as a win, because the conditions favoured it. It knew what it would be judged on: its rules were written after the first round and answer exactly the errors the others were penalised for. And once it caught and fixed its own error by itself, a Latin “p” inside a Bulgarian word. This round shows what a well-tuned process adds. Which model translates better is a different question.

One of Fable’s pipeline versions came out byte for byte the same as its plain translation, because the extra processing changed nothing. That is how two identical texts ended up in the blind comparison without anyone planning it.

What I learned about measuring itself

The same reader doesn’t repeat its findings

Some pipeline versions differ from the plain translations by only a few paragraphs, and the same reader read both. In practice that is the same text read twice.

One reviewer, the same text, read twice

Only 68% of the paragraphs with a remark on the first read have a remark on the second. On average 14 findings per text appear or vanish in paragraphs the pipeline never touched. So on 296 paragraphs the finding count has an error of about 10 to 15. The difference between 15 and 22 findings doesn’t exist. The difference between 15 and 92 does.

The second round gave a cleaner experiment. Fable’s two identical texts went into the blind comparison under different letters. The judges saw them as two different candidates and gave them identical scores on all six criteria. So the disagreement comes from splitting the reading into several sittings, and when a judge sees everything at once, it is precise. Hence the rule: score everything in one call.

How I almost published an error that doesn’t exist

What one rule in the instruction was worth

The first scoring instruction checked gender the way an English speaker would. In English the ship and the galley are both “it”, so a translation that says “ще го огледаме” (masculine) about the ship and “ще я потопя” (feminine) about the galley looked confused. By that rule almost every translation had a gender error, and for a while I had it written down as a finding. There is no such error. “Галера” is feminine, “кораб” is masculine, so both are right. Seven of the ten readers noticed it on their own, and that is how I noticed it too. After I rewrote the rule, “almost all” shrank to one real gender error in the blind comparison. The instruction has to carry the rules of the language even when most readers know them, because the rest will fill the table with errors that aren’t there.

What this study doesn’t prove

  • Everything rests on one story of 12,079 words. The next step is a longer book with the top three models.
  • Most numbers after the first round come from a single reader, and that reader repeats only 68% of itself.
  • The pipeline result is four out of four in the same direction. It is a signal until a second book says otherwise.
  • Thirteen candidates on a ten-point scale squash the top. At one judge three of them share the ten.
  • The stronger check of whether the models know the human translation, with a book that has no published translation, hasn’t been done yet.

So which model translates fiction best

Overall, gpt-6-astra showed the best results in every round.

AI literary translator ranking: points out of 10

gemini-3.8-flash came second, and it is by far the fastest of the eight.

I also saw that a translation pipeline can change the result noticeably, in either direction depending on the model. If I set out to translate fiction myself, I would do it with a pipeline, tuned to the model I pick.

The short version of this article is here: In Search of the Best A.I. Literary Translator.

Leave a Reply

Your email address will not be published. Required fields are marked *.

*
*