My wife reads faster than books get translated into Bulgarian. Some of the ones she wants don’t exist in Bulgarian and probably never will, because the market is small and the translation doesn’t pay for itself.

So I tried machine translation. It came out unreadable: sentences that are grammatically correct and mean nothing, and dialogue punctuated the English way. My wife got to page thirty and gave up.

I started with DeepL, which everyone calls the best. It turned out to be all talk, and later in the test it came last of ten.

The same happened with qwen3.8-max. I only included it because it is supposedly the best model for translation. It came second to last and was the only one that dropped parts of the text. All talk again.

How I tested the AI translators

So I decided to run my own study and find out which AI literary translator really does the job best. I took a Robert E. Howard story of about 12,000 words that also has a published human translation, and gave it to eight language models and to DeepL. Every one of them got the same instructions, word for word.

For the scoring I picked four models from four different makers. Each has a relative among the translators and may go a little easy on it, so with four the bias gets diluted, and I can also see whether their scores agree. I set a few rules as well. The judges don’t know who translated what, and they score all the translations at once, so they measure on the same scale. I also checked whether the models had seen the published translation during training, because then they would simply be remembering it. I found no sign of that.

Which AI literary translator came out on top

Here is how they ranked. One judge gave only an order without points, so the chart shows three.

What the pipeline adds

Based on these results I also tried a translation pipeline, which adds extra processing around the plain translation.

I ran the four best models again, this time through the pipeline. gpt-6-astra and gemini-3.1-pro gain one or two points, while gemini-3.8-flash and deepseek-v4-pro lose one. Each model was scored only against its own versions, so compare it only with itself.

The human translation, and the judge as a translator

I also scored the published human translation. It came in below the best machines, and the main reason is completeness. The translator cuts on purpose: of 125 departures from the original, 66 are deliberate choices, and the scoring counts every one of them as a loss, even when the cut works. It also loses points on language. It keeps using the short definite article where Bulgarian grammar needs the full one: “кимериеца”, “вашия капитан”, “пътя ти”. There is wrong verb government too, such as “пусни ни борда”, and that one appears twice. About ten technical errors remain in the text, such as a line-break hyphen in the middle of a word (“удивле-ние”), doubled words and Latin letters inside Cyrillic words. These most likely come from scanning the text, not from the translation. For Bulgarian it scores 6 out of 10, against 9 for the best model.

The translation is by Ivan Zlatarski, made around 1991 for a publisher, with no editing or proofreading. Where it appeared in print is still unclear.

Finally I had one of the judges, Fable 5.1, translate the story too, plain and through the pipeline. Two outside models scored it together with all the other translations at once, so the scale was shared. It came right after gpt-6-astra, but I don’t count that as a win, because it was the only one that knew in advance what it would be judged on.

So which model translates fiction best

Overall, gpt-6-astra showed the best results in every round.

gemini-3.8-flash came second, and it is by far the fastest of the eight.

I also saw that a translation pipeline can change the result noticeably, in either direction depending on the model. If I set out to translate fiction myself, I would do it with a pipeline, tuned to the model I pick.

Here you can read the detailed article about the research: In Search of the Best A.I. Literary Translator – Detailed.