The best translator is a hybrid translator

Building a better translator

I've noticed while doing this LLM translation quality research that models will often by idiomatic in about 70% of their choices, but fail in at least one part of the sentence. With coherence now being a practically "solved" metric, and idiomaticity being the target, I wanted to see if I could combine the best models to produce a translation greater than the max of its parts.

I built that, and made it open source. It turns out that you can! While its coherence is slightly lower than its peers (more on that in a moment), it is the most idiomatic model while also being far more consistent, with a much lower standard deviation across all three metrics. It works by taking the top 3-4 models for a given language (based on this research), translating with them, then having a judge model (currently GPT-4.1) consider the strengths and weaknesses of each translation and merge them together in an idiomatic way.

Benchmark saturation and going beyond coherence and rating

This benchmark has clearly reached saturation. The difference between models is stuck within a narrow range, and it's reaching the point where increases to idiomaticity are negatively correlated with coherence, because beyond a certain point coherence becomes more of a proxy about how literal the translation is than how accurate it is.

I was recently discussing this with someone else who's benchmarking LLM translation, and he approached this by directly comparing translations against each other and applying a a Bradley-Terry model, which is conceptually similar to ELO. I think that that's the best way forward, for future evaluations.

Using hybrid translation in Nuenki

Hybrid translation is far too slow for the low-latency translation Nuenki does as you browse the web, so I'm going to keep it to the translation utility. Paying users also get additional access to the utility, should you find it useful and use it so much that the ratelimits become a problem.