| Model | Overall Score | Coherence | Idiomaticity | Accuracy | Latency (ms) | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean | IQR | P90 | Mean | IQR | P90 | Mean | IQR | P90 | Median | P90 | ||
| quasar-alpha | 9.12 | 8.66 | 1 | 7 | 8.37 | 1 | 7 | 8.98 | 1 | 8 | 832 | 1280 |
| gpt-4o-2024-08-06 | 8.91 | 8.75 | 1 | 7 | 8.05 | 2 | 6 | 8.90 | 1 | 8 | 803 | 1288 |
| deepl | 8.81 | 8.59 | 1 | 7 | 8.07 | 2 | 6 | 8.67 | 0 | 7 | 226 | 322 |
| claude-3-5-sonnet-20241022 | 8.77 | 8.53 | 1 | 7 | 8.38 | 1 | 7 | 8.91 | 0 | 8 | 1387 | 3112 |
| gemma-3-27b-it | 8.59 | 8.48 | 1 | 7 | 8.31 | 1 | 7 | 8.92 | 1 | 8 | 1115 | 1605 |
| llama-3.3-70b-versatile | 8.54 | 8.36 | 1 | 7 | 7.72 | 2 | 6 | 8.64 | 1 | 7 | 334 | 1687 |
| lingvanex | 8.46 | 8.69 | 2 | 7 | 7.39 | 3 | 4 | 8.52 | 1 | 6 | 206 | 270 |
| gemini-2.0-flash-exp | 8.26 | 8.66 | 1 | 7 | 8.24 | 1 | 6 | 9.00 | 1 | 8 | 514 | 702 |
| gemma2-9b-it | 7.91 | 7.87 | 2 | 6 | 7.71 | 2 | 5 | 8.44 | 2 | 7 | 407 | 489 |
| llama-3.1-8b-instant | 6.12 | 6.03 | 6 | 2 | 6.84 | 3 | 3 | 7.59 | 2 | 4 | 266 | 818 |
| mistral-small-latest | 5.25 | 6.54 | 5 | 2 | 7.17 | 3 | 4 | 7.46 | 2 | 3 | 603 | 1659 |
Quasar Alpha
A few days ago a mysterious "Quasar Alpha" model appeared on openrouter. It's described as a "cloaked model provided to the community to gather feedback". There's a lot of speculation as to what it is - the million-token context is typical to Gemini models, but there's also some speculation that it could be from OpenAI or Qwen. It'd be interesting to see if the difference in latency from different places could be used to work out where the datacentre is located. Either way, it's quite impressive.
An unusually large improvement
Nuenki needs to do a lot of language translation, quickly (to avoid noticeable latency when browsing), and at a high quality - learning from mistakes can do more harm than good. In previous blog posts I've compared various different models as they came out, generally with small improvements in performance. Quasar Alpha is completely different to those incremental improvements. It's at the top of the leaderboard for practically every metric, particularly idiomaticity, something that small models tend to struggle on. That implies that it's quite a large model, and yet it also has an impressive token speed.
Methodology
I went over it in more detail in the first post, but effectively:
- Coherence is calculated by translating from English -> target language -> English 3 times, then having 3 blinded LLMs rate how close the original text is to the new text
- Idiomaticity is calculated by translating from English to the target langauge, then having 3 other (blinded) LLMs rate how idiomatic the translation is.
- Accuracy is calculated by translating from English to the target langauge, then having 3 other (blinded) LLMs rate how accurate the translation is.
- This is repeated across a sample of ~40 sentences, and the overall score is weighted towards coherence and interquartile range as the most reliable metrics.
Conclusion
Quasar Alpha's performance is incredibly impressive. If its pricing is comparable to Sonnet, and it's either open-source or served by a reputable western company, it'll probably replace Sonnet as Nuenki's tertiary translation method.
The latency and latency distribution is, interestingly, quite close to that of gpt-4o's. The benchmark directly calls OpenAI's servers, while Quasar goes through Openrouter, so it's probably a coincidence. Either way, I'm looking forward to its public release.