Measurements, not promises
Last updated: September 2026
VAYS runs its AI on your device by default. Claims like that are easy to make and hard to check, so this page shows how we test: real benchmarks, run on the exact builds we ship, with the caveats spelled out.
Polish medical speech: the hardest test we could find
ADMEDVOICE is a public corpus of Polish medical dictation published in Nature Scientific Data (2025, DOI 10.1038/s41597-025-05776-1, CC BY 4.0): radiology reports, oncology notes, referrals, resuscitation records. Medical speech is a worst case for any recognizer: specialist vocabulary, drug names, dosages and units.
We measured the exact speech stack VAYS runs on-device (NVIDIA Parakeet v3, fully offline via CoreML on Apple Silicon) on the natural part of the corpus. The numbers below come from the updated engine now live in iOS 1.9 and Mac 2.8. To our knowledge this is the first published measurement of this engine on Polish medical speech.
| Metric | VAYS on-device stack |
|---|---|
| Recordings | 9,797 |
| Audio | 16.3 h |
| Word error rate (WER) | 21.4% |
| WER, numbers normalized | 20.1% |
| Character error rate (CER) | 9.4% |
| Speed on an Apple Silicon Mac | 65x faster than realtime |
Difficulty varies a lot by specialty: referrals and recommendations land near 10% WER, while cardiology reaches 44% (dosages, parameters, units). Part of that tail is the corpus itself: we identified 80 recordings with mismatched audio and reference text in the source data; with those filtered out, cardiology improves to about 37%.
Reference points
For context, results reported on Polish medical speech by others. Protocols differ, so treat this as orientation, not a ranking.
| System | WER | Notes |
|---|---|---|
| VAYS (on-device, this measurement) | 21.4% | Runs offline on your Mac or iPhone |
| Whisper-medium fine-tuned by the corpus authors | 9.0% | Our measurement of their model on our sample; it saw part of this data in training, treat as a best case |
| ElevenLabs Scribe | 10.6% | Cloud service, audio leaves the device (arXiv:2603.02246) |
| Whisper large, no fine-tuning | 42.3% | Baseline reported in the same paper |
The engine keeps improving
In August 2026 we updated the speech engine and re-ran the full benchmark before shipping. Accuracy held (WER 21.43% vs 21.47%), speed improved by 28%, and one bug class disappeared completely: short Polish utterances occasionally came back with stray Cyrillic characters. That happened 22 times across the corpus on the old engine and zero times on the new one.
Method and honesty
- We measure the real CoreML stack on Apple hardware, not a reference GPU build. The speed figure comes from one Apple Silicon configuration; older machines will be slower.
- ADMEDVOICE is single-speaker dictation. Conversational audio, meetings and noisy rooms will score differently.
- WER counts every inserted, missing or altered word. A 21% WER text is imperfect but very usable as a draft, which is how VAYS treats transcripts: as material you review, not as ground truth.
- The corpus is public (CC BY 4.0). Protocol details and our measurement scripts are available on request.
Interested in what this means for medical documentation? Read about VAYS MED.