I spent days tuning the prompt, and what actually moved accuracy was changing the model
I went from 30.7 % to 100 % of days correct reading shift rosters. Almost none of that gain came from the prompt, and the metric that mattered wasn't accuracy.
I was building an extractor that reads photographed shift rosters and turns them into a calendar. It’s the central piece of Turna, and the one that decides whether the product is any use: the user isn’t technical, doesn’t want another app and will give it exactly one chance. If the first roster comes out wrong, there is no second attempt.
The first thing I did, before the app, was a test bench. Without it, “it’s better now” is an impression. With it, it’s a number.
The test bench
Three parts: roster images (fixtures), the truth for every day annotated by hand in JSON (expected), and two scripts, one that calls the model with the image and stores the result, another that compares against the truth and produces a report.
Fifteen rosters in total. Eight synthetic, generated with HTML/CSS and captured as images, to force specific cases: nights with a hand-over day, partial holidays, a WhatsApp screenshot with chat noise, an ambiguous row with two people of similar names where the model has to give up rather than guess. And seven real ones, found in Facebook groups: Spanish cafés and restaurants with split shifts and open-ended closing times, and hospitals in Peru and Colombia with codes that don’t mean what they mean in Spain.
The numbers
| Run | Model | Prompt | Upscale and consensus | Accuracy | Confident errors |
|---|---|---|---|---|---|
| 1 | gpt-4o | v1 | — | 30.7 % | 148 |
| 2 | gpt-4o | v2 | — | 35.8 % | 138 |
| 3 | gpt-5.1 | v3 | 2× + consensus | 68.8 % | 48 |
| 4 | gpt-5.4 | v3 | 2× + consensus | 100 % | 0 |
| 5 | gpt-5.5 | v2/v3 | — | 96.3 % | 7-8 |
| 6 | gpt-5.5 | v3 | 2× + consensus | 100 % | 0 |
| 7 | gpt-5.5 | v7, 15 fixtures | 2× + consensus | 100 % | 0 |
The first run scored 30.7 % of days correct. I spent a good while rewriting the prompt: more context, examples, more explicit instructions about the legend. It went up to 35.8 %. Five points.
Then I switched to a newer model generation without touching anything else, and it jumped to 68.8 %. A prompt tuned for the new model, plus two technical tweaks, closed at 100 % with zero confident errors. Between the first run and the last there are more than sixty points, and the prompt explains five.
The metric that actually mattered
Accuracy is the comfortable metric. The one that decides whether the product is any use is a different one: how many days it reads wrong and asserts with full confidence. In the table it’s the last column, and it’s the one that had to reach zero.
It doesn’t matter that 95 % is right if the remaining 5 % lands in the user’s calendar with no flag on it. Someone shows up on a Sunday they had off and uninstalls the app. A doubtful day that’s marked as doubtful is a minor nuisance. A wrongly read, silent day is the end of the relationship with that user.
The model, on its own, never managed to say it wasn’t sure. It returned confidence 1.0 even on the days it got wrong. The only safety net that worked was reading twice and comparing: if the two readings disagree, the day gets flagged as doubtful and the user is asked. It’s a dumb mechanism, and it’s the one holding the product up.
Upscaling the image 2×
The tweak that fixed the last remaining failures has nothing to do with language: doubling the image size before sending it.
The failures that were left weren’t character-recognition errors, they were alignment errors. In a dense table the model read the cell correctly and assigned it to the neighbouring column. At twice the resolution, the alignment holds. It’s the most underrated lever in the whole project, and the cheapest.
The rule that came out of a regression
A Peruvian fixture had an M/T code, morning and afternoon combined. I
taught the model that this meant two time ranges. It worked for that fixture
and broke another: it generalised MN, which is a 24-hour on-call, into two
blocks with a gap in the middle that didn’t exist.
That produced a rule that is now non-negotiable: a day has two blocks only if the document prints two separate time ranges, never by analogy with the code’s name. And another of the same kind: the legend printed on each document always wins over any general glossary. The same letter means different things at different workplaces, and a fixed Spain-Spanish glossary broke the reading of the Latin American rosters. That second mistake, in fact, was in the evaluator, not in the prompt: the v6 run went from 95.2 % to 98.7 % just by fixing who was right.
What I’m taking away
If you’re fighting a prompt and not getting anywhere, first check whether the ceiling is actually the prompt’s. Very often the work is in the input (the resolution), the model (the generation) or the architecture around it (the consensus), and the prompt is just where you look because it’s what you know how to touch.
And one process rule worth more than any finding: no prompt change goes in
without running all fifteen fixtures and confirming none of them drops
against the previous version. The M/T regression was caught by the bench,
not by me.