How photo translation works
What happens in the few seconds between pointing your phone at a sign and reading it in your own language. Step through it on a Tokyo station sign, then try spoiling the photo.
Clean, straight-on photo: every block reads.
Step 1 of 5
A photo is only pixels
To the computer, your photo starts as a grid of coloured dots. There are no letters yet, just light and dark shapes.
Why reading order is the hard part
Recognising a character is the easy bit. The hard bit is knowing which characters belong together and in what order. Japanese and Chinese can run horizontally or in vertical columns read right to left; Arabic and Hebrew run right to left with numbers running left to right inside them; Thai puts no spaces between words at all.
Get the order wrong and a perfectly recognised sign turns into nonsense. That is why PointTranslate asks the model for whole blocks in reading order, with coordinates, instead of a flat list of characters.
Travellers who photograph the night sky as often as street signs might enjoy ahaboo, whose narrated explainer shows why the Moon seems to change shape over a month.
What it will not do
- Certify anything. Machine translation is for understanding. Legal, medical and official documents need a certified human translator.
- Read what is not there. Text hidden by glare, fingers or a crop is not guessed.
- Keep your photo. Images are processed once and discarded; see privacy.
Ready to try it on something real? Open the image translator or the menu translator.
Questions
Is the camera translation done on my phone?
The camera, cropping and resizing happen on your phone. Reading and translating the text happens on an AI model reached through Vercel AI Gateway, because a model able to read dozens of scripts in context is far too large to run well inside a browser.
Why does it sometimes skip text?
The model is told to leave out text it cannot read reliably rather than invent it. Blur, glare, tiny print, very decorative lettering and steep angles are the usual causes. A closer, straighter photo almost always fixes it.
How is this different from classic OCR?
Classic OCR recognises characters one by one and hands the result to a separate translator, which never sees the image. A vision-language model reads and translates in one pass, so the layout and surroundings inform both the reading and the translation.