Text search in photos (OCR)
Text search reads the text in photos and makes it searchable. A search for “lease agreement” or “invoice 4471” finds the photo of the document, the screenshot, or the street sign that contains those words. It runs in the server process with ONNX Runtime. No data leaves the server, except the one-time model download.
Text search is off by default. An owner turns on the ocr_search feature flag in Admin → Features. Text search also needs the albums flag.
What is indexed#
- Photos in albums, and the poster frame of videos.
- Not indexed: sampled video frames, cinema, music, podcasts, and books.
- Gallery reads the 1920 px thumbnail, not the original file. Very small or low-contrast text is not found.
- Handwriting is not supported. The model is trained on printed text and returns wrong text for handwriting.
Privacy#
Text in a photo is searchable by everyone who can see the photo. A screenshot with a password, an address, or an account number is searchable by the same people who can already read it in the photo. Gallery stores the recognized text in the database (one row for each photo) and never writes it into the source files. A user never finds text from a photo that the user cannot see.
Delete text data#
- Turn the feature off. When an owner turns off
ocr_searchin Admin → Features, Gallery asks whether to delete all text data too. The default answer keeps the data. - Delete at any time. Admin → Text Search → Delete all text data deletes every recognized text row. It works while the feature is off. Models and settings stay. The action is written to the audit log (
ocr_data_delete). - Deleted items. When an item is deleted, its text is deleted with it.
Use it#
- Search palette and the mobile search. Matches appear in a “Text in photos” group, with the matching line. In smart mode they appear above “Best matches” as “Text matches”.
- Album filter. In normal (not smart) mode the album filter also shows text matches below the title matches.
- Advanced search (web). A “Text in photo” field, combined with the other conditions.
- Lightbox. A photo with recognized text gets a “Text” button (web) or a “Show text” menu item (mobile). It draws a box over each line. On the web, click a line and choose Copy line. On the mobile app, tap a line to copy it. Copy all text copies every line. Opening a result from the web search highlights the lines that matched.
- Item info (web). Shows the recognized text, with Copy and Regenerate for users who can edit the album.
- Album menu. Regenerate Text Index queues the album again (only what is missing, or everything).
A query needs at least 2 characters. Every word must be found (in any order), as a part of a word: “4471” finds “invoice 4471-B”. Capitals and accents do not matter (“cafe” finds “Café”). A phrase can wrap onto the next line. Only exact text matches. A misread character (0 for O) is not found.
Enable the feature#
- Make sure that ONNX Runtime loads. The release image includes it. The OCR models use the same library and the same
[SmartSearch] onnxRuntimeLibrarysetting as smart search (see Smart Search). - Turn on the
ocr_searchflag in Admin → Features. Gallery asks to start a backfill. - New photos are read after their thumbnails exist. To read the existing library, open Admin → Text Search and choose Generate missing. The models download on first use.
Regenerate all deletes the text of every item and reads it again. Use it after a change of model.
Models and languages#
Gallery downloads the models on first use into <BasePath>/data/models/ocr/<model id>/ and checks the SHA256 hash of each file. A file put there by hand with the right hash is used as is.
| Role | Built-in model | License | Size |
|---|---|---|---|
| Detector | PP-OCRv5 mobile (DB text detector) | Apache 2.0 | 4.6 MB |
Recognizer ch (default) |
PP-OCRv5 mobile: Latin (English), Chinese, Japanese | Apache 2.0 | 15.9 MB |
Recognizer latin |
PP-OCRv5 Latin mobile: accented European languages | Apache 2.0 | 7.5 MB |
The recognizer sets the scripts. Gallery uses one recognizer at a time. Choose another in Admin → Text Search → Settings (owners only). Cyrillic, Korean, Arabic, Thai, Greek, and Devanagari have no built-in recognizer: add one as a custom model (below). A different recognizer needs a new read of the library: Gallery offers Regenerate all after the change, and search ignores rows from the previous model until they are replaced.
Cost#
Measured on an M1 Pro, 8 threads, CPU only, 1440 px:
| Value | |
|---|---|
| Memory with models loaded | about 61 MB |
| Detect per photo | about 260 ms |
| Recognize (a photo with text) | about 0.9 s |
| A photo with no text | one detect pass, about 0.3 s |
| Backfill, 10k photos | about 1 to 3 hours on one worker, depending on how many have text |
| With 1 thread instead of 8 | about 4 to 5 times slower |
The “server” PP-OCRv5 variants are not built in: they were 20 to 40 times slower with no better accuracy. Models unload after 5 idle minutes.
Settings#
Stored in the database. Change them in Admin → Text Search (owners only):
| Setting | Default | Notes |
|---|---|---|
| Detector, Recognizer | PP-OCRv5 mobile, ch |
A built-in or custom model. |
| Maximum resolution | 1440 | Longest side of the image that is read. Accuracy was flat from 960 to 1920; speed is not. |
| Minimum detection score | 0.5 | Boxes below this score are dropped. |
| Minimum recognition score | 0.8 | Lines below this score are dropped. Lower it to find more text and accept more false text. |
| Unclip ratio | 1.6 | How far a detected box grows. |
| Threads | 0 (half the CPUs) | ONNX Runtime threads. |
| Idle minutes | 5 | Minutes without use before the models unload. |
Custom models#
An owner can add a detector and a recognizer by URL through the admin API (/api/admin/ocr/models). The admin panel does not have a form for this yet. A recognizer needs a dictionary file too. A custom detector must be a DB text detector. Gallery does not check the license of a custom model. The owner who adds it is responsible for the license for their own installation.
Search speed#
Search keeps recognized text in an in-memory index, one shard for each album, loaded the first time an album is searched and dropped when the least recently used shards pass 500. A search never reads the models.
Measured on a synthetic library of 1,000,000 photos in 500 albums (15% with text), searching every album, on SQLite:
| p50 | p95 | |
|---|---|---|
| Database scan (no index) | about 1.0 to 1.2 s | about 1.1 to 1.6 s |
| In-memory index, warm | 32 ms | 119 ms |
The first search after a restart loads each album it touches from the database. Loading all 500 albums took 1.5 to 3 minutes in the test. Searching one album, or a user who sees few albums, is much faster. Only SQLite was measured. ./gallery.sh ocr-perf repeats the test.
Clients#
| Client | Text search |
|---|---|
| Web (gallery-ui) | Full: palette, album filter, advanced search, lightbox overlay, item info, regenerate, admin panel. |
| Mobile app (gallery-app) | Search, album filter, lightbox overlay with tap to copy, Copy all text, Regenerate Text Index. No admin panel, no advanced search field. |
| TV app (gallery-tv, unreleased) | Search only: a “Text in photos” row. |
Limits#
- Text search uses the CPU only. Large libraries take hours to read the first time.
- Exact matching only. There is no fuzzy match for misread characters.
- No manual correction of recognized text.
- Video frames are not read.