Project Description
The Challenge
The task was not "what kind of object is this", but the fine-grained distinction of visually nearly identical objects within a very large catalog. This is exactly where generic AI models fail: a purchased embedding model returned consistently saturated similarity scores and, in critical cases, even rated the wrong match higher than the correct one. The result was no usable confidence signal. The core thesis: the very domain-agnosticism that makes a generic model robust prevents the sharp separation of almost identical variants.
Our Approach
We froze a self-supervised pre-trained image backbone and added a lean, self-trained head on top (metric learning with angular margin). Instead of retraining the large model from scratch, only the head learns the fine distinctions of the domain. The result is a compact vector per image, searched via a vector database (PostgreSQL with pgvector, HNSW).
Why This Stays Lean
- Efficient iteration: Training the head takes only seconds - the only expensive step is the one-time feature extraction. This enables very fast experimentation cycles.
- No GPU budget: Runs entirely on CPU with little memory - no standing graphics card required.
- On premise: Runs within the EU stack; the images never leave the client's own infrastructure.
- No costly labeling: Training labels are derived from existing metadata.
The Result
The trained head increases the hit rate significantly over the generic baseline. The strongest finding: even on object classes that never appeared in training, the matching improves significantly. The head therefore does not memorize examples but learns genuine distinguishing structure - it delivers precisely where the generic solution failed.
What We Honestly Measured and Discarded
Part of the substance is what did not work. Systematically tested and discarded: a single head for multiple attributes at once, elaborate hard-example mining, fusing two models, a small language model as a re-ranker - and above all more training data, which did not raise quality. Higher image resolution yielded barely any gain at many times the compute cost and was deliberately discarded.
The single biggest jump in the project was not a new model but the correction of a calibration error: a threshold was still set to the scale of the predecessor model, causing a large share of consensus decisions to break off. And we maintained measurement discipline - because single runs fluctuated on an identical test set, improvements were averaged over multiple runs rather than booking noise as success.
The Honest Trade-Off
Specialization buys sharpness at the price of domain sensitivity: on unfamiliar imagery, the model's discriminative power drops. This is not a flaw but the deliberate consequence of a model optimized for exactly one task.
Scope of Services
- Domain-specific fine-tuning on a frozen vision backbone
- Metric learning for the fine-grained distinction of nearly identical objects
- Significantly higher hit rate than generic embedding models
- Generalizes to object classes never seen during training
- CPU inference, no GPU operation required
- On premise within the EU stack - images never leave the infrastructure
- Vector search via pgvector and HNSW
- Training labels from existing metadata, no costly manual labeling
- Disciplined measurement approach with clear GO/NO-GO criteria
Challenge
Reliably telling apart nearly identical objects within a very large catalog. Generic AI models are distinction-blind here: they return saturated similarity scores and, in critical cases, rate the wrong match higher than the correct one. The very domain-agnosticism that makes a purchased model robust prevents the sharp separation of almost identical variants - and thus a reliable confidence signal for automatic matching.
Solution
A self-supervised pre-trained image backbone is frozen and complemented with a lean, self-trained head (metric learning with angular margin). Only the head learns the fine distinctions of the domain; training labels are derived from existing metadata. Each image produces a compact vector searched via a vector database (pgvector, HNSW). Training the head takes only seconds, operation runs entirely on CPU and on premise within the EU stack - without a GPU budget and without images ever leaving the client's own infrastructure.
Results
The hit rate rises significantly over the generic baseline - and, most valuably, even on object classes never seen during training the matching improves significantly. The head learns genuine distinguishing structure rather than memorizing examples. Stated honestly: several obvious extensions (more training data, model fusion, a downstream language model) were systematically tested and discarded; the single biggest jump was the correction of a calibration error, not a new model. Specialization buys its sharpness at the price of domain sensitivity on unfamiliar imagery - a deliberate, clearly named trade-off.
Ready for your project?
Let's realize your digital vision together.