NEWS & INSIGHTS

Matryoshka embeddings for image retrieval: shortlist with a prefix, rerank with the full vector

A practical reading of Matryoshka Representation Learning, its nested training objective and the ImageNet-1K retrieval results reported at NeurIPS 2022.

Concept diagram of image retrieval using a short vector prefix followed by a longer-vector comparison.
Concept diagram of image retrieval using a short vector prefix followed by a longer-vector comparison.

Comparing every product photograph in full detail can be expensive. Matryoshka Representation Learning trains several prefix lengths together so that even a short beginning of an image embedding can support retrieval. A hypothetical catalogue and the paper’s ImageNet-1K experiment show how short vectors can select candidates before full vectors reorder them.

1. How many numbers does an image search need?

Imagine receiving a photograph of a blue handled cup with only a date as its filename. Searching thousands of catalogue photographs requires comparing their content. Yet an initial pass need not examine every handle, rim and pattern at full detail.

An image embedding is a vector of numbers produced by a model; its dimension is the number of entries. Comparing vector distances can identify similar images. Repeating that comparison across a catalogue makes dimensionality part of the retrieval cost.

Kusupati and colleagues presented Matryoshka Representation Learning at NeurIPS 2022. Their approach allows different prefix lengths of the same embedding: a short representation for broad screening and a longer one for finer comparison.

2. Training multiple lengths within one output

In the image experiments, the model outputs 2,048 entries, with nested prefixes such as 8, 16 and 32 dimensions. A 16-dimensional vector and a 32-dimensional vector are not separate model outputs; both are taken from the beginning of that single output.

During training, each selected length predicts the image class and contributes a loss against the correct label. These losses are combined to update the shared model. Short-prefix predictions therefore influence learning rather than being ignored until after training.

The classification setup uses classifiers for the different lengths. Retrieval instead compares distances between the learned representations. For a 16-dimensional comparison, both query and catalogue images use their first 16 entries.

3. Shortlist catalogue images, then compare them again

Suppose a hypothetical catalogue contains bottles, mugs, paper cups and handleless cups. The uploaded blue-cup image is encoded, while catalogue vectors have already been computed with the same model. The first stage compares short prefixes and retains nearby candidates.

The second stage reads longer portions of those same vectors and recalculates distances for the retained candidates. Two blue cups may change order when finer differences become relevant. This illustration does not assume that any individual dimension represents a particular colour or component.

The first stage reduces the search space and the second refines its ranking. A desired cup omitted from the shortlist cannot reappear during reranking. Choosing the shortlist size therefore requires considering retrieval quality alongside computation.

4. What the ImageNet-1K experiment reports

Using ResNet50 representations, the researchers searched ImageNet-1K for images sharing a query’s class. One configuration selected 200 candidates with 16-dimensional prefixes and reranked them using all 2,048 dimensions. The official supplement’s Tables 8 and 11 report exact distance-search results.

For the same MRL representations, full 2,048-dimensional retrieval scored 65.20% mAP@10, while the 16-dimensional shortlist followed by full-vector reranking scored 65.27%. This metric summarises the ranking of relevant images in the top ten across queries. The close values show comparable ranking quality under these experimental conditions.

The corresponding retrieval-compute entries are 2,624 and 21 MFLOPs per query. MFLOPs count floating-point operations in millions. These figures do not by themselves measure a customer’s end-to-end response time, which also includes embedding generation and data access.

5. Choosing vector length and candidate count

Shorter prefixes reduce the number of entries compared per image. They can also discard useful candidates where many images look alike or distinctions are subtle. For the harder ImageNet-4K setting, the paper includes a first-stage length of 64 dimensions; 16 dimensions and 200 candidates are one experimental configuration, not universal defaults.

The meaning of relevance also depends on the service. Category search may accept the same cup in another colour, whereas SKU identification must distinguish different products with similar handles or patterns. Defining relevance first helps locate where quality deteriorates as settings change.

Reranking requires access to the candidates’ longer vectors. The size of the short-vector index and the location of full-vector storage are separate design choices. Recording arithmetic cost, vector-read latency and storage consumption separately makes their effects clearer.

6. Comparisons useful for a technical discussion

On the same catalogue, compare full-vector retrieval with prefix-based shortlisting. First record whether the relevant candidate survives; then record its position after reranking. This separates first-stage omissions from ranking problems in the second stage.

Include changed lighting, cropped photographs and different viewing angles. Good results on clean images need not hold for customer photographs. Besides mean latency, examine which conditions produce slow requests before changing candidate counts or vector access.

For a patent-oriented technical record, connect the lengths evaluated during training, the candidate-selection rule and the information reused in the second comparison. Controlled results on the same catalogue help distinguish the effects of representation learning from those of retrieval-system design.

The original figure labels and forms are preserved; the text explains the source material.

The product catalogue is hypothetical. The reported accuracy and computation figures describe the paper’s experiment, not guaranteed production performance.

NeurIPS 2022: Matryoshka Representation Learning

Final NeurIPS 2022 paper

Official supplement, including Tables 8 and 11

Google’s document-understanding patent: reading the layout of headings, text and images together

Lost in the Middle: why the position of an answer-bearing document matters

Explore IPLEX’s AI projects and consultation materials to discuss embedding models and retrieval pipelines.

Explore our AI and patent practice

Original Tistory article

Read the Korean source

This article reflects the information available when it was published. Contact us to discuss your circumstances.
Discuss this topic ↗All articles

Put your IP strategy into practice.