Needle, a visual search engine for 10,000 artworks

From a query to a visible artwork.

Find a work in the collection. Follow the search, the image and the decisions that make it appear.

Built with: React, TypeScript, HNSW, Web Workers, Node.js, Sharp and Docker.

Four results to explore

Select an artwork

Metadata queryHybrid retrievalRanked IDsVisible artwork

Prepared searches from Needle’s released engine and 10,000-record corpus. Select a query and inspect its results. Search freely in Needle ↗

Finding it is only half the work.

A ranked ID still needs an image, a place on screen and the right to replace the previous result.

01 / Search

A graph with an identity.

The index is prepared ahead of time. The worker checks the corpus checksum and encoder version before using it.

Change the corpus ↓

02 / Delivery

The image that fits.

A small tile and an inspector have different jobs. Each asks for the image size it needs.

Compare the bytes ↓

03 / Reuse

A cache that knows what changed.

Corpus, source, format and size each participate in the decision to reuse work.

Take a second visit ↓

04 / Rendering

A collection, not 10,000 elements.

The catalog stays in the model. Only a window of artwork needs to be mounted.

Move through the window ↓

Search and ownership

Prepare the graph.
Check that it belongs.

The corpus and graph load in parallel. Metadata encoding and retrieval run in a worker, while the interface keeps ownership of the current query.

01Hash corpus bytes
02Accept prepared graph
03Retrieve in worker
04Accept current request

Prepare once. Verify on arrival.

The corpus checksum and encoder version match. The worker can load the prepared graph, then return results carrying the current request ID.

The current query owns the result.

Interactive model of the release’s compatibility and request checks.

Moving work does not remove its cost.

Precomputation and workers solve different parts of the wait.

Decision: Load a compatible snapshot instead of building its graph on every visit. Keep parsing, encoding and retrieval outside the UI path.
Tradeoff: Transfer, hashing, encoding and worker-to-UI data movement still take time and memory.
Boundary: A delayed response can finish successfully and still be stale. Request identity decides whether it can be committed.

Worker initialization ↗Request ownership ↗

Image delivery

The museum image
is not the thumbnail.

One artwork. Three files. Change the size to see what travels over the network.

Queen Louise, Tile · AVIF — Queen Louise · Elizabeth S. Tucker

4.1 kB

160 × 294 px

A 160-pixel derivative for the small artwork tile. The request matches the place the image will appear.

Original JPEG88.3 kB

Tile · AVIF4.1 kB

Detail · AVIF14.4 kB

One bundled JPEG, resized without enlargement through Sharp 0.35.5. The detail stays at the 339 px source width. AVIF quality 55, effort 2. Encoded file sizes; excludes HTTP overhead. This is a reproducible sample, not the historical page benchmark.

Make the image path part of search.

A result is not ready while its image is still empty.

Request: The product requests 160, 320, 640 or 1280 pixels and tries AVIF, WebP and JPEG. Decoding is asynchronous, with responsive sizes and an explicit unavailable state.
Prepare: The container build prepares 120 opening previews through the same pipeline used at runtime. Other museum images arrive as needed.
Bound: Three simultaneous transforms, a pending-work limit, remote timeouts and a source-size budget prevent an image burst from becoming unlimited work.
Recover: Matching requests share work. Format fallback and source backoff bound retries when the museum source fails.

Derivative generation ↗Opening previews ↗

Cache identity

Reuse is a decision.

A graph belongs to a corpus. A derivative belongs to a source, a size, a format and a pipeline version.

01
Corpus
Download & hash
02
Graph
Load compatible snapshot
03
Derivative
Read disk or generate
04
Browser
Store the response

A prepared graph saves construction, not transfer. The browser still loads the corpus and checks compatibility. An uncached derivative needs a transform.

Illustrated request behavior from the release code. This diagram issues no live requests.

The release’s cache policies

Resource	Identity	HTTP behavior
Corpus / graph	Checksum + encoder version	Revalidate before reuse
HTML / pack manifest	Current release / active pack	Do not store
Local artwork	Versioned pack URL	One year · immutable
Local derivative	Source + size/mtime + width + format	One year · immutable
Remote derivative	Source + daily bucket + width + format	One day
Other modules	File size / mtime ETag	Five minutes unless filename is hashed

Fresh, validated and persistent are different.

Fresh: A fresh immutable response can avoid a request entirely. Changing its URL is essential when the underlying pack changes.
Validated: A matching conditional request can return 304. It still made a trip to the server.
Persistent: A disk cache needs a persistent mount to survive container replacement. The Dockerfile alone does not establish that configuration.

HTTP caching and ETags ↗

Rendering budget

10,000 records.
A window of artwork.

Scroll the model. The catalog count stays fixed while mounted elements follow the visible rows.

Scroll the collection model

Keep the records.
Mount the window.

The same row calculation used in Needle selects visible records, plus one extra row on either side.

20mounted elements
out of 10,000 records

A four-column model. Cells represent records, not artwork previews.

Visible window + overscanRows 1–5

00001

00002

00003

00004

00005

00006

00007

00008

00009

00010

00011

00012

00013

00014

00015

00016

00017

00018

00019

00020

Use the mouse wheel, arrow keys, or the jump control.

A phone needs its own budget.

Rendering, requests and retained memory each have a separate limit.

Window: The collection wall calculates rows from scroll position, viewport height and column count, with one row of overscan on either side.
Previews: The spatial view uses 28, 44 or 64 previews at the inspected viewport breakpoints. Detail images can retain a preview during loading.
Memory: The decoded-source map is capped at 96 entries. Reducing mounted elements alone would not bound retained image memory.

Row-window calculation ↗Viewport budgets ↗Image state ↗

The actual product

The collection, in context.

The walkthrough above opens up individual decisions. These captures show how they come together in Needle.

The real Needle collection wall with Queen Louise ranked first and the artwork inspector open. — Actual Needle release captured locally, 4 October 2026 · revision baa50f2 · query: Queen Louise. This capture does not measure the public host.

Stack, limits and verification.

10,000Catalog records

120Opening previews

3Simultaneous transforms

96Retained decoded sources

My role

Product design and performance engineering: search initialization, image delivery, cache identity and rendering budgets.

Search model

Field-weighted museum metadata and hybrid retrieval in this release. The case study does not claim the experimental neural retrieval system.

The graph still has to arrive.

Cold visit: Precomputation moves work earlier. A worker moves it away from the interface. The corpus and graph still have to cross the network.
Public host: Host startup is a separate clock from browser initialization. Local captures do not establish public-host latency or uptime.

Evidence with a boundary.

Verified source: The manifest declares 10,000 records and 120 opening images. The graph checksum matches the corpus. The mechanisms above are linked to the pinned release.
Image sample: The encoded file sizes are reproducible from one bundled artwork. They do not establish a whole-page transfer reduction.
Historical performance: The original before/after artifacts are absent from the public package. Startup, LCP, heap, scroll and slow-network gains are not quantified here.

Frozen pack manifest ↗Deployment package ↗