Wait for every model to finish
Simplifies the final presentation, but makes selection wait on slow or disagreeing semantic results.
Independent project · product design & engineering
Mirror AI
Point to what you want to understand.
At a glance
My ownership
I built the image interaction, local model integration, cache, and asynchronous state—and connected user corrections to Eval Lab.
Result
A working local-first prototype with persistent corrections and testable evidence handling.
Problem & key decision
The problem
A region can be correctly outlined but wrongly identified, while slow models make direct interaction wait.
Key decision
Keep the selected subject visible, read cached scene geometry for interaction, and reconcile identities separately from spatial evidence.
Inspect the interface & evidence →Product screenshot; the handoff separately reports 49 unit/component tests.
Mirror AI · image and text assistant
My roleProduct design · frontend engineering · local model integration
ContextIndependent local-first prototype
Built withTypeScript · Vite · browser workers · IndexedDB · COCO-SSD · Florence-2 · SAM · Tesseract · Ollama
I made images directly interactive while models work.
I built a local-first assistant that lets you explore an image directly: select a subject, ask about it, and correct what the model gets wrong. A contour reveals the subject under the pointer; selecting it opens a crop, interpretation, and focused questions while the original image stays central.
Correction → evaluation
A wrong answer becomes a test.
- 01 · Inspect
Give the answer a visible subject
Select a contour, inspect its crop and interpretation, or deliberately mark a missed region for targeted analysis.
- 02 · Correct
Preserve the user’s evidence
Append-only correction history takes precedence over weaker model results that arrive later.
- 03 · Evaluate
Keep enough context to revisit it
Eval Lab stores the question, answer, correction, and a compressed image with focus-region information locally. Saving a case does not retrain the model or prove a fix.
Technical decisions & failure states
The outline can be right while the answer is wrong.
Separate interaction from identification
Expose mapped regions immediately. Let semantic readings arrive progressively, with uncertainty and evidence-merging rules.
What I chose
I separated spatial geometry, semantic identity, and corrections in a versioned scene manifest. Known-region interactions use cached state; incoming model results pass arbitration rules before changing an identity.
The tradeoff
The interface must communicate incomplete identification. Local inference avoids a paid API requirement in the core path, but needs model downloads, memory, and suitable hardware. Saved-scene playback gives repeatable presentation, not a cold-start benchmark.
When models disagree
State Condition Implemented response
Wrong identity
A detector locates a region but assigns the wrong category.
Retain its output as evidence and reconcile independent semantic readings.
Disagreement
Models disagree or remain inconclusive.
Preserve uncertainty instead of forcing a precise label.
Late result
Weaker evidence arrives after a stronger identity.
Apply evidence-merging rules; reject stale runs.
Correction
The user challenges an interpretation.
Preserve append-only correction history and give the correction precedence.
Missing region
The initial map misses an area.
Allow deliberate region selection and targeted analysis.
Wrong answer
A response does not match the image.
Capture context and expected behaviour as an Eval Lab regression case.
What I verified—and what remains uncertain.
The supplied project handoff reports 49 passing unit/component tests across coordinate transforms, cache versions, selection, correction precedence, concurrent results, and segmentation handling. A retained cinematic report records 630 samples with no detected popup gaps, duplicates, or hover-label overflow.