Multicam Reasoning Memory: cross-camera identity that refuses to guess

Your cameras think together and remember. A person walks past camera 0, disappears, and reappears on camera 1 as a brand new track ID. This engine decides whether that new track is the same physical person, with evidence attached to the decision, and it is built to prefer splitting one person into two records over merging two people into one.

1.The problem: an identity dies at the edge of the frame

Every single-camera tracker resets to zero the moment a person leaves its field of view. A person walks past camera 0, the tracker assigns a track id such as cam0:t17, and the instant that person steps out of frame the track closes for good. Minutes later the same person walks into camera 1's frame and gets a brand new track id, cam1:t42, with no memory of ever having been cam0:t17. A cross-camera system now has exactly one decision to make, immediately and without a chance to take it back later without extra evidence: is this the same person passing through, or a different person who happens to arrive at a similar time?

The two ways to get this wrong are not equally costly. A wrong merge (treating two different people as one) poisons the memory for as long as it is retained, potentially weeks, because there is no clean way afterward to tell which parts of the combined history belong to which person: a query about either person now returns a blend of two unrelated lives. A missed link (treating one person as two) costs exactly one duplicate entity row, easy to spot in a count and, if a stronger identity signal turns up later such as a badge swipe or a recognized plate, easy to reconcile without touching anything that was already recorded. That asymmetry, not raw accuracy, is the actual design constraint: when cross-camera evidence is ambiguous, the system has to prefer minting a new identity over guessing a link, even though a naive accuracy metric would reward guessing whenever the guess happens to be right.

This is the decision every fleet operator, warehouse floor lead, or campus security desk already makes today, usually by eyeballing timestamps on a bank of monitors and hoping. Multicam Reasoning Memory makes the decision explicit, scored, and reversible instead of implicit and silent: every link is a scored gate with recorded evidence attached, every answer the memory gives carries the episode ids it stands on, and every link can be withdrawn later without losing any of the underlying observations.

To be clear about what this product is not: it is not a visual re-identification model and it does not extract face or body embeddings from pixels. It sits one layer above whatever perception stack is already producing per-camera tracks, whether that is a commercial VMS, an open-source tracker, or Frigate's own detections, and turns that tracker's output into a decision about identity continuity across cameras. When a sister product does supply a real embedding-based identity signal (a face-free cluster id, for instance), it plugs into the same identity_ref field a plate or a badge would use, converting a probabilistic transit-time guess into an exact link, for free, without any change to this engine's code.

2.How it works

Bounded memory. Raw events never persist as-is. Each incoming observation extends an episode, one row per entity times camera times contiguous-presence span, so storage grows as entities plus episodes plus links, never as raw event count. A retention policy ages episodes out by day, drops entities with nothing left, and deactivates their links and relations. The bound is asserted by a test, not assumed: a simulated month of per-second presence (20 visits a day, 60 events per visit, three cameras) run through daily pruning keeps the resulting SQLite file under 2 megabytes. The store is SQLite in WAL mode, stdlib only, and restart safe: the suite closes a store holding one entity with a recorded identity reference and a camera 2 episode carrying a position, reopens the same file as a brand new MemoryStore instance, and confirms the identity reference, the camera id, and the exact last-known position all come back unchanged, no separate export or backup step required.

Linking asymmetry. No link beats a wrong link, so the linker is built asymmetric on purpose. When an event carries an explicit identity_ref (a plate, a badge, an embedding cluster id from a sister product), the match is exact: certainty 1.0, no scoring involved. When it does not, candidates are entities recently seen on other cameras, scored by a transit-time prior, a per camera-pair Gaussian learned online with Welford's algorithm and updated only from ref-confirmed links, never from the linker's own probabilistic guesses, which would otherwise self-reinforce. A probabilistic link is only made if three gates clear together: the best score must clear a link threshold (0.5 by default), it must beat the runner-up candidate by a margin (0.2 by default), and a uniqueness guard must see at most one candidate above a plausibility bar (0.2 by default). A second plausible candidate at any margin refuses the link outright and mints a new entity instead. That third gate is the whole point: a margin-only linker, with the uniqueness guard effectively disabled, measured only 0.918 precision on the seed 1 site, because peaky Gaussian transit scores produce a wide, confident-looking margin even when the winner is the wrong person (two travelers 20 seconds apart at the same camera pair both score near the prior's peak; one just loses by a lot for the wrong reason). Adding the uniqueness guard at its shipped default took precision to 1.0 across every seed tested, at a real, deliberate cost in recall.

These are exactly the five fields of LinkerConfig; every measured number on this page uses the defaults above except the threshold sweep, which explicitly overrides only plausible_threshold and states every value it used.

Every link lands in the store as a claim with its evidence attached, and claims are reversible: an unlink call deactivates the claim and moves the track's episodes to a fresh entity, so the observations stay intact and only the identity assertion gets withdrawn. Concretely, in the test suite: a track linked at confidence 0.8 by the transit-prior method, with one recorded episode, gets unlinked, and the track comes back owned by a fresh entity id that is provably different from the one it was just merged into, while the active-link lookup for that track now correctly returns nothing for the old claim. Nothing about the underlying camera observation is touched; only the identity assertion sitting on top of it changes.

Provenance queries. Five query functions answer the questions an operator actually asks: where is this entity now, what is its history, who was at this camera in this window, what journeys happened between two cameras, and who co-occurred with whom. Every answer carries a provenance block, the exact episode ids and link evidence it stands on, the same "every alert explains itself" principle applied to memory instead of live alerting. An optional narration step can rephrase a computed answer through a small local language model, but the model is never a source of facts, only a sentence generator over numbers the query functions already computed. Concretely, from the test suite: an identity referenced twice on camera 0 (five seconds apart, aggregating into one episode) then once on camera 1 at t=60 produces a where_is answer of camera 1 with an episode id attached, a history answer listing cameras ["cam0", "cam1"] with two episode ids in its provenance, and a journeys answer for cam0 to cam1 finding exactly one journey with a transit time of 55.0 seconds, the same 55 seconds a human would get by subtracting the two timestamps by hand. Two co-located identities observed one second apart on the same camera also produce exactly one co_occurrences relation between them, queryable from either side.

Ground-truth benchmarking. A synthetic site generator simulates a chain of cameras with true transit times plus jitter, walks synthetic people across it emitting per-camera track events exactly like a real tracker would, and ships the ground truth (which person produced which track keys) alongside the event stream. A pairwise scorer then computes precision and recall against that truth over every pair of tracks that truly belong to the same person, so linking quality is measured, never eyeballed. The exact same scorer works against ground truth from a public dataset, which is how the WILDTRACK real-data numbers further down this page were produced.

The pipeline, end to end

This diagram traces one arriving track through the actual decision path implemented in the linker, not a generic diagram: an identity_ref match takes the exact-link shortcut on the left; a ref-less track goes through candidate search, scoring, and the three-gate check.

Left branch: an identity_ref match (plate, badge, embedding cluster id from a sister product) links exactly, no scoring. Right branch: a ref-less track is scored against candidates on other cameras by the transit-time prior and must clear all three gates to link; failing any one of them refuses the link and creates a new entity instead.

Quickstart

The whole surface is four commands. Ingest any tracker's JSONL export, ask the resulting memory a question, or run the synthetic end-to-end demo that produced every seed 1 number on this page:

# 1. Ingest any tracker's JSONL stream into a memory DB
fleetmind ingest --events events.jsonl --memory site.db

# 2. Ask the memory a question. answers carry provenance plus a plain-English sentence
fleetmind query where-is --memory site.db --entity ent_abc123
fleetmind query journeys --memory site.db --from-camera cam0 --to-camera cam1
fleetmind query who-was-at --memory site.db --camera cam2 --t-min 0 --t-max 3600
fleetmind query co-occurrences --memory site.db --entity ent_abc123

# 3. Run the synthetic end-to-end demo (ingest + scored linking + sample queries)
fleetmind demo --seed 1

# 4. Score any ingested memory against external ground truth (used for the WILDTRACK eval below)
fleetmind score --memory site.db --truth truth.json

The event contract requires only three fields per JSONL row: t (seconds), camera_id, and track_id. class, position_m, identity_ref, and free-form attributes are all optional. Full schema and public-dataset adaptation notes live in docs/INPUT_SCHEMA.md.

Integrations: Frigate NVR

Anyone already running Frigate, the self-hosted NVR, can point it at this engine and get cross-camera memory without a separate pipeline: a bridge subscribes to Frigate's own frigate/events MQTT topic, maps each detection into the same track-event contract every tracker uses here, and feeds it straight into the real linker and memory store. The standout behavior: Frigate's own recognized license plate or recognized face becomes an exact cross-camera link, for free, the moment it clears a configurable confidence gate, because a wrong certain link is exactly the failure mode this whole product exists to avoid. A recognition below that gate is not trusted as an identity; the observation still counts, it just falls back to the transit-prior path like any ref-less track.

Frigate publishes exactly one new event, zero or more update events, and exactly one terminal end event per track, all sharing the same id. Only new and end become track events here: new opens the presence, end closes it and, being the track's final and most complete state, is guaranteed to carry the best-known plate or face even if recognition only resolved partway through the track. update is dropped on purpose, it is Frigate's own zone-change and better-snapshot cadence, and its state is superseded by the later end anyway. false_positive detections are dropped regardless of type. Frigate's pixel boxes are not metric coordinates and a generic install has no camera calibration, so position_m is left unset by default rather than fabricated; a calibrated homography callback can supply real ground-plane coordinates if one is available.

pip install paho-mqtt   # optional, only needed for the live bridge

# Live: subscribe to Frigate's MQTT broker and link cross-camera memory as detections arrive
fleetmind frigate-bridge --mqtt-host 192.168.1.50 --memory site.db

# Capture-then-replay: write mapped events to JSONL, ingest later
fleetmind frigate-bridge --mqtt-host 192.168.1.50 --jsonl frigate_events.jsonl
fleetmind ingest --events frigate_events.jsonl --memory site.db

This integration has its own test suite (20 tests, tests/test_frigate.py) separate from the 14 core-engine tests, covering the event mapping on real Frigate payload shapes, identity precedence and confidence gating, and an end-to-end cross-camera link formed from a matching plate with no live MQTT broker required to run the tests.

3.The interactive demo

Every control below replays real, already-computed runs; nothing here is generated on the fly by your click, for the Enterprise-plan reason explained in the honesty notes above.

Explore the guard: precision versus recall, 11 real runs

Drag the slider to change the uniqueness guard's plausibility bar (plausible_threshold). Lower is stricter: more candidates count as plausible, so more links get withheld. Higher effectively disables the guard, falling back to the margin-only baseline described above. Every point is a real linking run against the same seed 1 site, not an interpolation between two endpoints.

Journey visualization: cam0 to cam1, seed 1

Every journey the memory answers carries the episode ids it was derived from. These are the three real transits the demo recovered for the cam0 to cam1 pair, drawn to scale:

Each point is a real entity from the committed seed 1 report, positioned along the cam0-to-cam1 transit-time axis; every one of these answers traces back to a specific episode id in the memory store, not a bare aggregate number. Entity labels are generic here since the underlying entity ids are randomly generated per run and are not stable identifiers.

Scaling the guarantee: 24 sites, 165,902 events, zero wrong links

The design bet of this product is that precision is non-negotiable and recall is the dial. The benchmark dataset's committed linking_report.json runs the linker over a grid of 24 synthetic sites (3 to 8 cameras, 20 to 100 people, 20% to 60% of people carrying an external identity_ref such as a plate or badge). The result across all 165,902 events: precision is 1.0 on every one of the 24 sites, minimum and maximum alike. Not one wrong link anywhere in the grid.

Recall is the price, and it moves exactly the way the design says it should: it scales with reference coverage (mean recall 0.20 at 20% refs, 0.39 at 40%, 0.45 at 50%, 0.60 at 60%), because ref-confirmed links are both direct links and the only training signal the transit-time prior is allowed to learn from. It also tends to rise with camera count (mean 0.36 at 3 cameras to 0.48 at 8) as more camera pairs accumulate learned priors. At the hardest corner (3 cameras, 20% refs) the linker links almost nothing, recall 0.048, rather than guess: every withheld link there is the uniqueness guard doing its job, and a bigger memory with a duplicate entity beats a poisoned one.

4.Measured results

Four sets of numbers back this page: the seed 1 synthetic demo, the 11-run threshold sweep behind the slider above, the benchmark dataset's 24-site scaling grid, and a real-data evaluation against the WILDTRACK academic pedestrian dataset. Every table below states exactly what was run and what file the numbers came from.

Seed 1 synthetic demo, in full

Source: evidence/demo_seed1.json, command PYTHONPATH=src .venv/bin/python -m fleetmind.cli demo --seed 1. Re-run this session (2026-07-09) on Python 3.12.13, macOS, CPU only: every number above reproduced exactly; only the two illustrative entity-id strings sampled by where_is/journeys differ, because entity ids are randomly generated per run and were never claimed to be stable.

Threshold sweep, all 11 runs

Not a repository evidence file: measured fresh this session (2026-07-09) with a short script that constructs fleetmind.synth.generate_site(seed=1) once and re-runs CrossCameraLinker(store, LinkerConfig(plausible_threshold=pt)) plus score_linking for each of the 11 threshold values shown, holding link_threshold=0.5 and margin_threshold=0.2 fixed. The rerun reproduced every one of the 11 precision/recall pairs already embedded in this page's sweep data exactly, so the numbers here and the ones driving the slider above are the same measurement, not two different runs that happen to agree.

24-site scaling grid, aggregates

The full per-site records (all 24 rows) are embedded in this page and drive the interactive explorer above; click any point there to read a site's complete record, including its linked and true pair counts and its memory size. The aggregate view:

Source: the aggregate block of linking_report.json in the multicam-reasoning-memory-benchmark dataset, embedded verbatim in this page's data script. Recall spans 0.0476 (the hardest corner: 3 cameras, 20 percent refs) to 0.6788 (the easiest: 8 cameras, 60 percent refs); precision's minimum and maximum across all 24 sites are both 1.0.

Real-data results: WILDTRACK

Perfect precision held on real data across every row below: the linker never merged two different WILDTRACK pedestrians into one identity, even in a much denser and more ambiguous scene than any synthetic run. The refs-0-percent row defines the operating envelope rather than reading as a weak score: the transit prior models travel time between non-overlapping cameras, and WILDTRACK's 7 cameras fully overlap the same courtyard, so there is no transit signal for the model to use at all. Faced with zero usable signal, the linker correctly refuses to link anything rather than guessing, which is exactly the behavior the uniqueness guard exists to produce.

WILDTRACK GT-replay leg, measured 2026-07-09; tracker-pipeline leg pending. This eval feeds the linker and memory directly from WILDTRACK's own ground-truth bounding boxes (split into per-camera track segments), not from a real object detector or tracker's output, so it is a test of CrossCameraLinker and score_linking, never of a perception model. A tracker-in-the-loop version of this same eval (a real detector and tracker feeding fleetmind instead of replayed ground truth) has not been run and is not represented anywhere on this page.

$ fleetmind ingest --events events_refs0.jsonl --memory mem_refs0.db
{
  "events": 42707,
  "decisions": {"new": 1663},
  "memory": {"entities": 1663, "episodes": 1663,
             "links_active": 1663, "relations": 56608,
             "db_bytes": 7405568}
}
$ fleetmind score --memory mem_refs0.db --truth truth.json
{
  "tracks_linked": 1663, "precision": 1.0, "recall": 0.0,
  "linked_pairs": 0, "true_pairs": 3903
}

$ fleetmind ingest --events events_refs40.jsonl --memory mem_refs40.db
{
  "events": 42707,
  "decisions": {"new": 991, "ref-new": 125, "ref-link": 547},
  "memory": {"entities": 1116, "episodes": 1648,
             "links_active": 1663, "relations": 48945,
             "db_bytes": 6541312}
}
$ fleetmind score --memory mem_refs40.db --truth truth.json
{
  "tracks_linked": 1663, "precision": 1.0, "recall": 0.4087,
  "linked_pairs": 1595, "true_pairs": 3903
}

Source: evidence/wildtrack/RESULTS.md, produced by evidence/wildtrack/convert_wildtrack.py against 400 annotated frames (2 fps, 200 seconds) of WILDTRACK's real 7-camera university courtyard sequence (EPFL CVLAB, CVPR 2018), then fleetmind ingest / fleetmind score exactly as shown. Not re-run this session: reproducing it requires re-fetching roughly 13 MB of WILDTRACK annotation JSON out of a 6.8 GB archive via the byte-range workaround documented in that file, so this table is read verbatim from the committed evidence rather than rerun. Decisions show zero prior-link events in either condition; none of Condition B's recall comes from the transit-prior mechanism, all of it comes from exact identity_ref matching (547 ref-link plus 125 ref-new decisions, 40 percent of tracks, zero false merges among them).

How the ground truth became track segments

track_id is built from per-camera visibility runs, never from WILDTRACK's own person id, which would leak the answer a real tracker never has. For each person and camera pair, frames where that person is visible are sorted and split into a new track segment whenever the gap between two consecutive visible frames exceeds 3.0 seconds, chosen empirically rather than tuned against the score: a scan of all 1,639 person-camera visibility runs found gaps of 0.5 to 3.5 seconds are common (annotation flicker or brief occlusion), while gaps beyond that are rare outliers, up to 44 seconds, that look like genuine departures and returns. The result is not sensitive to the exact threshold chosen: 1,663 segments at 3.0 seconds, versus 1,693 at a stricter 1.5 seconds and 1,653 at a looser 5.0 seconds.

Diagnosis: why refs-0-percent recall is exactly zero, not just low

Investigated directly rather than assumed. The linker only treats a track as a link candidate when the candidate's episode on another camera has already closed before the new track starts, that is its model of "a person leaves camera A, then later arrives at camera B." WILDTRACK's 7 cameras overlap the same courtyard, so the same person is typically visible on several cameras at once and there is rarely a genuine closed-then-arrives transition to score. Measured directly: with refs at 0 percent, 155 of the 1,663 new decisions had zero transit-prior candidates at all (nothing had closed yet); the remaining 1,508 saw hundreds of candidates each, for example 8 decisions saw exactly 1,023 candidates, and every single one was rejected by the uniqueness guard, because with 300-plus people in frame and no learned prior yet, an unlearned camera pair returns a flat, uninformative likelihood of 0.3 to every candidate and essentially all of them clear the 0.2 plausibility bar simultaneously. Zero transit priors were ever learned in this condition, since there was no identity_ref to learn from. With refs at 40 percent, only 2 of the 42 possible camera pairs ever accumulated any ref-confirmed transit sample at all, (cam5, cam0) with 6 samples averaging 1.83 seconds and (cam3, cam0) with 1 sample at 0.5 seconds, both near-zero "transit times" that are really cross-view detection jitter on a person in both views at once, not a real walk between cameras. Neither pair ever won a prior-link decision.

Memory footprint, across every measured run

Storage grows with entities, episodes, and links, never with raw event count, so the same engine handles a 20-person synthetic site and a 313-person real crowd without a different code path, only a bigger SQLite file.

Same three sources as the tables above (evidence/demo_seed1.json and evidence/wildtrack/RESULTS.md), collected side by side here for a single comparison. Both WILDTRACK runs process the identical 42,707 events; refs at 40 percent ends with fewer entities (1,116 versus 1,663) because 672 tracks got merged into 125 people's single entities instead of each staying its own, which is exactly the recall difference the table above already reports as linked pairs.

Bounded-memory regression test

A simulated month of continuous per-second presence (20 visits a day, 60 events per visit, three cameras, 30 days) run through daily pruning must aggregate into episodes rather than accumulate one row per event.

tests/test_fleetmind.py::test_month_of_events_stays_byte_bounded and tests/test_fleetmind.py::test_site_linking_precision_over_recall (the regression gate that locks in precision ≥ 0.95 on the seed 1 site), run this session via PYTHONPATH=src .venv/bin/python -m pytest tests/ -q (14 core tests plus the Frigate integration's own 20 tests, 34 total).

5.Honest limitations

• Timestamps are assumed clock-aligned (NTP) across cameras. There is no per-camera clock-offset estimation yet; that is the documented next step and would tighten the transit priors without touching the linking logic itself.
• Spatial position is provenance only. position_m is stored with every episode but the v1 linker does not use it; scoring runs purely on per-camera-pair transit time.
• The transit-time prior is built for non-overlapping camera chains. WILDTRACK's 7 cameras overlap the same courtyard, so its real-data eval exercises the exact-identity_ref path and the refusal behavior at real density, but it does not exercise or validate the transit-prior learning and scoring logic the synthetic benchmark measures; on that dataset the prior is structurally starved of the sequential arrive-after-departure evidence it needs and contributes zero prior-link decisions. Anyone citing "recall 0.41 on real data" should also say it is 100 percent reference-driven, not prior-driven.
• A fair non-overlapping real-data eval is still open. A dataset with genuinely disjoint camera coverage, such as a subset of the AI-City MTMC benchmark, would be needed to measure the transit-prior mechanism itself on real data; that pass has not been run.
• All synthetic numbers come from a procedural generator, not a real multi-camera deployment. The repository documents how to adapt WILDTRACK or AI-City ground truth into the engine's input format (see the FAQ), and WILDTRACK has now been run through it, but no live camera footage of real people has.
• No model weights ship with this repository. There is nothing to load from a Hugging Face model hub location; this is a code and library release, and the linker itself has no neural component to weight, only a Gaussian prior fit online from data.

6.FAQ