Detection pipeline
This page follows detection through the code: how motion and loud sounds open events, how the apdetect worker recognises objects, how an external DeepStack-style server fits in, and where every result is stored and published. It is written for developers; for the settings themselves see Motion and Object recognition.
The code lives in four packages and one command:
| Path | What it does |
|---|---|
internal/motion |
One ffmpeg per camera, cell differencing in Go, motion events, colour frames for recognition |
internal/audio |
One ffmpeg per camera with audio detection on, loudness events |
internal/detect |
Live recognition, tracking, event tagging, presence, the package watch, the external detector bridge |
internal/ai |
Runtime and model downloads, the worker protocol, pre- and post-processing, label mapping |
cmd/apdetect |
The worker process that runs ONNX Runtime |
The wiring between them is in cmd/nvrd/main.go.
Overview
Section titled “Overview”Every detector reads go2rtc’s restream, never the camera itself (see How it works). Motion is the trigger; object recognition rides on the motion decoder and adds no second decode.
go2rtc (the only connection to each camera) | cam_<id>_sub (or cam_<id> when there is no sub stream) cam_<id> | | v v motion ffmpeg --pipe:1 64x36 gray, 5 fps --> diffFrac audio ffmpeg --s16le 8 kHz--> rms | | | | v v | events (type "motion") events (type "audio") | +--pipe:3 RGB, 5 or 10 fps --> detect.Live --frames--> apdetect (ONNX Runtime) | <------- boxes -------' +--> change feed: motion.state, objects.live +--> event_objects, object_tracks +--> package watch --> events (type "package") +--> presence --> Home Assistant over MQTT
event opens --> detect.Engine.Enrich --> local: wait up to 1.5 s for live objects \-> bridge: frame.jpeg x burst --> POST to external server --> notify, webhooks, automations, HomeKitMotion detection
Section titled “Motion detection”motion.Detector (internal/motion/detector.go) supervises one ffmpeg per watched camera and turns frame differences into events.
Which cameras are watched
Section titled “Which cameras are watched”Detector.shouldDetect watches a camera when it is enabled, has a main RTSP URL, its Recording mode is not never (shown as Off), and either its Motion detection toggle is on or its mode is motion (Motion only recording cannot work without the trigger).
A second case is a live-only camera: recording mode never, but object recognition on. Reconcile then starts a decoder that feeds only the live motion outline and live recognition. It creates no events.
Reconcile runs at start, every 60 seconds (reconcileEvery) and whenever a camera is saved. Each running decoder carries a signature of threshold, zones JSON, source URL, colour frame size and rate, and the live-only flag; when the signature changes, the decoder restarts. Pre-roll, post-roll, cooldown and minimum duration are not in the signature: they are re-read on every reconcile and apply to the next event without a restart.
The decode
Section titled “The decode”Motion decodes the camera’s sub stream when it has one (motionSource): the grid it looks at is 64 by 36, so decoding a 4K main stream for it would waste a core. Without object recognition the command is:
ffmpeg -hide_banner -loglevel error -nostdin -threads 2 -rtsp_transport tcp \ -i rtsp://127.0.0.1:8554/cam_<id>_sub -an \ -vf fps=5,scale=64:36,format=gray -f rawvideo -pix_fmt gray -When recognition wants frames from the camera (FrameSink.FrameSize returns more than 0), the same decode is split into two outputs. With a model input side S of 640 and a colour rate R of 5, the command is:
ffmpeg -hide_banner -loglevel info -nostats -nostdin -threads 2 -rtsp_transport tcp \ -i rtsp://127.0.0.1:8554/cam_<id>_sub -an \ -filter_complex "[0:v]split=2[a][b];[a]fps=5,scale=64:36,format=gray[m];[b]fps=5,scale=1920:640:force_original_aspect_ratio=decrease:force_divisible_by=2,format=rgb24[c]" \ -map "[m]" -f rawvideo -pix_fmt gray pipe:1 \ -map "[c]" -f rawvideo -pix_fmt rgb24 pipe:3The grayscale grid arrives on stdout; the colour frames arrive on file descriptor 3, an os.Pipe passed through cmd.ExtraFiles. Because ffmpeg picks the colour size from the source’s aspect ratio, watchColourSize reads it from ffmpeg’s log line for Stream #1:0 (hence -loglevel info), and readColour then reads fixed-size frames. Colour frames fit within 3·S wide by S high (scale=3·S:S), so a 16:9 camera at S = 640 comes out about 1138 by 640, and a 32:9 panorama 1920 by 540.
Motion itself always runs at 5 frames a second. The colour rate R is 5 on a CPU and 10 when the detector runs on CUDA (Live.FrameRate).
Cell differencing
Section titled “Cell differencing”Each 64 by 36 frame (gridW, gridH in internal/motion/region.go) is compared with the one before it by diffFrac:
- A pixel counts as changed when its luma differs by more than 24 (
pixNoise). - Pixels under the ignore mask are skipped.
- The score is changed pixels divided by unmasked pixels.
- If the score is above the camera’s threshold, the frame is a motion tick.
The threshold is the camera’s motion_threshold, 0.02 by default (2 % of the unmasked picture). The web app’s Sensitivity choices map to fixed values in web/src/components/CameraModal.tsx:
| Sensitivity | Threshold |
|---|---|
| Low | 0.06 |
| Medium-low | 0.03 |
| Medium | 0.02 |
| High | 0.012 |
| Very high | 0.006 |
Every changed pixel also marks one cell of a coarser 32 by 18 grid (SearchW, SearchH). Those cells are unioned over the whole event and stored as events.motion_cells, a JSON array of cell indices. Grid Search (web/src/components/GridSearch.tsx) filters events by them. The same per-frame map goes to live recognition through FrameSink.OnMotion.
Motion zones are stored per camera in cameras.motion_zones_json as:
{ "cols": 32, "rows": 18, "ignore": [0, 1, 32, 33] }ignore holds flat cell indices (row*cols+col) of the editor grid. The zone editor (web/src/components/ZoneEditor.tsx) paints a 32 by 18 grid and saves it with PUT /api/v1/cameras/{id}/zones. parseIgnoreMask scales the editor grid onto the 64 by 36 detection grid, so each editor cell covers four detection pixels. Masked pixels are excluded from both the numerator and the denominator of the score.
Event lifecycle
Section titled “Event lifecycle”onTick opens an event on the first tick and extends it on later ones; finalizeStale, run every second, closes it.
- Open: an
eventsrow is inserted withtypemotion,start_tsset to the first tick minus the pre-roll, andend_tsNULL. A thumbnail is captured and theonEventcallback runs. - Extend: each tick updates the last tick time, the maximum score and the cell union.
- Close: when no tick has come for longer than the cooldown,
end_tsbecomes the last tick plus the post-roll, andFinalizeEventwrites the end, the maximum score andmotion_cells. - Discard: if a minimum duration is set and
end_ts - start_tsis shorter, the row and its thumbnail are deleted instead.
| Setting | Column | Default | Fallback in code |
|---|---|---|---|
| Pre-roll | preroll_ms |
5000 | prerollMs = 5000 |
| Post-roll | postroll_ms |
5000 | postrollMs = 5000 |
| Cooldown | cooldown_ms |
5000 | cooldown = 5 * time.Second |
| Ignore blips under | min_duration_ms |
0 (off) | none |
The pre-roll needs no buffer in the detector. The recorder writes segments all the time, also in motion mode, where the retention janitor later deletes only the segments that overlap no event (pruneNonMotion in internal/retention/janitor.go). The seconds before the first tick therefore still exist on disk. See Recording pipeline.
On startup Run calls store.FinalizeOrphanedEvents, which closes events a previous run left open by setting end_ts = start_ts + 10000.
Thumbnails
Section titled “Thumbnails”captureThumb stores a JPEG at <recordings>/<camera id>/events/<event id>.jpg and records the path with SetEventThumb. The picture comes from stream.Manager.SnapshotCached: a cached frame younger than 20 seconds, or a fresh one from go2rtc’s /api/frame.jpeg. The cache is warmed every 15 seconds, so a motion or audio thumbnail can be up to 20 seconds old and show the scene before the motion that opened the event. Package events store the analysed frame instead (see below).
Motion state
Section titled “Motion state”Independently of events, motionStates runs every 250 ms and tells the frame sink when a camera starts or stops moving. A camera counts as moving for 1.5 seconds after its last tick (motionHold). Live.OnMotionState publishes this as the ephemeral motion.state change-feed message ({"active": true}), which draws the live motion outline in the web app, and the package watch uses it too.
Supervision
Section titled “Supervision”If ffmpeg exits, detectCamera restarts it after a backoff that starts at 1 second and doubles to at most 30 seconds. A run that lasted more than 30 seconds resets the backoff to 1 second. On shutdown, Run waits for every per-camera goroutine, so each ffmpeg is killed and reaped by nvrd itself.
Audio detection
Section titled “Audio detection”internal/audio/detector.go mirrors the motion detector. It runs for cameras with Audio detection on (cameras.audio_detection), enabled, with a main URL and a recording mode other than never. It reads the main restream (RTSPURL), not the sub stream:
ffmpeg -hide_banner -loglevel error -nostdin -rtsp_transport tcp \ -i rtsp://127.0.0.1:8554/cam_<id> \ -vn -ac 1 -ar 8000 -f s16le -acodec pcm_s16le -The PCM is read in windows of 1600 samples (200 ms). rms returns the window’s root-mean-square level normalised to 0..1 (divided by 32768). A window above the threshold is a tick.
| Constant | Value |
|---|---|
sampleRate |
8000 Hz, mono |
windowSamples |
1600 (200 ms) |
defaultThresh |
0.06, used when audio_threshold is 0 |
prerollMs |
3000 |
postrollMs |
4000 |
cooldownMs |
4000 |
reconcileEvery |
10 s |
Audio events have type audio, a thumbnail like motion events, and no motion_cells. Their pre-roll, post-roll and cooldown are fixed; the per-camera motion settings do not apply. Audio events go to notifications, webhooks and automations, but not to object recognition. A stream without an audio track gives ffmpeg nothing to decode; it exits and is retried with the same backoff as motion.
Object recognition
Section titled “Object recognition”Local recognition is detect.Live (internal/detect/live.go) feeding frames to an ai.Manager (internal/ai/manager.go), which runs the apdetect worker. With the detection mode off, the default, nothing is downloaded and no worker runs.
Which cameras use it
Section titled “Which cameras use it”Engine.localCameras selects a camera when its own method (cameras.detect_method) is local, or empty and the global mode in the detection setting is local. The older value onnx is read as local (normMode). The camera must be enabled, and have its Motion detection toggle on, or recording mode motion, or recording mode never (the live-only case above).
In the camera editor this is the Object recognition field: Same as Settings, On (this server), Off for this camera or External detector.
The worker process
Section titled “The worker process”apdetect (cmd/apdetect/main.go) is a separate binary, installed as /usr/lib/aperture/bin/apdetect. nvrd finds it with helperPath: first on PATH, then next to its own executable, then that fixed path. Inference runs in its own process so that a crash in native code or a GPU driver cannot take recording down.
ai.StartWorker (internal/ai/worker.go) starts it with:
apdetect -lib <ai>/packs/<pack>/onnxruntime/lib/libonnxruntime.so \ -model <ai>/models/<model>.onnx -provider cpu -device 0 -threads <n> \ -score 0.250 [-packages <ai>/models/dfine-s-obj365.onnx] -arch yolox -classes coco80<ai> is the download folder, /var/lib/aperture/ai on an installed system (see Downloads and the files on disk). A custom model’s path is under <ai>/custom/ instead.
| Flag | Default | Meaning |
|---|---|---|
-lib |
Path to libonnxruntime.so |
|
-model |
Path to the .onnx model |
|
-arch |
yolox |
Model format: yolox or detr |
-classes |
coco80 |
Label set (coco80, coco91, obj365) or a file with one class name per line |
-provider |
cpu |
Execution provider: cpu or cuda |
-device |
0 | GPU index for CUDA |
-threads |
2 | Intra-op CPU threads per inference |
-score |
0.25 | Lowest score the worker reports |
-packages |
Optional separate package model | |
-package-score |
0.12 | Lowest package score reported |
-probe |
off | Load the model, run it once on a blank picture, describe inputs and outputs, exit |
The manager passes -score 0.25, or 0.12 when the main model is an all-in-one model, because parcels score low and nvrd applies its own thresholds afterwards. -threads comes from aiThreads in cmd/nvrd/main.go: half the CPUs the service may use (it reads the cgroup’s cpu.max, so a systemd CPUQuota= counts), clamped to 1..4.
Process safety in StartWorker:
Pdeathsig: SIGKILL, so the worker never outlives nvrd./proc/<pid>/oom_score_adjis set to 500. If the service runs short of memory, the kernel ends the worker before the recorder. The unit setsOOMPolicy=continue, so that kill does not stop the service.- The worker shares the
apertureservice’s cgroup, and so the memory and CPU limits indeploy/systemd/aperture.service. - Loading must finish within 2 minutes.
Closecloses stdin, waits 3 seconds, then kills. - The last six stderr lines are kept and attached to errors.
Manager.run restarts a worker that exits, with a backoff from 1 second doubling to 1 minute. If a CUDA worker fails to start, the manager records a note, switches to the CPU provider and starts again; the CUDA build of ONNX Runtime also runs on the CPU.
The ONNX Runtime binding
Section titled “The ONNX Runtime binding”internal/ai/ort binds the ONNX Runtime C API with purego at run time, so neither binary links against it and both build with CGO_ENABLED=0. It binds only what detection needs, by function-table index at C API version 23 (APIVersion), which later releases keep unchanged. Runtime.NewSession sets graph optimisation to all, intra-op threads to -threads and inter-op threads to 1. For CUDA it appends the CUDA execution provider with device_id, cudnn_conv_algo_search=HEURISTIC and do_copy_in_default_stream=1. A model must have exactly one input.
Downloads and the files on disk
Section titled “Downloads and the files on disk”Runtimes and models are downloaded on request into /var/lib/aperture/ai, a folder next to the data directory rather than inside it (filepath.Join(filepath.Dir(cfg.DataDir), "ai")). The layout, from ai.Store in internal/ai/install.go:
/var/lib/aperture/ai/ packs/<pack>/<artifact>/... unpacked libraries, each with a .sha256 marker models/<model>.onnx with <model>.onnx.sha256 beside it custom/ models you added (see below) tmp/ partial downloadsEvery artifact in internal/ai/catalog.go has a pinned URL, size and SHA-256. A download resumes a partial file, is retried up to four times, and is deleted if its checksum does not match. Archives are unpacked keeping only the files listed in Keep. Installing needs free space of three times the download size. The marker file holds the SHA-256. When a new Aperture release pins a newer runtime, the old marker shows it is an update and it is fetched without asking again (Store.updating). A missing model is also fetched without asking when its runtime is already installed.
| Pack | Contents | Download |
|---|---|---|
cpu |
ONNX Runtime 1.31.1 for Linux x64 | about 11 MB |
cuda |
ONNX Runtime 1.31.1 CUDA 12 build, CUDA runtime 12.9, cuBLAS, cuFFT, cuRAND, NVRTC, nvJitLink, cuDNN 9.28 | about 3.1 GB |
Only the NVIDIA driver must be installed on the machine; the CUDA libraries come from the pack and are put on LD_LIBRARY_PATH for the worker. Aperture never installs a driver. GET /api/v1/system/ai/gpu (internal/ai/gpusetup.go) inspects the system and returns the steps to run.
Models
Section titled “Models”| ID | Name in the app | Format | Labels | Input | Download | Rough ms per frame, CPU / GPU |
|---|---|---|---|---|---|---|
yolox-nano |
Fast | YOLOX | COCO 80 | 416 | 3.7 MB | 9 / 2 |
yolox-tiny |
Balanced | YOLOX | COCO 80 | 416 | 20 MB | 30 / 3 |
yolox-s |
Accurate | YOLOX | COCO 80 | 640 | 36 MB | 110 / 4 |
yolox-m |
Best | YOLOX | COCO 80 | 640 | 101 MB | 300 / 12 |
dfine-x-obj365 |
All-in-one | DETR | Objects365 | 640 | 252 MB | 900 / 24 |
dfine-l-obj365 |
All-in-one (lighter) | DETR | Objects365 | 640 | 126 MB | 500 / 16 |
dfine-x-coco |
Sharpest (COCO) | DETR | COCO 80 | 640 | 251 MB | 900 / 24 |
dfine-s-obj365 |
Packages (separate package model) | DETR | Objects365 | 640 | 42 MB | 150 / 8 |
The millisecond figures are the catalogue’s rough estimates (MsCPU, MsGPU), used to choose a default and show a hint. The YOLOX models are Apache-2.0. The D-FINE models are Apache-2.0 code; the Objects365 ones are trained on Objects365, whose terms may limit commercial use, and the app says so where it offers them. The three main-model D-FINE entries are marked GPUOnly: too slow on a CPU to be offered in the simple choice.
AllInOne models also find parcels, so a camera watching for packages needs no separate model with them, unless packageMode is separate.
Devices and defaults
Section titled “Devices and defaults”ai.Devices (internal/ai/devices.go) lists the hardware:
- CPU: always usable; its name comes from
/proc/cpuinfo, or from the hardware profile when one is saved. - NVIDIA: found on the PCI bus (vendor
0x10de) or through/proc/driver/nvidia/version. Usable only when the driver is loaded andlibcuda.so.1is installed. - AMD: listed, never usable; recognition runs on the CPU.
PickDevice honours an explicit cpu or cuda choice when that device is usable; Automatic takes the first usable device, a GPU before the CPU. DefaultModel picks dfine-x-obj365 on a GPU, yolox-nano on a CPU with at most two threads, and yolox-tiny otherwise.
If a GPU appears after setup and the device is automatic, Manager.Configure keeps recognising on the CPU with what is installed and reports an upgrade with its download size, instead of stopping.
Worker protocol
Section titled “Worker protocol”The protocol is defined in internal/ai/proto.go:
- On start the worker prints one JSON
Helloline:ready, the ONNX Runtimeversion, theproviderit got, theprovidersthe library offers, the modelinputside,warmupMs, whether a package model is loaded (packages), thearch, the number ofclasses, andcombinedwhen the main model finds parcels itself. On failure it prints{"ready": false, "error": "..."}and exits. - Each request is a 12-byte header followed by packed RGB: the magic
APDF(main model) orAPDP(package model), a little-endianuint32id,uint16width anduint16height, then width × height × 3 bytes. Frames larger than 4096 on a side are rejected (maxFrameSide). - Each reply is one JSON
Resultline:id,ms,boxesand optionallyerror. A box is{"c": class, "s": score, "x", "y", "w", "h"}, normalised to the frame with the origin at the top left.
The worker handles one frame at a time. nvrd matches replies to requests by id (Worker.DetectKind) and serialises writes. The worker exits when stdin closes.
Before it says it is ready, the worker warms the model up with one inference on a 64 by 36 blank frame, which on a GPU also selects kernels.
Pre-processing and decoding
Section titled “Pre-processing and decoding”The two model formats are handled differently:
- YOLOX (
internal/ai/yolox.go):LetterboxRectscales the frame to fit the square input without distortion, pads with grey 114, converts to planar BGR with values 0..255, and samples bilinearly so thin, distant people survive the shrink.DecodeYOLOXreads the[1, N, 5+classes]output across strides 8, 16 and 32, multiplies objectness by the best class score, maps boxes back to the frame, and runs non-maximum suppression per class at IoU 0.45. - DETR (D-FINE,
internal/ai/dfine.go):Stretchresizes to the square input without keeping the aspect ratio, as these models are trained, as planar RGB in 0..1.DecodeDETRtakes, per query, the best class among those Aperture reports, applies a sigmoid, converts centre boxes to corner boxes, and runs NMS at 0.6 in case one object came out as two classes mapping to the same label. The worker finds the outputs by name (logits,pred_logitsorscores;pred_boxesorboxes).
The input side is the model’s fixed side if it has one, otherwise 640.
Tiling wide cameras
Section titled “Tiling wide cameras”ai.Tiles splits a frame whose width is at least 1.9 times its height into overlapping 16:9 tiles spanning the full height: tile width min(w, h*16/9), count ceil(w/h/1.5) and at least 2, evenly spaced, the last flush right. The colour frame of a 32:9 panorama (1920 by 540) becomes three 960-wide tiles at x = 0, 480 and 960. The worker runs each tile, maps the boxes back to frame coordinates, and merges the tiles with NMS at 0.45 (tiled in cmd/apdetect/main.go). A frame under that ratio is one tile.
Label mapping
Section titled “Label mapping”Aperture reports one vocabulary: the 80 COCO classes by COCO index, plus PackageClass (1000) for parcels (internal/ai/labels.go). VocabMap maps a model’s class list onto it when the worker starts:
coco80maps by position.coco91maps COCO category ids (1 to 90 with gaps) to the 80 classes.obj365: the classes “Storage box”, “Luggage” and “Briefcase” becomepackage; every other name is lowercased and looked up through a synonym table (“SUV” and “van” becomecar, “pickup truck” becomestruck, “handbag/satchel” becomeshandbag, and so on) and then by COCO name.- Classes with no counterpart map to -1 and are never reported.
ai.Label turns an index back into the name (person, car…). For the separate package model, PackageVocab keeps only people, parcels, backpacks and handbags.
Frame pacing
Section titled “Frame pacing”Live keeps only the latest colour frame per camera, and only while the camera is being looked at (liveCam.looking): while it moves, while someone watches it live, or while Home Assistant follows its presence. Older frames are overwritten, never queued, so a slow detector drops frames instead of falling behind.
Live.run wakes on every new frame and at least every 100 ms. Live.next then picks, among the cameras with a frame newer than the last one sent and whose gap has passed, the camera that has waited longest, which shares the detector fairly across cameras.
| Constant | CPU | CUDA | Source |
|---|---|---|---|
| Colour frame rate delivered | 5 fps | 10 fps | Live.FrameRate |
| Gap between frames, camera moving | 300 ms | 100 ms | Engine.liveInterval |
| Gap, watched live but still | 700 ms | 200 ms | Engine.watchInterval |
| Gap, followed for presence only | 3 s | 1 s | Engine.presenceInterval |
| Frames in flight to the worker | 2 | 2 | maxInFlight |
| Timeout per frame | 10 s | 10 s | Live.detect |
A live viewer is counted by the WebSocket proxy: handleStreamWS in internal/api/stream_handlers.go, behind GET /api/v1/cameras/{id}/ws, calls Engine.Watch for the duration of the connection (see Live streaming).
Which objects count
Section titled “Which objects count”A box from the worker passes through three steps in Live.detect.
Candidates (candidates): the label must be allowed and its score must reach the threshold (Config.liveAllow), and the box must cover at least 0.0004 of the frame (2 % by 2 %). Without a label list, these labels are allowed: person, car, truck, bus, motorcycle, bicycle, dog, cat, bird, horse, bear, cow, sheep. A non-empty labels map in the detection setting is an allow-list, each label with its own threshold (0 means the default). The default threshold depends on label and sensitivity (threshold):
| Label | Low | Normal | High |
|---|---|---|---|
| person | 0.75 | 0.65 | 0.50 |
| bird, cat | 0.65 | 0.55 | 0.40 |
| others | 0.70 | 0.60 | 0.45 |
Tracking (trackLocked): each candidate takes the id of an earlier object with the same label, best score first, matched by IoU above 0.1 or, failing that, a centre distance smaller than the largest box side (matchScore). An object unseen for 3 seconds is forgotten (trackMemory). Ids are handed out per process.
Acceptance (acceptLocked) uses the union of the motion maps of the last 1.5 seconds (cellMemory):
- A person whose box overlaps motion counts, and keeps counting for a minute after (
stickyFor) while still recognised. - A person who never moved counts if their score is at least the person threshold plus 0.1 (
stillBonus). - Anything else counts only after its own box has moved while overlapping motion (
displaced: the centre moved by at least 1.5 % of the frame or a fifth of the box’s smaller side, or the area changed by more than 30 %), and then for a minute. Motion inside a parked car’s box, such as a person walking past or headlights, is not enough. - An object boxed a moment ago that the model missed in this frame keeps its last box for 1.2 seconds (
coastFor).
Parked vehicles
Section titled “Parked vehicles”parkedLocked counts the cars, trucks, buses and motorcycles seen in this frame that are not boxed and have been tracked for at least 3 seconds (parkedAfter). The count travels with each objects.live message as parked, and the live view marks such a camera with a “P”.
Live boxes
Section titled “Live boxes”After each recognised frame, Live.detect publishes the ephemeral change-feed message objects.live when there is something to show or to clear:
{ "type": "objects.live", "camera": "<camera id>", "data": { "ts": 1700000000000, "objects": [{ "id": 12, "label": "person", "score": 0.81, "x": 0.41, "y": 0.22, "w": 0.09, "h": 0.38 }], "parked": 0 }}When a camera stops moving, nobody watches it and Home Assistant does not follow it, its frame, tracks and boxes are dropped and an empty objects list is published. The change feed keeps the latest ephemeral message per type and camera and replays it to a client that connects.
In the web app, web/src/lib/liveState.ts applies these messages outside React Query, so only the affected tile re-renders, and web/src/components/LiveObjectsOverlay.tsx draws the boxes. Boxes disappear 1.5 seconds after the last update (OBJECT_TTL) and the parked mark after 2.5 seconds (PARKED_TTL), so a lost “clear” message cannot leave a box on screen.
Tagging the event
Section titled “Tagging the event”Recognition attaches what it sees to the camera’s open motion event.
- When a motion event opens, the
onEventcallback incmd/nvrd/main.gocallsEngine.EventOpened, which ties the event to the camera, and thenEngine.Enrich. - In local mode
Enrichwaits up to 1.5 seconds for the event’s first object (Live.waitFirst). Whatever was found is saved, and the labels are put on the in-memory event, so notification rules and automations that filter on objects can see them. - Then notifications, webhooks, automations and the HomeKit motion pulse run, in that order.
- While the event runs,
aggregateLockedkeeps the best sighting per label. It saves again at most every 2 seconds (saveInterval), and only when a label is new or its best score moved into another 0.05 step (sigOf). - Every recognised position is buffered as a track point and written to
object_tracksin batches every 2 seconds (pathLocked). - When the event closes,
Engine.EventClosedsaves the last unsaved objects and track points.
Each save goes through Engine.saveObjects and store.SaveEventDetections (internal/store/detection.go), which in one transaction replaces the event’s event_objects rows and sets events.top_label (the best-scoring label) and events.object_summary (up to four distinct labels, best first, such as person, car). It then emits detection.created.
The package watch
Section titled “The package watch”internal/detect/packages.go reports parcels delivered and picked up, on cameras listed in the detection setting’s packages array that also use local recognition. It cannot recognise a courier; it judges by what happens in the scene.
How parcels are found
Section titled “How parcels are found”packageMode decides where package boxes come from (ai.Settings.PackageMode):
live: the main model is all-in-one and finds parcels on every frame;Live.detecthands its boxes to the watch, at most one look a second (packages.observe). When no live frames are recognised, the scheduled looks below run on the main model.separate: the separate package modeldfine-s-obj365is loaded into the same worker and frames are sent with theAPDPmagic (Manager.DetectPackages).auto(the default): live when the main model is all-in-one, otherwise separate.
The separate check runs on colour frames offered by the motion decoder (packages.offer), one check at a time across cameras: 3 seconds after motion stops (pkgSettle), every 8 seconds while the camera moves or something is pending (pkgBusyEvery), and every 30 seconds otherwise (pkgIdleEvery).
Delivered, picked up, or part of the scene
Section titled “Delivered, picked up, or part of the scene”pkgCam.update keeps a list of parcel-like things standing in view:
- Candidates: package boxes scoring at least the threshold minus 0.1 (
pkgKeepMargin), between 0.0004 and 0.4 of the frame, not covered half or more by a person (person boxes need a score of 0.4). Boxes lying 60 % or more inside a better one are dropped as duplicates (distinct). - Bags: a candidate lying at least half inside a backpack or handbag box scoring 0.3 or more (
pkgBagScore) is marked a bag and can never become a delivery. - Scene: everything seen on the first look, and anything that appears when nothing has moved for the last 2 minutes (
pkgMotionWindow), is part of the scene and raises no event. - Delivered: a new parcel is confirmed when it has stood on its own, with no person within an arm’s length (
attended) and without moving more than half its size (shifted), on at least two looks spanning 30 seconds (pkgConfirmAfter). - Picked up: a confirmed parcel missed on two looks over at least 15 seconds (
pkgGoneAfter), not merely hidden by someone standing in front of it, with motion since it was last seen.
A new parcel needs the full threshold, which depends on sensitivity (packageThreshold):
| Sensitivity | Package threshold |
|---|---|
| Low | 0.35 |
| Normal | 0.25 |
| High | 0.18 |
The thresholds are low because a real flat box at night scores weakly; the motion rules do most of the work.
Package events
Section titled “Package events”packages.raise inserts an event of type package with start_ts at the parcel’s first sighting (delivered) or last sighting (picked up) and end_ts now. It saves one event_objects row labelled package with the summary package delivered or package picked up, writes the analysed frame as the thumbnail (JPEG quality 85), and sends the event to notifications, webhooks and automations. Delivered parcels still standing there are added to objects.live with a negative id. Engine.PackageWatches reports each camera’s last look, delivered parcels, scene items and best score for the settings page.
Custom models
Section titled “Custom models”You can add your own ONNX model with POST /api/v1/system/ai/models (Manager.AddCustomModel in internal/ai/custom.go):
- The upload is saved under
custom/tmp/and checked withapdetect -probe, using whichever runtime pack is installed (so recognition must have been set up once). GuessFormatdecides the format from the outputs: one[1, N, 5+classes]output is YOLOX; a[1, Q, 4]box output with a matching[1, Q, classes]logits output is DETR. Anything else is refused.GuessLabelsassumes COCO for 80 or 91 classes and Objects365 for 366 or 365; other counts need the class names pasted, one per line, in order.- A model none of whose classes map onto Aperture’s labels is refused.
- It is stored as
custom/custom-<hex>.onnxwithcustom-<hex>.jsonbeside it, pluscustom-<hex>.classes.txtwhen it has its own class list, which the worker reads through-classes.
A model in use cannot be removed (DELETE /api/v1/system/ai/models/{id}).
External detector
Section titled “External detector”The bridge (internal/detect/bridge.go) sends pictures to an inference server you already run, such as CodeProject.AI or DeepStack, instead of running a model in Aperture. It is the bridge mode, set globally in Settings › Detection under Use an external detector instead (CodeProject.AI, DeepStack), or per camera as External detector.
Unlike local recognition, it runs once per event, when the event opens:
Engine.EnrichcallsdetectBurst, which takesburstsnapshots 250 ms apart (Pictures per event, default 1, at most 5,MaxBurst). Each snapshot is go2rtc’s/api/frame.jpegof the main stream. A snapshot and its request share an 8-second deadline.- Each JPEG is posted as multipart form field
image(file nameframe.jpg) to the Detector URL. The HTTP client also has its own 15-second timeout, and reads at most 1 MiB of response. - The response is parsed by Response format:
deepstack:{"success": true, "predictions": [{"label", "confidence", "x_min", "y_min", "x_max", "y_max"}]}in pixels, normalised with the JPEG’s dimensions. A non-emptyerrorfield fails the call.generic:{"detections": [{"label", "confidence", "x", "y", "w", "h"}]}, already normalised to 0..1.
- Detections are filtered by
Config.allow: with a label list, only those labels, each at its own threshold orminConfidence; without one, every label atminConfidence, which defaults to 0.4. - The best detection per label across the burst is kept, saved to
event_objectswith the event’s start time, and the labels go on the event before notifications run.
The bridge backend is built lazily and cached until the mode, URL, response format or model path changes (Engine.backendFor). A failed request is logged and the next snapshot of the burst is tried. Bridge results have no track points, so playback shows them as a still box while the playhead is inside the event.
Trying it
Section titled “Trying it”Try it in Settings › Detection calls POST /api/v1/system/detection/test (Engine.Test). It takes one fresh snapshot (the sub stream first for local recognition, the main stream first for the bridge, each falling back to the other) and runs the camera’s effective method with a 12-second timeout. Thresholds are ignored so you see everything the model reports; for local recognition each detection carries passes, saying whether live recognition would report it. Local snapshots are scaled to fit 3·S by S first (toRGB), like live frames, so wide cameras are tiled the same way.
Where results land
Section titled “Where results land”Database
Section titled “Database”| Table | Written by | Key columns |
|---|---|---|
events |
motion, audio, package watch | id, camera_id, type (motion, audio, package), start_ts, end_ts (NULL while open), score, thumb_path, top_label, object_summary, motion_cells, seq |
event_objects |
SaveEventDetections |
id, event_id, label, confidence, bbox_x, bbox_y, bbox_w, bbox_h (normalised 0..1), ts |
object_tracks |
AddTrackPoints |
event_id, camera_id, obj (tracker id), label, confidence, x, y, w, h, ts |
event_objects and object_tracks reference events(id) with ON DELETE CASCADE, and the store opens SQLite with foreign_keys=ON, so deleting an event deletes its objects and paths. All timestamps are Unix milliseconds. The detection settings are one JSON value under the detection key of the settings table (detect.Config). The migrations that created these columns are 0005_motion_tuning.sql, 0010_detection.sql, 0014_camera_detection.sql, 0017_event_motion_cells.sql, 0018_camera_audio.sql, 0023_event_object_id.sql and 0036_object_tracks.sql. See Data model.
Change feed
Section titled “Change feed”The change feed is GET /api/v1/feed, a server-sent event stream (scope events:read). Messages are delivered only to subscribers allowed to see the camera.
| Type | Durable | Sent when |
|---|---|---|
event.created |
yes, id is the event’s seq |
an event opens (notify.Engine.OnEvent) |
event.closed |
yes, same seq |
a motion or audio event closes |
detection.created |
no | objects were saved for an event; refetch /events/{id}/detections |
motion.state |
no | a camera starts or stops moving |
objects.live |
no | live recognition has boxes to show or clear |
API routes
Section titled “API routes”| Route | Scope | Returns |
|---|---|---|
GET /api/v1/events/{id}/detections |
events:read |
The event’s event_objects |
GET /api/v1/cameras/{id}/tracks?from=&to= |
events:read |
Track points, at most one hour and 20000 points per request |
GET /api/v1/system/detection, PUT |
detection:read, detection:write (admin) |
The detection setting |
POST /api/v1/system/detection/test |
detection:write (admin) |
One test run |
GET /api/v1/system/ai |
detection:read (admin) |
Manager status, package watches, presence |
POST /api/v1/system/ai/enable, /cancel, /disable |
detection:write (admin) |
One-button setup, cancel a download, turn off |
DELETE /api/v1/system/ai/files |
detection:write (admin) |
Delete all downloads (recognition must be off) |
PUT /api/v1/system/detection validates the body: mode must be off, local or bridge; sensitivity low, normal or high; device auto, cpu or cuda; packageMode auto, live or separate; the model must exist; bridge mode needs a URL; minConfidence is clamped to 0..1 and burst to 1..5. The full reference is at API reference.
Playback
Section titled “Playback”web/src/components/DetectionOverlay.tsx fetches track points ten minutes at a time and moves each object’s box along its path as the video plays, interpolating between stored positions. A box stays 700 ms after its last position, and gaps longer than 2.5 seconds are not bridged. Events without a path, from the bridge or from before paths were stored, show their best sighting as a still box.
Home Assistant
Section titled “Home Assistant”The MQTT publisher (internal/ha/presence.go, internal/ha/publisher.go) turns recognition into presence entities per camera:
| Entity | Component | Device class | State topic |
|---|---|---|---|
| Person | binary_sensor |
occupancy |
aperture/<camera id>/person |
| People | sensor |
none | aperture/<camera id>/people |
| Vehicle | binary_sensor |
occupancy |
aperture/<camera id>/vehicle |
| Animal | binary_sensor |
occupancy |
aperture/<camera id>/animal |
| Parked vehicle | binary_sensor |
occupancy |
aperture/<camera id>/parked |
| Package | binary_sensor |
none | aperture/<camera id>/package |
Live.presence groups labels: person; vehicle for car, truck, bus, motorcycle and bicycle; animal for dog, cat, bird, horse, sheep, cow and bear. A group stays present for 5 seconds after it was last counted (presenceHold), so one missed frame does not make the sensor flap. A parked vehicle also counts as a vehicle. People is the number of people in the last recognised frame, at least 1 while Person is on. Package is on while a delivered parcel stands there, and the entity exists only on cameras watching for packages.
While connected, the publisher checks presence every second and publishes only changes. Every 5 seconds it calls Engine.KeepPresence with the enabled cameras it publishes, which makes recognition look at them even when nothing moves and nobody watches, every 3 seconds on a CPU and every second on CUDA, so Person stays on for someone sitting still. When the connection drops, that stops. A camera that recognition does not look at gets an empty retained discovery config, which removes its presence entities.
The per-camera motion binary_sensor is on while one of the camera’s events is open, or while the camera is moving according to live motion (Engine.Moving), which also covers live-only cameras that create no events. See Home Assistant.
Limits
Section titled “Limits”- CPU cost. Each watched camera runs one ffmpeg decode, cheap on a sub stream. Recognition adds the worker’s inference; on a CPU the larger models cost hundreds of milliseconds per frame, and the scheduler then drops frames rather than queueing them. Following presence for Home Assistant keeps recognition running on every published camera.
- Sub stream detail. Motion and live recognition both read the sub stream when there is one, so small or distant objects have only the sub stream’s pixels.
- Recognition needs motion detection. A camera that records continuously or on a schedule with Motion detection off gets no recognition at all.
- GPUs. Only NVIDIA GPUs with the driver and
libcuda.so.1installed are used. AMD GPUs are not supported. - Memory. The worker runs inside the service’s memory limit, and a CUDA worker with the package model loaded needs noticeably more memory than a CPU one. Under pressure it is killed first and restarted.
- External detector timing. The bridge sees only the snapshots taken when an event opens, so an object that arrives later in the event is missed.
- Packages. Delivery and pick-up are inferred from motion, people and time. A parcel set down while nothing else moves for two minutes is treated as part of the scene.
Related: Object recognition, Motion, Home Assistant, Data model.