Skip to content

Detection pipeline

This page follows detection through the code: how motion and loud sounds open events, how the apdetect worker recognises objects, how an external DeepStack-style server fits in, and where every result is stored and published. It is written for developers; for the settings themselves see Motion and Object recognition.

The code lives in four packages and one command:

Path What it does
internal/motion One ffmpeg per camera, cell differencing in Go, motion events, colour frames for recognition
internal/audio One ffmpeg per camera with audio detection on, loudness events
internal/detect Live recognition, tracking, event tagging, presence, the package watch, the external detector bridge
internal/ai Runtime and model downloads, the worker protocol, pre- and post-processing, label mapping
cmd/apdetect The worker process that runs ONNX Runtime

The wiring between them is in cmd/nvrd/main.go.

Every detector reads go2rtc’s restream, never the camera itself (see How it works). Motion is the trigger; object recognition rides on the motion decoder and adds no second decode.

go2rtc (the only connection to each camera)
|
cam_<id>_sub (or cam_<id> when there is no sub stream) cam_<id>
| |
v v
motion ffmpeg --pipe:1 64x36 gray, 5 fps --> diffFrac audio ffmpeg --s16le 8 kHz--> rms
| | |
| v v
| events (type "motion") events (type "audio")
|
+--pipe:3 RGB, 5 or 10 fps --> detect.Live --frames--> apdetect (ONNX Runtime)
| <------- boxes -------'
+--> change feed: motion.state, objects.live
+--> event_objects, object_tracks
+--> package watch --> events (type "package")
+--> presence --> Home Assistant over MQTT
event opens --> detect.Engine.Enrich --> local: wait up to 1.5 s for live objects
\-> bridge: frame.jpeg x burst --> POST to external server
--> notify, webhooks, automations, HomeKit

motion.Detector (internal/motion/detector.go) supervises one ffmpeg per watched camera and turns frame differences into events.

Detector.shouldDetect watches a camera when it is enabled, has a main RTSP URL, its Recording mode is not never (shown as Off), and either its Motion detection toggle is on or its mode is motion (Motion only recording cannot work without the trigger).

A second case is a live-only camera: recording mode never, but object recognition on. Reconcile then starts a decoder that feeds only the live motion outline and live recognition. It creates no events.

Reconcile runs at start, every 60 seconds (reconcileEvery) and whenever a camera is saved. Each running decoder carries a signature of threshold, zones JSON, source URL, colour frame size and rate, and the live-only flag; when the signature changes, the decoder restarts. Pre-roll, post-roll, cooldown and minimum duration are not in the signature: they are re-read on every reconcile and apply to the next event without a restart.

Motion decodes the camera’s sub stream when it has one (motionSource): the grid it looks at is 64 by 36, so decoding a 4K main stream for it would waste a core. Without object recognition the command is:

Terminal window
ffmpeg -hide_banner -loglevel error -nostdin -threads 2 -rtsp_transport tcp \
-i rtsp://127.0.0.1:8554/cam_<id>_sub -an \
-vf fps=5,scale=64:36,format=gray -f rawvideo -pix_fmt gray -

When recognition wants frames from the camera (FrameSink.FrameSize returns more than 0), the same decode is split into two outputs. With a model input side S of 640 and a colour rate R of 5, the command is:

Terminal window
ffmpeg -hide_banner -loglevel info -nostats -nostdin -threads 2 -rtsp_transport tcp \
-i rtsp://127.0.0.1:8554/cam_<id>_sub -an \
-filter_complex "[0:v]split=2[a][b];[a]fps=5,scale=64:36,format=gray[m];[b]fps=5,scale=1920:640:force_original_aspect_ratio=decrease:force_divisible_by=2,format=rgb24[c]" \
-map "[m]" -f rawvideo -pix_fmt gray pipe:1 \
-map "[c]" -f rawvideo -pix_fmt rgb24 pipe:3

The grayscale grid arrives on stdout; the colour frames arrive on file descriptor 3, an os.Pipe passed through cmd.ExtraFiles. Because ffmpeg picks the colour size from the source’s aspect ratio, watchColourSize reads it from ffmpeg’s log line for Stream #1:0 (hence -loglevel info), and readColour then reads fixed-size frames. Colour frames fit within 3·S wide by S high (scale=3·S:S), so a 16:9 camera at S = 640 comes out about 1138 by 640, and a 32:9 panorama 1920 by 540.

Motion itself always runs at 5 frames a second. The colour rate R is 5 on a CPU and 10 when the detector runs on CUDA (Live.FrameRate).

Each 64 by 36 frame (gridW, gridH in internal/motion/region.go) is compared with the one before it by diffFrac:

  1. A pixel counts as changed when its luma differs by more than 24 (pixNoise).
  2. Pixels under the ignore mask are skipped.
  3. The score is changed pixels divided by unmasked pixels.
  4. If the score is above the camera’s threshold, the frame is a motion tick.

The threshold is the camera’s motion_threshold, 0.02 by default (2 % of the unmasked picture). The web app’s Sensitivity choices map to fixed values in web/src/components/CameraModal.tsx:

Sensitivity Threshold
Low 0.06
Medium-low 0.03
Medium 0.02
High 0.012
Very high 0.006

Every changed pixel also marks one cell of a coarser 32 by 18 grid (SearchW, SearchH). Those cells are unioned over the whole event and stored as events.motion_cells, a JSON array of cell indices. Grid Search (web/src/components/GridSearch.tsx) filters events by them. The same per-frame map goes to live recognition through FrameSink.OnMotion.

Motion zones are stored per camera in cameras.motion_zones_json as:

{ "cols": 32, "rows": 18, "ignore": [0, 1, 32, 33] }

ignore holds flat cell indices (row*cols+col) of the editor grid. The zone editor (web/src/components/ZoneEditor.tsx) paints a 32 by 18 grid and saves it with PUT /api/v1/cameras/{id}/zones. parseIgnoreMask scales the editor grid onto the 64 by 36 detection grid, so each editor cell covers four detection pixels. Masked pixels are excluded from both the numerator and the denominator of the score.

onTick opens an event on the first tick and extends it on later ones; finalizeStale, run every second, closes it.

  • Open: an events row is inserted with type motion, start_ts set to the first tick minus the pre-roll, and end_ts NULL. A thumbnail is captured and the onEvent callback runs.
  • Extend: each tick updates the last tick time, the maximum score and the cell union.
  • Close: when no tick has come for longer than the cooldown, end_ts becomes the last tick plus the post-roll, and FinalizeEvent writes the end, the maximum score and motion_cells.
  • Discard: if a minimum duration is set and end_ts - start_ts is shorter, the row and its thumbnail are deleted instead.
Setting Column Default Fallback in code
Pre-roll preroll_ms 5000 prerollMs = 5000
Post-roll postroll_ms 5000 postrollMs = 5000
Cooldown cooldown_ms 5000 cooldown = 5 * time.Second
Ignore blips under min_duration_ms 0 (off) none

The pre-roll needs no buffer in the detector. The recorder writes segments all the time, also in motion mode, where the retention janitor later deletes only the segments that overlap no event (pruneNonMotion in internal/retention/janitor.go). The seconds before the first tick therefore still exist on disk. See Recording pipeline.

On startup Run calls store.FinalizeOrphanedEvents, which closes events a previous run left open by setting end_ts = start_ts + 10000.

captureThumb stores a JPEG at <recordings>/<camera id>/events/<event id>.jpg and records the path with SetEventThumb. The picture comes from stream.Manager.SnapshotCached: a cached frame younger than 20 seconds, or a fresh one from go2rtc’s /api/frame.jpeg. The cache is warmed every 15 seconds, so a motion or audio thumbnail can be up to 20 seconds old and show the scene before the motion that opened the event. Package events store the analysed frame instead (see below).

Independently of events, motionStates runs every 250 ms and tells the frame sink when a camera starts or stops moving. A camera counts as moving for 1.5 seconds after its last tick (motionHold). Live.OnMotionState publishes this as the ephemeral motion.state change-feed message ({"active": true}), which draws the live motion outline in the web app, and the package watch uses it too.

If ffmpeg exits, detectCamera restarts it after a backoff that starts at 1 second and doubles to at most 30 seconds. A run that lasted more than 30 seconds resets the backoff to 1 second. On shutdown, Run waits for every per-camera goroutine, so each ffmpeg is killed and reaped by nvrd itself.

internal/audio/detector.go mirrors the motion detector. It runs for cameras with Audio detection on (cameras.audio_detection), enabled, with a main URL and a recording mode other than never. It reads the main restream (RTSPURL), not the sub stream:

Terminal window
ffmpeg -hide_banner -loglevel error -nostdin -rtsp_transport tcp \
-i rtsp://127.0.0.1:8554/cam_<id> \
-vn -ac 1 -ar 8000 -f s16le -acodec pcm_s16le -

The PCM is read in windows of 1600 samples (200 ms). rms returns the window’s root-mean-square level normalised to 0..1 (divided by 32768). A window above the threshold is a tick.

Constant Value
sampleRate 8000 Hz, mono
windowSamples 1600 (200 ms)
defaultThresh 0.06, used when audio_threshold is 0
prerollMs 3000
postrollMs 4000
cooldownMs 4000
reconcileEvery 10 s

Audio events have type audio, a thumbnail like motion events, and no motion_cells. Their pre-roll, post-roll and cooldown are fixed; the per-camera motion settings do not apply. Audio events go to notifications, webhooks and automations, but not to object recognition. A stream without an audio track gives ffmpeg nothing to decode; it exits and is retried with the same backoff as motion.

Local recognition is detect.Live (internal/detect/live.go) feeding frames to an ai.Manager (internal/ai/manager.go), which runs the apdetect worker. With the detection mode off, the default, nothing is downloaded and no worker runs.

Engine.localCameras selects a camera when its own method (cameras.detect_method) is local, or empty and the global mode in the detection setting is local. The older value onnx is read as local (normMode). The camera must be enabled, and have its Motion detection toggle on, or recording mode motion, or recording mode never (the live-only case above).

In the camera editor this is the Object recognition field: Same as Settings, On (this server), Off for this camera or External detector.

apdetect (cmd/apdetect/main.go) is a separate binary, installed as /usr/lib/aperture/bin/apdetect. nvrd finds it with helperPath: first on PATH, then next to its own executable, then that fixed path. Inference runs in its own process so that a crash in native code or a GPU driver cannot take recording down.

ai.StartWorker (internal/ai/worker.go) starts it with:

Terminal window
apdetect -lib <ai>/packs/<pack>/onnxruntime/lib/libonnxruntime.so \
-model <ai>/models/<model>.onnx -provider cpu -device 0 -threads <n> \
-score 0.250 [-packages <ai>/models/dfine-s-obj365.onnx] -arch yolox -classes coco80

<ai> is the download folder, /var/lib/aperture/ai on an installed system (see Downloads and the files on disk). A custom model’s path is under <ai>/custom/ instead.

Flag Default Meaning
-lib Path to libonnxruntime.so
-model Path to the .onnx model
-arch yolox Model format: yolox or detr
-classes coco80 Label set (coco80, coco91, obj365) or a file with one class name per line
-provider cpu Execution provider: cpu or cuda
-device 0 GPU index for CUDA
-threads 2 Intra-op CPU threads per inference
-score 0.25 Lowest score the worker reports
-packages Optional separate package model
-package-score 0.12 Lowest package score reported
-probe off Load the model, run it once on a blank picture, describe inputs and outputs, exit

The manager passes -score 0.25, or 0.12 when the main model is an all-in-one model, because parcels score low and nvrd applies its own thresholds afterwards. -threads comes from aiThreads in cmd/nvrd/main.go: half the CPUs the service may use (it reads the cgroup’s cpu.max, so a systemd CPUQuota= counts), clamped to 1..4.

Process safety in StartWorker:

  • Pdeathsig: SIGKILL, so the worker never outlives nvrd.
  • /proc/<pid>/oom_score_adj is set to 500. If the service runs short of memory, the kernel ends the worker before the recorder. The unit sets OOMPolicy=continue, so that kill does not stop the service.
  • The worker shares the aperture service’s cgroup, and so the memory and CPU limits in deploy/systemd/aperture.service.
  • Loading must finish within 2 minutes. Close closes stdin, waits 3 seconds, then kills.
  • The last six stderr lines are kept and attached to errors.

Manager.run restarts a worker that exits, with a backoff from 1 second doubling to 1 minute. If a CUDA worker fails to start, the manager records a note, switches to the CPU provider and starts again; the CUDA build of ONNX Runtime also runs on the CPU.

internal/ai/ort binds the ONNX Runtime C API with purego at run time, so neither binary links against it and both build with CGO_ENABLED=0. It binds only what detection needs, by function-table index at C API version 23 (APIVersion), which later releases keep unchanged. Runtime.NewSession sets graph optimisation to all, intra-op threads to -threads and inter-op threads to 1. For CUDA it appends the CUDA execution provider with device_id, cudnn_conv_algo_search=HEURISTIC and do_copy_in_default_stream=1. A model must have exactly one input.

Runtimes and models are downloaded on request into /var/lib/aperture/ai, a folder next to the data directory rather than inside it (filepath.Join(filepath.Dir(cfg.DataDir), "ai")). The layout, from ai.Store in internal/ai/install.go:

/var/lib/aperture/ai/
packs/<pack>/<artifact>/... unpacked libraries, each with a .sha256 marker
models/<model>.onnx with <model>.onnx.sha256 beside it
custom/ models you added (see below)
tmp/ partial downloads

Every artifact in internal/ai/catalog.go has a pinned URL, size and SHA-256. A download resumes a partial file, is retried up to four times, and is deleted if its checksum does not match. Archives are unpacked keeping only the files listed in Keep. Installing needs free space of three times the download size. The marker file holds the SHA-256. When a new Aperture release pins a newer runtime, the old marker shows it is an update and it is fetched without asking again (Store.updating). A missing model is also fetched without asking when its runtime is already installed.

Pack Contents Download
cpu ONNX Runtime 1.31.1 for Linux x64 about 11 MB
cuda ONNX Runtime 1.31.1 CUDA 12 build, CUDA runtime 12.9, cuBLAS, cuFFT, cuRAND, NVRTC, nvJitLink, cuDNN 9.28 about 3.1 GB

Only the NVIDIA driver must be installed on the machine; the CUDA libraries come from the pack and are put on LD_LIBRARY_PATH for the worker. Aperture never installs a driver. GET /api/v1/system/ai/gpu (internal/ai/gpusetup.go) inspects the system and returns the steps to run.

ID Name in the app Format Labels Input Download Rough ms per frame, CPU / GPU
yolox-nano Fast YOLOX COCO 80 416 3.7 MB 9 / 2
yolox-tiny Balanced YOLOX COCO 80 416 20 MB 30 / 3
yolox-s Accurate YOLOX COCO 80 640 36 MB 110 / 4
yolox-m Best YOLOX COCO 80 640 101 MB 300 / 12
dfine-x-obj365 All-in-one DETR Objects365 640 252 MB 900 / 24
dfine-l-obj365 All-in-one (lighter) DETR Objects365 640 126 MB 500 / 16
dfine-x-coco Sharpest (COCO) DETR COCO 80 640 251 MB 900 / 24
dfine-s-obj365 Packages (separate package model) DETR Objects365 640 42 MB 150 / 8

The millisecond figures are the catalogue’s rough estimates (MsCPU, MsGPU), used to choose a default and show a hint. The YOLOX models are Apache-2.0. The D-FINE models are Apache-2.0 code; the Objects365 ones are trained on Objects365, whose terms may limit commercial use, and the app says so where it offers them. The three main-model D-FINE entries are marked GPUOnly: too slow on a CPU to be offered in the simple choice.

AllInOne models also find parcels, so a camera watching for packages needs no separate model with them, unless packageMode is separate.

ai.Devices (internal/ai/devices.go) lists the hardware:

  • CPU: always usable; its name comes from /proc/cpuinfo, or from the hardware profile when one is saved.
  • NVIDIA: found on the PCI bus (vendor 0x10de) or through /proc/driver/nvidia/version. Usable only when the driver is loaded and libcuda.so.1 is installed.
  • AMD: listed, never usable; recognition runs on the CPU.

PickDevice honours an explicit cpu or cuda choice when that device is usable; Automatic takes the first usable device, a GPU before the CPU. DefaultModel picks dfine-x-obj365 on a GPU, yolox-nano on a CPU with at most two threads, and yolox-tiny otherwise.

If a GPU appears after setup and the device is automatic, Manager.Configure keeps recognising on the CPU with what is installed and reports an upgrade with its download size, instead of stopping.

The protocol is defined in internal/ai/proto.go:

  1. On start the worker prints one JSON Hello line: ready, the ONNX Runtime version, the provider it got, the providers the library offers, the model input side, warmupMs, whether a package model is loaded (packages), the arch, the number of classes, and combined when the main model finds parcels itself. On failure it prints {"ready": false, "error": "..."} and exits.
  2. Each request is a 12-byte header followed by packed RGB: the magic APDF (main model) or APDP (package model), a little-endian uint32 id, uint16 width and uint16 height, then width × height × 3 bytes. Frames larger than 4096 on a side are rejected (maxFrameSide).
  3. Each reply is one JSON Result line: id, ms, boxes and optionally error. A box is {"c": class, "s": score, "x", "y", "w", "h"}, normalised to the frame with the origin at the top left.

The worker handles one frame at a time. nvrd matches replies to requests by id (Worker.DetectKind) and serialises writes. The worker exits when stdin closes.

Before it says it is ready, the worker warms the model up with one inference on a 64 by 36 blank frame, which on a GPU also selects kernels.

The two model formats are handled differently:

  • YOLOX (internal/ai/yolox.go): LetterboxRect scales the frame to fit the square input without distortion, pads with grey 114, converts to planar BGR with values 0..255, and samples bilinearly so thin, distant people survive the shrink. DecodeYOLOX reads the [1, N, 5+classes] output across strides 8, 16 and 32, multiplies objectness by the best class score, maps boxes back to the frame, and runs non-maximum suppression per class at IoU 0.45.
  • DETR (D-FINE, internal/ai/dfine.go): Stretch resizes to the square input without keeping the aspect ratio, as these models are trained, as planar RGB in 0..1. DecodeDETR takes, per query, the best class among those Aperture reports, applies a sigmoid, converts centre boxes to corner boxes, and runs NMS at 0.6 in case one object came out as two classes mapping to the same label. The worker finds the outputs by name (logits, pred_logits or scores; pred_boxes or boxes).

The input side is the model’s fixed side if it has one, otherwise 640.

ai.Tiles splits a frame whose width is at least 1.9 times its height into overlapping 16:9 tiles spanning the full height: tile width min(w, h*16/9), count ceil(w/h/1.5) and at least 2, evenly spaced, the last flush right. The colour frame of a 32:9 panorama (1920 by 540) becomes three 960-wide tiles at x = 0, 480 and 960. The worker runs each tile, maps the boxes back to frame coordinates, and merges the tiles with NMS at 0.45 (tiled in cmd/apdetect/main.go). A frame under that ratio is one tile.

Aperture reports one vocabulary: the 80 COCO classes by COCO index, plus PackageClass (1000) for parcels (internal/ai/labels.go). VocabMap maps a model’s class list onto it when the worker starts:

  • coco80 maps by position.
  • coco91 maps COCO category ids (1 to 90 with gaps) to the 80 classes.
  • obj365: the classes “Storage box”, “Luggage” and “Briefcase” become package; every other name is lowercased and looked up through a synonym table (“SUV” and “van” become car, “pickup truck” becomes truck, “handbag/satchel” becomes handbag, and so on) and then by COCO name.
  • Classes with no counterpart map to -1 and are never reported.

ai.Label turns an index back into the name (person, car…). For the separate package model, PackageVocab keeps only people, parcels, backpacks and handbags.

Live keeps only the latest colour frame per camera, and only while the camera is being looked at (liveCam.looking): while it moves, while someone watches it live, or while Home Assistant follows its presence. Older frames are overwritten, never queued, so a slow detector drops frames instead of falling behind.

Live.run wakes on every new frame and at least every 100 ms. Live.next then picks, among the cameras with a frame newer than the last one sent and whose gap has passed, the camera that has waited longest, which shares the detector fairly across cameras.

Constant CPU CUDA Source
Colour frame rate delivered 5 fps 10 fps Live.FrameRate
Gap between frames, camera moving 300 ms 100 ms Engine.liveInterval
Gap, watched live but still 700 ms 200 ms Engine.watchInterval
Gap, followed for presence only 3 s 1 s Engine.presenceInterval
Frames in flight to the worker 2 2 maxInFlight
Timeout per frame 10 s 10 s Live.detect

A live viewer is counted by the WebSocket proxy: handleStreamWS in internal/api/stream_handlers.go, behind GET /api/v1/cameras/{id}/ws, calls Engine.Watch for the duration of the connection (see Live streaming).

A box from the worker passes through three steps in Live.detect.

Candidates (candidates): the label must be allowed and its score must reach the threshold (Config.liveAllow), and the box must cover at least 0.0004 of the frame (2 % by 2 %). Without a label list, these labels are allowed: person, car, truck, bus, motorcycle, bicycle, dog, cat, bird, horse, bear, cow, sheep. A non-empty labels map in the detection setting is an allow-list, each label with its own threshold (0 means the default). The default threshold depends on label and sensitivity (threshold):

Label Low Normal High
person 0.75 0.65 0.50
bird, cat 0.65 0.55 0.40
others 0.70 0.60 0.45

Tracking (trackLocked): each candidate takes the id of an earlier object with the same label, best score first, matched by IoU above 0.1 or, failing that, a centre distance smaller than the largest box side (matchScore). An object unseen for 3 seconds is forgotten (trackMemory). Ids are handed out per process.

Acceptance (acceptLocked) uses the union of the motion maps of the last 1.5 seconds (cellMemory):

  • A person whose box overlaps motion counts, and keeps counting for a minute after (stickyFor) while still recognised.
  • A person who never moved counts if their score is at least the person threshold plus 0.1 (stillBonus).
  • Anything else counts only after its own box has moved while overlapping motion (displaced: the centre moved by at least 1.5 % of the frame or a fifth of the box’s smaller side, or the area changed by more than 30 %), and then for a minute. Motion inside a parked car’s box, such as a person walking past or headlights, is not enough.
  • An object boxed a moment ago that the model missed in this frame keeps its last box for 1.2 seconds (coastFor).

parkedLocked counts the cars, trucks, buses and motorcycles seen in this frame that are not boxed and have been tracked for at least 3 seconds (parkedAfter). The count travels with each objects.live message as parked, and the live view marks such a camera with a “P”.

After each recognised frame, Live.detect publishes the ephemeral change-feed message objects.live when there is something to show or to clear:

{
"type": "objects.live",
"camera": "<camera id>",
"data": {
"ts": 1700000000000,
"objects": [{ "id": 12, "label": "person", "score": 0.81, "x": 0.41, "y": 0.22, "w": 0.09, "h": 0.38 }],
"parked": 0
}
}

When a camera stops moving, nobody watches it and Home Assistant does not follow it, its frame, tracks and boxes are dropped and an empty objects list is published. The change feed keeps the latest ephemeral message per type and camera and replays it to a client that connects.

In the web app, web/src/lib/liveState.ts applies these messages outside React Query, so only the affected tile re-renders, and web/src/components/LiveObjectsOverlay.tsx draws the boxes. Boxes disappear 1.5 seconds after the last update (OBJECT_TTL) and the parked mark after 2.5 seconds (PARKED_TTL), so a lost “clear” message cannot leave a box on screen.

Recognition attaches what it sees to the camera’s open motion event.

  1. When a motion event opens, the onEvent callback in cmd/nvrd/main.go calls Engine.EventOpened, which ties the event to the camera, and then Engine.Enrich.
  2. In local mode Enrich waits up to 1.5 seconds for the event’s first object (Live.waitFirst). Whatever was found is saved, and the labels are put on the in-memory event, so notification rules and automations that filter on objects can see them.
  3. Then notifications, webhooks, automations and the HomeKit motion pulse run, in that order.
  4. While the event runs, aggregateLocked keeps the best sighting per label. It saves again at most every 2 seconds (saveInterval), and only when a label is new or its best score moved into another 0.05 step (sigOf).
  5. Every recognised position is buffered as a track point and written to object_tracks in batches every 2 seconds (pathLocked).
  6. When the event closes, Engine.EventClosed saves the last unsaved objects and track points.

Each save goes through Engine.saveObjects and store.SaveEventDetections (internal/store/detection.go), which in one transaction replaces the event’s event_objects rows and sets events.top_label (the best-scoring label) and events.object_summary (up to four distinct labels, best first, such as person, car). It then emits detection.created.

internal/detect/packages.go reports parcels delivered and picked up, on cameras listed in the detection setting’s packages array that also use local recognition. It cannot recognise a courier; it judges by what happens in the scene.

packageMode decides where package boxes come from (ai.Settings.PackageMode):

  • live: the main model is all-in-one and finds parcels on every frame; Live.detect hands its boxes to the watch, at most one look a second (packages.observe). When no live frames are recognised, the scheduled looks below run on the main model.
  • separate: the separate package model dfine-s-obj365 is loaded into the same worker and frames are sent with the APDP magic (Manager.DetectPackages).
  • auto (the default): live when the main model is all-in-one, otherwise separate.

The separate check runs on colour frames offered by the motion decoder (packages.offer), one check at a time across cameras: 3 seconds after motion stops (pkgSettle), every 8 seconds while the camera moves or something is pending (pkgBusyEvery), and every 30 seconds otherwise (pkgIdleEvery).

Delivered, picked up, or part of the scene

Section titled “Delivered, picked up, or part of the scene”

pkgCam.update keeps a list of parcel-like things standing in view:

  • Candidates: package boxes scoring at least the threshold minus 0.1 (pkgKeepMargin), between 0.0004 and 0.4 of the frame, not covered half or more by a person (person boxes need a score of 0.4). Boxes lying 60 % or more inside a better one are dropped as duplicates (distinct).
  • Bags: a candidate lying at least half inside a backpack or handbag box scoring 0.3 or more (pkgBagScore) is marked a bag and can never become a delivery.
  • Scene: everything seen on the first look, and anything that appears when nothing has moved for the last 2 minutes (pkgMotionWindow), is part of the scene and raises no event.
  • Delivered: a new parcel is confirmed when it has stood on its own, with no person within an arm’s length (attended) and without moving more than half its size (shifted), on at least two looks spanning 30 seconds (pkgConfirmAfter).
  • Picked up: a confirmed parcel missed on two looks over at least 15 seconds (pkgGoneAfter), not merely hidden by someone standing in front of it, with motion since it was last seen.

A new parcel needs the full threshold, which depends on sensitivity (packageThreshold):

Sensitivity Package threshold
Low 0.35
Normal 0.25
High 0.18

The thresholds are low because a real flat box at night scores weakly; the motion rules do most of the work.

packages.raise inserts an event of type package with start_ts at the parcel’s first sighting (delivered) or last sighting (picked up) and end_ts now. It saves one event_objects row labelled package with the summary package delivered or package picked up, writes the analysed frame as the thumbnail (JPEG quality 85), and sends the event to notifications, webhooks and automations. Delivered parcels still standing there are added to objects.live with a negative id. Engine.PackageWatches reports each camera’s last look, delivered parcels, scene items and best score for the settings page.

You can add your own ONNX model with POST /api/v1/system/ai/models (Manager.AddCustomModel in internal/ai/custom.go):

  1. The upload is saved under custom/tmp/ and checked with apdetect -probe, using whichever runtime pack is installed (so recognition must have been set up once).
  2. GuessFormat decides the format from the outputs: one [1, N, 5+classes] output is YOLOX; a [1, Q, 4] box output with a matching [1, Q, classes] logits output is DETR. Anything else is refused.
  3. GuessLabels assumes COCO for 80 or 91 classes and Objects365 for 366 or 365; other counts need the class names pasted, one per line, in order.
  4. A model none of whose classes map onto Aperture’s labels is refused.
  5. It is stored as custom/custom-<hex>.onnx with custom-<hex>.json beside it, plus custom-<hex>.classes.txt when it has its own class list, which the worker reads through -classes.

A model in use cannot be removed (DELETE /api/v1/system/ai/models/{id}).

The bridge (internal/detect/bridge.go) sends pictures to an inference server you already run, such as CodeProject.AI or DeepStack, instead of running a model in Aperture. It is the bridge mode, set globally in Settings › Detection under Use an external detector instead (CodeProject.AI, DeepStack), or per camera as External detector.

Unlike local recognition, it runs once per event, when the event opens:

  1. Engine.Enrich calls detectBurst, which takes burst snapshots 250 ms apart (Pictures per event, default 1, at most 5, MaxBurst). Each snapshot is go2rtc’s /api/frame.jpeg of the main stream. A snapshot and its request share an 8-second deadline.
  2. Each JPEG is posted as multipart form field image (file name frame.jpg) to the Detector URL. The HTTP client also has its own 15-second timeout, and reads at most 1 MiB of response.
  3. The response is parsed by Response format:
    • deepstack: {"success": true, "predictions": [{"label", "confidence", "x_min", "y_min", "x_max", "y_max"}]} in pixels, normalised with the JPEG’s dimensions. A non-empty error field fails the call.
    • generic: {"detections": [{"label", "confidence", "x", "y", "w", "h"}]}, already normalised to 0..1.
  4. Detections are filtered by Config.allow: with a label list, only those labels, each at its own threshold or minConfidence; without one, every label at minConfidence, which defaults to 0.4.
  5. The best detection per label across the burst is kept, saved to event_objects with the event’s start time, and the labels go on the event before notifications run.

The bridge backend is built lazily and cached until the mode, URL, response format or model path changes (Engine.backendFor). A failed request is logged and the next snapshot of the burst is tried. Bridge results have no track points, so playback shows them as a still box while the playhead is inside the event.

Try it in Settings › Detection calls POST /api/v1/system/detection/test (Engine.Test). It takes one fresh snapshot (the sub stream first for local recognition, the main stream first for the bridge, each falling back to the other) and runs the camera’s effective method with a 12-second timeout. Thresholds are ignored so you see everything the model reports; for local recognition each detection carries passes, saying whether live recognition would report it. Local snapshots are scaled to fit 3·S by S first (toRGB), like live frames, so wide cameras are tiled the same way.

Table Written by Key columns
events motion, audio, package watch id, camera_id, type (motion, audio, package), start_ts, end_ts (NULL while open), score, thumb_path, top_label, object_summary, motion_cells, seq
event_objects SaveEventDetections id, event_id, label, confidence, bbox_x, bbox_y, bbox_w, bbox_h (normalised 0..1), ts
object_tracks AddTrackPoints event_id, camera_id, obj (tracker id), label, confidence, x, y, w, h, ts

event_objects and object_tracks reference events(id) with ON DELETE CASCADE, and the store opens SQLite with foreign_keys=ON, so deleting an event deletes its objects and paths. All timestamps are Unix milliseconds. The detection settings are one JSON value under the detection key of the settings table (detect.Config). The migrations that created these columns are 0005_motion_tuning.sql, 0010_detection.sql, 0014_camera_detection.sql, 0017_event_motion_cells.sql, 0018_camera_audio.sql, 0023_event_object_id.sql and 0036_object_tracks.sql. See Data model.

The change feed is GET /api/v1/feed, a server-sent event stream (scope events:read). Messages are delivered only to subscribers allowed to see the camera.

Type Durable Sent when
event.created yes, id is the event’s seq an event opens (notify.Engine.OnEvent)
event.closed yes, same seq a motion or audio event closes
detection.created no objects were saved for an event; refetch /events/{id}/detections
motion.state no a camera starts or stops moving
objects.live no live recognition has boxes to show or clear
Route Scope Returns
GET /api/v1/events/{id}/detections events:read The event’s event_objects
GET /api/v1/cameras/{id}/tracks?from=&to= events:read Track points, at most one hour and 20000 points per request
GET /api/v1/system/detection, PUT detection:read, detection:write (admin) The detection setting
POST /api/v1/system/detection/test detection:write (admin) One test run
GET /api/v1/system/ai detection:read (admin) Manager status, package watches, presence
POST /api/v1/system/ai/enable, /cancel, /disable detection:write (admin) One-button setup, cancel a download, turn off
DELETE /api/v1/system/ai/files detection:write (admin) Delete all downloads (recognition must be off)

PUT /api/v1/system/detection validates the body: mode must be off, local or bridge; sensitivity low, normal or high; device auto, cpu or cuda; packageMode auto, live or separate; the model must exist; bridge mode needs a URL; minConfidence is clamped to 0..1 and burst to 1..5. The full reference is at API reference.

web/src/components/DetectionOverlay.tsx fetches track points ten minutes at a time and moves each object’s box along its path as the video plays, interpolating between stored positions. A box stays 700 ms after its last position, and gaps longer than 2.5 seconds are not bridged. Events without a path, from the bridge or from before paths were stored, show their best sighting as a still box.

The MQTT publisher (internal/ha/presence.go, internal/ha/publisher.go) turns recognition into presence entities per camera:

Entity Component Device class State topic
Person binary_sensor occupancy aperture/<camera id>/person
People sensor none aperture/<camera id>/people
Vehicle binary_sensor occupancy aperture/<camera id>/vehicle
Animal binary_sensor occupancy aperture/<camera id>/animal
Parked vehicle binary_sensor occupancy aperture/<camera id>/parked
Package binary_sensor none aperture/<camera id>/package

Live.presence groups labels: person; vehicle for car, truck, bus, motorcycle and bicycle; animal for dog, cat, bird, horse, sheep, cow and bear. A group stays present for 5 seconds after it was last counted (presenceHold), so one missed frame does not make the sensor flap. A parked vehicle also counts as a vehicle. People is the number of people in the last recognised frame, at least 1 while Person is on. Package is on while a delivered parcel stands there, and the entity exists only on cameras watching for packages.

While connected, the publisher checks presence every second and publishes only changes. Every 5 seconds it calls Engine.KeepPresence with the enabled cameras it publishes, which makes recognition look at them even when nothing moves and nobody watches, every 3 seconds on a CPU and every second on CUDA, so Person stays on for someone sitting still. When the connection drops, that stops. A camera that recognition does not look at gets an empty retained discovery config, which removes its presence entities.

The per-camera motion binary_sensor is on while one of the camera’s events is open, or while the camera is moving according to live motion (Engine.Moving), which also covers live-only cameras that create no events. See Home Assistant.

  • CPU cost. Each watched camera runs one ffmpeg decode, cheap on a sub stream. Recognition adds the worker’s inference; on a CPU the larger models cost hundreds of milliseconds per frame, and the scheduler then drops frames rather than queueing them. Following presence for Home Assistant keeps recognition running on every published camera.
  • Sub stream detail. Motion and live recognition both read the sub stream when there is one, so small or distant objects have only the sub stream’s pixels.
  • Recognition needs motion detection. A camera that records continuously or on a schedule with Motion detection off gets no recognition at all.
  • GPUs. Only NVIDIA GPUs with the driver and libcuda.so.1 installed are used. AMD GPUs are not supported.
  • Memory. The worker runs inside the service’s memory limit, and a CUDA worker with the package model loaded needs noticeably more memory than a CPU one. Under pressure it is killed first and restarted.
  • External detector timing. The bridge sees only the snapshots taken when an event opens, so an object that arrives later in the event is missed.
  • Packages. Delivery and pick-up are inferred from motion, people and time. A parcel set down while nothing else moves for two minutes is treated as part of the scene.

Related: Object recognition, Motion, Home Assistant, Data model.