Skip to content

Object recognition

Aperture can recognise what moves in front of a camera: people, vehicles, animals and delivered parcels. It runs on your own server, on the CPU or an NVIDIA GPU, draws labelled boxes in the live view, and tags events so notifications and automations can react to a person rather than a leaf. This page covers turning it on, choosing a model and hardware, what counts as a detection, packages, your own models and the alternative of an external detection server.

Object recognition is off on a new install. Nothing is downloaded and nothing runs until you turn it on.

Motion detection is always the trigger. While a camera sees motion, Aperture hands its latest picture to a detection model a few times a second. The model runs in a separate worker process, apdetect, using Microsoft’s ONNX Runtime. What the model finds is drawn live, attached to the camera’s current event, and used by notification rules and automations.

The decoded pictures come from the same ffmpeg process that does motion detection, so recognition costs no second decode of the camera stream. If the worker crashes, recording and motion detection carry on and Aperture restarts the worker.

For the internals (the worker protocol, scheduling and the tracker), see Detection pipeline.

  1. Open Settings › Detection.
  2. The card shows where recognition would run (your CPU and how many threads, or your NVIDIA GPU), which model it would use and roughly how long each picture takes.
  3. Select Enable Object Recognition & Download Required Models. The button shows the download size. If the files are already there, it reads Enable Object Recognition.
  4. Wait for the download and unpacking to finish. The card shows progress and speed; Cancel stops it. A download that fails part-way resumes where it left off when you try again; one you cancel starts over.
  5. When the card shows Running, recognition is active. It reads “waiting for motion” until a camera moves, then shows the time per picture and how many pictures a second it is processing.

Recognition looks at every camera that has motion detection on (and at cameras whose recording is Off, for the live view only; see Which cameras). The Cameras row under the card lists them.

Turn off stops recognition and keeps the downloaded files, so turning it on again is instant.

Out of the box, Aperture reports three groups. You can narrow them under Recognise; at least one group stays selected.

Group Labels
People person
Vehicles car, truck, bus, motorcycle, bicycle
Animals dog, cat, bird, horse, bear, cow, sheep

Parcels are handled by the package watch, which you turn on per camera.

Aperture speaks one vocabulary: the 80 COCO object classes plus “package”. Models trained on other class lists are mapped onto it by name. A model’s “van”, “SUV” or “taxi” becomes car, “pickup truck” becomes truck, “pedestrian” becomes person, “puppy” becomes dog; Objects365’s “Storage box”, “Luggage” and “Briefcase” become package. Classes with no counterpart (furniture, plants, food) are never reported. Through the API (labels in PUT /api/v1/system/detection) you can also list other COCO labels, such as backpack.

The Model row offers a simple choice. Automatic picks for your hardware: All-in-one on a usable NVIDIA GPU, Fast on a CPU with two threads or fewer, and Balanced on any other CPU.

Choice Model Input Download Per picture, CPU Per picture, GPU Notes
Fast YOLOX-Nano 416 px 3.5 MB about 9 ms about 2 ms For small CPUs
Balanced YOLOX-Tiny 416 px 19 MB about 30 ms about 3 ms The default on a CPU
Accurate YOLOX-S 640 px 34 MB about 110 ms about 4 ms Sharper, for strong CPUs
Best YOLOX-M 640 px 97 MB about 300 ms about 12 ms People, vehicles and animals
All-in-one D-FINE-X (Objects365) 640 px 240 MB about 900 ms about 24 ms Also finds parcels; the default on a GPU; needs an NVIDIA GPU

The times are rough figures for a desktop CPU and a mid-range NVIDIA GPU; the settings page shows the time measured on your machine once it runs. A bigger input sees small and far-away things better. Choosing a model that isn’t downloaded yet fetches it by itself, and recognition keeps running with the old one until the new one is in place.

Advanced: hardware, every model, your own models, downloads lists two more models, both for an NVIDIA GPU:

Model Download Per picture, GPU Notes
All-in-one (lighter): D-FINE-L (Objects365) 120 MB about 16 ms The all-in-one model at half the size
Sharpest (COCO): D-FINE-X (COCO) 240 MB about 24 ms The most accurate for people, vehicles and animals; no parcels

The YOLOX models are Apache-2.0. The D-FINE code is Apache-2.0; the Objects365 models were trained on the Objects365 dataset, whose terms may limit commercial use. The settings page notes this next to those models.

In Advanced you also choose the hardware under Runs on (Automatic, CPU or NVIDIA GPU), then select Save, or Switch & download when the choice needs files.

Recognition runs on the CPU on any machine. An NVIDIA GPU is much faster, which allows the larger models and more pictures a second.

Aperture uses at most half the machine’s CPU threads, at most half the service’s CPU quota, and never more than four. With the systemd unit the installer sets up (a quota of six CPUs), that is at most three threads. The card shows the number, for example “CPU, 3 threads”.

What you need:

  • An NVIDIA graphics card. The bundled CUDA 12.9 and cuDNN 9.28 libraries cover every generation from Pascal (GTX 10 series) to Blackwell (RTX 50 series).
  • The NVIDIA driver installed on the machine, including its CUDA library (libcuda.so.1). This is the only thing you install yourself.
  • On a virtual machine, the card passed through to the VM (PCI passthrough).

Aperture downloads everything else: the GPU build of ONNX Runtime and NVIDIA’s redistributable CUDA runtime, cuBLAS, cuFFT, cuRAND, NVRTC, nvJitLink and cuDNN libraries. Together that is about 2.9 GB to download, plus the model. AMD graphics cards are detected but not supported; recognition runs on the CPU instead.

To set up the driver, use the guide built into the page:

  1. In Settings › Detection, select Set up an NVIDIA GPU (or the card’s name). The guide also opens by itself when a card is present but not usable yet.
  2. It checks the machine and lists the steps that are still to do, with the exact commands for Debian 12 and 13 and Ubuntu 22.04, 24.04 and 26.04: whether the card is visible (on a VM, how to pass it through), kernel headers, NVIDIA’s package repository, the right driver for your card (open kernel modules for RTX 20 series and newer, the 580 branch for older cards), turning off the built-in nouveau driver, a restart, and a final check. For other distributions it points to NVIDIA’s installation guide.
  3. Run the commands in a terminal on the server, top to bottom. Aperture never installs drivers itself. Select Check again after each step.
  4. Once the driver works, select Use the GPU & download. The download includes the CUDA libraries; recognition keeps running on the CPU until they are in place.

If you add a GPU after setting up on the CPU, recognition keeps running on the CPU and the card offers Use the GPU & download instead of stopping.

If the GPU fails to start (a driver mismatch, too little graphics memory), Aperture says why (“Couldn’t use the GPU (…). Running on the CPU instead.”) and continues on the CPU with the GPU build of the runtime.

Recognition rides on motion detection. A camera is watched when:

  • recognition is on in Settings › Detection, or the camera’s own setting turns it on, and
  • the camera has Motion detection on, or records in Motion only mode, or its recording is Off.

Each camera’s Detection settings (in Devices, on the camera’s Settings tab) have an Object recognition choice: Same as Settings, On (this server), Off for this camera or External detector. Use it to leave out an indoor camera, for example.

A camera whose Recording mode is Off is watched for the live view only: it shows boxes but creates no events, so nothing is notified or recorded.

In the live view, each recognised object gets a labelled box that follows it. The Show object boxes button (on the dashboard and in the player) hides or shows them. Recognition only reports an object when it is likely to matter:

  • People count when they move. Someone who walked in and sat down keeps their box while they are still recognised, for a minute after they last moved. Someone who was standing still in view all along is boxed only if the model is more sure of them than usual (0.1 above the normal threshold).
  • Vehicles and animals count once they have moved themselves, and keep their box for a minute after. Motion merely inside their box is not enough: someone walking past a parked car, or headlights sweeping over it, does not box the car.
  • Boxes smaller than 2 % × 2 % of the picture are ignored.
  • A box the model misses in one picture stays up briefly (1.2 seconds) instead of blinking.

A vehicle (car, truck, bus or motorcycle) that has stood in view without moving for 3 seconds or more is parked. It gets no box; instead the camera shows a blue “P” badge (with a count when there is more than one). The badge follows the Show object boxes switch too.

Parcels the package watch has confirmed as delivered are boxed while they stand there.

When a motion event opens, Aperture waits up to 1.5 seconds for the first recognised object, so the event and its notification can say what moved (“Person · Front door”). While the event goes on, the best sighting of each label is added to it, and when it closes the rest is saved. The event’s summary lists up to four labels, most confident first.

  • In Alarms and the event strip in Playback, events show their labels and boxes, and can be filtered by People, Vehicles, Animals and Packages. Smart shows events with a recognised object, Basic those with plain motion.
  • In Playback, the boxes are drawn over the recording at the playhead, following the path recognition recorded for each object.
Situation On a CPU On an NVIDIA GPU
Pictures decoded for recognition 5 a second 10 a second
Camera moving: at most one picture every 300 ms 100 ms
Someone watches the camera live, nothing moves 700 ms 200 ms
Home Assistant follows presence, nothing moves, nobody watches 3 s 1 s
Nothing moves, nobody watches, Home Assistant not connected not looked at not looked at

At most two pictures are with the detector at a time, and cameras take turns: the one that has waited longest goes next. When the model is slower than the cameras ask for, each camera gets fewer pictures; nothing piles up.

Pictures are scaled to the model’s input height (up to three times as wide), so a camera with a wide panoramic picture keeps enough detail. Pictures wider than about 1.9 : 1 are split into overlapping 16 : 9 tiles before they reach the model. Recognition sees the stream motion detection decodes, which is the sub stream when the camera has one. A very small sub stream limits how far away a person can be recognised.

Setting What it does
Cameras Lists the cameras recognition watches, and why the others are left out.
Model See Choose a model.
Recognise People, Vehicles, Animals. “Leaves, shadows and furniture never count.”
Sensitivity Low, Normal (default), High. Higher finds more, and is wrong more often.
Packages The cameras that watch for parcels, and how. See Packages.
Try it Pick a camera and select Try: recognition runs on a live picture now and lists what it found, plus what it also saw but did not count.

The sensitivity sets how sure the model must be before an object counts (scores run from 0 to 1):

Object Low Normal High
Person 0.75 0.65 0.50
Cat, bird 0.65 0.55 0.40
Any other vehicle or animal 0.70 0.60 0.45
A new parcel (package watch) 0.35 0.25 0.18

The package watch tells you when a parcel is left at the door and when it is taken. Each delivery and each pick-up is an event of type package, with a picture, so it shows in Alarms and can trigger notifications, webhooks and automations.

  1. Under Packages in Settings › Detection, select the cameras that see your door. Only cameras that recognition watches are offered.
  2. Choose how parcels are found:
Mode How it works
Automatic Live when the main model is an all-in-one model, otherwise Separate check.
Live The all-in-one main model finds parcels in every picture it already looks at. A parcel is noticed as it is put down. Needs the All-in-one model.
Separate check A small extra model, D-FINE-S (Objects365, 40 MB), looks every few seconds. Works with any main model and on a CPU. The first camera you select downloads it.

The separate check looks 3 seconds after motion stops, every 8 seconds while the camera moves or something is pending, and every 30 seconds when all is quiet. In live mode, the live pictures count as looks, at most one a second.

How a delivery is told apart from everything else:

  • A delivery is something parcel-like that turns up after motion (within the previous two minutes), then stands on its own, with nobody within arm’s reach and without moving, for 30 seconds over at least two looks.
  • Anything that stands there when watching begins (a storage box, a bin, a doormat the model takes for a parcel) is part of the scene and never a delivery. So is something that appears while nothing has moved for a while.
  • Bags people carry or set down (handbags, backpacks, shopping bags) are never deliveries.
  • A pick-up is a delivered parcel that is missing in two looks for at least 15 seconds, with nobody standing in front of it, after the camera saw motion.

Under the mode choice, each camera shows its state, for example “1 parcel waiting · looked 5 s ago” or “no parcel there”. Aperture cannot recognise a courier by uniform or logo; the open models can’t do that, so the delivery is told by what happens instead.

When the Home Assistant integration is connected, each camera that recognition watches gets presence entities: person, number of people, vehicle (moving or parked), parked vehicle, animal, and package (on cameras watching for packages). To keep them true while someone sits still, recognition then also looks at those cameras when nothing moves (see How often it looks). A group stays present for 5 seconds after it was last seen, so a missed picture doesn’t make the entity flap.

Every file Aperture downloads is pinned to an exact URL and SHA-256 checksum in the source (internal/ai/catalog.go), so what runs is exactly what was published.

What Where from Size
ONNX Runtime 1.31.1 for the CPU Microsoft’s releases on GitHub 11 MB
ONNX Runtime 1.31.1 GPU build, CUDA 12.9 libraries, cuDNN 9.28 Microsoft’s releases on GitHub, developer.download.nvidia.com about 2.9 GB
YOLOX models The YOLOX project’s releases on GitHub 3.5 to 97 MB
D-FINE models huggingface.co/onnx-community, pinned to a commit 40 to 240 MB
  • A file that does not match its checksum is deleted and the setup stops with “the file did not match its published checksum”.
  • A download broken off by the network resumes from where it stopped. Each file is tried up to four times.
  • Before downloading, Aperture checks for three times the download size in free space (unpacked libraries take about twice their archive).
  • The files live in /var/lib/aperture/ai (next to the data folder, not inside it, so they don’t count against your recording storage): packs/ for the runtimes, models/ for the models, custom/ for your own models and tmp/ for partial downloads.
  • When an Aperture update pins a newer runtime, it is fetched by itself on the next start. Recognition keeps running with the old files until the new ones are in place.
  • With recognition off, Advanced offers Remove downloads, which deletes the runtimes and catalogue models (your own models stay). With it on, the same place shows how much the files use and how much space is free.

The downloads go out over HTTPS. If the server reaches the internet only through a proxy, set HTTPS_PROXY in /etc/aperture/aperture.env and restart Aperture (see Configuration).

You can add a detection model of your own, for example one fine-tuned on your yard.

  1. Set up object recognition once, so a runtime is installed: checking a model needs it.
  2. In Settings › Detection, open Advanced: hardware, every model, your own models, downloads and select Add your own model (.onnx).
  3. Choose the file, give it a name if you like, and select Add model. After the upload, Aperture loads the model once (“Checking the model…”) to learn its format and classes.
  4. If it cannot tell the classes, it asks for Class names, one per line, in the model’s order. Paste them and select Add model again.
  5. The model appears under Your models. Select it and Save (or Switch & download).

Your model then works like a catalogue model: you can switch to it while recognition runs, or save it while recognition is off and turn recognition on with it. It is already on the server, so only a missing runtime is downloaded. Its entry reads “Your model” and shows no time per picture until it has run; once it runs, the main card shows the measured time.

What a model must look like:

Requirement
File ONNX, up to 2 GB.
Input One picture, shape 1 × 3 × H × W, float. A fixed square input is used at its size; a dynamic one runs at 640 × 640.
YOLOX format One output of shape [1, N, 5 + classes], as YOLOX’s own ONNX export writes it: raw grid predictions for strides 8, 16 and 32, objectness and class scores already through a sigmoid, not yet decoded or merged. The input is the picture letterboxed into the square (padded with grey 114 on the right and bottom), in BGR order, values 0 to 255.
DETR format D-FINE, RT-DETR and RF-DETR models as onnx-community exports them: class logits [1, Q, classes] and boxes [1, Q, 4] (centre x, centre y, width, height, 0 to 1). Outputs named logits, pred_logits or scores, and pred_boxes or boxes, are found by name. The input is the picture stretched to the square, RGB, values 0 to 1.
Classes 80 classes are read as COCO, 91 as COCO by category id, 366 as Objects365 with “None” first, 365 as Objects365 without it. Any other count needs the names pasted.

Aperture tells the format from the outputs, not from what you say. Class names are mapped onto Aperture’s labels as described in What it recognises; at least one must match a person, vehicle, animal or parcel, or the model is refused. A model with a parcel class (“package”, “parcel”, “box”, “cardboard box”, “carton”) finds parcels itself, like the all-in-one models.

A model in use cannot be removed; choose another first. The trash button next to it removes it.

Instead of Aperture’s own recognition, you can send pictures to an inference server you already run that speaks the DeepStack / CodeProject.AI detection API, for example on another machine with its own accelerator.

It works differently from local recognition: when a motion event opens, Aperture takes as many snapshots as Pictures per event says (1 to 5) from the camera’s main stream, a quarter of a second apart, sends each to the server, and attaches the most confident detection of each label to the event. There are no live boxes, no package watch, no presence for Home Assistant, and no tracking. Notifications for the event wait until the server has answered (each picture may take up to 8 seconds).

To use one:

  1. Run the server. With CodeProject.AI in Docker, for example:

    services:
    detector:
    image: codeproject/ai-server:cpu # or a CUDA variant
    restart: unless-stopped
    ports:
    - "32168:32168"
    volumes:
    - cpai-data:/etc/codeproject/ai
    volumes:
    cpai-data:
  2. In Settings › Detection, open Use an external detector instead (CodeProject.AI, DeepStack).

  3. Enter the Detector URL, for example http://192.168.1.10:32168/v1/vision/detection.

  4. Choose the Response format and how many Pictures per event (1 to 5; more finds more, and costs more).

  5. Select Use this detector. Stop using it turns it off again.

A single camera can also use the external server while the others use local recognition: set its Object recognition to External detector.

What Aperture sends and expects:

  • A POST with multipart/form-data, the JPEG in the field image. The request times out after 15 seconds.
  • DeepStack / CodeProject.AI responses: {"success": true, "predictions": [{"label": "person", "confidence": 0.91, "x_min": 120, "y_min": 40, "x_max": 260, "y_max": 400}]}, boxes in pixels.
  • Generic (normalized boxes) responses: {"detections": [{"label": "person", "confidence": 0.91, "x": 0.1, "y": 0.05, "w": 0.12, "h": 0.5}]}, boxes from 0 to 1.

Detections count at a confidence of 0.4 or more. If Recognise was narrowed while local recognition was on, that choice also limits the external server. Per-label confidences can be set through the API: minConfidence and labels (label to minimum confidence) in PUT /api/v1/system/detection. Labels are taken as the server names them, without the mapping local recognition does, so notification rules must use the server’s names.

The Try it button belongs to local recognition. To test the external server, call POST /api/v1/system/detection/test with {"cameraId": "<camera id>"}: it returns everything the server reported for a fresh snapshot, without thresholds.

  • The worker is a separate process. A fault in it, in the model or in a GPU driver cannot stop recording; Aperture restarts it with a growing pause (up to a minute). Under memory pressure the kernel ends the worker before Aperture itself.
  • On an NVIDIA GPU, the worker takes about 1.1 GB of system memory (with the separate package model), besides graphics memory.
  • The systemd unit caps the whole service (recording, motion detection, recognition and the web app) at 5 GB of memory, throttling from 4.5 GB, and six CPUs. Recognition shares that budget. On a small machine, prefer the Fast or Balanced model.
  • CPU use grows with the number of cameras moving at once and with the model’s time per picture. The settings card shows both the measured time per picture and the current pictures a second.
  • One detector serves all cameras. Busy scenes on many cameras at once share its pictures (see How often it looks).
  • The download fails. The card shows the reason. Select the button again: the download resumes. The server needs HTTPS access to GitHub, developer.download.nvidia.com (GPU only) and huggingface.co (D-FINE models). “Not enough disk space” means three times the download must be free in /var/lib/aperture.
  • The GPU is not used. Read the note on the card: it says why, for example a missing driver, a missing libcuda.so.1, or the GPU failing to start. Open the NVIDIA guide and select Check again. After installing the driver, restart the machine; Aperture then offers Use the GPU & download.
  • “waiting for motion” and no boxes. Recognition only looks while a camera moves or someone watches it. Check that the camera is listed under Cameras and that its Motion detection is on.
  • A lamp post, statue or poster is boxed as a person. Only when the model is very sure of something that never moved. Set Sensitivity to Low or choose a larger model.
  • Far-away people are missed. Choose Accurate or Best (640 px input) or, on a GPU, All-in-one or Sharpest (COCO). Check that the camera’s sub stream is not tiny.
  • The wrong labels, or animals you don’t care about. Narrow Recognise, and use object filters in notification rules.
  • A notification says “Motion detected” though a person was there. The person was recognised more than 1.5 seconds after the event opened. See the caution under Events, Alarms and Playback.
  • Parcels are not reported. The camera must be selected under Packages and its state line must show recent looks. A parcel must stand on its own for 30 seconds; anything there when watching began is part of the scene. On a CPU, choose Separate check.

Related: Motion and audio detection · Notifications · Home Assistant · Detection pipeline