TECHNICAL NOTE · NIMBI VISION
ABSTRACT
This note describes how Nimbi Vision assesses inhaler technique with the iPhone's camera and microphone. Part I covers how the devices and the hand are seen; Part II covers how each remaining step is confirmed.
SENSORS
Front TrueDepth camera, microphone
FRAMEWORKS
ARKit · Vision · Core ML · AVFoundation
COMPUTE
Detector, hands, face and sound run on the phone
DEVICE
iPhone with Face ID
PART I · SEEING
§1 Processing pipeline
§2 Object detection
§3 Detection filtering
§4 Shake detection
PART II · THE STEPS
§5 Lip seal
§6 Actuation
§7 Breath-hold
§8 The Coach
§1 PROCESSING PIPELINE
ARKit tracks the face in every camera frame. Alongside it, frames are passed to an object detector and a hand-pose model. A new frame is taken only when the previous one has finished, so results describe the current moment rather than a queue of older frames, and the newest result is sent to the interface.
taken(k) ⇔ t_k ≥ t_busy
t_busy = t_k + τ_orient + max(τ_detector, τ_hand+face) + τ_tracker
interface ← newest result
Fig. 1 Frames offered, processed and skipped. Schematic, slowed six times.
§2 OBJECT DETECTION
The detector is a Core ML model trained by transfer learning on images of the inhaler and spacer. The training set also includes images with neither device, so that the model learns to reject other objects.
IoU(A, B) = |A ∩ B| / |A ∪ B|
AP_c = ∫₀¹ precision_c(r) dr, a box is correct ⇔ IoU ≥ 0.5
mAP = ½ (AP_inhaler + AP_spacer)
Fig. 2 How the detector reads a frame. Schematic.
On held-out images the detector reaches a mean average precision of 0.951 at an IoU of 0.5 (inhaler 0.943, spacer 0.959).
Fig. 3 Training loss and held-out scores.
§3 DETECTION FILTERING
A single frame can produce a false detection, for example on a hand, a mouse or a sleeve. A detection is therefore accepted only when it has sufficient confidence, a plausible size relative to the frame and the face, contact with a hand or the other device, and consistency over recent frames. Accepted boxes are smoothed between frames.
g(b) = [p ≥ p*] · [A_min ≤ A ≤ A_max] · [A ≤ κ · A_face] · [held(b) ∨ joined(b)]
tracked(t) ⇔ Σ_{k<n} g(t − k) ≥ m
b̄_t = b̄_{t−1} + α (b_t − b̄_{t−1})
Fig. 4 Hand-labelled boxes by size and shape against the size limits. Rejected boxes are examples.
§4 SHAKE DETECTION
A hand-pose model locates 21 joints on each hand. The distance from the wrist to the middle knuckle gives the size of the palm, which serves as the unit of distance, so measurements do not depend on how far the child is from the camera. The palm's position is analysed for reversals of direction: a shake is a run of rapid reversals of sufficient amplitude, while slower movements such as carrying the inhaler are rejected.
ℓ = ‖p_wrist − p_middleMCP‖
turn at t_e ⇔ |y(t) − y(t_e)| ≥ k · ℓ
shake ⇔ N consecutive turns with Δt_min ≤ Δt ≤ Δt_max
Fig. 5 Reversals detected in a shake (left) and in carrying the inhaler (right).
§5 LIP SEAL
The seal is checked in two stages. First the mouthpiece must be at the mouth. The centre of the lips is taken from the fitted face mesh and projected onto the screen, and it has to lie inside the spacer’s detected box, or at the mouthpiece end of the inhaler when no spacer is used. It must stay there for a short period, so that a device passing in front of the face is not counted.
Only then are the lips read. Face tracking reports the shape of the mouth as coefficients between 0 and 1. A seal is recognised when any of three of them, describing lips pushed forward or pressed together, rises above its threshold. The depth camera measures the face in three dimensions, so the reading holds at arm’s length.
atLips(t) ⇔ m(t) ∈ B_spacer ∨ ‖m(t) − e(t)‖ ≤ r
sealed(t) ⇔ atLips held for T_s ∧ (funnel > f* ∨ pucker > p* ∨ close > c*)
Fig. 6 The mouthpiece reaches the mouth, stays, and the lip coefficients cross their thresholds. Schematic.
§6 ACTUATION
Pressing the canister moves it only a few millimetres, which a camera cannot see reliably. It does make a sound: a short, broadband hiss. Nimbi listens for it with the microphone. The signal is cut into short frames, about a hundred a second, and two quantities are computed for each: its loudness, and its zero-crossing rate, which is high for a hiss and low for voiced speech.
Loudness is compared with a threshold that follows the background level of the room, so the same rule works in a quiet bedroom and a noisy kitchen. A burst counts as a puff when it is loud enough, lasts a plausible time, and is hiss-like for most of its length. Speech, including the Coach’s own voice, is loud but not hiss-like, and is rejected. The audio is analysed on the phone and is not stored.
L(t) = RMS of frame t, z(t) = zero crossings per second
θ(t) = f(noise floor)
puff ⇔ L > θ for t₁ ≤ Δt ≤ t₂ ∧ share of frames with z > z* ≥ h*
Fig. 7 Speech and a puff on the same recording: both are loud, only the puff is hiss. Schematic.
§7 BREATH-HOLD
After the puff the child breathes in slowly and holds. The Coach counts aloud to ten, and the count keeps pace with the voice: it waits while a number is being spoken, so none is skipped. Short gaps in tracking are tolerated, so a moment of lost tracking does not end the hold.
The duration reached is recorded as it was measured, whether that is four seconds or ten.
hold = Σ Δt, Δt ≤ Δt_max
the count advances only while the Coach is silent
recorded = seconds reached
Fig. 8 A slow breath in, then ten counted seconds.
§8 THE COACH
The Coach is the part the child meets. The phone is propped up, Nimbi speaks each step, and nothing needs to be tapped. Each step is tied to the sensor that can confirm it: the detector for the devices, hand tracking for the shake, face tracking for position and seal, the microphone for the puff, and a timer for the hold. When a step is confirmed, the Coach moves to the next.
A step that is not confirmed is not treated as a mistake. The Coach gives one spoken reminder with a single visual hint. If the step is still not seen, it moves on and records the step as not seen. The record keeps the result of every step of every session, and this is what the technique chart in the visit summary is built from.
step k: wait for confirm_k(t)
not confirmed within T_k → one reminder
still not confirmed → record “not seen”, continue
session = (seen₁, …, seen₈)
Fig. 9 A practice session. One step is not seen, is reminded once, and is recorded as not seen.
IN SUMMARY