Events

One channel for what happens, two reads for what moves at capture rate.

One channel

Everything the library reports goes through options.onEvent, called with a single object carrying a type. realtime:* says where the library is, token:error why a token could not be had, and measure:* carries the measurement — what the server computes, and what the local detector and counters say about it. Your handler is called inside a try: what it throws is logged to the console and stops nothing.

EventPayloadWhen
realtime:attachedwidth, heightstart() has attached the source and loaded what it needed; only the session is left to open. Re-emitted when the source changes.
realtime:startedmeasureIdThe session is open and capture is running. measureId names the measure it records — a string, or null when the server did not announce it in time; getMeasureId() returns it too.
realtime:stoppedreasonThe measurement stopped: requested once stop() has finished, aborted after abort(), destroy() or a stop() that had to cut, transport-closed when the link dropped or the server closed the session (that includes a session cut for lack of credits — see the FAQ — and the library does not pass the server’s close code on), capture-failed when the capture chain itself failed (a realtime:error carrying the cause comes just before). One per session.
realtime:destroyed—The last event of an instance: nothing follows.
realtime:errormessageStarting failed — the source, the models, the workers, the session. Also, on a running instance, a source swap that designates nothing, a model turned on that fails to load, or the capture chain failing — realtime:stopped then follows with capture-failed.
token:errorreasonThe token could not be obtained.
measure:bvpindexBase, bvpOne pulse-waveform point, at capture rate.
measure:breathingindex, valueOne breathing-waveform point, per measured torso frame.
measure:variablesvariablesThe vitals, once a second.
measure:presenceface, body, faceBox, bodyBoxA subject entered or left the frame.
measure:statsstatsThe counters, once a second while a session runs.
RealtimeMeasure.create({
    onEvent: event => {
        switch (event.type) {
            case "measure:bvp":       chart.push(event.indexBase, event.bvp); break
            case "measure:breathing": breathing.push(event.index, event.value); break
            case "measure:variables": panel.show(event.variables); break
            case "measure:presence":  hint.toggle(event.face); break
            case "realtime:error":    banner.show(event.message); break
        }
    }
})

What the vitals carry

measure:variables carries the same shape the server computes: a heart rate, a breathing rate and where it was read from, variability indices, a quality score, and two flags worth reading before you display anything.

FieldWhat to do with it
heartRate, breathingRateThe two numbers a user expects: beats per minute, and breaths per minute. Either can be null when the signal allows no reading — test before formatting; so can heartRateIntervals and breathingSource.
breathingSource, breathingQuality, breathingWindowSWhere the breathing rate was read, how clearly it stood out (from 0 to 1), and over how many seconds.
stars, sqiSignal quality, as a whole number of stars from 0 to 5 and as a raw index from 0 to 1.
maturefalse until ten seconds have been measured. Say “measuring”, not a number.
confidentfalse when quality does not clear the bar, or when frames went missing — roughly 40 % of the last ten seconds or more — so that the signal is largely filled in rather than measured, even if its quality looks fine. The field does not say which of the two. Show it as provisional.
sdnn, rmssd, pnn50, sd1, sd2, lfHfVariability — in milliseconds, except pnn50, a percentage, and lfHf, a ratio — null when quality did not allow computing it, which takes a higher bar than confident asks for: confident: true with sdnn: null is normal.
heartRateIntervals, beats, rejectedThe interval-based rate, in beats per minute, and how many beat-to-beat intervals it kept and dropped.
timeThe capture time of the frame these values were computed for, in milliseconds since the session opened. It is an integer.
healthIndexReserved: it needs the subject’s age, which a realtime session has no way to receive today, so it is always null.

Nothing arrives at all until the window is long enough — about seven seconds into the measurement, the first two being discarded — so silence early on means not yet, never nothing to measure. And a variability field at null means no measurement — never zero, which would be the reading of a perfectly regular heart.

The breathing rate is read one of two ways, and breathingSource says which. motion is the rise and fall of the chest, measured on the torso frames — where breathing actually happens. bvp is the slow modulation the pulse wave carries. It is the fallback for a session that sends no torso, and also what a session that does send one reports while the torso reading has not concluded: that reading needs a signal of its own, confirmed over time, and starts over once the torso has been out of frame for more than two seconds. breathingSource can therefore change from bvp to motion — and back — during a measurement. The two are not worth the same, so a display that shows the number without reading its source shows two different things under one label. breathingQuality, between 0 and 1, says how clearly the rhythm stood out of the signal, and breathingWindowS the interval it was read over — several are tried at each publication and the one the signal resolves best on is kept, so it changes during a measurement. Both are null when the source is bvp.

measure:bvp carries indexBase, which identifies the capture it belongs to, counted from zero at the start of each session. It cannot be inferred from arrival order: a frame with no face is never sent, so the sequence has gaps. Those gaps are the honest picture of a measurement — a face that left the frame, a moment of movement — and drawing them as gaps rather than closing the line is what keeps a reader from believing the signal was continuous.

measure:breathing is the same idea on the torso: one point per measured torso frame, index naming that frame and skipping the same way — a capture in which no torso was found sends nothing. Its value is the cumulative vertical displacement of the chest, in 128ths of the height of the torso crop — a unit that does not change with the crop’s size — and only its variation means anything: the origin is wherever the first frame happened to sit, and it drifts slowly. Scale a chart to the minimum and maximum of what it shows, never to zero, and read a rhythm in it rather than a level. The rate itself is in the vitals.

Nothing is computed in the browser

The waveform points — pulse and breathing alike — arrive as the server computed them. Deriving your own heart rate or breathing rate from them would produce a second number, different and with no authority behind it.

Reading the detection, rather than being told about it

What the detector sees moves at capture rate. Turning that into an event would put thirty messages a second on a channel that, in a mobile integration, crosses a WebView bridge — so it is a read instead:

function drawOverlay() {
    const { source, face, body } = measure.getDetection()
    if (face !== null) {
        face.box              // { left, top, width, height } in source pixels — the crop actually sent
        face.normalizedBox    // the same, between 0 and 1: multiply by your display size
        face.landmarks        // 478 points, { x, y, z }, normalized
    }
    // … draw, then:
    requestAnimationFrame(drawOverlay)
}

Call it from your own render loop. The box comes in both frames of reference because it is eight numbers; the points come normalized only — the face alone has 478 of them, three numbers each — and source ({ width, height }) is all you need to place them in pixels. The torso box is not the bounding box of its points: it is built on the shoulders and hips alone — a raised wrist would otherwise stretch it across the whole image — widened by a small margin, kept inside the image, and held still until the subject really moves, a steady crop being what lets the chest’s movement be read. landmarks gives you all thirty-three points all the same.

The detector runs from the moment the session opens, which realtime:started announces, until whichever gesture ends the measurement — stop(), abort() or destroy(). Outside that interval — including while start() is still loading or waiting for the handshake — face and body are null.

measure:presence still tells you when a subject enters or leaves, which is the part you would otherwise have to poll for, and it carries faceBox/bodyBox — { box, normalizedBox } as in getDetection(), or null for a part that left the frame — the box at that instant, in both frames of reference. It is not tracked afterwards: that is what the read above is for.

Counters

getStats() returns cumulative counters. The same object arrives as measure:stats once a second while a session runs, so you rarely need to call it yourself.

FieldWhat it counts
frames, droppedFrames captured, and the times capture fell behind its cadence and skipped ahead.
detections, faceDetected, bodyDetectedDetection passes, and how many of them found a face, an upper body.
faceFrames, faceBytes, bodyFrames, bodyBytesFrames sent and bytes written on the wire, per stream, after compression.
savedBytes, faceSavedBytes, bodySavedBytesBytes the compression spared, in all and per stream.
queued, queuedPeakMessages waiting to be written, and the highest that queue has been during the session.
latencyLast, latencyAverage, latencyMax, latencySamplesThe round trip of a frame to its pulse-waveform point, in milliseconds — last, average, worst — and how many round trips they rest on.

Cumulative, never a rate: a rate needs a window, and that window is a display decision. Subtract two snapshots if you want throughput. And cumulative over the instance, not the session: a second start() carries on from where the first left off — take a snapshot at realtime:started and subtract it to get one session’s figures. Only the queue belongs to the session.

Two of them are easy to misread. savedBytes is the sum of faceSavedBytes and bodySavedBytes, so counting all three counts the saving twice. And latencySamples says how many round trips the three latency figures rest on: zero means nothing has come back yet, which a display has to tell apart from a latency of zero.

A queue that does not come back down

queued and queuedPeak are the early sign that capture is producing faster than the link absorbs — visible well before latency shows it. Lowering fps is the lever.