Wire protocol

For integrations that stream from a server rather than a browser: the transport, the streams, and the byte layout.

Alpha — read this first

Everything on this page is alpha: it can change without a deprecation window, and a change may require you to rewrite. The library documented in the rest of this section hides it precisely so that most integrations never have to read it.

Talk to us before you build against it. If you can run our module in a browser, do that instead.

There is one case this page exists for: the video does not pass through a browser you control. A teleconsultation whose media server already holds the stream, a recorded file replayed for validation, a mobile application with its own capture pipeline — those cannot load an ES module, and they talk to the transport directly.

The transport

WebTransport, which is HTTP/3 over QUIC. One session per measurement, opened against the analysis server of the environment you measure — the same host the library derives on its own, on the standard HTTPS port:

https://api-webtransport.<your environment host>/measures?accessTokenValue=<token>&visual=<…>

Which host that is depends on your environment:

EnvironmentSession address
Productionhttps://api-webtransport.saphere.ai/measures
Testhttps://api-webtransport.test.saphere.ai/measures

A server-to-server integration is still not something you switch on alone: tell us the shape of your source before you start.

The access token travels in the URL, as the accessTokenValue query parameter: a browser cannot set a header on a WebTransport handshake, and the interface is the same for everyone. It is minted exactly as for the library, with the permission measurement.realtime, and the handshake consumes it.

One path, and a session that declares what it will send

There is a single address. What a session will send is declared in its query, one parameter per modality — which is what lets a modality be added without an address being added.

https://api-webtransport.<your environment host>/measures
    ?accessTokenValue=<token>
    &visual=<what the camera sends>
    &format=<container>          # video only
    &timeOrigin=<milliseconds>   # optional, images only
    &loggerId=<uuid>             # optional
visualYou sendWhat the server does
images-partsRGB crops — a face, a torsomeasures them as they are
images-fullwhole RGB framesfinds the face, crops, measures
video-partsa video of a cropcuts it into frames, measures them as they are
video-fulla video of whole framescuts, finds the face, crops, measures

format says the container — webm, mp4, or h264 for a raw Annex B stream — and belongs to a video session only: declared on an images session it is refused, because nothing would read it.

timeOrigin is optional, and belongs to an images session only: the origin of your timestamps, in milliseconds, as a whole number from 0 to 2147483647 — 0 when it is absent. The server subtracts it from the time of every frame as it arrives, so that the times it sends back count from your origin rather than from wherever your clock happened to stand: with timeOrigin=2000, a frame captured at 2010 is a frame at 10, and so are the hr point and the variables computed from it. Take it from the clock your frames are timed with, just before you open the session — no frame may be older than the origin. Declared on a video session it is refused: there the times are read from your container, not sent. Every time the server keeps or returns is on that scale, starting at 0 — including the duration the API gives, afterwards, for a realtime measure.

loggerId is optional and only informs: a random UUID that identifies the instance of your page, and that the server writes on the log lines of the session so that a session can be traced back to the page that opened it. A value that is not a UUID is ignored — never a reason to refuse — and nothing the server does depends on it.

voice is the next modality. It is part of the vocabulary already, and a session that declares it is refused — nothing measures a voice yet, and serving such a session on its visual half alone would let you believe a measurement is happening where none is.

Everything is checked at the handshake, before anything is written, so no measurement exists and no token is spent for a session that would have measured nothing. The checks run family by family, in the order of the table: capacity, the shape of the token, what the session declares, then the token and the account. The first family that refuses answers; inside a family, a session with two faults may get either status.

StatusWhen
503The server is at capacity or under memory pressure. Nothing has been read yet: treat it as try again shortly, never as a credential problem.
401No accessTokenValue, or one that is not a well-formed token.
400visual missing or unknown; format missing or unknown on a video session; format declared on an images session; timeOrigin that is not a whole number from 0 to 2147483647, or declared on a video session.
501voice declared; a whole-frame session while the server cannot detect faces; a video session while it cannot decode video.
401The token is expired, already spent, revoked, minted without measurement.realtime, or its account has been deactivated.
403Realtime measurement is not available on your account: no cost has been set for it. Or the credit left for the month does not cover the first slice, which a session pays when it opens.

Only a session that opens spends its token, and its first slice of credit. A browser does not expose these statuses — its ready promise rejects the same way for all of them — so they are for a client of your own to read.

Input: one stream per source

You open a unidirectional stream — the server only reads it — write what your session has not already said, and then send. On an images session, one message per frame, back to back; on a video session, the container bytes as they are.

The byte layout of an images streamA stream opens with one byte naming what it carries — and only when the session announced crops — then carries one message per frame: index, timestamp, flags, width, length, and the payload itself.ONCE, WHEN THE STREAM OPENScontent1 Bwhich part it carries — and only whenthe session announced cropsTHEN ONE MESSAGE PER FRAMEindex4 Btime4 Bflags1 Bwidth2 Blength4 Bpayloadas many as length saysno separator, no padding: a reader advances by what length announced
A stream declares one thing, and only when there is a choice: which part it carries, when the session announced crops. A session that already said everything opens straight into its messages — and nothing separates them, a reader advancing by the length each one announced.
Header fieldBytesValue
content10 face, 1 body — images-parts and video-parts only

Every integer is unsigned and big-endian.

Message fieldBytesValue
index4the capture this frame belongs to. It may skip.
time4the capture timestamp, in milliseconds. It never goes back within a stream, and it is never older than the session’s timeOrigin.
flags1bit 0: the payload is brotli-compressed. The other seven are reserved.
width2the width of this frame, from 1 to 1920 — the height it implies may not exceed 1920 either, and the frame may not hold more pixels than 1920 × 1080
length4the payload length, from 1 byte to 10 MiB

The payload is 8-bit RGB, interleaved — R, G, B for each pixel, no alpha, not BGR. index and time are unsigned 32-bit integers: count time from the start of your capture, not from the Unix epoch, whose milliseconds no longer fit in 32 bits (the ceiling is 4294967295 ms, about 49.7 days).

Written as code, one message of an images session:

// The stream of an `images-parts` session opens, once, with its `content` byte: `0` face, `1` body.
function encodeMessage({ index, time, width, rgb }) {    // rgb: width × height × 3 bytes
    const message = new Uint8Array(15 + rgb.length)
    const view = new DataView(message.buffer)            // big-endian, which is the default
    view.setUint32(0, index)                             // the capture this frame belongs to
    view.setUint32(4, time)                              // its capture timestamp, in milliseconds
    view.setUint8(8, 0)                                  // flags: bit 0 clear, the payload is not compressed
    view.setUint16(9, width)
    view.setUint32(11, rgb.length)                       // the payload length
    message.set(rgb, 15)
    return message
}

Five things about it are not guessable:

  • The height does not travel. A decompressed RGB payload weighs width × height × 3, so the height follows from the length — and what that buys is not a byte, it is the right to change size from one frame to the next: a crop follows a face that moves. A length that does not make whole rows is refused rather than guessed.
  • index is the capture, and it skips. A frame with no face is never sent, so the sequence has gaps. Do not renumber to close them: the gap is what says a moment was not measured.
  • time is the capture timestamp, not the send time. It is what the measurement reads a heart rate from. Deriving it from a frame counter divides the pulse by whatever cadence you assumed, and the error is silent — the quality score will not catch it.
  • Compression is per message. The flag is read on every frame, so an encoder may stop compressing under load without renegotiating anything.
  • One stream per content. A second stream of the same content ends the session: it would open a second measurement of the same thing, and interleave two signals with no error raised. That holds even once the first stream has ended: a content cannot be opened twice in one session. A stream of another content is accepted — that is how a face and a torso travel together — while a session of whole frames carries a single stream in all.

A malformed stream ends the session, it is not skipped. An unknown content byte, a width or a height out of bounds, an empty or oversized payload, a decompressed length that does not make whole rows, a timestamp that goes back or that is older than the timeOrigin the session declared, a stream that ends in the middle of a message, a second stream of the same content: each closes the whole session, with close code 1 and the reason measure stream failed. Code 1 is the server’s, and it is not only for malformed streams: a failure on the server’s own side ends the session the same way. When the server itself stops — for a restart, say — it closes every open session with code 0 and the reason Server shutting down: open a new one, with a new token. When you close a session yourself, the library’s own convention is 2 for a requested stop, 3 for an abort and 4 when its own capture chain failed.

Credit is paid in advance, one 60-second slice at a time: the first when the session opens — a session whose credit does not cover it is refused at the handshake, with 403 — then one at the start of every further minute the session stays open, whether it receives data or not. A session whose credit no longer covers the next slice is closed by the server, with close code 5 and the reason credit exceeded — see the credit system in the FAQ. Nothing already received is lost: what was measured stays measured. Nothing is charged at the close, since the slice under way is already paid; the shortest session costs one slice.

A video session carries no messages

After its header — the content byte on a video-parts session, nothing at all on a video-full one — a video session takes the container bytes, as they are: no per-frame framing, since the container already has its own. The server cuts the frames out itself, and four of its rules matter to you:

  • mp4 means fragmented mp4. An ordinary mp4 carries its index at the end, so a stream of one yields no frames at all. What a browser’s recorder produces is already fragmented; a file pushed from disk usually is not.
  • The cadence is a ceiling, not a target. The server keeps at most 30 frames per second and follows your source below that — the decoder never duplicates a frame to reach a rate. The timestamps it uses are the ones your container carries.
  • Nothing is resized. Frames are decoded at your source’s size, which is also why no dimensions are declared. A source that changes resolution mid-stream is refused, and so is a frame with a side beyond 1920 or more pixels than 1920 × 1080 — a portrait 1080 × 1920 source is accepted, a square 1920 × 1920 one is not.
  • Leading negative timestamps are rebased; unreadable ones, and ones that go back, are refused. A container with an edit list — an iOS capture, typically — legitimately starts a few frames before zero: the scale is shifted so nothing precedes zero, every gap preserved, and the gaps are all a heart rate reads. A frame whose timestamp cannot be read refuses the session rather than guessing.

Output: one stream per kind

The server opens the return streams, one per kind. Each begins with a single byte naming it, then carries length-prefixed JSON messages — four bytes of big-endian length, then the payload.

ByteStreamWhat it carries
0hrthe pulse waveform: { index, time, value }
1brthe breathing waveform: { index, time, value }
2variablesthe vitals, once a second — the same shape the library reports
3sessionthe identifier of the measure: { measureId }

session comes first. The server opens it as soon as the session is established, writes a single message and closes it. measureId names the measure this session records: hand it to your own server, which reads the measure with its API key — its essentials, and its outputs once recorded.

The list of kinds is open. A kind your reader does not know is read to its end and thrown away, never treated as an error: kinds may be added, and a client that stops on an unknown one would break the day they are.

There is no discriminator inside a message: the stream said its kind once, at its start. That is what lets a reader subscribe to the vitals alone and never decode a waveform it does not draw. The reader must still read that stream, and throw it away: a stream nobody reads eventually holds back the others.

The two signals have the same shape, deliberately: a reader that can draw one can draw the other, and a signal added later teaches it nothing new. A point says three things — where it sits in its signal, when it was captured, and what it is worth.

They do not arrive at the same rhythm, because they are not read from the same thing. The pulse gives one point per measured face frame; breathing gives one point per measured torso frame — it is the rise and fall of the chest, where breathing actually happens. Both indexes skip, and for the same reason: a capture with no face to measure — or no torso — produces no point. A session that sends no torso therefore has no breathing waveform to draw — and neither has a session of whole frames, images-full or video-full: the server finds the face in a whole frame, not the torso. Its breathing rate then comes from the slow modulation of the pulse wave instead — variables says which of the two you are reading.

A breathing point’s value is a displacement, not a level: the cumulative vertical movement of the chest, in 128ths of the height of the torso crop it was measured on — a unit that does not change with the crop’s size. Its origin is wherever the first frame happened to sit, and it drifts slowly, so only the variation means anything. Scale a chart to the minimum and maximum of what it shows, never to zero.

A video session, end to end

What follows is the whole client, in one page. It is browser code — WebTransport is a browser API — and a server integration differs in one line: Node has no WebTransport of its own, and @fails-components/webtransport provides a client with the same interface (import { WebTransport } from "@fails-components/webtransport").

Sending

One session, one unidirectional stream, no header at all — the session declared everything in its URL — then your video as it comes.

const origin = "https://api-webtransport.<your host>"
const url = `${origin}/measures`
    + `?visual=video-full`                 // whole frames, in a container: the server finds the face
    + `&format=webm`                       // the container — declared here, not on the wire
    + `&accessTokenValue=${encodeURIComponent(token)}`

const session = new WebTransport(url)
const unanswered = new Promise((_, reject) =>
    setTimeout(reject, 10_000, new Error("the handshake was not answered in 10 seconds")))
await Promise.race([session.ready, unanswered])   // the wait is bounded, as the library bounds it

const writer = (await session.createUnidirectionalStream()).getWriter()

// No header: a session that sends whole frames has nothing left to declare, and its container is in the
// URL. The frame size is your source's, and the server reads it from the stream itself.

// Straight to the container bytes. Nothing frames them: it is your video, byte for byte.
const camera = await navigator.mediaDevices.getUserMedia({ video: true, audio: false })
const mimeType = ["video/webm;codecs=h264", "video/webm"].find(type => MediaRecorder.isTypeSupported(type))
const recorder = new MediaRecorder(camera, { mimeType, videoBitsPerSecond: 2_500_000 })
recorder.ondataavailable = async event => {
    if (event.data.size === 0) return
    await writer.ready               // backpressure: QUIC will ask you to wait
    await writer.write(new Uint8Array(await event.data.arrayBuffer()))
}
recorder.start(200)                  // a fragment every 200 ms

The wait on ready is bounded at ten seconds, as the library bounds it: some browsers neither accept the handshake nor refuse it — see what the browser has to provide.

Ending is two steps, in this order. await writer.close() ends the video — and the measurement is not over: frames are still in the decoder and their results are still coming. Close the session once you have read what you came for.

Receiving

The server opens session first, as soon as the session is established; then one stream per kind, at the first value of that kind: the pulse waveform within a second or so, the vitals after about seven seconds, the breathing waveform as soon as a torso frame has been measured — which, on a whole-frame session like this one, is never. So accept incoming streams for the whole session, and read each one alongside the others — a loop that awaits one stream to its end never sees the next.

const KINDS = ["hr", "br", "variables", "session"]

const incoming = session.incomingUnidirectionalStreams.getReader()
for (;;) {
    const { value, done } = await incoming.read()
    if (done) break
    readOutputStream(value, onMessage)   // not awaited: the streams are read at once
}

And the reader itself. QUIC delivers chunks, not messages: a boundary falls in the middle of a frame as happily as anywhere else, and the kind byte can arrive on its own. Hence the buffer.

async function readOutputStream(stream, onMessage) {
    const reader = stream.getReader()
    let buffer = new Uint8Array(0)
    let kind = null

    for (;;) {
        const { value, done } = await reader.read()
        if (done) return

        const merged = new Uint8Array(buffer.length + value.length)
        merged.set(buffer, 0)
        merged.set(value, buffer.length)
        buffer = merged

        if (kind === null) {
            if (buffer.length < 1) continue
            kind = KINDS[buffer[0]] ?? `unknown(${buffer[0]})`
            buffer = buffer.subarray(1)   // the kind byte is read once, and once only
        }

        for (;;) {
            if (buffer.length < 4) break
            const length = new DataView(buffer.buffer, buffer.byteOffset)
                .getUint32(0, false)
            if (buffer.length < 4 + length) break

            const payload = buffer.subarray(4, 4 + length)
            onMessage(kind, JSON.parse(new TextDecoder().decode(payload)))
            buffer = buffer.subarray(4 + length)
        }
    }
}

What comes back

function onMessage(kind, message) {
    if (kind === "hr")        chart.push(message.time, message.value)
    if (kind === "br")        breathing.push(message.time, message.value)
    if (kind === "variables") panel.show(message)
    if (kind === "session")   measureId = message.measureId   // for your server, afterwards
    // any other kind: ignore it — the list is open
}

hr arrives at the rate of the measured face frames, br at the rate of the measured torso frames, and variables once a second of capture time (a file pushed faster than real time gets them faster) — nothing at all until the window is long enough, so silence early on means not yet, never nothing to measure.

Field of variablesWhat to do with it
heartRate, breathingRateThe two numbers a user expects: beats per minute, and breaths per minute. Either can be null when the signal allows no reading — test before formatting; so can heartRateIntervals and breathingSource.
breathingSource, breathingQuality, breathingWindowSWhere the breathing rate was read — motion from the torso, bvp from the pulse wave — how clearly it stood out (from 0 to 1), and over how many seconds. The last two are null for bvp. A session that does send a torso can report bvp too, while the torso reading has not concluded — it takes tens of seconds, and starts over after more than two seconds without a torso — so breathingSource can change during a measurement.
breathingRate timingA torso-read rate needs about 30 seconds: the server refuses to name a rhythm until a reading from ten seconds earlier agrees with the current one. A rhythm that merely drifts is followed; a chest that is not breathing gives no torso reading rather than a number; and more than two seconds without a torso starts the reading over.
stars, sqiSignal quality, as a whole number of stars from 0 to 5 and as a raw index from 0 to 1.
maturefalse until ten seconds have been measured. Say “measuring”, not a number.
confidentfalse when quality does not clear the bar, or when frames went missing — roughly 40 % of the last ten seconds or more — so that the signal is largely filled in rather than measured, even if its quality looks fine. The field does not say which of the two. Show it as provisional.
sdnn, rmssd, pnn50, sd1, sd2, lfHfVariability — in milliseconds, except pnn50, a percentage, and lfHf, a ratio — null when quality did not allow computing it.
heartRateIntervals, beats, rejectedThe interval-based rate, in beats per minute, and how many beat-to-beat intervals it kept and dropped.
timeThe capture time of the frame these values were computed for, in milliseconds — the same scale as the time of the hr and br points, your timeOrigin already subtracted. It is an integer.
healthIndexReserved: it needs the subject’s age, which a session has no way to receive today, so it is always null.

A variability field at null means no measurement — never zero, which would be the reading of a perfectly regular heart.

hr and br each carry index, the capture their point belongs to, and both skip: a capture with no face to measure — for br, no torso — produces no point. Draw those gaps as gaps. Closing the line would tell a reader the signal was continuous when it was not, and the index is the only thing that says otherwise. A frame that reaches the measurement but cannot be measured there still produces a point, at the previous value: a flat stretch is not a gap, and nothing on the wire tells the two apart. On a video session you send no index: the one that comes back is the rank, from 0, of each frame the server kept — not a position in your source.

Do not recompute

The waveform points are the server’s, computed on the whole signal it holds. Deriving your own heart rate from the slice you received would produce a second number, different and with no authority behind it.

Five things to get right

  • Nothing is declared about size, and nothing is resized. Frames are decoded at your source’s own size — no side beyond 1920 and no more pixels than 1920 × 1080, beyond which the session is refused. Landscape 1920 × 1080 and portrait 1080 × 1920 both fit. Sending a smaller source is still the cheapest way to use less bandwidth and processing, on your side and on ours.
  • mp4 means fragmented mp4. An ordinary one carries its index at the end, so a stream of it yields no frames at all. A browser recorder already produces fragments; a file pushed from disk usually does not.
  • The cadence is a ceiling, and it has a floor. At most 30 frames a second, and your source’s rate below that — the decoder never duplicates a frame, so a 25-frame source is measured at 25. But the measurement itself runs on a grid of 30 steps a second: when two frames are more than 50 ms apart — any source below 20 frames a second, and a 20-frame source as soon as its cadence jitters — it fills the gap by repeating the current frame, and those repeats count as missing frames for confident. Send 25 frames a second or more.
  • Timestamps come from your container, frame by frame, and they are what a heart rate is read from. A container written with a made-up or constant time base yields a plausible pulse that is simply wrong, and the quality score will not catch it.
  • Write with backpressure. await writer.ready before each write. Without it, a slow link turns into memory in your own process, and the failure appears somewhere it did not start.

What is not settled yet

Being alpha, this is where it moves: the video path accepts no per-frame timestamps of its own, the server finds no torso in a whole frame — breathing is read from the chest only when you send its crop — and composing a face from a whole frame you also send is not implemented: a whole frame is measured by the server’s own detection, or not at all.