Wire protocol

There is one case this page exists for: the video does not pass through a browser you control. A teleconsultation whose media server already holds the stream, a recorded file replayed for validation, a mobile application with its own capture pipeline — those cannot load an ES module, and they talk to the transport directly.

The transport

WebTransport, which is HTTP/3 over QUIC. One session per measurement, opened against the analysis server of the environment you measure — the same host the library derives on its own, on the standard HTTPS port:

https://api-webtransport.<your environment host>/measures?accessTokenValue=<token>&visual=<…>

Which host that is depends on your environment:

A server-to-server integration is still not something you switch on alone: tell us the shape of your source before you start.

The access token travels in the URL, as the accessTokenValue query parameter: a browser cannot set a header on a WebTransport handshake, and the interface is the same for everyone. It is minted exactly as for the library, with the permission measurement.realtime, and the handshake consumes it.

One path, and a session that declares what it will send

There is a single address. What a session will send is declared in its query, one parameter per modality — which is what lets a modality be added without an address being added.

https://api-webtransport.<your environment host>/measures
    ?accessTokenValue=<token>
    &visual=<what the camera sends>
    &format=<container>          # video only

format says the container — webm, mp4, or h264 for a raw Annex B stream — and belongs to a video session only: declared on an images session it is refused, because nothing would read it.

voice is the next modality. It is part of the vocabulary already, and a session that declares it is refused — nothing measures a voice yet, and serving such a session on its visual half alone would let you believe a measurement is happening where none is.

Everything is checked at the handshake, before anything is written, so no measurement exists and no token is spent for a session that would have measured nothing. The checks run in this order, and the first one that fails answers:

Only a session that opens spends its token. A browser does not expose these statuses — its ready promise rejects the same way for all of them — so they are for a client of your own to read.

Input: one stream per source

You open a unidirectional stream — the server only reads it — write what your session has not already said, and then send. On an images session, one message per frame, back to back; on a video session, the container bytes as they are.

Every integer is unsigned and big-endian.

Five things about it are not guessable:

• The height does not travel. A decompressed RGB payload weighs width × height × 3, so the height follows from the length — and what that buys is not a byte, it is the right to change size from one frame to the next: a crop follows a face that moves. A length that does not make whole rows is refused rather than guessed.
• index is the capture, and it skips. A frame with no face is never sent, so the sequence has gaps. Do not renumber to close them: the gap is what says a moment was not measured.
• time is the capture timestamp, not the send time. It is what the measurement reads a heart rate from. Deriving it from a frame counter divides the pulse by whatever cadence you assumed, and the error is silent — the quality score will not catch it.
• Compression is per message. The flag is read on every frame, so an encoder may stop compressing under load without renegotiating anything.
• One stream per content. A second stream of the same content ends the session: it would open a second measurement of the same thing, and interleave two signals with no error raised. A stream of another content is accepted — that is how a face and an upper body travel together — while a session of whole frames carries a single stream in all.

A malformed stream ends the session, it is not skipped. An unknown content byte, a width or a height out of bounds, an empty or oversized payload, a decompressed length that does not make whole rows, a timestamp that goes back, a stream that ends in the middle of a message, a second stream of the same content: each closes the whole session, with close code 1 and the reason measure stream failed. Code 1 is the server’s; when you close a session yourself, the library’s own convention is 2 for a requested stop and 3 for an abort.

A video session carries no messages

After its header — the content byte on a video-parts session, nothing at all on a video-full one — a video session takes the container bytes, as they are: no per-frame framing, since the container already has its own. The server cuts the frames out itself, and four of its rules matter to you:

• mp4 means fragmented mp4. An ordinary mp4 carries its index at the end, so a stream of one yields no frames at all. What a browser’s recorder produces is already fragmented; a file pushed from disk usually is not.
• The cadence is a ceiling, not a target. The server keeps at most 30 frames per second and follows your source below that — it never duplicates a frame to reach a rate. The timestamps it uses are the ones your container carries.
• Nothing is resized. Frames are decoded at your source’s size, which is also why no dimensions are declared. A source that changes resolution mid-stream is refused, and so is a frame wider than 1920 or taller than 1080 — a portrait 1080 × 1920 source included.
• Leading negative timestamps are rebased; unreadable ones, and ones that go back, are refused. A container with an edit list — an iOS capture, typically — legitimately starts a few frames before zero: the scale is shifted so nothing precedes zero, every gap preserved, and the gaps are all a heart rate reads. A frame whose timestamp cannot be read refuses the session rather than guessing.

Output: one stream per kind

The server opens the return streams, one per kind. Each begins with a single byte naming it, then carries length-prefixed JSON messages — four bytes of big-endian length, then the payload.

There is no discriminator inside a message: the stream said its kind once, at its start. That is what lets a reader subscribe to the vitals alone and never decode a waveform it does not draw.

The two signals have the same shape, deliberately: a reader that can draw one can draw the other, and a signal added later teaches it nothing new. A point says three things — where it sits in its signal, when it was captured, and what it is worth.

They do not arrive at the same rhythm, because they are not read from the same thing. The pulse gives one point per measured face frame; breathing gives one point per measured torso frame — it is the rise and fall of the chest, where breathing actually happens. Both indexes skip, and for the same reason: a frame in which the server found nothing produces no point. A session that sends no torso therefore has no breathing waveform to draw — and neither has a session of whole frames, images-full or video-full: the server finds the face in a whole frame, not the upper body. Its breathing rate then comes from the slow modulation of the pulse wave instead — variables says which of the two you are reading.

A breathing point’s value is a displacement, not a level: the cumulative vertical movement of the chest, in 128ths of the height of the torso crop it was measured on — a unit that does not change with the crop’s size. Its origin is wherever the first frame happened to sit, and it drifts slowly, so only the variation means anything. Scale a chart to the minimum and maximum of what it shows, never to zero.

A video session, end to end

What follows is the whole client, in one page. It is browser code — WebTransport is a browser API — and a server integration differs in one line: Node has no WebTransport of its own, and @fails-components/webtransport provides a client with the same interface (import { WebTransport } from "@fails-components/webtransport").

Sending

One session, one unidirectional stream, no header at all — the session declared everything in its URL — then your video as it comes.

const origin = "https://api-webtransport.<your host>"
const url = `${origin}/measures`
    + `?visual=video-full`                 // whole frames, in a container: the server finds the face
    + `&format=webm`                       // the container — declared here, not on the wire
    + `&accessTokenValue=${encodeURIComponent(token)}`

const session = new WebTransport(url)
await session.ready

const writer = (await session.createUnidirectionalStream()).getWriter()

// No header: a session that sends whole frames has nothing left to declare, and its container is in the
// URL. The frame size is your source's, and the server reads it from the stream itself.

// Straight to the container bytes. Nothing frames them: it is your video, byte for byte.
const camera = await navigator.mediaDevices.getUserMedia({ video: true, audio: false })
const mimeType = ["video/webm;codecs=h264", "video/webm"].find(type => MediaRecorder.isTypeSupported(type))
const recorder = new MediaRecorder(camera, { mimeType, videoBitsPerSecond: 2_500_000 })
recorder.ondataavailable = async event => {
    if (event.data.size === 0) return
    await writer.ready               // backpressure: QUIC will ask you to wait
    await writer.write(new Uint8Array(await event.data.arrayBuffer()))
}
recorder.start(200)                  // a fragment every 200 ms

Ending is two steps, in this order. await writer.close() ends the video — and the measurement is not over: frames are still in the decoder and their results are still coming. Close the session once you have read what you came for.

Receiving

The server opens one stream per kind, at the first value of that kind: the pulse waveform within a second or so, the vitals after about seven seconds, the breathing waveform as soon as a torso frame has been measured — which, on a whole-frame session like this one, is never. So accept incoming streams for the whole session, and read each one alongside the others — a loop that awaits one stream to its end never sees the next.

const KINDS = ["hr", "br", "variables"]

const incoming = session.incomingUnidirectionalStreams.getReader()
for (;;) {
    const { value, done } = await incoming.read()
    if (done) break
    readOutputStream(value, onMessage)   // not awaited: the three are read at once
}

And the reader itself. QUIC delivers chunks, not messages: a boundary falls in the middle of a frame as happily as anywhere else, and the kind byte can arrive on its own. Hence the buffer.

async function readOutputStream(stream, onMessage) {
    const reader = stream.getReader()
    let buffer = new Uint8Array(0)
    let kind = null

    for (;;) {
        const { value, done } = await reader.read()
        if (done) return

        const merged = new Uint8Array(buffer.length + value.length)
        merged.set(buffer, 0)
        merged.set(value, buffer.length)
        buffer = merged

        if (kind === null) {
            if (buffer.length < 1) continue
            kind = KINDS[buffer[0]] ?? `unknown(${buffer[0]})`
            buffer = buffer.subarray(1)   // the kind byte is read once, and once only
        }

        for (;;) {
            if (buffer.length < 4) break
            const length = new DataView(buffer.buffer, buffer.byteOffset)
                .getUint32(0, false)
            if (buffer.length < 4 + length) break

            const payload = buffer.subarray(4, 4 + length)
            onMessage(kind, JSON.parse(new TextDecoder().decode(payload)))
            buffer = buffer.subarray(4 + length)
        }
    }
}

What comes back

function onMessage(kind, message) {
    if (kind === "hr")        chart.push(message.time, message.value)
    if (kind === "br")        breathing.push(message.time, message.value)
    if (kind === "variables") panel.show(message)
}

hr arrives at the rate of the measured face frames, br at the rate of the measured torso frames, and variables once a second — nothing at all until the window is long enough, so silence early on means not yet, never nothing to measure.

A variability field at null means no measurement — never zero, which would be the reading of a perfectly regular heart.

hr and br each carry index, the capture their point belongs to, and both skip: a frame the server measured nothing in produces no point. Draw those gaps as gaps. Closing the line would tell a reader the signal was continuous when it was not, and the index is the only thing that says otherwise.

Five things to get right

• Nothing is declared about size, and nothing is resized. Frames are decoded at your source’s own size — at most 1920 wide and 1080 high, beyond which the session is refused, a portrait 1080 × 1920 source included. Sending a smaller source is still the cheapest way to spend less, on your side and on ours.
• mp4 means fragmented mp4. An ordinary one carries its index at the end, so a stream of it yields no frames at all. A browser recorder already produces fragments; a file pushed from disk usually does not.
• The cadence is a ceiling. At most 30 frames a second, and your source’s rate below that. Nothing is ever duplicated to reach a number, so a 25-frame source is measured at 25.
• Timestamps come from your container, frame by frame, and they are what a heart rate is read from. A container written with a made-up or constant time base yields a plausible pulse that is simply wrong, and the quality score will not catch it.
• Write with backpressure. await writer.ready before each write. Without it, a slow link turns into memory in your own process, and the failure appears somewhere it did not start.