How the measurement works

How light carries a pulse

Blood absorbs light. Not all of it, and not evenly, but enough for a camera to notice.

At each heartbeat, a little more blood fills the tiny vessels just under the skin. That blood absorbs part of the light falling on the face, so slightly less light is reflected back. The skin darkens a little as the pulse arrives, and lightens again as it passes.

The change is far too small for anyone to see. A camera sensor can record it, on two conditions: you have to know which parts of the face to look at, and you have to average over many pixels and many images.

This technique is called photoplethysmography, which means “writing the volume with light”. With a sensor in contact with the skin, it is what the clip on a hospital patient’s finger does. At a distance, with an ordinary camera and daylight, it is called remote photoplethysmography, or rPPG. That is what Saphere does.

From images to a pulse wave

Four things happen, in this order.

A model finds the face. Not to recognise anyone, only to know where the face is and how it is turned. This model runs inside the browser, on every image.

The steadiest patches of skin are selected. Not the whole image, and not even the whole face. The measurement uses the places where the skin is thin, well supplied with blood and moves little — the forehead above all. The eyes, the mouth and the hair are left out, because they move, and movement is noise.

Each image is cropped and lightened. The widget keeps only the face, and only the single colour channel that carries the pulse signal. That is a quarter of the data of a colour image, and it is why a measurement sends far less than a video would. This is why a measurement sends far less data than a video, and why the amount sent does not depend on how close the person sits.

The numbers, put end to end, draw a wave. Over about thirty seconds, they trace a regular curve. How often it repeats gives the heart rate. The spacing between beats gives the variability. A slower movement carried on top of it gives the breathing rate. The shape of the wave, combined with what you told us about the person, gives the remaining indicators.

Why the requirements are what they are

Each requirement listed in what Saphere is follows directly from the paragraphs above.

Light must fall on the face, not behind it. The signal is reflected light. If a bright window sits behind the person, the camera adjusts its exposure for the window and the face falls into shadow. What the measurement needs is then squeezed into a handful of dark values, and nothing can bring it back.

The light must not change during the capture. A cloud passing, a screen changing colour, someone switching a lamp on: each writes into the signal a variation larger than the pulse, which cannot be told apart from it.

The person must stay still. When the face moves, the selected patches slide onto other parts of the skin, receive light from another angle and land on another part of the sensor. The variation this produces is far larger than the one the heartbeat produces.

The face must be uncovered and inside the frame. Skin the camera cannot see contributes nothing. A face partly outside the frame leaves fewer usable patches, and a noisier average.

The full journey, end to end

A live measurement through the widget goes through five stages.

1 — The widget obtains a token. At the start of each measurement it asks your application for a short-lived access token. Your server issues it using your API key. The widget never sees the key itself.

2 — The camera opens and the person is guided into position. The face-detection model runs on the device. Nothing has left it yet.

3 — The capture runs for about thirty seconds. Each image is reduced in the browser, then sent to the measurement server, which acknowledges every one. A slow network stretches this out but loses nothing, and the widget tells you when it happens.

4 — The server computes. Once the last image is acknowledged, the widget signals the end of the capture and waits. The computation happens on the server.

5 — The result comes back over the same connection. It is checked, then handed to your onEvent handler as measure:result. It is also stored, so you can read the same measurement again later through the API.

For a recorded file, the first four stages become a single HTTP call: you send the video or the images, the server does the reduction itself, and its answer is the finished measurement.

Where the data goes

Two different things leave the device, and it is worth knowing exactly what each one is.

During the capture, about nine hundred small images are sent. Each one is cropped to the face — the rest of the picture is discarded on the device — and reduced to a single colour channel, the one that carries the pulse. That is one byte per pixel instead of four, which is why a measurement is far lighter than a video of the same length.

Once, halfway through the capture, a single full photograph is sent. It is what the variables read from the face need: visual age, and body mass index estimated from the face. It is a real picture of what the camera sees at that moment.

Afterwards, the finished measurement is stored under your account, along with the signal it was computed from and any userData you supplied. It stays readable through the API until you delete it.