Satellite Protocol

What a board has to speak to be a Satellite. Any board that speaks it is added in Settings > Devices as type Satellite. It needs no driver and no plugin. A Raspberry Pi speaks it, and so can an ESP32.

board    --POST /api/devices/<name>/voice-->   Sapphire   what it heard
board    --POST /api/devices/<name>/press-->   Sapphire   a button set to send her a message
board    --POST /api/devices/<name>/ask----->   Sapphire   work that is hers to do: stop, backup
board    --GET  /api/devices/<name>/events-->  Sapphire   what its light should show
Sapphire --GET /health, POST /audio/speak--->  board      status, say, light, ...

Everything is plain HTTP with a bearer key, on the local network only.

The board says what it is

GET /health is the one address every board must answer.

{"ok": true, "name": "den", "board": "waveshare-s3-audio", "firmware": "0.1.0",
 "has": ["speaker", "mic", "wake"],
 "plays": {"type": "audio/wav", "rate": 16000, "channels": 1},
 "volume": 85, "uptime_s": 61,
 "wakeword": {"enabled": true, "running": true, "model": "hey_sapphire", "format": "tflite"},
 "link": {"connected": true}}

has takes these names: speaker, mic, light, wake, camera, power, storage, screen, keyboard, buttons, sensors, firmware. A name Sapphire does not know is left out and logged. storage is said only while the card is mounted: no card, no Backup tab. keyboard is a board that types at her instead of listening (a pocket terminal, tmp/pocket-esp32): it gets the mic's key and chat settings, and her reply goes to its screen as she writes it (below).

plays has one type today, audio/wav, which is always 16 bit PCM. rate is 8000 to 48000. channels is 1 or 2. A small board plays this with no decoder and almost no memory.

A board that states neither is taken as an early Pi: it has everything, and it gets the sound as the voice engine made it (ogg, wav or mp3), as a form file.

What a board takes

Sapphire shows a setting or an action only on a board that says it takes it (_takes in the satellite driver; the engine keeps the list with the device, so it holds while the board is off):

• each key in /health screen is a screen setting it takes (brightness, dim, dim_after_s, off_after_min, flip, rotation, invert, mirror, home)
• screen with w, h and format: it takes a picture
• mic.max_s: it takes the longest question
• led.looks, a list of look names: the only looks it takes. A board that lists none takes them all

What Sapphire asks of the board

A board answers only the addresses of what it has.

/audio/speak carries the sound as the request body, with its Content-Type, when the board stated plays. The board can play it while it arrives, or take it whole first and play it from memory (Waveshare 0.7.7: its /health then says sound: {"queued_kb", "late_count", "late_total_ms", "late_last_ms", "heals", "heal_last"}, where late is playing longer than the sound is long, and a heal is a sound chip found reset and set up again before a sentence). Either way it answers once the sound is taken. A board that keeps a queue and has no room holds its answer until there is (Waveshare: while what is queued plays, plus 10 s), so a reply made faster than she speaks waits at the door; a stop meanwhile answers it stopped.

Her reply comes one sentence at a time. While she is still writing, each finished sentence is rendered, trimmed to a breath of silence at each end (a sentence rendered alone carries ~0.3 s before and ~0.4 s after; joined, that was a hole at every joint) and sent as its own /audio/speak, in order, the next one only after the board has answered the last. The first sound arrives a second or two after her first sentence, not after her whole reply. A board keeps its speaker open across the sentences of one reply (the Pi body feeds them into one aplay; opening a Bluetooth speaker costs half a second, which was a hole between every two sentences) and answers each /audio/speak once the sound is taken, so the next sentence is already there before the last has finished. Between two sentences the turn is still hers: when the sound has ended the board goes back to the brain's last cue from the events stream (thinking, tool) and only to idle when that stream says so. A stopped answer ends the rest of that reply; nothing more is sent for it, and the board refuses what was already on its way. A board whose driver cannot take sound this way hears the reply whole, at the end, as before.

A refusal is any status from 400 up with {"detail": "the reason"}. The reason is shown as it is. 401 or 403 means the key was wrong.

What the light shows, first wins: a state of the board (listening, thinking, tool, speaking, error); then what she set with /led, held for its time, or dark after {"state": "off"} until {"state": "idle"}; then, inside the hours, the resting look, and outside them the night look, where the color off is dark. A state always shows, through a blackout and at any hour, so the ring says when she is listening. The ESP32 firmware (0.7.0) is stricter about off: a ring turned off, by her or by its own button, is dark through every state except the camera's warning, and stays off across a restart (led.off in /health); its screen still shows the state. A backup landing on its card shows as a soft orange pulse. The hours are by the board's own local time; the same from and until means always on. Built in, before any save: dark from 00:00 to 08:00, tool purple, speaking cyan. The Pi body (0.7.2) and the ESP32 firmware keep this order, and both state plays.

What the board sends to Sapphire

Sapphire's address is HTTPS with her own certificate, for example https://192.168.1.101:8073.

Its own address is learned. Every call below carries the board's key, so the address it calls from is where it lives: Sapphire writes that into the device's url when the device has none (a board set up from the browser) or when the host changed (a new DHCP lease), keeping the scheme and port it had. Only an address on the local network is believed, and a device that still answers the health keeper at the address it has is not moved: the keeper is asked to look again, and the new address is taken once two probes in a row have missed (about a minute). So an address typed behind a gateway or a proxy stays, and a caller that only holds the board's key cannot draw Sapphire's traffic its way while the board is up. A board therefore never needs its address typed, and a board that moves is followed. Drivers opt in with learns_address in their SPEC.

New keys take effect when the board brings them. Setting a board up again (POST /api/devices/provision for a name that exists) mints keys that wait, in memory, for 15 minutes: the device keeps working on its old ones. The first call that carries the new key (voice.key_ok → engine.promote) makes them the device's keys, renames the device when the board was given a new name, and records the board's id as its fingerprint. A name held by a board that cannot be proven to be this one needs replace: true, which the page asks for.

Setting a board up. Sapphire's own firmware takes its settings over its USB console, one line in and one >> {json} line back: setup {json} with name, wifi_ssid, wifi_password, key (the key Sapphire sends), voice_key (the key the board sends), sapphire (her address) and cert (her certificate, PEM); a field left out keeps its old value, and the board saves and restarts. show answers what it has (ready, ip, firmware, the names of what is set). scan answers {"networks": [{"ssid", "rssi"}]} so a flasher can offer the WiFi as a list. The Devices page's flasher speaks this over Web Serial after writing the firmware; setup_board.py speaks it from a terminal.

Its wake word fired

Two boards in one room hear the same wake word, and so does Sapphire's own microphone. Before it has anything to send, a board may ask for the wake:

POST /api/devices/den/wake
Authorization: Bearer <voice key>

Ask alongside the recording, never before it: the round trip must not delay the start of listening. A board that does not ask is still answered once (a repeat of what another listener heard is dropped at /voice). If Sapphire's own microphone takes the wake a moment after a board was told yours, the board's light stream carries standdown (below).

What it heard

After its wake word, the board records until the speaker stops, then sends it:

POST /api/devices/den/voice
Authorization: Bearer <voice key>
Content-Type: audio/wav

<the recording>

A form with the file field audio works too.

The best recording is 16000 Hz, one channel, 16 bit. Limit: 30 requests a minute.

A piece of the room

While a capture goes on (POST /capture), the board sends every piece as it fills:

POST /api/devices/den/room
Authorization: Bearer <voice key>
Content-Type: audio/wav
X-Capture: 184233
X-Chunk: 7
X-Fires: [[203520,991],[371200,978]]
X-Fire-Count: 2
X-Test: 0
X-Model: hey_marcus

<the piece>

X-Capture names the capture (the board's clock at its start), X-Chunk the piece's number in it, X-Fires the wake fires that fell in the piece as [sample offset, score x 1000] (up to 40; X-Fire-Count counts them all), X-Test whether fires were only counted, X-Model the wake word model that was listening.

Who collects is voice.room_sink: the Wakeword Maker's room watch keeps every piece as a room take and sorts the fires. Limit: 30 requests a minute, so pieces are 5 to 60 seconds.

What was typed

A board with a keyboard sends the words instead of a recording:

POST /api/devices/pocket/text
Authorization: Bearer <voice key>
Content-Type: text/plain; charset=utf-8

what time is it

JSON {"text": "..."} works too. Up to 2000 characters.

A button was pressed

A board with buttons says so in /health (buttons, above). In Settings > Devices each way of pressing each button (short, long, double) is left to the board, or set to one of the jobs the board said it can do, or to tell: send Sapphire a message. The board learns what is set from its events stream, when the stream opens and at every save:

data: {"state": "buttons", "bound": {"boot": {"short": "tell", "double": "listen"}}}

The board keeps bound in memory only: before its stream has opened, and on a board Sapphire never reached, every press is the board's own. A long press is one held 0.8 s and fires while still held. A short press fires on release; only on a button with a double in bound does it wait 0.3 s for a second press first. Never give a press a job that restarts the board: a button held through a restart is the chip's download mode on an ESP32.

POST /api/devices/pocket/press
Authorization: Bearer <voice key>
Content-Type: application/json

{"button": "boot", "how": "short"}

Jobs that are Sapphire's. Some work a press asks for cannot be done on the board: a backup is made by Sapphire. The board lists such a job in can like any other, and when the press is that job it posts {"button": "boot", "how": "short", "job": "backup"}. What the Devices page set for the press wins; job counts only for a press left to the board. The one job so far is backup: a fresh backup, sealed, onto that device's own card (it needs storage). Any other word is accepted: false.

The board asks for a job

A board may ask for such work without a press behind it:

POST /api/devices/pocket/ask
Authorization: Bearer <the key the satellite sends>

{"job": "stop"}

answered {"ok": true, "accepted": true, "job": "stop", "said": "Stopped"}. said is a line short enough for a small screen. The jobs:

• stop: she was stopped at the board (a tap, a button) while her turn was going on. Sapphire ends that turn, spoken and written, and drops the questions from that board still waiting behind it. A board whose speak answers once the sound is taken (a play queue) asks too when it cut sound with no turn going on (an announcement): that sound was answered long ago, so nothing else would tell Sapphire. A board that answers when the sound has played needs no ask for that: the sentence it answers stopped says the same. The wake word said over her needs no ask either: claiming the wake ends her turn.
• backup: a fresh backup, sealed, onto that board's own card.

An unknown job is {"ok": false, "error": "..."}. A Sapphire without the door answers 404. A new job is one function and one name in voice.ASKS. | {"ok": false, "error": "..."} | The reason. With "busy": true, three questions already wait |

Her reply, for a screen. Her words never ride the events stream: a stream that falls behind drops a line, and a dropped line of prose is garbage on a screen. The stream carries a doorbell, {"state": "text", "msg", "rev", "have", "done"}: which reply, its revision, how many characters there are, whether she is finished. The board pulls what it lacks:

GET /api/devices/pocket/text?msg=3f9a1c&from=120&max=700
Authorization: Bearer <voice key>

{"msg": "3f9a1c", "rev": 0, "from": 120, "text": "...", "have": 410, "done": false}

Her words reach a screen with the common emoji turned into the faces they came from (:), <3, +1) and the rest left out (voice.small_screen): a small board's font has none. A doorbell comes at most four times a second while she writes, and once more with done. A missed doorbell is made good by the next one. rev goes up when her final text differs from what streamed (a tool round joined, thinking stripped): the board starts that reply over from from=0. max is held to

2000. With no msg, the latest reply: what a board that just connected should show. The board keeps the last four replies' worth of pulls honest; an older msg answers with the latest, whose id the board will not match.

The board picks its chat

A board with a picker on its screen asks which chats it may show, and says which one it talks in:

GET /api/devices/pocket/chats
Authorization: Bearer <the key the satellite sends>

{"ok": true, "chat": "default", "chats": ["default", "kitchen", "trinity"]}

POST /api/devices/pocket/chat
Authorization: Bearer <the key the satellite sends>

{"name": "kitchen"}

answered {"ok": true, "chat": "kitchen"}. The pick is the device's Talks in chat setting, the same as typing it in Settings > Devices: what the board sends lands there from now on, and the name comes back to its screen through POST /screen {"chat"} for a board that states screen.chat in /health. The chat open in the browser is not touched. The list is the one chat the board talks in unless Browse chats is on for the device in Settings > Devices (off by default, so a board's key reaches no chat but its own; a pick of another name is then {"ok": false}). With it on: newest first, at most 40 plus the one it talks in, plain chats only: no private chat (not even as chat, which is then ""), nothing archived, no game or story session. A name that is not in the list is {"ok": false, "error": "..."}. A Sapphire without the doors answers 404; a board then shows the one name it is sent.

The board reads its chat

A board with a screen opens on the conversation, not on a blank page:

GET /api/devices/pocket/history?before=&max=1500
Authorization: Bearer <the key the satellite sends>

{"ok": true, "chat": "kitchen", "before": 38, "total": 41,
 "messages": [{"who": "her", "text": "Soup, and the bread from yesterday."},
              {"who": "you", "text": "what's for dinner"}, ...]}

Newest first, whole messages, at least one, until max bytes (200 to 8000; 1500 if left out); each message is cut to 3000 bytes with .... before in the answer is the cursor for the next older pull (?before=38); 0 means nothing older. The words only: her thinking, file markers and the line that names the device are left out, as are tool rounds. A private chat reads back as {"chat": "", "messages": [], "before": 0}. The reader (firmware/cyd28) pulls this when it links, when its chat changes, and when you page back past what it holds.

What its light should show

The board holds one stream open:

GET /api/devices/den/events
Authorization: Bearer <voice key>

It answers server-sent events. Each is one line of JSON after data:.

A line that starts with : arrives every 20 seconds and means nothing. When the stream ends, open it again. A board only ever hears about its own turns.

Reference for AI

• Driver: core/devices/drivers/satellite.py. It asks /health on every status, and at most once a minute before it speaks (ABOUT_FRESH).
• has is kept by the engine on the device row (parts[].has, engine.capabilities). status() of any driver may answer it.
• Conversion: core/devices/voice.py fit(audio, kind, plays), checked by wanted(plays). The only place her voice is converted for a device.
• A board from the browser: interfaces/web/static/views/settings-tabs/device-flash.js (two lanes: esptool-js over Web Serial with shared/md5.js for the write check, or core/devices/flasher.py on Sapphire's computer through /api/devices/flash/{ports,chip,start,status,ask,close}; then the console line), the chip's MAC as the device fingerprint (engine.provision, /health.mac, console show.mac), POST /api/devices/provision (engine.provision: the row with no url, two keys, her address: the body's sapphire or GET /api/devices/here's guess from net.local_ips, tunnels and bridges excluded; ssl_utils.cert_pem), GET /api/devices/firmware[/{board}/{part}] (core/devices/firmware.py, index + ESP Web Tools manifests, cached). The address: engine.learned, called by _device_key on every device door.
• Doors: core/routes/devices.py devices_voice (body or form, AUDIO_BODIES) and devices_events; typed words devices_text (POST) and devices_text_read (GET), which ride voice.typed, voice.Reply and voice.reply_text.
• A board's chat picker: voice.chats and voice.pick_chat (devices_chats, devices_chat): the same chat setting the Devices page saves, then glass.sync_soon. Its history: voice.history (devices_history) over session_manager.get_display_messages_for.
• Buttons: the buttons setting is a bindings field (engine._bound, engine._menu), its menu is the driver's bindings() from the board's /health; voice.bound is what the stream carries (routes.light_stream at open, the driver's apply at a save), voice.pressed the press door (devices_press). The chat every lane uses: voice.chat_for, which makes a chat that is named and does not exist. Firmware: main/buttons.c in each board.
• Backups onto a board: core/devices/storage.py makes a SatelliteTarget (in the driver) for every device whose has says storage; core/backup_targets.ship seals once, PUTs, rotates by the device's keep fields, drops the openers. Never plaintext: Target.check (Sapphire) + the board's own magic check.
• tests/test_docs_satellite_protocol.py runs this page's health example through the real driver and checks every address in the table.