Operations

Running a Soma service: storage, restarts, failures, and what happens when it goes wrong. Part of the guide to the Soma programming language.

Tested facts about the process, not intentions. Version: soma --version (the website's /version.json says which release is published; a served program has no such route).

What kills the process, what does not

Event	Effect
A handler raises (require, invariant, transition, `fail`, division by zero, a String reaching an `Int` parameter)	That request is answered with a 4xx/5xx JSON body (table below); every write and transition of the request is rolled back; the process stays up
Runaway recursion in a handler	Answered `500 {"kind": "stack_overflow"}` at depth 512; request threads have a 64 MB stack so the guard fires before the OS does; the process stays up
A `[native]` handler panics (a `buf_get` past the end, an `idiv` by zero)	Caught at the boundary: an ordinary `try`-catchable error with the same kind the interpreter would raise (`index` → 400, `division_by_zero` → 400); the process stays up. Int overflow does not panic: it promotes to BigInt exactly as interpreted code does
`think()` fails or times out	Kind `llm`, rolled back like any error; the request is answered, not hung: one provider round-trip is capped at 60 s (`timeout_ms` in the options map, or `SOMA_LLM_TIMEOUT_MS`), retried up to 3 times on 429/5xx — never after a timeout (the provider may still be generating and billing that reply); `timeout` and `timeout_ms` are the same option
A scheduled `every` / `after` block raises	Logged, rolled back (writes and transitions — it is a handler invocation), next tick runs
Port already answering, `soma check` errors, an unreadable file	`soma serve` refuses to start and exits 1 — it never serves a program that does not check (`--no-check` overrides)
Out of memory, SIGKILL, `kill -9`, SIGTERM	The process dies at once (no draining: an in-flight client gets an empty reply); committed handlers are in `.soma_data/soma.db` (SQLite); the handler in flight is lost as a whole — its writes sat in one uncommitted SQLite transaction
Disk full while writing	Expected (not exercised): the SQLite write fails, the request is rolled back and answered 500
A slow upstream (`http_post` to a service that hangs)	The handler holds the process-wide lock for the whole call: every other request waits. Every http builtin has a timeout (default 30 s; pass `map("timeout", ms)`) — keep it short; the upstream is not cancelled
A slow handler (a quadratic loop, a huge `to_json`, a loop of 100 000 `slot.set`)	Handlers run one at a time: every other request and every scheduler tick WAITS for it — there is no per-request time limit. `soma verify` proves termination, not speed. A persistent slot write is an SQLite statement inside the handler's transaction (about 0.1 ms measured under serve; more on a slow disk; an invariant that reads `size` adds a COUNT of the slot per write — keep size invariants on small slots): a rebuild of hundreds of thousands of keys in one handler holds the process for a minute or more. Keep handlers short; batch bulk loads outside the request path; put a proxy timeout in front
The program changed and `.soma_data/` is older	A renamed slot is a new empty slot (the old rows stay in the file); a slot whose value TYPE changed gives back the old values with their old type; an invariant added later is not checked against stored values (verify proves it for future writes only); a state-machine instance stored in a state the new machine no longer declares takes no transition, `*` edges included. `soma serve` first runs SQLite's `quick_check` and refuses to start on a damaged file (restore a backup or move `.soma_data` aside); a file that is not a database stops any command with the same advice. `soma serve` and `soma run` audit the database at start-up and print one `warning: stored data: …` line per problem: instances in undeclared states, values an invariant refuses (List slots and `size` clauses included), values of another type than declared, a slot re-declared List ↔ Map. `soma serve` (which owns its data directory) also reports slot data no slot declares any more and tables of cells the program no longer declares (renamed/removed) — `soma run` does not, since several programs often share one directory. Migrate (below) or delete `.soma_data/`

Addresses and ports

soma serve app.cell -p 8080 binds 127.0.0.1:8080. Nothing is reachable from the network until you pass --host 0.0.0.0 (or put a reverse proxy in front — recommended: TLS, auth and headers are the proxy's job; Soma reads none).
The event bus for emit between processes binds port +2 (8082), the WebSocket endpoint port +1 (8081) only when a cell declares on ws(...). Both follow --host.
serve probes the port first: a process already answering there is an error (exit 1), not a silent bind beside it.
The dashboard is /__soma/ on the same port; static files are served from ./static/ only.

Exit codes

Command	0	1
`soma check`	no errors (warnings allowed)	at least one error
`soma verify`	`VERIFY OK` (vacuous when the program has no state machine — it says so)	`VERIFY FAILED …` (also when `soma check` fails)
`soma test`	every assertion passed	any failure, or `soma check` fails, or no test cell
`soma run`	the handler returned	the handler raised, or the program does not check, or the handler name is unknown
`soma serve`	(runs until stopped)	cannot start: port taken, check errors, bind failure
`soma deploy`	provider CLI succeeded	the CLI is missing or failed (the Dockerfile is still generated)
`soma docs`, `soma example`	printed	unknown topic / no match

--json on check, verify, test, describe, example prints machine-readable stdout (also for a fatal error such as an unreadable file); diagnostics go to stderr. soma verify --strict turns every ⚠ into a failure.

HTTP answers for a raised error

Body: {"error": "<message>", "kind": "<kind>"}.

kind	status	raised by
`not_found`	404	`fail("not_found", …)`
`unauthorized`, `unauthenticated`	401	`require token_ok else unauthorized`
`rate_limited`, `too_many_requests`	429	`require hits < 10 else rate_limited`
`guard_failed`, `forbidden`, `approval_required`	403	a transition guard; `fail("forbidden")`; `approve()` with nobody to answer
`invalid_transition`, `conflict`	409	`transition()` off the machine; `fail("conflict")`
`invariant`, `ensure`	422	a memory invariant refusing a write; `ensure`
`json`, `division_by_zero`, `type`, `index`	400	a non-JSON body for `body: Map`; arithmetic; a wrong-typed argument or a value that does not fit the slot's declared type; a list index out of range (also from a `[native]` buffer)
your own `require … else Tag` / `fail("tag", …)`	400	the program refused the request
`stack_overflow`, `llm`, `budget`, `undefined_variable`, `undefined_function`, `no_handler`	500	the program itself is wrong or the world failed

A handler that returns normally answers 200 with its value as JSON (() is null, a String is {"result": "…"}), or the status inside a response(status, body) map.

Limits

Int is arbitrary precision (i64 fast path, BigInt beyond); Float is f64; 7 / 2 is 3.5.
Recursion depth: 512 frames, then stack_overflow.
Request bodies: a declared body past 256 MB is refused (413), and a JSON body holding more than ~1 000 000 values (objects and lists weigh more) is refused (413) before it is parsed; paths have no configured cap (20 KB paths were served). Put tighter caps on the proxy.
Handlers run one at a time (a process-wide lock): correct under contention, no parallelism inside one process. Throughput is not a goal.
forall properties in tests walk every value up to 20 000, then sample with a fixed seed.
One process, one SQLite file (.soma_data/soma.db, created beside the program); no replication unless a scale section and a bus join are configured (experimental).

Migrating stored data

There is no migration command: a migration is a handler, run once with soma run against the same .soma_data/ (it shares the database with a running soma serve, and like every handler it is one transaction).

Added slot / field: old rows read (). Backfill in a one-shot handler (for id in skus.keys { if prio.get(id) == () { prio.set(id, 2) } }), or default at read time (prio.get(id) ?? 2).
Renamed state: run a copy of the program that still declares the old state and an edge out of it (picked -> packed), with a _migrate handler calling transition(id, "packed") for each stuck id; then serve the new program. soma run migrate.cell _migrate.
Renamed slot: keep the old slot declared next to the new one for one release, copy in _migrate, then drop it. The audit names the orphaned rows until then.
Re-typed slot: read, convert, set — or rename the slot.
Tightened invariant: stored values that violate it are served but can not be written back; fix them in _migrate or keep the old bound.

Make _migrate idempotent (check before writing) and back up .soma_data/soma.db first.

Environment variables

Variable	Effect
`SOMA_LLM_KEY` (or `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`)	the provider key for `think()`; without one `soma test` mocks and `soma serve` raises kind `llm`
`SOMA_LLM_MOCK=echo` \| `fixed:<text>` \| `rules:<file.json>`	`think()` never reaches a provider (overrides `[agent] mock` in soma.toml); `soma serve` prints `llm: MOCK …` at start-up when the program calls think; an `echo` reply is cut at max_tokens (~4 characters per token) like a provider's, a `fixed:` reply over max_tokens raises kind `llm` as a scripted `mock think` does; `rules:` reads a JSON list of `{"match": "…", "cell": "…", "reply": …}` — the first rule whose `match` is in the prompt (and `cell` is the calling agent) answers, none: an echo
`SOMA_LLM_MOCK_LATENCY_MS=2000`	a mocked `think()` waits that long (outside the lock in a `[task]` handler): tests concurrency and hordes offline
`SOMA_LLM_RPM` / `SOMA_LLM_TPM`	provider limits (requests / tokens per minute) shared by every `think()` of the process — override `[agent] rpm` / `tpm`; 0 = none
`SOMA_LLM_TIMEOUT_MS`	one provider round-trip cap (default 60 000)
`SOMA_APPROVE=always` \| `never`	answers `approve()` when no terminal is attached (`soma serve` fails closed otherwise: 403 `approval_required`)
`PORT`	not read — pass `-p`

Other environment variables are not readable from a program. A secret or a config value (an admin token) goes in a file read once in on start(): let t = read_file("secrets/admin_token.txt") then require type_of(t) == "String" && len(trim(t)) >= 16 else NoSecret — on a missing file read_file returns an {error} Map, and a check such as t != () or an interpolation ("Bearer {t}") would turn that error text into a token anyone can guess (fail closed: the start-up then fails). Relative read_file / write_file paths resolve against the directory soma was started in, not the .cell file's — keep that file out of static/. File builtins — read_file, read_csv, load / include / load_template, read_files included — refuse a path with a .. segment, and a write that would replace a .cell source, soma.toml, soma.lock or .soma_data (kind path); a path built from client input still needs a fixed directory and a validated name.

Between processes

emit reaches every cell of the same process synchronously. Across processes it needs the bus: a [peers] table in soma.toml (other = "host:PORT+2") on the sending side, and on the RECEIVING side the events it accepts from other processes: [bus] accept = ["paid"] (an event this program emits itself is accepted too; anything else — any other handler — is refused). The event reaches every cell with on paid. --join host:bus-port forms an experimental cluster and also forwards committed emit events to connected cluster members. Selected Map slots replicate independently of application events. See cluster.md for the protocol, recovery and the exact limits; avoid configuring the same connection through both [peers] and cluster discovery. An event sent while the peer is down is logged bus: event '…' NOT delivered (also after a linked peer disconnects); soma run opens no bus, so an emit meant for [peers] is reported not delivered there too. A JSON request body or bus event with more than 1 000 000 values is refused (413 / connection closed) before it is parsed. Each [peers] link is supervised: a peer that is down at start-up, that restarts, or that was dropped for reading too slowly is reconnected (every 1 s, backing off to 30 s; peer: … linked again). Two processes that list each other exchange each event once (a link opens with HELLO, and the side that has its own link to the other only receives on it); a [peers] address that is this process's own bus port is refused. A bus connection must send its first line within 10 s, and at most 256 are open at once. There is no loop detection ACROSS processes: A.on tick { emit pong } with B.on pong { emit tick } runs forever — guard such chains with a hop count or an id already seen. Events emitted while it is down are logged NOT delivered — they are not queued. This is the experimental corner of Soma; single-process is the supported shape. Delivery is fire-and-forget: an event emitted while no peer is connected (or to a peer that dropped) is NOT delivered — the sender logs bus: event '…' NOT delivered (naming each [peers] peer whose link is down, even while other links are up), and the handler still commits. emit goes to EVERY connected link — the [peers] you list and the processes linked to you; there is no addressing: each receiver keeps only the events in its [bus] accept list and drops (and logs) the rest, so keep event names distinct per receiver and expect refusals in the logs of the others. To move value between processes, keep an outbox slot on the sender (the transfer, with an id), have the receiver deduplicate by id and emit an acknowledgement, and let an every tick re-send what is unacknowledged. The bus port has no authentication: any local process can send an accepted event — firewall it.

Persistence

[persistent] slots and state-machine instances live in .soma_data/soma.db next to the .cell file (wherever the command is run from), shared by soma serve and soma run; soma run --fresh deletes it first (when other programs' cells also keep tables there, only this program's own tables are reset). soma test starts from empty storage every run. Back up the file; there is no migration tool — a renamed slot is a new, empty slot. Values round-trip exactly: an Int beyond 64 bits comes back as that Int, a variant as a variant, () as (). A write that does not fit the slot's declared value type (Map<String, Int> given a String or 1.0; Map<String, Pay> given a plain map) is refused with kind type before it commits; an Int written to a Float slot is stored as a Float. One process holds one connection to the database and each handler runs inside BEGIN IMMEDIATE … COMMIT, which also serializes handlers across processes.

Deploying on Linux

There is no published Linux binary yet; build from source (Rust stable + GMP):

apt-get install -y build-essential libgmp-dev m4 git
git clone --depth 1 --branch v<version> https://github.com/soma-dev-lang/soma
cd soma/compiler && cargo build --release
# ./target/release/soma serve /srv/app/app.cell -p 8080 --host 0.0.0.0

soma deploy --target fly|cloudflare|aws generates a multi-stage Dockerfile that does exactly this (build stage rust:1-bookworm, runtime debian:bookworm-slim + libgmp10) and invokes the provider CLI; a missing CLI exits 1 after generating the files. A systemd unit is a one-liner: ExecStart=/usr/local/bin/soma serve /srv/app/app.cell -p 8080, WorkingDirectory=/srv/app (the database lives there), Restart=always.

Logs

One line per request on stderr: POST /pay/x → 409 2ms invalid transition …. Start-up prints the cell, its handlers, the database path, the bind address and the dashboard URL. No log files are written by Soma; use the supervisor's.