soma.cell

Blue/green cutover — there is no state where nobody serves traffic

Blue/green deployment's nightmare is the gap: blue is drained, green isn't ready, and for ninety seconds production serves nothing. It is the load-balancer cousin of the airlock problem — two doors that must never both be shut. Salesforce's May 2021 outage had this shape: a DNS change rolled out in a way that briefly left no servers answering, and the tooling needed those very servers to roll back. In this controller the gap is not a state. Every state names exactly which pool is live: blue_live, shifting_to_green (BOTH serve), green_live, shifting_to_blue (BOTH serve). A "both dark" state does not exist in the machine, and `soma verify` proves the reachable set is exactly these four. The traffic split is invariant-walled to 0..100 percent, so no weight calculation can send 130% or -30% of traffic anywhere.

A Soma program from the devops part of the verified corpus: 114 lines, cells BlueGreen, CutoverTests. It passes soma check with soma 3.2.4.

How it is built

The cell BlueGreen declares 5 signals in its face: start_shift(lb: String) -> Map, complete_shift(lb: String) -> Map, cancel_shift(lb: String) -> Map, serving_pools(lb: String) -> String and set_green_weight(lb: String, pct: Int) -> Map; keeps 1 memory slot: green_pct (Map<String, Int>, persistent and consistent) under the invariant green_pct >= 0 && green_pct <= 100. Its state machine cutover starts in blue_live and allows 6 transitions: blue_live -> shifting_to_green, shifting_to_green -> green_live, shifting_to_green -> blue_live, green_live -> shifting_to_blue, shifting_to_blue -> blue_live and shifting_to_blue -> green_live. Handlers: start_shift (line 48), complete_shift (line 57), cancel_shift (line 69), serving_pools (line 79) and set_green_weight (line 86).

The file also holds 1 test cell (CutoverTests) with 10 assertions, run by soma test.

Generated from soma describe on this file.

The source

blue_green_cutover.cell

// ============================================================================
// BLUE/GREEN CUTOVER — there is no state where nobody serves traffic
// ============================================================================
//
// Blue/green deployment's nightmare is the gap: blue is drained, green
// isn't ready, and for ninety seconds production serves nothing. It is
// the load-balancer cousin of the airlock problem — two doors that must
// never both be shut. Salesforce's May 2021 outage had this shape: a
// DNS change rolled out in a way that briefly left no servers
// answering, and the tooling needed those very servers to roll back.
//
// In this controller the gap is not a state. Every state names exactly
// which pool is live: blue_live, shifting_to_green (BOTH serve),
// green_live, shifting_to_blue (BOTH serve). A "both dark" state does
// not exist in the machine, and `soma verify` proves the reachable set
// is exactly these four. The traffic split is invariant-walled to
// 0..100 percent, so no weight calculation can send 130% or -30% of
// traffic anywhere.
// ============================================================================

cell BlueGreen {
    face {
        signal start_shift(lb: String) -> Map
        signal complete_shift(lb: String) -> Map
        signal cancel_shift(lb: String) -> Map
        signal serving_pools(lb: String) -> String
        signal set_green_weight(lb: String, pct: Int) -> Map
    }

    memory {
        // percent of traffic on the green pool; blue gets the rest.
        green_pct: Map<String, Int> [persistent, consistent]
        invariant green_pct >= 0 && green_pct <= 100
    }

    // Note what is ABSENT: any state with zero live pools. A shift can
    // complete or cancel, but the lights never go out.
    state cutover {
        initial: blue_live
        blue_live -> shifting_to_green
        shifting_to_green -> green_live
        shifting_to_green -> blue_live
        green_live -> shifting_to_blue
        shifting_to_blue -> blue_live
        shifting_to_blue -> green_live
    }

    on start_shift(lb: String) {
        let s = get_status(lb)
        let next = "shifting_to_green"
        if s == "green_live" { next = "shifting_to_blue" }
        let r = try { transition(lb, next) }
        if r.error != () { return map("ok", false, "reason", "{r.error}") }
        return map("ok", true, "state", next, "pools", serving_pools(lb))
    }

    on complete_shift(lb: String) {
        let s = get_status(lb)
        let target = "green_live"
        if s == "shifting_to_blue" { target = "blue_live" }
        let r = try { transition(lb, target) }
        if r.error != () { return map("ok", false, "reason", "{r.error}") }
        let pct = 100
        if target == "blue_live" { pct = 0 }
        green_pct.set(lb, pct)
        return map("ok", true, "state", target)
    }

    on cancel_shift(lb: String) {
        let s = get_status(lb)
        let back = "blue_live"
        if s == "shifting_to_blue" { back = "green_live" }
        let r = try { transition(lb, back) }
        if r.error != () { return map("ok", false, "reason", "{r.error}") }
        return map("ok", true, "state", back)
    }

    // every state answers with at least one live pool — by construction
    on serving_pools(lb: String) {
        let s = get_status(lb)
        if s == "blue_live" { return "blue" }
        if s == "green_live" { return "green" }
        return "blue+green"
    }

    on set_green_weight(lb: String, pct: Int) {
        let w = try { green_pct.set(lb, pct) }
        if w.error != () { return map("ok", false, "reason", "{w.error}") }
        return map("ok", true, "green_pct", pct)
    }
}

cell test CutoverTests {
    rules {
        // a full cutover: blue -> both -> green. Always somebody serving.
        assert serving_pools("lb1") == "blue"
        assert start_shift("lb1").pools == "blue+green"
        assert complete_shift("lb1").state == "green_live"
        assert serving_pools("lb1") == "green"

        // THE GAP: jump pools without the overlap window. Not an edge.
        assert_fails transition("lb1", "blue_live")

        // a shift can be cancelled — back to the pool that never dropped
        assert start_shift("lb1").ok == true
        assert cancel_shift("lb1").state == "green_live"

        // traffic weights live inside [0, 100], enforced by memory
        assert set_green_weight("lb1", 50).ok == true
        assert_fails green_pct.set("lb1", 130)
        assert_fails green_pct.set("lb1", -30)
    }
}

What the toolchain says

Run on a fresh copy of the program at build time; the outputs are pasted unedited.

$ soma check blue_green_cutover.cell
✓ All checks passed.
$ soma test blue_green_cutover.cell
test CutoverTests ...
  ✓ assert serving_pools("lb1") == "blue"
  ✓ assert start_shift("lb1").pools == "blue+green"
  ✓ assert complete_shift("lb1").state == "green_live"
  ✓ assert serving_pools("lb1") == "green"
  ✓ assert_fails transition("lb1", "blue_live") — raised invalid transition: green_live → blue_live. Current state: 'green_live'. Valid targets: [shifting_to_blue]
  ✓ assert start_shift("lb1").ok == true
  ✓ assert cancel_shift("lb1").state == "green_live"
  ✓ assert set_green_weight("lb1", 50).ok == true
  ✓ assert_fails green_pct.set("lb1", 130) — raised memory invariant violated on 'green_pct': green_pct >= 0 && green_pct <= 100 — rejected write of 130 (key "lb1"); the slot is unchanged
  ✓ assert_fails green_pct.set("lb1", -30) — raised memory invariant violated on 'green_pct': green_pct >= 0 && green_pct <= 100 — rejected write of -30 (key "lb1"); the slot is unchanged

10 tests: 10 passed, 0 failed
$ soma verify blue_green_cutover.cell
Verifying blue_green_cutover.cell...

State machine 'cutover': 4 states, initial 'blue_live'
  States: [blue_live, green_live, shifting_to_blue, shifting_to_green]

  ✓ 4 states, 6 transitions
  ✓ all states reachable from 'blue_live'
  ✓ reactive machine: no terminal state, and every reachable state can return to 'blue_live'
  ✓ no deadlocks
  ✓ liveness: every reachable state can return to 'blue_live' (reactive machine)
  ⚠ dynamic transition targets: handler `start_shift` uses a dynamic (computed) transition target; handler `complete_shift` uses a dynamic (computed) transition target; handler `cancel_shift` uses a dynamic (computed) transition target — the state these handlers move to is computed at run time (from input or an LLM answer), so the machine's safety is checked at run time for them, not proven
  ✓ termination: all 5 handlers structurally terminate
  ⚠ refinement (partial — dynamic targets fall back to runtime check): handler `start_shift` calls transition() with a computed target — which edge it takes is checked at run time (invalid_transition), not proven; write the target as a literal state to have it verified
  ⚠ refinement (partial — dynamic targets fall back to runtime check): handler `complete_shift` calls transition() with a computed target — which edge it takes is checked at run time (invalid_transition), not proven; write the target as a literal state to have it verified
  ⚠ refinement (partial — dynamic targets fall back to runtime check): handler `cancel_shift` calls transition() with a computed target — which edge it takes is checked at run time (invalid_transition), not proven; write the target as a literal state to have it verified
  ✓ refinement (partial — dynamic targets fall back to runtime check): handler `start_shift` ⟶ <dynamic>
  ✓ refinement (partial — dynamic targets fall back to runtime check): handler `complete_shift` ⟶ <dynamic>
  ✓ refinement (partial — dynamic targets fall back to runtime check): handler `cancel_shift` ⟶ <dynamic>

BlueGreen/invariants:

  ✓ invariant green_pct >= 0 && green_pct <= 100 — writer 'complete_shift' proven by induction (writes pct)
  ⚠ invariant green_pct >= 0 && green_pct <= 100 — runtime-checked (computed values): set_green_weight → green_pct because `pct` is a parameter (narrow it: `require pct >= 0 && pct <= 100 else …`)

10 passed, 5 warnings, 0 failures

  Temporal properties for 'cutover':
  ✓ deadlock_free — no deadlocks in any reachable state
Temporal: 1 passed, 0 failed
VERIFY OK

Builtins used

get_status, map, transition

Source file: /corpus/devops/blue_green_cutover.cell · examples/corpus/devops/blue_green_cutover.cell