soma.cell

soma eval

Grade a handler on a dataset: soma eval app.cell --handler classify --dataset cases.jsonl.

soma eval is a command of soma, the toolchain of the Soma programming language (compiler, checker, verifier, test runner and server in one binary). Its options, as soma 3.2.4 prints them:

$ soma eval --help
Grade a handler on a dataset: soma eval app.cell --handler classify --dataset cases.jsonl

Usage: soma-v3.2.4 eval [OPTIONS] --handler <HANDLER> --dataset <DATASET> <FILE>

Arguments:
  <FILE>  Path to the .cell source file

Options:
      --handler <HANDLER>      The handler to call with each case's input (`classify`, or `Cell.classify`)
      --dataset <DATASET>      JSON lines: {"input": …, "expected": …, "name"?: …} — input is the argument (a list: the arguments)
      --grade <GRADE>          exact | contains | handler:Cell.judge (called as judge(input, answer, expected) -> Bool) [default: exact]
      --threshold <THRESHOLD>  The pass rate the run must reach (0 to 1) [default: 1]
      --json                   Output as JSON (for agents)
      --stdlib <STDLIB>        Path to stdlib directory (default: auto-detect)
  -h, --help                   Print help