Tokenizer — split text into normalized word tokens
Lowercases the input, replaces punctuation with spaces, then splits on whitespace and drops empty fragments. Demonstrates replace/lowercase/split and list filtering with a lambda. Run: soma run tokenizer.cell --signal tokenize "Hello, World! Hello..."
A Soma program from the text part of the verified corpus: 45 lines, cells Tokenizer, TokenizerTest. It passes soma check with soma 3.2.4.
How it is built
The cell Tokenizer declares 2 signals in its face: tokenize(text: String) -> List<String> and count_tokens(text: String) -> Int. Handlers: tokenize (line 18) and count_tokens (line 30).
The file also holds 1 test cell (TokenizerTest) with 6 assertions, run by soma test.
The source
tokenizer.cell
// ============================================================================
// Tokenizer — split text into normalized word tokens
// ============================================================================
//
// Lowercases the input, replaces punctuation with spaces, then splits on
// whitespace and drops empty fragments. Demonstrates replace/lowercase/split
// and list filtering with a lambda.
//
// Run: soma run tokenizer.cell --signal tokenize "Hello, World! Hello..."
cell Tokenizer {
face {
signal tokenize(text: String) -> List<String>
signal count_tokens(text: String) -> Int
}
// Normalize: lowercase, strip common punctuation, split on spaces.
on tokenize(text: String) {
let clean = lowercase(text)
clean = replace(clean, ",", " ")
clean = replace(clean, ".", " ")
clean = replace(clean, "!", " ")
clean = replace(clean, "?", " ")
clean = replace(clean, ";", " ")
clean = replace(clean, ":", " ")
let parts = split(clean, " ")
return parts |> filter(w => len(w) > 0)
}
on count_tokens(text: String) {
return len(tokenize(text))
}
}
cell test TokenizerTest {
rules {
assert count_tokens("Hello, World!") == 2
assert count_tokens("one two three four") == 4
assert count_tokens(" ") == 0
assert tokenize("Hello, HELLO!")[0] == "hello"
assert tokenize("a... b... c")[2] == "c"
assert count_tokens("Mix; of: punctuation? yes!") == 4
}
}What the toolchain says
$ soma check tokenizer.cell
✓ All checks passed.$ soma test tokenizer.cell
test TokenizerTest ...
✓ assert count_tokens("Hello, World!") == 2
✓ assert count_tokens("one two three four") == 4
✓ assert count_tokens(" ") == 0
✓ assert tokenize("Hello, HELLO!")[0] == "hello"
✓ assert tokenize("a... b... c")[2] == "c"
✓ assert count_tokens("Mix; of: punctuation? yes!") == 4
6 tests: 6 passed, 0 failed$ soma verify tokenizer.cell
Verifying tokenizer.cell...
No state machine in this program: nothing to prove beyond `soma check` (invariants and lifecycles are what verify proves).
VERIFY OK (vacuous — no state machine)