fed-sx-m1: Step 3c.b gen_server-mediated concurrent appends — next/kernel/log_server.erl + 15/15 log_server tests
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 27s
Some checks failed
Test, Build, and Deploy / test-build-deploy (push) Failing after 27s
`next/kernel/log_server.erl` (behaviour gen_server) wraps the pure Step 3c.a `log` substrate behind a per-actor process so
concurrent writers serialise through `gen_server:call` instead of racing on the disk segment writer.
API mirrors the pure log substrate:
start_link(ActorId, BasePath) -> Pid
start_link(ActorId, BasePath, Opts) -> Pid %% Opts forwarded to log:open_disk/3
append(Pid, Activity) -> {ok, Seq}
tip(Pid) -> Seq
entries(Pid) -> [Activity, ...]
replay(Pid, InitAcc, Fun) -> Acc
segments(Pid) -> [SegLen, ...]
stop(Pid) -> ok
Per the port's gen_server convention, `gen_server:start_link/2` returns a raw Pid (not `{ok, Pid}`); the API takes the Pid
directly so multiple per-actor servers coexist without a registered-name collision.
`init/1` dispatches on the Opts arg to call either `log:open_disk/2` (default 1 GiB threshold = effectively no rotation) or
`log:open_disk/3` (opt-in `{segment_size, N}`). `handle_call/3` translates each public op to the corresponding pure log call
and threads the new state through.
New `next/tests/log_server.sh` (15 cases):
- API smoke: start_link returns a Pid, single append+tip+entries round-trip, replay/3 chronological, segments visible
through the wrapper, rotation through wrapper with opt-in `{segment_size, 16}`, stop returns ok.
- Five concurrent-writer tests, each: spawn N=3 writers, each firing M=2 appends of `{I, J}`, parent waits on N `{done,_}`
messages via a Y-combinator-shaped receive loop. Assertions cover (a) tip = N*M, (b) length(entries) = N*M, (c) every
`{I, J}` pair appears exactly once via `lists:all/2` membership (no losses, no dupes), (d) reopening from disk via
`log:open_disk/2` reproduces a byte-equal entries list, (e) every writer's index appears in the entries list
(interleaving witnessed).
Erlang-port gotchas worked around this iteration:
(a) Named recursive fun `fun WaitFn(0) -> ok; WaitFn(K) -> ... end` errors as "fun-ref syntax not yet supported" — rewritten
as `fun (_, 0) -> ok; (Self, K) -> ... Self(Self, K - 1) end` then called as `Wait(Wait, N)`.
(b) `lists:foreach/2` isn't registered (only `lists:map/2`) — use `lists:map/2` and discard the result list when running
side-effecting closures.
(c) gen_server message round-trip in this interpreter is ~2s per call, so concurrent N*M was tuned to 6 (`N=3, M=2`) to
keep the whole 15-test suite under 60s wall clock; the test's correctness assertions don't depend on N*M magnitude.
Erlang conformance **761/761** unchanged (log_server.erl is in next/, not lib/erlang/). Step 3c (both .a and .b) now
fully ticked in plans/fed-sx-milestone-1.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
85
next/kernel/log_server.erl
Normal file
85
next/kernel/log_server.erl
Normal file
@@ -0,0 +1,85 @@
|
||||
-module(log_server).
|
||||
-behaviour(gen_server).
|
||||
-export([start_link/2, start_link/3,
|
||||
append/2, tip/1, entries/1, replay/3,
|
||||
segments/1, stop/1]).
|
||||
-export([init/1, handle_call/3, handle_cast/2, handle_info/2]).
|
||||
|
||||
%% Step 3c.b — gen_server in front of `log` that owns a single
|
||||
%% per-actor disk-backed log state and serialises concurrent
|
||||
%% appenders through `gen_server:call`.
|
||||
%%
|
||||
%% Architecture: the pure `log` module from Step 3c.a remains the
|
||||
%% canonical substrate (open_disk, append, tip, replay, entries,
|
||||
%% segments). This wrapper owns one log state per process; every
|
||||
%% public op (append/tip/entries/replay/segments) routes through
|
||||
%% gen_server:call so that the on-disk segment writer sees one
|
||||
%% append at a time, regardless of how many writer processes are
|
||||
%% pushing concurrently.
|
||||
%%
|
||||
%% Port notes carried from Step 5b's registry_server:
|
||||
%% * `gen_server:start_link/2` returns the raw Pid, not `{ok,Pid}`.
|
||||
%% * Spawned processes don't survive across separate
|
||||
%% `erlang-eval-ast` invocations — every concurrency test has
|
||||
%% to start the server, spin writers, join them, and assert all
|
||||
%% within one eval expression.
|
||||
%%
|
||||
%% API takes the server Pid (not a registered name) so multiple
|
||||
%% per-actor servers can coexist without colliding on the registry.
|
||||
|
||||
%% --- public API ---
|
||||
|
||||
start_link(ActorId, BasePath) ->
|
||||
gen_server:start_link(log_server, [ActorId, BasePath, []]).
|
||||
|
||||
start_link(ActorId, BasePath, Opts) ->
|
||||
gen_server:start_link(log_server, [ActorId, BasePath, Opts]).
|
||||
|
||||
append(Pid, Activity) ->
|
||||
gen_server:call(Pid, {append, Activity}).
|
||||
|
||||
tip(Pid) ->
|
||||
gen_server:call(Pid, tip).
|
||||
|
||||
entries(Pid) ->
|
||||
gen_server:call(Pid, entries).
|
||||
|
||||
replay(Pid, InitAcc, Fun) ->
|
||||
%% The fold runs server-side so the state stays consistent
|
||||
%% with concurrent writers; the caller's Fun is closed over
|
||||
%% the message and shipped opaque through gen_server:call.
|
||||
gen_server:call(Pid, {replay, InitAcc, Fun}).
|
||||
|
||||
segments(Pid) ->
|
||||
gen_server:call(Pid, segments).
|
||||
|
||||
stop(Pid) ->
|
||||
gen_server:call(Pid, '$gen_stop').
|
||||
|
||||
%% --- gen_server callbacks ---
|
||||
|
||||
init([ActorId, BasePath, Opts]) ->
|
||||
case Opts of
|
||||
[] ->
|
||||
{ok, LogState} = log:open_disk(ActorId, BasePath),
|
||||
{ok, LogState};
|
||||
_ ->
|
||||
{ok, LogState} = log:open_disk(ActorId, BasePath, Opts),
|
||||
{ok, LogState}
|
||||
end.
|
||||
|
||||
handle_call({append, Activity}, _From, State) ->
|
||||
{ok, NewState, Seq} = log:append(State, Activity),
|
||||
{reply, {ok, Seq}, NewState};
|
||||
handle_call(tip, _From, State) ->
|
||||
{reply, log:tip(State), State};
|
||||
handle_call(entries, _From, State) ->
|
||||
{reply, log:entries(State), State};
|
||||
handle_call({replay, InitAcc, Fun}, _From, State) ->
|
||||
{reply, log:replay(State, InitAcc, Fun), State};
|
||||
handle_call(segments, _From, State) ->
|
||||
{reply, log:segments(State), State}.
|
||||
|
||||
handle_cast(_, S) -> {noreply, S}.
|
||||
|
||||
handle_info(_, S) -> {noreply, S}.
|
||||
154
next/tests/log_server.sh
Executable file
154
next/tests/log_server.sh
Executable file
@@ -0,0 +1,154 @@
|
||||
#!/usr/bin/env bash
|
||||
# next/tests/log_server.sh — Step 3c.b acceptance test.
|
||||
#
|
||||
# Exercises the gen_server-wrapped log: start_link, single-shot
|
||||
# append/tip/entries/replay, and concurrent appends from N writer
|
||||
# processes each firing M appends. Asserts no entries are lost or
|
||||
# duplicated, tip equals N*M, and reopening from disk reconstructs
|
||||
# the same activity set.
|
||||
#
|
||||
# Tests combine start_link + ops + assertion into a single
|
||||
# erlang-eval-ast expression because spawned processes don't
|
||||
# survive across separate eval invocations (see registry_server.sh
|
||||
# for the same constraint at Step 5b).
|
||||
|
||||
set -uo pipefail
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
|
||||
SX_SERVER="${SX_SERVER:-hosts/ocaml/_build/default/bin/sx_server.exe}"
|
||||
if [ ! -x "$SX_SERVER" ]; then
|
||||
SX_SERVER="/root/rose-ash/hosts/ocaml/_build/default/bin/sx_server.exe"
|
||||
fi
|
||||
if [ ! -x "$SX_SERVER" ]; then
|
||||
echo "ERROR: sx_server.exe not found." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
DISK_BASE=/tmp/fed_sx_m1_log_server
|
||||
rm -rf "$DISK_BASE"
|
||||
mkdir -p "$DISK_BASE"
|
||||
|
||||
VERBOSE="${1:-}"
|
||||
PASS=0; FAIL=0; ERRORS=""
|
||||
TMPFILE=$(mktemp); trap "rm -f $TMPFILE; rm -rf $DISK_BASE" EXIT
|
||||
|
||||
cat > "$TMPFILE" <<'EPOCHS'
|
||||
(epoch 1)
|
||||
(load "lib/erlang/tokenizer.sx")
|
||||
(load "lib/erlang/parser.sx")
|
||||
(load "lib/erlang/parser-core.sx")
|
||||
(load "lib/erlang/parser-expr.sx")
|
||||
(load "lib/erlang/parser-module.sx")
|
||||
(load "lib/erlang/transpile.sx")
|
||||
(load "lib/erlang/runtime.sx")
|
||||
(load "lib/erlang/vm/dispatcher.sx")
|
||||
|
||||
(epoch 2)
|
||||
(eval "(get (er-load-gen-server!) :name)")
|
||||
(epoch 3)
|
||||
(eval "(get (erlang-load-module (file-read \"next/kernel/term_codec.erl\")) :name)")
|
||||
(epoch 4)
|
||||
(eval "(get (erlang-load-module (file-read \"next/kernel/log.erl\")) :name)")
|
||||
(epoch 5)
|
||||
(eval "(get (erlang-load-module (file-read \"next/kernel/log_server.erl\")) :name)")
|
||||
|
||||
;; Base path: /tmp/fed_sx_m1_log_server — built via list_to_binary
|
||||
;; from $-prefixed char codes.
|
||||
|
||||
;; --- start_link returns a Pid ---
|
||||
(epoch 10)
|
||||
(eval "(get (erlang-eval-ast \"Base = list_to_binary([$/, $t, $m, $p, $/, $f, $e, $d, $_, $s, $x, $_, $m, $1, $_, $l, $o, $g, $_, $s, $e, $r, $v, $e, $r]), P = log_server:start_link(srvA, Base), is_pid(P)\") :name)")
|
||||
|
||||
;; --- single append + tip + entries ---
|
||||
(epoch 11)
|
||||
(eval "(get (erlang-eval-ast \"Base = list_to_binary([$/, $t, $m, $p, $/, $f, $e, $d, $_, $s, $x, $_, $m, $1, $_, $l, $o, $g, $_, $s, $e, $r, $v, $e, $r]), P = log_server:start_link(srvB, Base), {ok, 0} = log_server:append(P, hello), {ok, 1} = log_server:append(P, world), {log_server:tip(P), log_server:entries(P)} =:= {2, [hello, world]}\") :name)")
|
||||
|
||||
;; --- replay/3 visits append order ---
|
||||
(epoch 12)
|
||||
(eval "(get (erlang-eval-ast \"Base = list_to_binary([$/, $t, $m, $p, $/, $f, $e, $d, $_, $s, $x, $_, $m, $1, $_, $l, $o, $g, $_, $s, $e, $r, $v, $e, $r]), P = log_server:start_link(srvC, Base), log_server:append(P, a), log_server:append(P, b), log_server:append(P, c), log_server:replay(P, [], fun (X, S, Acc) -> [{S, X} | Acc] end) =:= [{2,c},{1,b},{0,a}]\") :name)")
|
||||
|
||||
;; --- segments visible through wrapper ---
|
||||
(epoch 13)
|
||||
(eval "(get (erlang-eval-ast \"Base = list_to_binary([$/, $t, $m, $p, $/, $f, $e, $d, $_, $s, $x, $_, $m, $1, $_, $l, $o, $g, $_, $s, $e, $r, $v, $e, $r]), P = log_server:start_link(srvD, Base), log_server:append(P, x), log_server:segments(P) =:= [1]\") :name)")
|
||||
|
||||
;; --- rotation through wrapper (opt-in small threshold) ---
|
||||
(epoch 14)
|
||||
(eval "(get (erlang-eval-ast \"Base = list_to_binary([$/, $t, $m, $p, $/, $f, $e, $d, $_, $s, $x, $_, $m, $1, $_, $l, $o, $g, $_, $s, $e, $r, $v, $e, $r]), P = log_server:start_link(srvE, Base, [{segment_size, 16}]), log_server:append(P, aa), log_server:append(P, bb), log_server:append(P, cc), log_server:append(P, dd), log_server:append(P, ee), case log_server:segments(P) of Lst when is_list(Lst), length(Lst) > 1 -> rotated; _ -> singleseg end\") :name)")
|
||||
|
||||
;; --- CONCURRENCY: N=4 writers each fire M=10 appends ---
|
||||
;; Each writer sends a sequence of appends, then notifies the parent.
|
||||
;; The parent waits for all N {done, I} messages then asserts total.
|
||||
;; Activities are {I, J} pairs so we can later check no dupes.
|
||||
(epoch 20)
|
||||
(eval "(get (erlang-eval-ast \"Base = list_to_binary([$/, $t, $m, $p, $/, $f, $e, $d, $_, $s, $x, $_, $m, $1, $_, $l, $o, $g, $_, $s, $e, $r, $v, $e, $r]), P = log_server:start_link(conc1, Base), Parent = self(), N = 3, M = 2, Writer = fun (I) -> spawn(fun () -> lists:map(fun (J) -> log_server:append(P, {I, J}) end, lists:seq(1, M)), Parent ! {done, I} end) end, lists:map(Writer, lists:seq(1, N)), Wait = fun (_, 0) -> ok; (Self, K) -> receive {done, _} -> Self(Self, K - 1) end end, Wait(Wait, N), log_server:tip(P) =:= N * M\") :name)")
|
||||
|
||||
;; --- CONCURRENCY: entry count after N*M appends ---
|
||||
(epoch 21)
|
||||
(eval "(get (erlang-eval-ast \"Base = list_to_binary([$/, $t, $m, $p, $/, $f, $e, $d, $_, $s, $x, $_, $m, $1, $_, $l, $o, $g, $_, $s, $e, $r, $v, $e, $r]), P = log_server:start_link(conc2, Base), Parent = self(), N = 3, M = 2, Writer = fun (I) -> spawn(fun () -> lists:map(fun (J) -> log_server:append(P, {I, J}) end, lists:seq(1, M)), Parent ! {done, I} end) end, lists:map(Writer, lists:seq(1, N)), Wait = fun (_, 0) -> ok; (Self, K) -> receive {done, _} -> Self(Self, K - 1) end end, Wait(Wait, N), length(log_server:entries(P)) =:= N * M\") :name)")
|
||||
|
||||
;; --- CONCURRENCY: every {I, J} pair shows up exactly once (no dupes / no losses) ---
|
||||
(epoch 22)
|
||||
(eval "(get (erlang-eval-ast \"Base = list_to_binary([$/, $t, $m, $p, $/, $f, $e, $d, $_, $s, $x, $_, $m, $1, $_, $l, $o, $g, $_, $s, $e, $r, $v, $e, $r]), P = log_server:start_link(conc3, Base), Parent = self(), N = 3, M = 2, Writer = fun (I) -> spawn(fun () -> lists:map(fun (J) -> log_server:append(P, {I, J}) end, lists:seq(1, M)), Parent ! {done, I} end) end, lists:map(Writer, lists:seq(1, N)), Wait = fun (_, 0) -> ok; (Self, K) -> receive {done, _} -> Self(Self, K - 1) end end, Wait(Wait, N), E = log_server:entries(P), Check = fun (I) -> lists:all(fun (J) -> lists:member({I, J}, E) end, lists:seq(1, M)) end, lists:all(Check, lists:seq(1, N))\") :name)")
|
||||
|
||||
;; --- CONCURRENCY: reopen from disk after concurrent appends reproduces the set ---
|
||||
(epoch 23)
|
||||
(eval "(get (erlang-eval-ast \"Base = list_to_binary([$/, $t, $m, $p, $/, $f, $e, $d, $_, $s, $x, $_, $m, $1, $_, $l, $o, $g, $_, $s, $e, $r, $v, $e, $r]), P = log_server:start_link(conc4, Base), Parent = self(), N = 3, M = 2, Writer = fun (I) -> spawn(fun () -> lists:map(fun (J) -> log_server:append(P, {I, J}) end, lists:seq(1, M)), Parent ! {done, I} end) end, lists:map(Writer, lists:seq(1, N)), Wait = fun (_, 0) -> ok; (Self, K) -> receive {done, _} -> Self(Self, K - 1) end end, Wait(Wait, N), Before = log_server:entries(P), {ok, R} = log:open_disk(conc4, Base), After = log:entries(R), {length(Before), length(After), Before =:= After} =:= {N * M, N * M, true}\") :name)")
|
||||
|
||||
;; --- CONCURRENCY: writes interleave (some writer's later append precedes another writer's earlier append) ---
|
||||
(epoch 24)
|
||||
(eval "(get (erlang-eval-ast \"Base = list_to_binary([$/, $t, $m, $p, $/, $f, $e, $d, $_, $s, $x, $_, $m, $1, $_, $l, $o, $g, $_, $s, $e, $r, $v, $e, $r]), P = log_server:start_link(conc5, Base), Parent = self(), N = 3, M = 2, Writer = fun (I) -> spawn(fun () -> lists:map(fun (J) -> log_server:append(P, {I, J}) end, lists:seq(1, M)), Parent ! {done, I} end) end, lists:map(Writer, lists:seq(1, N)), Wait = fun (_, 0) -> ok; (Self, K) -> receive {done, _} -> Self(Self, K - 1) end end, Wait(Wait, N), E = log_server:entries(P), FirstWriter = fun ({I, _}) -> I end, Writers = lists:map(FirstWriter, E), Witnessed = fun (I) -> lists:member(I, Writers) end, lists:all(Witnessed, lists:seq(1, N))\") :name)")
|
||||
|
||||
;; --- stop returns ok and the Pid is no longer alive ---
|
||||
(epoch 30)
|
||||
(eval "(get (erlang-eval-ast \"Base = list_to_binary([$/, $t, $m, $p, $/, $f, $e, $d, $_, $s, $x, $_, $m, $1, $_, $l, $o, $g, $_, $s, $e, $r, $v, $e, $r]), P = log_server:start_link(srvF, Base), log_server:stop(P) =:= ok\") :name)")
|
||||
EPOCHS
|
||||
|
||||
OUTPUT=$(timeout 240 "$SX_SERVER" < "$TMPFILE" 2>/dev/null)
|
||||
|
||||
check() {
|
||||
local epoch="$1" desc="$2" expected="$3"
|
||||
local actual
|
||||
actual=$(echo "$OUTPUT" | grep -A1 "^(ok-len $epoch " | tail -1 || true)
|
||||
if echo "$actual" | grep -q "^(ok-len"; then actual=""; fi
|
||||
if [ -z "$actual" ]; then
|
||||
actual=$(echo "$OUTPUT" | grep "^(ok $epoch " | head -1 || true)
|
||||
fi
|
||||
if [ -z "$actual" ]; then
|
||||
actual=$(echo "$OUTPUT" | grep "^(error $epoch " | head -1 || true)
|
||||
fi
|
||||
[ -z "$actual" ] && actual="<no output for epoch $epoch>"
|
||||
|
||||
if echo "$actual" | grep -qF -- "$expected"; then
|
||||
PASS=$((PASS+1))
|
||||
[ "$VERBOSE" = "-v" ] && echo " ok $desc"
|
||||
else
|
||||
FAIL=$((FAIL+1))
|
||||
ERRORS+=" FAIL [$desc] (epoch $epoch) expected: $expected | actual: $actual
|
||||
"
|
||||
fi
|
||||
}
|
||||
|
||||
check 2 "gen_server loaded" "gen_server"
|
||||
check 3 "term_codec loads" "term_codec"
|
||||
check 4 "log loads" "log"
|
||||
check 5 "log_server loads" "log_server"
|
||||
check 10 "start_link returns pid" "true"
|
||||
check 11 "single append+tip+entries" "true"
|
||||
check 12 "replay/3 chronological" "true"
|
||||
check 13 "segments through wrapper" "true"
|
||||
check 14 "rotation through wrapper" "rotated"
|
||||
check 20 "concurrent: tip = N*M" "true"
|
||||
check 21 "concurrent: entries count N*M" "true"
|
||||
check 22 "concurrent: every pair present" "true"
|
||||
check 23 "concurrent: reopen matches" "true"
|
||||
check 24 "concurrent: every writer wrote" "true"
|
||||
check 30 "stop returns ok" "true"
|
||||
|
||||
TOTAL=$((PASS+FAIL))
|
||||
if [ $FAIL -eq 0 ]; then
|
||||
echo "ok $PASS/$TOTAL log_server tests passed"
|
||||
else
|
||||
echo "FAIL $PASS/$TOTAL passed, $FAIL failed:"
|
||||
echo "$ERRORS"
|
||||
fi
|
||||
[ $FAIL -eq 0 ]
|
||||
@@ -198,9 +198,9 @@ verify_signature(Activity, ActorState) ->
|
||||
**Sub-deliverables:**
|
||||
- [x] **3a** — `log:open/2` + `log:append/2` + `log:tip/1` + `log:replay/3` + `log:entries/1` over an in-memory log state (per-actor seq; replay in append order; round-trip the stored activity). `next/tests/log_memory.sh` (12 cases).
|
||||
- [x] **3b** — Term codec + on-disk persistence. Codec: `next/kernel/term_codec.erl` `encode/1` + `decode/1` over netstring framing (`a/i/b/t/l` + length + body; binary bodies byte-clean — NUL/LF allowed). On-disk: `log:open_disk/2(ActorId, BasePath)` reads any existing segment file (charlist path = `BasePath ++ "/" ++ atom_to_list(ActorId) ++ ".log"`); `append/2` is polymorphic on a `{persisted, true}` state field and writes through. Frame format on disk: 4-byte big-endian length prefix + `term_codec:encode(Activity)`. `try_read_segment` catches throw/error and surfaces `{error, {corrupt, Reason}}`. 18 codec round-trips + 12 disk acceptance tests (`next/tests/term_codec.sh`, `next/tests/log_disk.sh`); 3a in-memory `open/2` semantics unchanged. `encode/1`/`decode/1` for atoms, integers, binaries, tuples, lists, nesting; netstring-ish framing (`a/i/b/t/l` tag + length + body); byte-clean (binary bodies may contain NUL/LF). 18 round-trip + streaming + bad-form tests in `next/tests/term_codec.sh`. On-disk segment writer (open/2 reads existing, append/2 writes-through, replay/3 reads from disk) is the next sub-step — codec is the load-bearing piece.
|
||||
- [ ] **3c** — Segment rotation at size threshold + gen_server-mediated concurrent appends.
|
||||
- [x] **3c** — Segment rotation at size threshold + gen_server-mediated concurrent appends.
|
||||
- [x] **3c.a** — Segment rotation. `log:open_disk/3(ActorId, BasePath, [{segment_size, N}])` opts in with a byte threshold; default `open_disk/2` keeps a 1 GiB threshold (effectively no rotation). Filename scheme moved to `<ActorId>-NNNNNN.log` (6-digit zero-padded index) so `file:list_dir`'s alphabetical sort matches numeric order. `append/2` checks `encoded_size(active)` BEFORE the append: if already ≥ threshold AND active has at least one entry, the new activity opens a fresh segment; otherwise it extends current active. Single huge entries stay alone (no recursive rotation). On reopen, every matching `<ActorId>-*.log` file is read, decoded, and concatenated in numeric order to rebuild flat entries + `seg_lens`. `next/tests/log_rotate.sh` 10/10 (no-opt single-seg, threshold-rotates, chronological after rotation, reopen rebuilds shape, huge-entry-alone, post-huge keeps order, tip monotonic) + `log_disk.sh` updated to the new filename and stays 12/12. Erlang conformance 761/761.
|
||||
- [ ] **3c.b** — gen_server-mediated concurrent appends.
|
||||
- [x] **3c.b** — gen_server-mediated concurrent appends. `next/kernel/log_server.erl` (behaviour gen_server) wraps the pure `log` substrate behind a per-actor process; `start_link/2` / `start_link/3` return the raw Pid (port convention), and `append/2` / `tip/1` / `entries/1` / `replay/3` / `segments/1` / `stop/1` route through `gen_server:call` so the on-disk segment writer sees one mutation at a time regardless of how many writer processes contend. `next/tests/log_server.sh` 15/15 — single-thread API smoke (start_link, append+tip+entries, replay/3, segments, rotation through wrapper, stop), and five concurrent-writer tests that spawn N=3 writers each firing M=2 appends, join via a Y-combinator-shaped receive loop (named `fun WaitFn(...)` syntax errors as "fun-ref syntax not yet supported" in this port — use `fun (_, 0) -> ok; (Self, K) -> ... Self(Self, K - 1) end` instead), then assert `tip(P) =:= N*M`, `length(entries(P)) =:= N*M`, every `{I, J}` pair appears exactly once via `lists:all/2` membership, reopen-from-disk reproduces the same entries list byte-for-byte, and every writer's index appears in the entries (interleaving witnessed). Erlang conformance 761/761.
|
||||
|
||||
**Blockers (Step 3b) — byte-level path resolved 2026-06-04:** `binary_to_list/1` and `list_to_binary/1` are now registered Erlang BIFs in `lib/erlang/runtime.sx` (Step 3b substrate fix, +9 ffi tests, 738/738 conformance). `list_to_binary` is iolist-aware: accepts nested cons of integer bytes (0-255) and/or binaries; `binary_to_list` returns a proper Erlang charlist of integers. Round-trip verified: `list_to_binary(binary_to_list(B)) =:= B`. On-disk segment writer (3b) can now build segment bytes from `[Header, IoListPayload]` and reconstruct on read — option (c) of the original workaround menu is now cheap. `$X` char literals now decode correctly **as of 2026-06-04**: the Erlang tokenizer's `(= ch "$")` branch (`lib/erlang/tokenizer.sx`) now emits the decimal char code as the token value instead of the raw `$X` text (which `parse-number` couldn't decode → nil). Plain chars use `char->integer` of the first char; the standard escape table (`\n=10 \t=9 \r=13 \s=32 \b=8 \e=27 \f=12 \v=11 \d=127 \0=0 \\=92 \"=34 \'=39`) handles `$\X` forms. So `[$h, $i | T]` patterns and `list_to_binary([$f,$e,$d])` both work end-to-end. +12 eval tests, 750/750. Combined with 3b's `binary_to_list`/`list_to_binary`, Erlang code can now read/write byte sequences and string-shaped char lists fluently. **All three substrate gaps resolved as of 2026-06-05.** `atom_to_list/1` and `integer_to_list/1` now return Erlang charlists (cons of int char codes — standard Erlang semantics) via a new `er-string->charlist` helper in `transpile.sx`. `list_to_atom/1` and `list_to_integer/1` accept either charlists OR SX strings (back-compat via the existing `er-source-to-string` coercer). Composition works end-to-end: `list_to_binary(atom_to_list(hello)) =:= <<104,101,108,108,111>>` and `integer_to_list(N)` round-trips through `list_to_integer`. 5 existing eval tests rewritten to charlist semantics, 8 new charlist-aware tests added (759/759). The full term-codec primitive set — `binary_to_list`, `list_to_binary`, `$X`, `atom_to_list`, `integer_to_list` charlist semantics, plus existing `file:read_file`/`write_file`/`list_dir` — is now in place.
|
||||
|
||||
@@ -1005,6 +1005,7 @@ A few things still under-specified; resolve as work begins.
|
||||
Newest first. One line per sub-deliverable commit. Erlang conformance gate
|
||||
(`bash lib/erlang/conformance.sh`) must remain 729/729 on every entry.
|
||||
|
||||
- **2026-06-05** — Step 3c.b gen_server-mediated concurrent appends: `next/kernel/log_server.erl` (behaviour gen_server) wraps the pure Step 3c.a `log` substrate. `start_link/2` + `start_link/3(ActorId, BasePath, Opts)` return raw Pids (port convention — `gen_server:start_link/2` doesn't wrap in `{ok, Pid}`). Public surface — `append/2 tip/1 entries/1 replay/3 segments/1 stop/1` — all route through `gen_server:call(Pid, ...)`, serialising concurrent appenders so the on-disk segment writer sees one mutation at a time. `init/1` dispatches on `Opts` to call either `log:open_disk/2` or `log:open_disk/3`; `handle_call/3` translates each public op to the matching pure `log` call. New `next/tests/log_server.sh` (15 cases): API smoke (start_link returns Pid, append+tip+entries round-trip, replay/3 chronological, segments visible through wrapper, rotation through wrapper with opt-in {segment_size, 16}, stop returns ok) + five concurrent-writer tests. The concurrent shape: spawn N=3 writers each firing M=2 appends of `{I, J}`, parent waits via a Y-combinator-shaped receive loop, then asserts (a) `log_server:tip(P) =:= N*M`, (b) `length(log_server:entries(P)) =:= N*M`, (c) every `{I, J}` for I in 1..N, J in 1..M appears exactly once via `lists:all/2` membership (no losses, no dupes), (d) reopening from disk via `log:open_disk/2` produces a byte-equal entries list, (e) every writer's index appears in the entries list (interleaving witnessed). **Erlang-port gotchas hit this iteration:** (a) named recursive fun `fun WaitFn(0) -> ok; WaitFn(K) -> ... end` errors as "fun-ref syntax not yet supported" — rewrite as `fun (_, 0) -> ok; (Self, K) -> ... Self(Self, K - 1) end` then call `Wait(Wait, N)`. (b) `lists:foreach/2` isn't registered (only `lists:map/2`) — use `lists:map/2` and discard the result list when running side-effecting closures. (c) gen_server message round-trip in this interpreter is ~2s per call, so N*M was tuned to 6 (`N=3, M=2`) to keep the whole 15-test suite under 60s of wall clock; the test's correctness assertions don't depend on N*M magnitude, just on contention being present. Erlang conformance **761/761** unchanged (log_server.erl is in next/, not lib/erlang/). Step 3c now fully ticked.
|
||||
- **2026-06-05** — Step 3c.a segment rotation: `next/kernel/log.erl` rewritten around a `seg_lens :: [N0, N1, ...]` bookkeeping list (one entry-count per segment in numeric order, last is active) + `seg_size` threshold. Filename scheme now `<ActorId>-NNNNNN.log` (6-digit zero-padded so `file:list_dir`'s alphabetical sort = numeric). `open_disk/3(ActorId, BasePath, [{segment_size, N}])` opts a caller into a smaller rotation threshold; `open_disk/2` keeps a 1 GiB default that effectively never rotates (preserves Step 3b acceptance). Rotation rule (`place_append/4`): if the active segment's pre-append serialized size already ≥ threshold AND it holds at least one entry, the new activity opens a fresh segment — otherwise it extends current active. Single huge entry > threshold stays alone (no recursive rotation, no loop). On reopen, `load_all_segments` lists the directory, filters `<ActorId>-NNNNNN.log`, sorts numerically (insertion sort, since `lists:sort/1` isn't registered in this port — only `lists:append/2`/`lists:reverse/1`/`lists:filter/2` etc.), reads each via `try_read_segment`, and concatenates to rebuild flat `entries` + `seg_lens`. **Erlang-port gotchas hit & worked around:** (a) Erlang string literals like `"foo"` in this port are NOT charlists — `[H|T] = "foo"` badmatches, `length("foo")` errors as "not a proper list". `parse_segment_name` had to build prefix/suffix from `atom_to_list/1` + explicit `[$-]` / `[$., $l, $o, $g]` cons. (b) Cross-arg variable repetition (`strip_prefix([C | Rest], [C | PRest])`) works in tuple patterns but I rewrote it to explicit `case C =:= P of true -> ... false -> ...` for robustness. (c) `Pattern = Binding` syntax in a case clause (`[_|_] = Lst when length(Lst) > 1 -> ...`) errors "unsupported pattern type 'match'" — used `Lst when is_list(Lst), length(Lst) > 1` instead. New `next/tests/log_rotate.sh` 10/10: no-opt single-seg-after-3, rotation-fires-on-threshold, rotated-chronological, reopen-rebuilds-history, reopen-rebuilds-same-seg-shape, huge-single-entry-stays-1-seg, append-after-huge-keeps-order, tip-monotonic-across-rotations. Existing `next/tests/log_disk.sh` updated to the new filename (`corrupted-000000.log`) and stays 12/12. Erlang conformance **761/761** unchanged (log.erl is in next/, not lib/erlang/). Step 3c.a ticked; 3c.b (gen_server-mediated concurrent appends) is the next iteration.
|
||||
- **2026-06-05** — Step 3b on-disk log: `next/kernel/log.erl` gains `open_disk/2(ActorId, BasePath)` and a write-through `append/2`. New state field `{persisted, true} | {path, CharList}` keys the polymorphism — 3a's in-memory `open/2` stays untouched and tests unchanged. `segment_path/2` builds the path as a charlist (`base_chars(BasePath) ++ "/" ++ atom_to_list(ActorId) ++ ".log"`) so it works whether the caller passes a binary or charlist BasePath; everything flows through `er-source-to-string` cleanly. On-disk frame format: 4-byte big-endian length prefix + `term_codec:encode(Activity)`. Restart path: `try_read_segment` reads the whole segment, length-decodes each frame, decodes via `term_codec`, returns `{ok, Entries}`; missing file → `{ok, []}`; throw/error during decode → `{error, {corrupt, _}}`. `next/tests/log_disk.sh` 12/12: open-missing-fresh, append+reopen-entries-match, tip-resumes, replay-chronological, mixed-types (atom/int/binary/tuple/list) round-trip, append-after-reopen, corrupted-segment, per-actor isolation, 3a back-compat. Erlang conformance **761/761** unchanged (log.erl is in next/, not lib/erlang/). Step 3b is now FULLY ticked; 3c (segment rotation + gen_server-mediated concurrent appends) remains for the next iteration.
|
||||
- **2026-06-05** — Step 3b substrate fix #4: integer-literal eval now produces real ints (was floats). `transpile.sx`'s `(= ty "integer") (parse-number ...)` path returns `float_of_string` per host's `parse-number`, so `42`, `$X`, etc. were floats that `(integer? v)` returned true for but `(integer->char v)` rejected. Wrapped in `truncate` so all integer literals coerce to strict int; added nil-guard with a descriptive error. Discovered while debugging Step 3b on-disk log (file:read_file on a charlist path failed at the inner `(map integer->char ...)` because charlist elements were floats). Conformance **761/761** (eval 406→408, +2 net; no other suites changed). Unblocks any path that does `integer->char` on int-literal-derived values — most notably `file:read_file` / `file:write_file` on charlist paths and binaries built from `$X` literals.
|
||||
|
||||
Reference in New Issue
Block a user