Skip to content

Roadmap — the plan and the record

What is left, what is blocked, and who is on it lives on the board: https://github.com/orgs/NeuroCognitiveImagingLab/projects/3 — that is authoritative. This file is not a tracker, and per-increment "status" is not maintained here any more.

This file is for the two things a board is bad at:

  • The plan — what each increment is, why they are ordered this way, and what "done" means.
  • The record — what was built, and more usefully what broke and what it taught us. The detail lives in the subsystem docs (ACTICHAMP.md, PUSH.md, RESTING.md, BIDS.md, …); this is the through-line between them.

Why the split (2026-07-16): this header used to carry a hand-maintained status line claiming "354 passing, 1 skipped, 2 failing". CLAUDE.md §0 carried its own copy claiming "395 passing, 0 failing". The suite actually stood at 399 passed, 1 skipped, 0 failing — so both copies were wrong, in different directions, and nothing surfaced it. That is this project's signature failure wearing a different hat: a description drifting from the thing it describes, invisibly, exactly as sub-902/ses-002 recorded runs their own metadata could not describe (docs/HANDOFF_2026-07-15_ses-002-review.md).

The lesson generalizes: do not keep a second copy of a fact that something else already knows. The test count is knowable by running pytest -q, so it is not written down here. What is left is knowable from the board, so it is not written down here either.

Incremental build. Each increment is independently useful, ends in something runnable, and respects the manifest contract + compliance guardrails. Nothing is scaffolded until the increment is agreed.

Real participant collection stays gated on REB amendments — see docs/REB_AMENDMENTS.md for the substance, and the board's reb-blocker label for what that currently stops.

Increment 0 — Foundation & repo (before feature work)

  • Agree these planning docs.
  • Create the private zeuron-inc repo; add .gitignore + .bidsignore that exclude BIDS/, sourcedata/, and all data/PII (COMPLIANCE R1).
  • Lift ssvep-toolbox-handoff/ssvep_core/ → src/ssvep/classification/; move schemas/ to the tree root; keep DESIGN_PRINCIPLES.md + report templates under docs/.
  • Finalize environment.yml (add pyglet + PySide6 to the ssvep env); wire package skeleton
  • tests + lint (no data).
  • Draft REB amendments A1–A3 (OpenBCI / gold-cup / dry headset) for Aaron to submit — see docs/REB_AMENDMENTS.md. No collection with that hardware until approved.
  • Exit criteria: import ssvep.classification works; smoke tests pass; repo clean of data.

Increment 1 — Stimulus builder + runtime ← FIRST BUILD

The prioritized capability. Two halves around the manifest.

1a. Stimulus builder (stim/ + ui/ Design mode) - ✅ Done (headless core): stim/flicker.py (luminance waveforms + renderability/ photosensitivity/dropped-frame validation, unit-tested incl. an FFT check that per-frame luminance produces the target frequency); stim/builder.py (UI-agnostic Protocol model → schema-valid manifest, with design-time validation); stim/renderer.py (pyglet vsync-locked FlickerRenderer consuming a manifest); io/manifest.py (schema validate/ load/save). scripts/build_example_protocol.py emits run-manifests/example_ssvep_9class.json. - ✅ Design GUI done: ui/design.py (PySide6) — target table + grid generator, timing/ acquisition forms, live validation, Measure-refresh button (stim/display.py empirically times vsync swaps; validation uses the measured rate), Save/Load, and Preview (launches the renderer as a subprocess = process-isolated agent). Run: ssvep-design. Offscreen smoke-tested. - ✅ On hardware (partly done, 2026-07-03): the renderer runs fullscreen on the real GPU (NVIDIA RTX 4090 laptop @ 240 Hz — confirmed hardware GL, not llvmpipe) and presented a full live session. ⚠️ Running renderer + LSL bridge + recorder on the one laptop dropped ~6.9 % of frames — offload acquisition or lighten the render host before any real timing run. - ✅ Visual-latency onset offset (2026-07-14): ssvep.DEFAULT_ONSET_OFFSET_S = 0.14 — one shared constant for design + analysis. Timing.onset_offset_s is added on top of the operator's stim_s (a 4 s stimulation flickers for 4.14 s, Timing.presented_stim_s) so the operator still gets a full window of usable steady state; the designer exposes it beside Cue/Stim/ITI. Analysis drops the lead-in and still charges it to ITR (DESIGN_PRINCIPLES #10). Legacy manifests without the field present exactly stim_s. - Blocked on parts/hardware → #10: flik-o-meter frame-timing verification; Measure-refresh + gamma/LUT confirmation ride along with it. A student build sheet (breadboard → soldered, PDF) now exists for the engineering student doing the build (2026-07-15, hardware/flikometer/). - Exit: design a 9-target 36–44 Hz protocol, preview the flicker in the GUI, save a validated manifest; frame-timing verified with the flik-o-meter (pending the build).

1b. Runtime / session runner (runtime/ + ui/ Run mode) - ✅ Done (headless core, tested): io/xdf.py — a minimal XDF writer verified to round-trip through pyxdf (EEG + markers + timestamps). runtime/: session.py (manifest → deterministic, seeded trial plan + markers), markers.py (schema-valid structured markers; JSON on the LSL stream), lsl_io.py (resolve/verify stream vs manifest, MarkerOutlet, LSLRecorder → XDF behind a testable source interface), auth.py (operator whitelist), provenance.py (provenance + consent gating; observer-only ⇒ discard), runner.py (SessionRunner: de-identified sub-XXX paths + self-describing session sidecar), openbci.py (OpenBCI Cyton → LSL BrainFlow bridge). brainflow added. - ✅ Trial-presentation loop (done): runtime/presentation.py expands a plan into a frame-accurate schedule (cue → stim → iti, cycling targets); the renderer's session mode (--session, or GUI Preview session) highlights the cued target each trial, flickers all targets (phase reset at onset), emits markers via MarkerOutlet (--markers), and keeps an always-available Esc abort. Schedule/marker logic unit-tested; visuals verify on hardware. - ✅ Electrode-impedance check (done; validated live on the dry headset): runtime/impedance.py (lead-off config_board commands, injected-tone RMS→kΩ, saved ImpedanceSnapshots, montage mirroring), viz.plot_impedance (luminance-balanced montage map, auto-scale ≥100 kΩ; no threshold yet), runtime/openbci.stream_impedance/measure_impedance_once (+ --impedance CLI), and the first Run-mode surface ui/run.py (ssvep-run: live montage, per-block re-check, Done → snapshot). Snapshots persist to the session sidecar + the recording's manifest montage. Core/UI unit-tested; the CLI one-shot read (measure_impedance_once) is validated live on the dry headset (Cyton COM5, 170–1030 kΩ, 2026-07-03). The GUI's continuous stream_impedance loop is exercised in normal use on both the actiCHamp (where it was hardened — see below) and the Cyton: it updates the montage live while you adjust electrodes, and values persist on Done, which is the design, not a gap (2026-07-16). See docs/IMPEDANCE.md. - ⚠️ "Continuous" here means the setup-time loop — there is no during-run impedance, and there cannot be. The Cyton exposes no continuous impedance stream, and the read works by enabling the ADS1299 lead-off drive: the channels are not recording EEG during it, because the board is injecting a tone. Measuring impedance mid-run would inject that tone into the recording. This note exists because the sentence that used to be here was read the other way and turned into a to-do (closed: #14). - ✅ Impedance check no longer fails silently (2026-07-14): the sub-904 "impedance did nothing" (no values, yet it claimed to save) was not a broken read — _ImpedanceWorker.run() called the stream fn with no try/except, so any exception killed the QThread silently. It now surfaces the error in the status line, and a busy open is retried (the single-client amp is still being released right after a recording). - ✅ End-to-end session validated on hardware (2026-07-03): a throwaway orchestrator ran the LSLRecorder concurrently with the renderer (markers over LSL) → XDF + sidecar via SessionRunner, on the dry headset. A 27-trial 9-class run epoched with zero heuristics (forward JSON markers) and decoded (FBCCA 25.9 % @4 s vs 11.1 % chance, p≈0.025 — weak-but-real dry-headset SSVEP; SNR positive at 37/39/40 Hz). This meets the 1b exit criterion. A startup race dropped block_start/first cue (epoch-defining stim_on/stim_off intact). - ✅ Run-mode GUI + orchestrator (done; used for real recordings): runtime/orchestrator.py (record_session) is the productionized session path — it launches the EEG bridge and the paradigm's renderer (renderer_command, chosen from the manifest paradigm), pumps EEG + markers into the LSLRecorder → XDF + sidecar, and surfaces both live via on_eeg/on_marker. ui/run.py is now the full operator flow in one window: 1 · Design → 2 · Setup → 3 · Impedance → 4 · Session. The session is no longer CLI/script-only — sub-902/903/904 were recorded through this GUI. - ✅ Setup-tab live signal preview (2026-07-10): stream the board before recording (the sub-902 lesson: you couldn't see the signal until stimulation started) — scrolling EEG + PSD with the stimulus frequencies marked, best-channel target SNR, and a quality banner flagging railing/flat channels (signal_quality.railing_flags, which caught PO7 pinned at the ADC rail). Mutually exclusive with the lead-off impedance drive (shared board). - ✅ Hardware selection lives on the Design tab only (2026-07-14): Setup no longer re-picks the board; "Detect devices" became "Resolve connection", which only finds the port/IP/USB-index for the designed board. The physical unit is still recorded in provenance. - ✅ Electrode type + prep captured (2026-07-10): optional acquisition.electrode_type/prep in the schema + Design tab, overridable "as run" at the bench; SessionRunner stamps them into provenance and merges them into the recording's manifest, so analysis/BIDS see what physically ran (fixes the sub-902 gap where wet gold-cup vs dry was indistinguishable). - ✅ Push-to-server (2026-07-15): runtime/push.py mirrors the finished run (XDF + sidecar + the session's impedance log) to ncil.psychology.dal.ca over SFTP at the end of every run — opt-in on the Setup tab, off by default, local files always kept. Transport is the system OpenSSH client (zero new dependencies; reuses the operator's key/known_hosts). Consent is re-read from the sidecar at push time — on the automatic path too — so observer-only, and any session whose consent can't be confirmed, is never pushed (R5). - The server is the source of truth. The toolbox lists the remote tree (ssh … find) and diffs against local; it keeps no ledger. A ledger can't know about files copied up by hand — which is how the 902/903/904 pilots got there — and a first draft duly invented a 37-file "backlog" that was already on the server in full. - Only the finished run goes automatically. Absence from the server isn't evidence a file belongs there, so everything else is offered in a Review / push… picker (new/changed/same per row; consent-blocked rows greyed and untickable; sub-000 listed but unticked). - Paths mirror 1:1, which is what forced the sourcedata/ layout fix below. - Verified against the real server 2026-07-15: 34 of 37 local files correctly recognised as already present, 0 ticked by default. docs/PUSH.md, REB row A11 (non-blocking transport divergence). - ✅ Raw XDF now written to sourcedata/ (2026-07-15): SessionRunner.output_base() writes <out_dir>/sourcedata/sub-XXX/ses-YYY/eeg/ instead of the subject top level, and _impedance_log_path follows it. A raw .xdf at the subject level makes BIDS/ fail validation; discover_runs and the NCIL server already assumed sourcedata/. The repo's BIDS/ was migrated (37 files) and the empty top-level subject dirs removed; migrate_toplevel_to_sourcedata is now legacy-only. - ✅ ses-002 review + 8 fixes (2026-07-15): re-analysing sub-902/ses-002 found that every run recorded something its own metadata could not describe — 3 markers silently lost per run, measured_refresh_hz null on every recording ever made, no cap-vs-montage check (the resting run recorded 32 floating channels), two presets rendering 2.75× apart under one protocol name, the resting run relabelled task-ssvep by the converter, actiCHamp impedance broken by a zero settle, a cry-wolf mains threshold, and Run mode crashing on start. All fixed + gated. deg is now the canonical stimulus unit and the lab viewing geometry is on every manifest. Full write-up: docs/HANDOFF_2026-07-15_ses-002-review.md. - ✅ Hardware left the run design (#147, 2026-09-11). Build run had an Acquisition group (device/sampling rate/headset/electrode type/prep) from the start — asked for at design time because that is where the rest of the form already lived, not because a run design is where hardware belongs. It never varied per run in practice, so it was a plausible-looking field that got saved unedited: the recorded default, not a considered choice. acquisition is now optional in run_manifest.schema.json and on RunSpec; Build run emits none at all. It is set once, protocol-wide, on Build Protocol's "Set up acquisition hardware" (renamed from "Set acquisition for all runs…", which is also where §2.1's "block" collision with a block of trials was fixed), and correctable as-run on Set up Session — hardware does not vary within a session, so a Cyton-vs-actiCHamp comparison is two sessions, not two runs of one sharing a protocol. lock_blocker() refuses to lock a protocol whose embedded runs still have no acquisition set. Electrode type stopped being typed at all: it is now derived from the headset (builder.headset_electrode_type), because a second field that restates what the first already implies will eventually disagree with it — the sub-902 wet/dry metadata gap was exactly that. PREPS is now skin prep only (alcohol/NuPrep/none); what the electrodes are made of is the headset's business. The same change moved Load protocol… / Resolve / Suggest ses # / Administer questionnaires / 🔒 Lock back from Run Session onto Set up Session — #126 had put them on Run Session so that tab (and the old QA-check tab) never needed a separate visit, but the lock reading fields on a different tab than the one it freezes was the wrong split; Run Session keeps the checklist, the live panes, and the (renamed) Start selected run button, i.e. everything that only matters once a session is actually recording. Lesson: a field asked for at the wrong altitude gets a plausible default that nobody edits, and a second field restating a first is a standing invitation for the two to disagree — the fix in both cases was to derive or relocate, not to add a validation rule catching the disagreement after the fact. - Remaining → board: break-time symptom checkpoints (#20); the Cyton timing error — a fixed ~5–6 ms offset, fatal >30 Hz, harmless at alpha, and measuring it with the flik-o-meter could calibrate it out (#9, the highest-value hardware experiment outstanding); actiCHamp impedance cap-on-head test (#11 — fixed but never verified against a real electrode); rerun resting on a correctly-declared montage (#13). ⚠️ Real participant collection with OpenBCI/dry-headset awaits REB amendments A1–A3 (#3); the alpha-band protocol additionally needs A8 (#4). - Exit: ✅ met on hardware (session → XDF → zero-heuristic epoching → decode), now through the GUI rather than a throwaway orchestrator. - 🆕 Hardware on hand (2026-07-04): OpenBCI WiFi shield → enables 500/1000 Hz sampling (the RFduino dongle is hard-capped at 250 Hz). Should lift the high-frequency-band decoding that sat near chance at 250 Hz (Increment 2) by capturing 2nd/3rd harmonics. The shields are detected and streaming (2026-07-07, below), but the sampling-rate comparison itself hasn't been run — worth doing once collection is unblocked. - ✅ Multi-amp support hardware-verified (2026-07-07): board auto-detection (runtime/discovery.py) confirmed across 4 Cyton dongles + 4 WiFi shields — FTDI (0x0403,0x6015) with per-dongle serials; shields OpenBCI-*@192.168.4.1 with a REST API (/board+/all → 8/16-ch, MAC, firmware). Now wired into the Setup tab (Detect pick-list, auto-fill, board-mismatch warning), with the physical unit (serial/MAC) recorded in provenance and first-connect flakiness absorbed by openbci.prepare_with_retry. WiFi packet loss is intermittent (0% clean but 3–6% on a bad run) → surface as per-session QC. g.tec Unicorn Hybrid Black validated (--board unicorn, serial UN-2024.10.06): 8-ch @ 250 Hz, clean link, via BrainFlow's bundled Unicorn.dll — no g.tec software install, no paid Python API; pair once via the Suite (persists across power-cycle); now a selectable Design device. Catalog tool scripts/probe_device.py (→ git-ignored derivatives/device_catalog.json). See docs/UNICORN_INTEGRATION.md. Remaining: Unicorn in the GUI Detect list + its electrode-quality readout for the Impedance tab; REB amendment covers the Unicorn too before collection. - ✅ actiCHamp is a selectable 64-ch device (2026-07-14): board="actichamp" routes to the clean-room ctypes driver rather than BrainFlow (recording launches python -m ssvep.runtime.actichamp --stream), with the full actiCAP 64Ch Standard-2 montage auto-locked, and Setup-tab impedance + signal preview served by actichamp.stream_impedance/stream_eeg_preview. See "Beyond" below. - ⚠️ Impedance is Cyton-only (2026-07-14): the Run-mode check uses the Cyton ADS1299 lead-off drive + the OpenBCI formula. The Unicorn has no lead-off path via BrainFlow and reports a bogus ~0 kΩ on every channel regardless of real contact — needs a Unicorn-specific method or an explicit "unsupported on this board" state. docs/IMPEDANCE.md.

Increment 2 — Offline analysis + reporting (analysis/ + ui/ Analyze mode)

  • ✅ Done (tested, incl. on real pilot data): analysis/epoching.py (XDF → Epochs; forward JSON-marker path + quarantined legacy pilot slash-marker parser), pipeline.py (window-sweep decode → canonical metrics.json; per-condition/band decoding with chance per band; best-channel SNR), stimulus_viz.py (stimulus-layout + trial-timeline figures from the manifest), report.py (metrics.json/.md + figures + index.html; outputs to git-ignored derivatives/). scripts/analyze_recording.py.
  • ✅ Real-data validation: reproduces the pilot's mid-band (31–42 Hz) 9-class result — e.g. sub-007 ses-001 100 % @4s (chance 11.1 %), sub-006 ~69 %; alpha ~80–83 %; high-freq near chance (expected at 250 Hz sampling). Matches the handoff's "mid 88 % @4s".
  • ✅ Occipital-ROI + re-reference policy (2026-07-14) — the big decode win. Calibration-free CCA/FBCCA decodes a 64-ch reference-free actiCAP at near chance across all channels, but ~68 % @4s once restricted to an occipital ROI with a common-average reference (sub-904: 22 % → 68 %; sub-902 unchanged at 86 % @3s). New ssvep.spatial: rereference(), select_channels(), occipital_roi() (canonical occ-8, not naive most-posterior-by-y) and occipital_decode_plan(manifest) — large reference-free montages get CAR + ROI, small/already- referenced montages are untouched. The same policy drives live and offline (DESIGN_PRINCIPLES #8), and is recorded in metrics.json.
  • ✅ TRCA works after band-passing — and beats FBCCA (2026-07-14). This corrects an earlier wrong conclusion: TRCA had been tested on unfiltered data, where mV-scale reference-free drift dominates the covariance and the spatial filter locks onto drift, not the SSVEP. Band-passed, on sub-904 (actiCAP occ-8 + CAR, 9-class, leave-one-block-out CV): TRCA 70.8 % @1s → 77.8 % @2–4s (peak ITR 42.7 b/min) vs FBCCA 25.0 % @1s → 68.1 % @4s (15.7 b/min) — TRCA at 1 s already beats FBCCA at 4 s. Reported beside FBCCA and gated on measured phase-locking (pipeline.phase_locking_gain_db), with leave-one-block-out CV because an uncross-validated TRCA number is leakage. FBCCA remains the reported default + the live decoder (TRCA needs per-user calibration).
  • 🔴 Phase-locking is an acquisition-path property, and Cyton fails it above ~30 Hz (2026-07-14; frequency-scoped 2026-07-15 — see the defect list below and DESIGN_PRINCIPLES #9). The renderer resets flicker phase per trial, so trials are phase-locked by design — but whether that survives to file depends on the amp's marker↔EEG timestamps. Measured directly: actiCHamp (direct ctypes) +5.7 dB (PLV≈0.68) → TRCA 77.8 %; Cyton (chunked BrainFlow→LSL) +1.4 dB (PLV≈0.35) → TRCA at chance (FBCCA, being phase-invariant, still hits 86 %). Same PC, monitor and renderer for both ⇒ the jitter is in the amp's timestamps, not the display. Fixing the Cyton path would unlock ~2.7× ITR on that hardware.
  • ✅ Channel search for high-density montages (2026-07-14): analysis.channel_search — greedy forward selection (O(pool·k)) over the posterior pool with an honest split-half number beside the fixed occipital ROI. Auto-runs only above spatial.BIG_MONTAGE (16 ch) and is informational — it never changes the decode. On sub-904 the fixed ROI won (68.1 % vs 65.3 % split-half), while the full-data pick (66.7 %) looked better than it generalises — so the fixed ROI stays the default (DESIGN_PRINCIPLES #8: don't per-recording-tune the montage).
  • ✅ Readable montage maps + quality as maps (2026-07-14): viz.plot_channel_array draws discs in data coordinates at 0.45× nearest-neighbour spacing (never overlapping at any channel count) and the canvas scales with the montage — fixes the unreadable 64-ch map. Per-channel quality (amplitude std, mains ratio) is now montage maps instead of a 64-row table.
  • ✅ Window sweep derived from the data (2026-07-14): 0.5 s steps up to the usable stimulation rather than a hardcoded 1 s tuple.
  • ✅ The batch is a package function, and it takes a selector (2026-07-21, #67). BIDS/code/02_run_offline_pipeline.py could only analyse the entire dataset — finishing one visit meant reprocessing every participant ever collected, and there was no way to redo one person with a different --method. The loop moved into ssvep.analysis.batch (run_batch, select_runs, analyse_run); the script is now argparse over it with --sub/--ses (either entity form), and the GUI's Analyze tab calls the same function — the two-entry-point split that made scripts/analyze_recording.py a second-class result is not recreated. Selection filters bids.discover_runs rather than narrowing its glob, so discovery keeps one definition of a run; --ses without --sub is refused; an empty match exits non-zero listing what does exist (a silent "0 analysed" reads as success, which is exactly how a typo'd ID becomes "my data is fine"); and the run log records the selector, so a log in logs/ is never ambiguous about its coverage.
  • ✅ Analyze mode exists in the GUI (2026-07-21, first cut of #30). A sixth tab (ssvep.ui.analyze.AnalyzePanel): scan the BIDS root, tick a session in a sub → ses → run tree (paradigm read from each manifest, reports already on disk marked), Analyze on a worker thread, open the report. It is a selector over disk, not a form — the typo failure mode #67 guards against on the CLI can't be typed here at all. The tab's root follows Set up Session's output directory, so "analyse the session I just recorded" needs no path. Finishing shows the results: the batch writes a summary page and the GUI opens it — the first version required a click nobody expected, which is a finished analysis you can't see. (That per-batch summary_<stamp>.html was replaced by the persistent index below; the "it must open by itself" part stands.)
  • ✅ Analysing derives the BIDS dataset, not just the derivatives (2026-07-30). Every path into the pipeline reads the XDF in sourcedata/, so nothing downstream ever asked for the BrainVision copy — and the GUI, which never called 01_xdf_to_bids.py, quietly skipped it. Result: sub-001/002/003 had complete derivatives/ssvep-analysis/ output over a dataset with no sub-XXX/…/eeg/*.vhdr at all — derivatives of nothing, and not a BIDS dataset. run_batch now runs that conversion over its selection first (ensure_bids=True), and ssvep.io.bids grew the "is it already there?" check it needed: is_converted (header and signals present, and no older than the XDF — a newer source means the derived copy describes data that has changed) plus ensure_converted. A conversion failure is logged and the run is analysed anyway: the XDF is the source of truth, so a broken export is not a reason to withhold a result. The Analyze tab shows the state per run in a BIDS column before you press anything — the whole failure was that it was invisible. Lesson: a derived artefact with no downstream consumer will be skipped by anything that doesn't check for it; make the check part of the step that produces its neighbours.
  • ✅ The derivatives became navigable, and a dataset (2026-07-31, #80 + #18). Analysis produced exactly one level of artefact — a per-run report, flat in one directory — plus a summary_<stamp>.html per invocation. That summary looked like the missing higher level, which is why it read as done, but it was a receipt for one batch: built from the in-memory BatchResult, so it listed only that selection; timestamped, so a re-analysis left another one beside the reports and none was the entry point; and blind to the runs of a session it hadn't just analysed. Replaced by a persistent index.html → sub-XXX/index.html → sub-XXX/ses-YYY/index.html → run report, with outputs moved under sub-XXX/ses-YYY/ and a dataset_description.json (DatasetType: derived, GeneratedBy with the toolbox version + git commit, SourceDatasets) making the raw→report provenance machine-readable (#18). Two design choices carry the weight: the index is rebuilt by scanning disk (ssvep.analysis.derivatives.build_index), the same derive-from-disk rule Run mode's session checklist uses — so it covers everything analysed so far, survives a crashed batch, and updates in place instead of accumulating; and run order is the declared run_index from each recording's session_position stamp (#43), not the filename, so a session page shows the order the participant actually experienced. Old flat trees are migrated (moved, not re-analysed) on the next batch. Lesson: a page that summarises an invocation will be mistaken for a view of the data — the difference only shows up on the second run, which is exactly when nobody is looking.
  • ✅ One operator path named, and the debug script stopped answering questions it can't (#69, 2026-07-31). scripts/analyze_recording.py predates the batch and was never folded into it. It is not redundant — --pilot is the only reader for the sub-005–008 slash markers, and --reref/--roi are how the 64-ch-at-chance vs ~68%-on-occipital-8 finding was made (DESIGN_PRINCIPLES #8) — so it stays, marked developer-only in its header, at runtime on stderr, and in the docs; README and WINDOWS_TESTING now name 02_run_offline_pipeline.py / the Analyze tab as the operator path. The part that was actually dangerous is fixed: it had no paradigm dispatch, so a resting recording was decoded as SSVEP and reported accuracies against frequencies never presented — it didn't fail, it produced an answer-shaped wrong result. It now refuses and names the batch. Lesson: a tool kept for its sharp edges still needs a guard on the case where it will confidently mislead; "developer only" is not a substitute for one, because the developer is exactly who will believe the output.
  • Later → board: group/multi-condition report (#31) — the analytic aggregation the index now has a natural home for; what remains of Analyze mode beyond run-and-open-report (#30).

Increment 3 — Online (real-time) decoding

  • ✅ Harness built + validated on synthetic (2026-07-05): runtime/online.py LiveDecoder buffers the live EEG stream + structured markers and decodes each trial (stim_on→stim_off) with the calibration-free CCA/FBCCA engine — the validated math, new harness only (DESIGN_PRINCIPLES). runtime/orchestrator.py surfaces live EEG/markers via on_eeg/on_marker; the Run GUI's Session tab shows scrolling EEG waveforms + trigger lines (pyqtgraph) and a live per-trial decode panel (cued→predicted, running accuracy vs chance). Unit-tested; end-to-end on the synthetic board (3/3 trials decoded, ~4 s windows, chance-level as expected for synthetic).
  • ✅ Decode policy now shared with offline (2026-07-14): LiveDecoder gained ch_labels/reref/decode_channels (inert defaults) and the Run view drives it FBCCA + the manifest's occipital/re-reference policy — so the live decoder and the offline pipeline apply the same spatial policy instead of diverging (the fix that took sub-904 from 22 % to 68 % offline).
  • ✅ Paradigm-aware live panel (2026-07-15): on a resting run the Session tab swaps the CCA decode panel (meaningless — nothing to decode) for live occipital alpha reactivity, so a dead posterior montage shows up in the first eyes-closed block rather than in analysis after the participant has gone home (the sub-903/904 lesson).
  • ✅ Decodes real SSVEP on hardware (confirmed 2026-07-16): the live panel was watched working across the sub-902/903/904 runs. Recorded here because the evidence is only that — someone watched it. The panel is display-only, so no session file can say what its own decoder reported (#34 fixes that; it is the same self-description gap the ses-002 review was about, one layer up).
  • Remaining → board: save the live decode into the recording (#34); live SNR-spectrum display (#21); FBCCA band auto-config + a CCA/FBCCA method toggle (#22); smoothing / early-stopping (#23). Reuses the validated math; only these harness pieces are new.

Increment 4 — BIDS + data management

  • 🟡 Converter built (2026-07-10): manifest-driven XDF→BIDS-EEG via mne-bids (ssvep.io.bids, scripts/bids_convert.py, BIDS/code/) — BrainVision + sidecars, sample-accurate events.tsv, per-headset acq- scoping, participants.tsv (no demographics), impedance in channels.tsv; round-trip tested (tests/test_bids.py). Applied to sub-000/901/902. See docs/BIDS.md.
  • ✅ Reproducible offline runner (2026-07-14): BIDS/code/02_run_offline_pipeline.py applies the occipital/re-reference policy per run and writes a timestamped run log (versions, git SHA, per-run policy + result) to derivatives/logs/; it routes task-rest to PAF + alpha reactivity with the same report.html/metrics.json shape (2026-07-15).
  • 🟡 The official validator became runnable — which is not the same as conformance being checked (2026-08-06, #16). bids-validator-deno (MIT; vendors the Deno runtime via pip, so no separate Deno/npm install) is in environment.yml, and bids-validator-deno BIDS/ is documented in docs/BIDS.md. But the only run so far was against a fresh checkout — the tracked de-identified scaffold and no sub-*/ — so its first finding was literally "there are no subject directories". The tool executes; the converter's output has still never been validated. Lesson, and the reason this bullet is 🟡 and not ✅: enabling a check and passing it are two different claims, and the run that produces reassuring output on an empty tree is exactly the one that invites collapsing them.
  • Remaining → board: validate a populated tree + remediate the findings (#110) — the conformance gate this increment actually rests on; a GUI validate button wrapping the same command (#98); consent-flag-aware export/sharing tooling (#17); fill reference/ground when hardware specifies (#19). (BIDS/derivatives/ as a formal derivatives dataset, #18, is not outstanding — it landed 2026-07-31 and is recorded under Increment 2, beside the derivatives/index.html work it shipped with. It sat in this list for a week after it was done.)

Increment 5 — Remote analysis frontend (optional, when prioritized)

  • Browser analysis UI over the same core, served from the Canadian compute server with GitHub-ID auth + whitelist + TLS + CA/US IP allowlist (COMPLIANCE R3/R4). This is the "grow into C" step the boundary was designed for.

Beyond the core increments

  • ✅ Resting state is a manifest paradigm (2026-07-15) — it now records, cues, and reports. Tone-cued eyes-open/eyes-closed blocks (ssvep.stim.resting + tones + resting_renderer); ssvep.analysis.resting_paf estimates the individual peak alpha frequency (to steer SSVEP flicker away from PAF + harmonics) and the eyes-closed/eyes-open alpha-reactivity ratio (posterior signal-quality check). Doubles as the SSVEP negative control (no flicker → decoder at chance). Expressed as manifest v1.2: paradigm: "ssvep" | "resting" (absent ⇒ ssvep, so every older manifest still reads), stimuli: [] (the absence of flicker is the design), block structure in design.resting, task rest ⇒ BIDS task-rest. The runtime picks its renderer from that field and analysis picks its pipeline from it — never inferred from the data, and never from "stimuli is empty, must be resting". Everything downstream (bridge, recorder, XDF, sidecar, consent, provenance, impedance) is unchanged, which is the point: there is no resting-specific run path left to forget to wire a recorder into.
  • ⚠️ The cautionary tale worth remembering (docs/RESTING.md): the first version was a Design-tab dialog that side-stepped the manifest — it saved nothing (bare renderer subprocess, no recorder; a whole sub-904 run was presented and no data could ever have been written), bypassed subject/consent/provenance/impedance/lock, and was silent on every machine (pyglet 2.1 moved AudioFormat; a bare except swallowed the ImportError, and the test only asserted callable(play_tone)). Also fixed: the final block_end was pinned at exactly total_frames, which the draw loop never reaches, so the last block never closed and analysis silently dropped it — a third of the eyes-closed data PAF rests on. Resting runs are now refused a silent start (probe_audio() preflight).
  • 🔴 REB: resting recording + auditory cueing are not in the approved protocol (V6 is visual flicker only) → amendment A10, blocking for collection — though it is the one entry that lowers participant risk (it removes photic exposure entirely). Nothing was collected out-of-protocol: the old path saved nothing.
  • 🟢 Brain Products actiCHamp driver (validated on hardware 2026-07-14): clean-room ctypes binding (ssvep.runtime.actichamp), direct-USB → LSL (no Recorder/RDA). Streams end-to-end on a real 64-ch actiCHamp: ABI confirmed (ResolutionEeg 0.0488 µV/bit, 74-int32 frames, counter +1/frame ⇒ no dropped samples), holds a steady ~508 Hz, and it is the one acquisition path measured to be phase-locked (+5.7 dB ⇒ TRCA 77.8 %). Three hardware bugs found and fixed: the rate enum (only 10/50/100 kHz are native; codes 3–15 silently fell back to 10 kHz → software anti-aliased decimation), µV coding is signed (a 2026-07-13 floating-input test wrongly indicated offset-binary — open inputs rail to +full-scale, indistinguishable from midscale), and DLL discovery (the DLL lives in the BrainVision Amplifier SDK dir, not the Recorder's). Impedance read path verified no-cap (0x7FFFFFFF not-connected sentinel → NaN ⇒ "n/a"); remaining: confirm sensible kΩ with a cap on a head.
  • ⚖️ Licensing (2026-07-13): the BrainVision Amplifier SDK is research-only, non-redistributable, non-commercial — commercial use of the SDK or of any app processing BP-hardware data needs a written BP agreement. So the BP amp modules are optional, user-supplied plugins outside the free-for-commercial core; we load a user-supplied DLL and never vendor a vendor binary (.gitignore now blocks *.dll/*.sys/*.bit/*.lib). Lab-internal research use is within the grant.
  • V-Amp/Refa8 have no Python driver — use OpenViBE → LSL. See docs/ACTICHAMP.md.

  • ✅ Generic by default — the site-profile layer (2026-09-14, #165 / #182 / #193). Deployment facts (institution, ethics reference, push destination, operator, mains frequency) were compiled into the code; they now come from a git-ignored config/site.local.json, and a fresh clone behaves generically — no institution stamped into BIDS, no push destination, a blank operator, mains recorded as unknown rather than assumed 60 Hz. The inversion is the load-bearing part: if the generic path is not the default it only runs when someone deliberately tests it, so it rots — which is precisely why sharing the toolbox was hard. Now every pytest -q exercises it, pinned by an autouse fixture and asserted by a test, so the day generic stops being the default path is a red test rather than a slow drift. One deliberate asymmetry: a missing profile means generic (fine), but a profile that is present and unparseable fails loudly — silently degrading a lab machine to generic would strip the ethics stamp off real participant data, a compliance failure that looks like success. Restrict-only is proved structurally, not just functionally: a test walks every property name in the schema and fails on anything matching photosens/abort/consent/bypass/override, so a future key that could weaken a safety rule fails at design time rather than depending on someone thinking to test for it.

  • ✅ The snapshot publisher, and the guard that earned its keep (2026-09-14, #154 / #180 / #181). scripts/publish_snapshot.py generates the teaching and public repositories from a tag as a single parentless commit — generated, never edited, because an editable mirror is this project's signature failure at repository scale. --dry-run is the default. The part worth remembering is the pre-publish guard, which re-scans the staged tree independently of the exclusion list rather than trusting curation, and matches shapes as well as literals (\b20\d{2}-\d{4}\b alongside this deployment's own file number, so a future ethics file number trips it identically).

  • ⚠️ On its first real run it refused, and it was right. 13 files in the kept tree carried the REB file number or lab identifiers — not the excluded ones, but files the teaching repo is supposed to ship, including operator-facing UI strings telling a student their work ran under a Dalhousie approval that does not cover them. A curated exclusion list would have published every one. Half cleared themselves when the site profile landed; the rest were prose, fixed in #188 by stating the constraint rather than the approval — "editing an approved instrument is an amendment matter under your ethics approval" is true everywhere, while naming one board's specific file number is true at one institution only. Review caught two further defects a passing guard structurally cannot: a citation to a file both profiles exclude, and a sentence whose "a different approval" lost its referent once nothing was named. Render operator-facing text and read it; do not infer it from a green check.

  • Two fail-open holes were found in the guard itself before it shipped: BIDS/participants.tsv was not in the path check (a force-added participants table would have passed, since its rows trip no content pattern), and files over the scan limit were silently treated as clean. Both now refuse. A guard that skips what it cannot conveniently read is not a guard.

  • ⚠️ Local test runs in a git worktree were testing the wrong checkout (2026-09-14, #184). pip install -e . records an absolute path, so pytest inside a worktree collected that branch's tests and executed the main checkout's source — exiting 0 and looking clean. The dangerous direction is a test that passes because main happens to satisfy it; worse, a "verify the test fails before the fix" check is meaningless, because reverting the worktree's source changes nothing about what is imported. CI was unaffected and remained authoritative throughout, so nothing shipped unverified — but several PR descriptions asserted local evidence their environment could not produce. Now a tests/conftest.py check fails the session with both shells' fix commands spelled out. Silence is not success: a wrong answer that looks like a right one is worth more engineering than a loud failure.

  • 🔴 A wrong 8-channel carrier cannot be detected in software (2026-09-14, #179 → #195). #179 was filed believing the montage preflight would catch an operator selecting the wrong electrode carrier once the OpenBCI bridge published channel labels. It cannot: record_run derives the labels it hands the bridge from the same effective manifest that verify_montage_or_raise compares them against, so both sides change together and always match. The actiCHamp path is not identical — record_run passes it --n-eeg rather than --labels, so its bridge publishes E1…En fallback labels that a real montage does not match — but it shares the underlying limitation, because the root cause is physical — a Cyton is eight channels of whatever someone plugged in, and gold-cup electrodes carry no identity — so no comparison between a declaration and a stream header can establish which electrode is on which site. Since the two shipped occipital montages now have disjoint labels, the mislabelling would be total, and analysis would faithfully report results for positions never used because the manifest spine forbids inferring design from data. Publishing labels still buys something real (a stale or foreign LSL stream on the same name, a channel-count divergence) and that is what it is now documented as. The mitigation for the rest has to be procedural — an operator assertion at lock time — and must not become a click-through, since an acknowledgement everyone dismisses manufactures a false record that someone checked.

  • ⚠️ Two tests that did not test what they claimed (2026-09-14, #183 / #176). test_car_recovers_target_under_large_common_mode passes with CAR disabled: its fixture adds one identical drift array to every channel, and a rank-1 interferer is exactly what FBCCA's own spatial filter nulls for free — amplitude is not what defeats a spatial filter, rank is. CAR is load-bearing for high-density amps (the sub-903/904 pilot went from near chance without it to ~68 % with it plus an occipital ROI), and its only dedicated test would have passed if CAR broke. Separately, proving live decoding equals offline decoding (#53) surfaced that LiveDecoder's inclusive t0 <= t <= t1 and build_epochs' half-open slicing disagree by up to one sample at a trial boundary — 4 ms at 250 Hz, the same order as the Cyton timing error behind #9, and a growing fraction as #23 shrinks the decision window. A preprocessing test that does not fail when you remove the preprocessing is not testing it.

Stimulus expressiveness, design generation & per-individual optimization

(Merged in from the former STIMULUS_ROADMAP.md on 2026-07-16 — recoverable from git history. It was a second roadmap whose relationship to this one was never stated, and it had drifted: it still described the appearance gap its own Phase A had closed, and still called Increment 3 "not built". One roadmap, so there is one place for a phase to be marked done.)

A workstream running alongside the increments rather than after them, aimed at the toolbox's scientific goal: make SSVEP experiments fast to build, able to sample the stimulus parameters the literature manipulates, and able to express not just static protocols but systematic parameter manipulation to find the best decoding for an individual.

Why this was ordered A → B → C → D: A/B/C are pure/offline and testable headlessly, so they were built first; D needs the online loop, an amp, and a scientific decision. Each phase keeps the manifest authoritative and bumps the schema version.

The foundation it builds on

The manifest-as-contract is the right backbone for parameter manipulation: it already carries design.conditions[] (each with stimulus_freqs_hz + free-form params) and an optional explicit trial_order; SessionPlan iterates conditions; analysis decodes per condition (run_by_condition). Multi-condition designs were representable and half-plumbed before any of this work — what was missing was generating and adapting them. Calibration-free CCA/FBCCA also means decoding from trial 1 with no training, which is what makes closed-loop adaptation possible at all.

Phase A — close the appearance gap — ✅ DONE (2026-07-02)

The gap, worth remembering because it is the pilot's lesson again: the manifest could not reproduce what was on screen. flicker.py supported depth/duty/mean, but the manifest's flicker block carried only type/freq/phase and the renderer hard-coded depth=1, mean=0.5; the manifest allowed shape ∈ {rect,circle,image,checkerboard} and a color, but the renderer drew a grayscale Rectangle and silently ignored both; gamma and background were renderer arguments, not manifest fields. So the exact on-screen appearance was not recoverable from the recording — a self-describing recording that could not describe its own stimulus.

Closed: flicker carries contrast/duty; per-target pattern + pattern_reversal + spatial_freq_cpd (stim/patterns.py draws checkerboard/grating); color/shape honoured; display carries gamma, background_rgb, and optional luminance_cd_m2 / gamma_lut. GL appearance correctness still wants a human visual check (docs/WINDOWS_TESTING.md) — the math and manifest round-trips are headless-tested.

Phase B — ssvep-design UX overhaul — ✅ DONE (2026-07-02)

Tooltips on every field (units, ranges, effects); live layout preview (no more blind coordinate entry); validated dropdowns + colour picker with inline error highlighting on the offending cell; row duplicate/reorder; phase-scheme auto-fill (JFPM phase = k·0.35π); presets for canonical paradigms; live geometry readout in deg and px.

Phase C — design generation: factorial sweeps / multi-condition — ✅ CORE DONE (2026-07-02)

design.factors (named factors → level lists) in the schema; stim/design.py expands a base protocol × factor grid → conditions[], each stamped with its levels in params, with the factors recorded verbatim so the recording stays self-describing; design.method ∈ {static, factorial, staircase, bayesopt} records how conditions were generated. analysis/optimization.py ranks conditions by accuracy / ITR / SNR per participant — the deliverable that answers "which settings decode best for this person".

Remaining: the GUI Sweep panel (pick factors + levels + repetitions; preview condition count and estimated duration). Not on the board — see the note at the end of this section.

Phase D — adaptive / psychophysical optimization — not started

The scientifically ambitious core, and the reason for the rest. - Adaptive engine — an interface propose_next(history) -> params, with staircase (transformed up/down, e.g. 2-down/1-up → ~71 % threshold) on a difficulty parameter (stim duration, contrast, size, inter-target frequency spacing) using decode correctness as the response; Bayesian adaptive threshold (QUEST/Psi) fitting a logistic psychometric function; and optimization (bandit / Bayesian optimization over the grid) maximizing accuracy or ITR. - Closed-loop runner — run trial → decode → update adaptive → re-render next trial with new params. The session renderer already rebuilds per trial, so it can take per-trial updates. - Per-individual calibration mode — a fast frequency/param scan that picks the best-SNR target set and writes a personalized protocol. - All adaptive state + chosen params must be stamped into the manifest/provenance, or a session stops being self-describing the moment it starts adapting.

⚠️ Its stated blocker is gone. Phase D was written as "needs Increment 3 + the OpenBCI amp"; Increment 3 is built and the amps are validated. What actually gates it now is the scientific decision — which parameter to adapt, against which objective (accuracy vs ITR vs threshold) — plus the REB envelope for any sweep that changes the stimulus (A5–A9). It is not blocked on code.

Schema evolution this workstream drove

Block Added Status
flicker contrast, duty ✅ v1.1
stimulus color, shape honoured; pattern, pattern_reversal, spatial_freq_cpd ✅ v1.1
display gamma, background_rgb, optional luminance_cd_m2, gamma_lut ✅ v1.1
design factors, method ∈ {static, factorial, staircase, bayesopt} ✅ v1.1 (adaptive values reserved for D)
provenance chosen/adaptive params, psychometric fit results ⬜ Phase D

Nothing here is tracked. The Sweep panel and Phase D have no issues — they are plan, not work in flight. If either becomes real work it belongs on the board, not in this list.

Deferred directions and standing decisions

The backlog itself now lives on the board — what used to be listed here is issues #9 onward, where it can be prioritized, assigned, and closed instead of quietly aging in a list. What stays here is the handful of entries that are decisions rather than work: things deliberately not being done, and why.

  • Two occipital montages; the 10-10 one is the default; the saved run manifests deliberately left alone (2026-09-12, revised 2026-09-13). The lab's occipital array was defined in absolute centimetres measured on one lab head, which is only correct on a head that size — so ahead of the public release it was restated in the 10-10 system's own units (whole percentages of the participant's naso-inion and preauricular arcs; the rounding moves every electrode by at most 1.5 mm and a test holds that bound), the wire colours were brought onto OpenBCI's standard Cyton ribbon order, and reference/ground moved to the earlobes. Drawing it over a 64-channel 10-10 layout is what prompted the second montage: two pairs of its channels are nearest to the same standard site, so the array cannot be named in 10-10 terms even approximately. The Occipital 10-10 montage (P1 PO3 O1 POz Oz P2 PO4 O2, docs/MONTAGE_1010.md) puts the same eight electrodes and the same wiring on standard sites, and is now DEFAULT_HEADSET: it is what a new user expects, what a cap already provides, and what the literature reports. The denser layout (docs/MONTAGE_FREE_ELECTRODE.md) is a deliberate choice with a stated case, not a fallback — the pilot data was collected on it. Neither document is framed as the other's appendix; that is why MONTAGE.md was renamed rather than kept as the obvious "the montage" file. What was NOT done, on purpose: the 30 files in run-manifests/ still carry the old wire colours and a scalp reference. All 17 saved protocols are pre-1.2 — they reference those files by path and pinned fingerprint — so editing them would raise ProtocolDriftError on every one, at resolve time, with a participant in the chair. That is the exact failure #115 was opened to remove. Instead the correction is applied where a recording is actually described: the session's as-run acquisition derives the carrier, montage and reference/ground fresh (builder.headset_reference_ground), and builder.canonical_headset maps the two pre-rename carrier names forward on read. Old files stay readable and say what they said; new recordings are right.

  • Cyton timing error → no TRCA above ~30 Hz (2026-07-14; re-scoped 2026-07-15). Kept in the record because it is the most consequential thing we know about our own hardware and the reasoning is worth not re-deriving. The chunked BrainFlow→LSL bridge delivers only +1.4 dB phase-locking (PLV≈0.35) vs the actiCHamp's +5.7 dB on the same PC/monitor/renderer — putting TRCA at chance on Cyton 40 Hz recordings while phase-invariant FBCCA still reaches 86 %. The failure is frequency-scoped, not absolute. sub-902/ses-002 ran the same Cyton and headset in two bands 9 min apart: −0.58 dB at 36–44 Hz (TRCA 21 %, chance) vs +7.14 dB at 8–12 Hz (TRCA 96 %, 74 bpm). Scrambled timestamps would kill alpha too — what fits is a fixed ~5–6 ms timing error (phase error = 2π·f·Δt ⇒ ≈85° at 40 Hz, ≈21° at 10 Hz). Rough rule: an amp is TRCA-capable below ~30/Δt_ms Hz. Measuring it — and finding out whether it is a constant latency that can be calibrated out, or true jitter — is #9.

  • Godot renderer evaluation — only if a rich game engine ever becomes core to Gambit. Not Unity (proprietary/paid ⇒ breaks the free-for-commercial rule, no timing gain, no in-house expertise); Godot (MIT) is the one to evaluate if it comes up. See CLAUDE.md §3.
  • Eye-tracking — deprioritized. It is in the REB but not on the current Zeuron/Gambit roadmap. If revived: an additional LSL stream + hybrid decoding, gaze coordinates only, never webcam frames (COMPLIANCE R7).