BIDS conversion & the `BIDS/` dataset¶

How raw acquisitions become a BIDS-EEG dataset, and how to regenerate/analyze it. The BIDS/ tree is git-ignored (participant data can never enter git by construction — see docs/DATA_GOVERNANCE.md; the Dalhousie-specific binding is COMPLIANCE R1); only the code below is version-controlled.

Layout¶

BIDS/
├── dataset_description.json, README, CHANGES, .bidsignore
├── participants.json                              # the column dictionary (static, tracked)
├── participants.tsv                               # sub-XXX + species/group, demographics = n/a
│                                                  #   GIT-IGNORED: grows a row per participant,
│                                                  #   regenerated by the converter
├── task-ssvep_events.json, channels.json         # inherited sidecars (event + impedance columns)
├── code/                                          # dataset-local runners (see below)
├── sourcedata/sub-XXX/ses-YYY/eeg/                # ← RAW: *.xdf + *.session.json + *_impedance.json
└── sub-XXX/ses-YYY/eeg/                           # ← DERIVED (BIDS): BrainVision + sidecars
        *_eeg.vhdr/.vmrk/.eeg  *_eeg.json  *_channels.tsv  *_events.tsv
        (*_electrodes.tsv + *_coordsystem.json only where the montage has 10-20 positions)

The XDF + embedded manifest in sourcedata/ are the source of truth. The subject-level BrainVision is derived and can be regenerated at any time; never hand-edit it.

The converter (`ssvep.io.bids`)¶

convert_dataset(BIDS_ROOT) (or python scripts/bids_convert.py):

Migrates any raw files still at the subject level down into sourcedata/ (idempotent). As of 2026-07-15 the runner writes into sourcedata/ directly (RunRecorder.output_base), so this step only matters for legacy trees recorded before that change — a raw .xdf at the subject level is not a BIDS datatype and makes the root fail validation.
For each run: reads the XDF (EEG µV + structured markers) and its session.json manifest → mne.io.RawArray (µV→V) → BrainVision via mne-bids, with:
channel names = the manifest montage's scalp labels;
a standard_1020 montage only when all labels are 10-20 sites (so the g.tec Unicorn runs get electrodes.tsv/coordsystem.json; the OpenBCI colour-labelled dry runs do not);
sample-accurate events: each marker is snapped to its nearest EEG sample via the recorded LSL timestamps (not time×nominal_rate), so events stay aligned despite clock drift;
a rich events.tsv (trial_type, target_freq_hz = decode label, condition/block/trial/ target_id), an impedance column in channels.tsv, and device/SSVEP fields in _eeg.json.
Writes the dataset-level files and a participants.tsv (no demographics are invented).

The `task-<x>` entity comes from the manifest, per run¶

Each run is labelled from its own manifest.run.task (bids.task_of; legacy experiment.task is still read), and a task-<x>_events.json is written for every task present in the dataset. A dataset holds more than one paradigm: task-ssvep and task-rest live side by side.

Why this is spelled out. convert_dataset(task=…) used to apply one caller-wide default to every run, so sub-902/ses-002's resting run landed as task-ssvep despite its manifest saying rest (2026-07-15). That is precisely the "analysis infers design instead of reading the manifest" failure the manifest spine exists to prevent — CLAUDE.md §2. The task= argument still exists, but only for tests/legacy callers; leave it None.

The `acq-<board>` entity¶

Runs in one session used different front-ends with different montages (OpenBCI Cyton colour-labels vs Unicorn 10-20). Because electrodes.tsv is scoped by [ses][acq][space] (not by run), each run carries acq-<board> (cyton, cytonwifi, unicorn) so the montage sidecars attach to the headset that produced them instead of one wrong session-level file. It also self-documents the amplifier in every filename.

Running it — the standard procedure for new data¶

Whenever new recordings arrive in sourcedata/, run both runners in order — this is the reproducible, documented way to "review new data", not eyeballing a raw XDF:

conda activate ssvep
python code/01_xdf_to_bids.py          # or: python scripts/bids_convert.py BIDS
python code/02_run_offline_pipeline.py # epoch → FBCCA decode → metrics.json → report → derivatives/

01 creates/updates the top-level sub-XXX/ BIDS folders + sidecars from sourcedata/. 02 writes each run's metrics.json + HTML report to derivatives/ssvep-analysis/, applies the per-recording spatial decode policy from the manifest (ssvep.spatial.occipital_decode_plan — large reference-free caps decode from an occipital ROI with CAR; stored as reref/decode_channels in the metrics), and writes a timestamped run log to derivatives/ssvep-analysis/logs/ recording toolbox + dependency versions, the git commit, and the spatial policy + result per run. The log + derivatives are the review record. --no-auto-spatial reverts 02 to the legacy all-channels/no-reref decode (which reads at chance on the 64-ch actiCAP — see docs/ACTICHAMP.md, DESIGN_PRINCIPLES #8).

BIDS/code/ holds thin, dataset-local runners that call the version-controlled, unit-tested package (tests/test_bids.py), so the dataset is self-describing on the NAS while the logic stays in git.

`02` does `01`'s work for the runs it analyses — it no longer assumes you ran it¶

Analysis reads the XDF (sourcedata/) directly, so nothing in the pipeline ever needed the BrainVision copy — and so it got skipped. Analysing from the GUI (which never invoked 01) filled derivatives/ for subjects that had no sub-XXX/…/eeg/*.vhdr at all: a dataset of derivatives with no BIDS-standard raw data to be a derivative of.

run_batch now derives the BIDS view of its selected runs first (ensure_bids=True, the default), converting only runs that lack an up-to-date one:

"Already there" is checked, not assumed — ssvep.io.bids.is_converted wants the run's .vhdr + .eeg to exist and to be no older than its XDF. A source newer than the copy derived from it means the copy describes data that has since changed, so it is re-derived. Conversion is idempotent; the cost is paid once per recording.
A converter code change is also checked, not just the data (#113). The mtime check above only catches a source XDF that changed — it cannot see that the converter changed, so a bug fixed in the conversion logic never used to reach a run that was already converted (its XDF hadn't moved, so is_converted reported it up to date forever). Every conversion now also stamps a *.converter.json marker beside the BrainVision header with a hash of ssvep/io/bids.py's own source (ssvep.io.bids._converter_fingerprint — not ssvep.__version__, which is a static "0.0.0" and would never move); is_converted compares that stamp against the fingerprint of the code running now, and a run with no marker at all (converted before this existed) is treated the same as a mismatch. The marker is a plain file next to the derived output, not a new key in the *_eeg.json BIDS sidecar, so it never reaches the schema the official validator checks — it is listed in .bidsignore like the other convenience sidecars. ⚠️ Operational consequence: the first 01_xdf_to_bids.py run after this lands reconverts every already-converted run in the tree once, because none of them yet carry the marker — expected and intended, not a regression.
Whole-tree metadata stays whole-tree. When anything is converted, the dataset-level files (dataset_description.json, README, participants.tsv, task-<x>_events.json) are rewritten from every run on disk, not just the selection — otherwise analysing one session would drop the task labels of the runs it didn't touch.
A failed conversion does not withhold an analysis. The XDF is the source of truth; a run whose BIDS view can't be derived is a broken export, logged as BIDS: … could not be converted, and still analysed.

01 remains the way to (re)derive the whole tree in one go — including migrating any legacy top-level raw files down into sourcedata/, which 02 does not do.

Analysing one session, not the whole dataset¶

02 takes a selector, so finishing a visit does not mean re-analysing every participant ever collected (#67):

python code/02_run_offline_pipeline.py --sub 902 --ses 002   # one session
python code/02_run_offline_pipeline.py --sub sub-902         # one subject, all sessions
python code/02_run_offline_pipeline.py                       # the whole dataset (unchanged default)

Both entity forms are accepted (902 and sub-902) because operators type both. --ses without --sub is refused — "session 002" is only meaningful inside a participant — and a selector matching nothing exits non-zero listing the subjects/sessions that do exist, because a silent "0 analysed" reads as success and is how a typo'd ID becomes "my data is fine". The run log records the selector used, so a log in logs/ is never ambiguous about its coverage.

The selection and the batch live in the package (ssvep.analysis.batch), not in the script: the script is argparse over run_batch, and the GUI's Analyze tab (below) calls the same function.

From the GUI: the Analyze tab¶

ssvep-toolbox's Analyze tab is the same batch with a selector you cannot typo. It scans the BIDS root's sourcedata/ and shows a checkable sub-XXX → ses-YYY → run tree with each run's paradigm (read from its manifest), whether its BIDS raw data has been derived, and whether it already has a Report; tick a session, press Analyze, and it runs ssvep.analysis.batch.run_batch on a worker thread — same derivatives/ssvep-analysis/ output, same run log. The tab's BIDS root follows the Set up Session tab's output directory, so "analyse the session I just recorded" needs no path typed.

Because the tab calls run_batch, analysing also converts: any ticked run without an up-to-date BIDS view is converted first (status: Converting to BIDS i/n: <run>…, which is where the first minutes of a fresh session go), and the finished status says how many were converted. This is the whole point of the BIDS column — the conversion state is visible before you press anything, rather than something you have to remember to have done from a terminal.

When it finishes, the index opens by itself. Every batch rebuilds derivatives/ssvep-analysis/index.html and the GUI opens it in your browser (Open index opens it any time — see below; the CLI logs its path). One tab, not one per run: an 8-run session would otherwise bury the browser. Open report, or double-clicking a run in the tree, opens a single run's *_report.html directly.

The derivatives index (#80)¶

Analysis outputs are navigable, not a flat pile. Each run's report and metrics are written to derivatives/ssvep-analysis/sub-XXX/ses-YYY/ — the relative path of the raw recording they came from — and three levels of generated index.html link them:

derivatives/ssvep-analysis/
├── dataset_description.json      # this is a derivatives dataset (#18) — see below
├── index.html                    # every subject → its sessions
├── sub-902/
│   ├── index.html                # every session → its runs
│   └── ses-002/
│       ├── index.html            # every run, in protocol order, + its headline result
│       ├── sub-902_ses-002_task-ssvep_run-001_report.html
│       └── sub-902_ses-002_task-ssvep_run-001_metrics.json
└── logs/offline-pipeline_<stamp>.log

Three things about that page are deliberate:

It is rebuilt from disk, not from the batch. ssvep.analysis.derivatives.build_index scans the tree for *_report.html + *_metrics.json and rewrites every level. So it covers everything analysed so far rather than the last selection, a batch that died part-way still leaves it truthful, and re-analysing a run updates its row instead of adding a page. It replaces the old per-batch summary_<stamp>.html, which described one invocation and accumulated one file per run of the pipeline — three pages that each looked like the entry point.
Run order is declared, not inferred. The # column is the run's run_index from its session_position sidecar stamp (#43) — its derived place in the session protocol — so the page reflects the order the participant actually experienced, not alphabetical filenames. A run with no sidecar (ad-hoc, or pre-#43) is still listed, tagged ad-hoc, ordered after the declared ones.
Links are relative. The pages open by double-clicking off local disk or the NAS; no server.

A first analysis on an existing tree migrates the old flat layout: reports and metrics are moved into sub-XXX/ses-YYY/ (moved, not regenerated) and the obsolete summary_*.html are removed. It is idempotent and logged.

It is a formal derivatives dataset (#18)¶

derivatives/ssvep-analysis/dataset_description.json declares DatasetType: "derived", a GeneratedBy block naming the pipeline, the toolbox version, and the git commit that produced the outputs, and a SourceDatasets link back to the raw dataset. Previously the directory held analysis outputs with nothing but its name to say what had made them or from what.

One deliberate deviation from strict BIDS-derivatives filenaming, recorded in the description itself: outputs sit at sub-XXX/ses-YYY/ without the eeg/ datatype level, because an HTML report spans a whole run rather than describing one datatype's data.

Notes / limits¶

BrainVision stores samples on the nominal clock (250/500 Hz); events are sample-indexed, so epoching is exact even though the wall-clock length differs from the effective rate by ~0.2 %.
The full BIDS validator now runs via bids-validator-deno (in environment.yml's pip section — ships the JS bundle plus a deno wheel that vendors the Deno runtime, so no separate Deno/npm/node install and nothing fetched to obtain the tool itself; the launcher grants Deno --allow-net unconditionally, and the bundled validator can still reach out — for remote HED schemas when a dataset declares a HED version, or for the BIDS schema itself under --schema stable/latest):
```
conda activate ssvep
bids-validator-deno BIDS/
```
Before this, this repo relied only on mne-bids' writer + a read_raw_bids round-trip test. Run this before any release/share.
On a checkout with no local data, a non-zero exit is expected and is not a conformance failure. The tracked scaffold is de-identified metadata only — sub-*/ is git-ignored — so the validator reports SUBJECT_FOLDERS ("no subject directories") and, at [ERROR] level, SIDECAR_WITHOUT_DATAFILE for channels.json / participants.json. Those are sidecars whose data files are not on this machine, which is the compliance design working, not a defect.
⚠️ Run it locally and keep its output local. The validator prints sub-* paths; summarising findings elsewhere means rules, counts and sub-XXX codes only (COMPLIANCE R2/R4).

What the validator found, and what we did about it¶

Validated against the real tree: 2026-08-06, 13 subjects / 738 files / 145 MB, exit 0 — zero errors, the converter's output is valid BIDS-EEG (#110). Every remaining finding is a warning; here is the disposition of each, so a future run can tell a known-and-accepted warning from a new one. The first three rows were fixed in the converter and the tree re-derived from sourcedata/; the rest are deliberate or open.

⚠️ A converter fix used to not reach runs that are already converted. is_converted() compared only mtimes, so it couldn't see that the converter changed — clearing these findings needed a forced re-derive by hand. Fixed by #113: is_converted() now also stamps and checks a converter-version marker (see above), so a future converter fix reconverts every affected run automatically on the next 01/02 run, with no manual re-derive.

Finding	Disposition
`SIDECAR_FIELD_OVERRIDE` (5 keys × 56 files)	Fixed — and it was not cosmetic. mne-bids writes a run-level `_events.json` repeating every column with generic boilerplate, and BIDS inheritance is nearest-wins key by key* — so our real column documentation in `task-<x>_events.json` (the `trial_type` level map, `target_freq_hz` as the decoding class label) was written to disk and then shadowed on every run. A BIDS reader got "The type, category, or name of the event" instead of the design. The run-level file now carries only `StimulusPresentation`, which genuinely varies per run.
`SIDECAR_KEY_RECOMMENDED` (12 keys × 173 files)	Mostly fixed, from the manifest — `TaskDescription`, `Instructions`, `SoftwareVersions`, `CapManufacturersModelName`, `MISCChannelCount`, `Institution`, and `StimulusPresentation` (with the display's refresh rate, which decides which square-wave frequencies were presentable at all — and prefers the measured* rate over the nominal one when the builder recorded it).
`EVENTS_TSV_MISSING` (2 runs)	Not a converter defect — aborted runs (one is ~8 s of EEG with the marker stream present and nothing ever pushed). The data is real, so it is kept. What was a defect is that they converted silently; `ConversionReport.eventless` now reports them, on every pass and not just the first, and the pipeline logs a warning. Currently `sub-002` ssvep run-006 and `sub-955` rest run-001.
`TOO_FEW_AUTHORS`	Open — Aaron's call. `Authors` lists the lab as a single collective author. Naming individuals is an authorship decision, not a conversion one, and investigator names are not participant data either way.
`JSON_KEY_RECOMMENDED` → `HEDVersion`	Deliberate — do not "fix" it. We use no HED tags, and declaring a HED version is exactly what makes the validator fetch a remote HED schema at validation time. Leaving it unset is why validation runs with no network activity despite the launcher's blanket `--allow-net`. Silencing a cosmetic warning would buy an outbound request while the validator reads a tree of participant data.
`SIDECAR_KEY_RECOMMENDED` → `HeadCircumference`	Deliberate. A physical measurement of the participant — the kind of indirect identifier COMPLIANCE R3 keeps out of the tree — and it buys no analysis here.
`SIDECAR_KEY_RECOMMENDED` → `SubjectArtefactDescription`	Deliberate. Per-participant freeform that nothing records. A fabricated `"n/a"` is not neutral: BIDS reads it as absence of major artifacts, a claim nobody has made.
`SIDECAR_KEY_RECOMMENDED` → `CogAtlasID` / `CogPOID`	Deliberate. No Cognitive Atlas or CogPO term corresponds to this paradigm; a wrong URI is worse than an absent one.

The same rule governs the whole table: a recommended key gets filled when the manifest already knows the answer, and is left absent when filling it would mean inventing one. "n/a" is a claim, not a blank — SoftwareRRID and OperatingSystem are omitted for exactly that reason (and an "n/a" RRID is a schema violation, which the validator does catch as an error).

Reference/ground are unknown for these amplifiers (n/a); fill in if the hardware specifies them.

BIDS conversion & the BIDS/ dataset¶