Validation and benchmarks

What is (and is not) claimed

A field may claim a better status than unvalidated only by citing a passing evidence manifest (benchmarks/evidence/), and tools/check_validation_registry.py checks every claim mechanically. Schema 1.0.0 fields are all unvalidated. In schema 1.1.0 (ecg-records 0.2.0) four record-level intervals and, since ecg-records 0.3.2, the per-lead t_offset are benchmark_validated; the heart rate and the P, QRS and T frontal axes are indirectly_validated (agreement with GE 12SL no worse than a second commercial algorithm's); the other 71 published fields are unvalidated. None of this is a clinical-performance claim.

Each field's status is in the record (/<field>/validation/status) and in every query result (result.validation.tier). It comes from the validation-evidence registry shipped for the record's schema family (ecgfeat/schemas/ecg-record/<1.0|1.1>/validation-evidence.json). The strict reader rejects any record that claims more than its family's registry allows (validation_above_registry).

Record status Query tier (result.validation.tier) Meaning
unvalidated unvalidated no passing evidence
indirectly_validated partially_validated supported by evidence against another algorithm or a related endpoint
benchmark_validated validated a manifest shows the field meets a stated threshold against an expert reference on a named dataset
known_problem unvalidated known to be unreliable; the debug-only four-class ST morphology is capped here

The registry also records a validation ceiling per field: the best status the field could ever reach with evidence. See the schema references (1.1, 1.0) and the validation policy.

Record-level intervals (schema 1.1.0)

Criterion: IEC 60601-2-25:2011, 201.12.4.102.3.2, global intervals, evaluated the CSE way: after removing the 8 differences farthest from the initial mean, the mean difference and its standard deviation must lie within the limits. Differences are record minus reference, in ms. Every run used the published entry point (ecg_record, default configuration) on a clean commit of the 0.3.2 release, whose engine carries the per-lead fiducials round of 2026-10-16; record-level intervals moved within noise against 0.2.0, and the compiled wheel reproduces the source tree byte for byte.

Field Limits (mean / SD) LUDB 1.0.1 vs experts PTB-XL hold-out vs GE 12SL Status
p_duration_ms ±10 / 15 −0.1 / 11.8 (n=167) ✔ +3.5 / 16.9 (n=2,675) ✘ benchmark_validated
pr_interval_ms ±10 / 10 +2.9 / 5.5 (n=165) ✔ +0.7 / 10.7 (n=2,657) ✘ benchmark_validated
qrs_duration_ms ±10 / 10 +2.8 / 6.3 (n=192) ✔ −0.1 / 7.8 (n=2,992) ✔ benchmark_validated
qt_interval_ms ±25 / 30 +9.9 / 9.9 (n=192) ✔ +0.6 / 13.5 (n=2,983) ✔ benchmark_validated
  • LUDB (200 records, 500 Hz, 12 leads, adults and children, many rhythms) is the expert reference: record-level values are reconstructed from the per-lead expert annotations. It is a proxy for the CSE database, not a conformance test.
  • PTB-XL (a pinned, patient-disjoint 3,000-record hold-out) is compared with GE 12SL from PTB-XL+ 1.0.1: indirect evidence, since the reference is another algorithm. P duration and PR fall outside the limits against 12SL. For context, Glasgow Uni-G against 12SL on the same records gives P −0.4 / 14.9 ms and PR −4.4 / 12.5 ms: the two machine algorithms also disagree on PR by more than the limit.
  • QT values whose feature state is not observed (about 30%: the QT reportability gate) are included above; restricted to observed values, LUDB gives +9.1 / 6.6 ms and PTB-XL −0.0 / 8.7 ms.
  • Coverage (published values / records with a reference): LUDB 99–100%, PTB-XL 98–100%.

Heart rate and frontal axes (schema 1.1.0, ecg-records 0.3.2)

The standard defines no limit for these, so the criterion is the inter-machine baseline: on the same PTB-XL hold-out, the record must agree with GE 12SL at least as closely as a second established algorithm, Glasgow Uni-G (PTB-XL+ 1.0.1), agrees with 12SL: CSE-trimmed SD not above the Uni-G–12SL SD, and |mean| not above max(|Uni-G–12SL mean|, one unit). The reference is another algorithm, so these fields are indirectly_validated.

Field Record vs 12SL (mean / SD, n) Uni-G vs 12SL (mean / SD) Status
heart_rate_bpm +0.0 / 0.8 /min (n=2,992) ✔ −0.5 / 1.2 /min indirectly_validated
p_axis_deg +0.8 / 11.3° (n=2,589) ✔ +0.7 / 14.1° indirectly_validated
frontal_qrs_axis_deg −0.5 / 5.6° (n=2,991) ✔ +0.1 / 6.7° indirectly_validated
t_axis_deg −0.1 / 11.8° (n=2,921) ✔ −0.9 / 17.7° indirectly_validated

Axis differences are wrapped to [−180°, 180°). Coverage: heart rate 100%, axes 95–100% of the records with a reference.

Per-lead fiducials (schema 1.1.0, ecg-records 0.3.2)

Every published beat × lead cell is paired with the expert annotation of the same lead on LUDB (record minus expert, ms, all leads pooled); the criterion is the CSE Working Party delineation tolerance on the standard deviation of the error. The per-lead fiducials round of 2026-10-16 roughly halved the P errors; t_offset is the first per-lead cell within its tolerance.

Cell Mean / SD (n) Sensitivity Tolerance (SD) Status
t_offset +4.8 / 28.1 ms (n=19,393) ✔ 99.4% 30.6 ms benchmark_validated
p_onset −3.5 / 20.4 ms (n=16,555) ✘ 99.7% 10.2 ms unvalidated
p_offset −1.8 / 19.2 ms (n=16,546) ✘ 99.6% 12.7 ms unvalidated
qrs_onset +0.2 / 10.8 ms (n=21,800) ✘ 100% 6.5 ms unvalidated
qrs_offset −2.2 / 12.2 ms (n=21,800) ✘ 100% 11.6 ms unvalidated

For comparison, 0.2.0–0.3.1 gave SD 37.2 (P onset), 34.1 (P offset), 12.0 (QRS onset), 15.5 (QRS offset) and 38.0 ms (T offset).

Not validated

Endpoint Result Criterion
per-lead P onset / P offset / QRS onset / QRS offset on LUDB see the table above: SD 20.4 / 19.2 / 10.8 / 12.2 ms CSE delineation tolerances on the SD (10.2 / 12.7 / 6.5 / 11.6 ms): not met
QTc Bazett / Fridericia vs 12SL (CSE-trimmed) mean +1.7 / +1.1, SD 17.0 / 15.5 ms none: Uni-G publishes one corrected QT of unstated formula, so there is no inter-machine baseline
median-beat R / S / Q / T amplitude per lead vs 12SL, all leads pooled, cells where both report the wave (µV) mean +4.3 / +3.9 / −7.0 / +7.3, SD 80.6 / 81.7 / 52.9 / 70.4 (n=33,849 / 23,728 / 11,828 / 34,295) inter-machine baseline (Uni-G vs 12SL mean +6.7 / +2.6 / −7.7 / +0.4, SD 78.6 / 54.2 / 42.2 / 73.8): not met (T: SD within, mean bias +7.3 µV above the 1 µV allowance)
median ST at J per lead vs 12SL (µV) mean +2.6, SD 39.8 (n=33,341) none (Uni-G publishes ST at 80 ms only)

Both algorithms report the same wave (present or absent) in 98.2% (R), 93.4% (S), 92.6% (Q), 95.3% (T) and 92.9% (ST) of the cells. Amplitude accuracy cannot be assessed on LUDB (normalized signals); the 12SL comparison is the first amplitude evidence and is reported here so the gap is visible.

These stay unvalidated. The manifests, raw outputs (runs/), tool and dataset hashes and every number above are in benchmarks/evidence/; the evaluation tools in benchmarks/evidence/tools/ use the LUDB annotation harness of the research repository and the mounted datasets.

Regression gates

The repository guards against unintended changes in what the library measures. These gates bound regressions from an accepted baseline. They are not claims of clinical accuracy.

  • Golden corpus (snapshot_regression.py, details): 156 sentinel and 1,063 full-tier records from seven public datasets (LUDB, QTDB, EDB, PTB-XL, BUT-PDB, NSTDB, GUDB). The frozen ecg-records-0.2.0 baseline holds the SHA-256 of every record profile that release produces (ecg-records-0.1.0 keeps the previous release's). Any change to a published value shows up as a difference. The older phase0-52a339c baseline preserves the pre-migration outputs: the legacy export reproduces them byte for byte, and every record value equals its legacy value through the crosswalk.
  • Contract, property and fixture tests: schema and invariant checks, Hypothesis property tests of the record builder, the pinned 10 s reference record and the summary byte contract (30,000 bytes in schema 1.1; 24,000 in 1.0).
  • Live crosswalk (tools/check_record_crosswalk.py): on the 108 default standard and limited sentinel cases, every published value equals the legacy export of the same engine run under independently re-implemented rules.
  • Forward compatibility: the released 0.1.0 reader loads 0.2.0 records in strict mode.
  • Performance budget (tools/check_performance.py): import time, first and warm extraction time, and peak memory on the reference record.
  • Layer contracts (import-linter): the record API, the engine, the pipeline and the plotting package stay separated.

Known weaknesses

  • P-wave delineation is the weakest endpoint in the benchmarks, especially at short RR intervals or when the P wave overlaps the preceding T wave.
  • LUDB distributes amplitude-normalized signals, so amplitude accuracy cannot be assessed on LUDB.
  • The four-class ST morphology was never validated and is anti-correlated with ischemia. It is not published (debug-only, capped at known_problem).

See also Limitations and intended use.