Open Experiment Standard
OES v0.1.0 · Draft

Specification

OES is a versioned JSON document, validated with JSON Schema, with optional attached CSV / Parquet / HTML / PNG artifacts. JSON Schema is a good fit because it already defines Core and Validation specs for describing and validating JSON documents.

The key design principle: separate what was planned, what happened, how it was analyzed, what was concluded, and what should be shown to humans.

Top-level structure

{
  "schemaVersion": "0.1.0",
  "objectType": "experiment",
  "experiment": {},
  "design": {},
  "variants": [],
  "metrics": [],
  "analysis": {},
  "results": {},
  "scorecard": {},
  "decision": {},
  "qualityChecks": [],
  "artifacts": [],
  "provenance": {},
  "extensions": {}
}

Sections

  1. Section 1
    Envelope and standard metadata

    Portability and version safety for the document itself.

  2. Section 2
    Experiment identity and documentation

    The human-facing context that makes a result meaningful later.

  3. Section 3
    Design and protocol

    The intended experimental design — not just what happened to be measured.

  4. Section 4
    Variants

    Each variant — independently identifiable, documentable, and presentable.

  5. Section 5
    Metric definitions

    Snapshot metric identity AND calculation — definitions change.

  6. Section 6
    Analysis configuration

    How the scorecard was computed — reproducibility starts here.

  7. Section 7
    Results

    Machine-readable, granular, and per-metric — not just an overall summary.

  8. Section 8
    Scorecard

    A curated, presentation-ready view — not a dump of every metric.

  9. Section 9
    Decision record

    Experiments fail as institutional memory when the result exists but the decision does not.

  10. Section 10
    Quality checks

    Trust checks as a first-class part of the standard.

  11. Section 11
    Data lineage and provenance

    What allows another system to trust — or reproduce — the result.

  12. Section 12
    Artifacts and package format

    Charts, CSVs, SQL, notebooks — bundled or linked.

Conformance summary

  • Conforming documents MUST set schemaVersion and objectType and SHOULD validate against the published JSON Schema for that version.
  • Importers MUST safely ignore unknown fields and unknown extensions.* namespaces.
  • Producers SHOULD snapshot metric definitions and code versions into the document — references alone are not sufficient because those references change over time.
  • Quality checks SHOULD be included whenever the producing system knows their result; absence of a check is not the same as a passing check.