tflo
testing golden-tests validation quality

Golden tests — catching regressions in floating-point math

When an operator's output has to match a reference implementation to six decimal places, unit tests won't cut it. Here's how tflo uses golden vectors to lock in correctness.

10 min read

You change one line in an RSI implementation. The math looks right. The tests pass. But when you compare the output to a reference chart, it’s off by 0.3 at period 22.

That’s the problem golden tests solve. You capture the exact output of a known-good implementation and lock it in as a fixture. Every change is measured against that fixture. If the output shifts by more than 1e-6, the test fails.

tflo uses this approach to validate the technical-analysis indicators in its finance plugin. Those indicators have a single, well-known source of truth — the TA-Lib C library — which makes them an ideal proving ground for the golden-vector technique. The harness lives in the tflo-fintech crate, under tflo-fintech/tests/golden/. The technique itself is domain-agnostic: any operator with a reference implementation can be locked down the same way. Here’s how it works.

The golden vector format

A golden vector is a JSON file with input data, expected output, and metadata about where the reference came from.

{
  "metadata": {
    "indicator": "rsi_talib_count",
    "source": "talib",
    "version": "0.4.0",
    "provenance": "Generated from TA-Lib C library using Python bindings",
    "license": "BSD"
  },
  "params": {
    "period": 14
  },
  "input": [44.0, 44.25, 44.5, 43.75, 44.5, 44.25, 44.0, 43.5, 43.25, 43.0, 43.25, 43.5, 43.75, 44.0, 44.25],
  "expected_output": [null, null, null, null, null, null, null, null, null, null, null, null, null, null, 50.978],
  "warmup_samples": 15
}

The expected_output array is the same length as input. Values before warmup are null. After warmup, each position contains the exact floating-point value the reference produced at that step.

For multi-output indicators — MACD, Bollinger Bands, Stochastic — expected_output becomes an array of arrays, one per output channel:

{
  "expected_output": [
    [null, null, ..., 0.5, 0.6, ...],
    [null, null, ..., 0.4, 0.5, ...],
    [null, null, ..., 0.1, 0.1, ...]
  ]
}

Loading and running

The GoldenVector struct reads these files and validates their structure — input length matches output length, warmup count is consistent, parameters are present.

let vector = GoldenVector::load("tflo-fintech/tests/golden/fixtures/rsi/rsi_talib_14.json")?;
let results = GoldenRunner::run(&vector)?;

GoldenRunner::run dispatches to the right operator implementation based on the metadata string — "rsi_talib_count" routes to Wilder’s RSI (rsi_wilder_n), "ema_talib_count" routes to the TA-Lib compatible EMA, and so on for SMA, WMA, MACD, deviation bands, Stochastic, Williams %R, CCI, Z-Score, and many more.

It constructs the graph using the same .tflo() API you’d use in production — the core ops come from tflo-core, the finance indicators from tflo-fintech:

use tflo_core::prelude::*;
use tflo_ops::prelude::*;
use tflo_fintech::prelude::*;

let computed: Vec<_> = records
    .iter()
    .cloned()
    .tflo(|t| {
        t.timestamp(|x| x.idx);
        let price = t.prop(|x| x.close);
        price.rsi_wilder_n(period)
    })
    .collect();

Then pads the front with None for the warmup period, appends the computed values, and returns a Vec<Option<f64>>.

Validating the results

The validate function compares expected vs actual output with a configurable tolerance:

use super::{GoldenVector, GoldenRunner, validate};

let vector = GoldenVector::load(fixtures_path)?;
let results = GoldenRunner::run(&vector)?;
let expected = vector.expected_output_single()?;
let result = validate(&expected, &results, 1e-6);

println!("Passed: {}", result.passed);
println!("Samples compared: {}", result.samples_compared);
println!("Max diff: {}", result.max_diff);
println!("Mean diff: {}", result.mean_diff);

The ValidationResult tells you:

  • passed — did everything stay within tolerance?
  • samples_compared — how many non-warmup positions were compared
  • samples_matched — how many positions matched within tolerance
  • max_diff — the largest absolute deviation found
  • mean_diff — the average deviation across all samples
  • mismatches — the first 10 positions that differed, with expected vs actual values and the difference

The default tolerance is 1e-6, which is tight enough to catch off-by-one smoothing errors but loose enough to pass across hardware and compiler variants. CCI gets a slightly higher tolerance of 1e-5 because its formula produces larger raw values. ADX and TRIX use 0.01 due to their recursive smoothing chains. MACD uses 1.0 internally because TA-Lib’s fast/slow EMAs use a slightly different seeding convention than the standalone EMA function — the difference decays over the series.

What this catches

Golden tests catch the subtle bugs that unit tests miss:

  • Smoothing method swaps — using simple average where Wilder’s RMA is expected. The difference is ~0.3 on volatile data. A unit test that checks “RSI is between 0 and 100” passes. A golden test that checks “RSI at index 14 equals 50.978” fails.

  • Off-by-one in warmup — if SMA(20) warms up at 19 samples instead of 20, the first output is wrong for every input. The first null vs Some(value) mismatch flags it immediately.

  • Initialization drift — EMA seeded with first value vs seeded with SMA(5). The difference compounds over time. A golden test across 500 samples catches the drift curve.

  • NaN propagation changes — if a code change accidentally turns NaN into 0.0 or skips a NaN check, the golden output diverges at the first bad position and never recovers.

The fixture library

A complete set of 55 golden vectors lives in tflo-fintech/tests/golden/fixtures, organized by indicator. Every one passes bit-exact against TA-Lib at the 1e-6 tolerance:

fixtures/
├── adx/              # Periods 14, 20
├── atr/              # Periods 14, 20
├── bollinger_bands/  # Multiple configs
├── cci/              # Periods 14, 20
├── cmo/              # Periods 14, 20
├── dema/             # Periods 10, 20
├── ema/              # Periods 5, 12, 26
├── kama/             # Periods 10, 30
├── linearreg_slope/  # Periods 14, 20
├── macd/             # Multiple configs
├── mfi/              # Period 14
├── minus_di/         # Periods 14, 20
├── mom/              # Periods 10, 20
├── obv/              # Single config
├── plus_di/          # Periods 14, 20
├── ppo/              # Multiple configs
├── roc/              # Periods 10, 20
├── rsi/              # Periods 7, 14, 21
├── sma/              # Periods 5, 20, 50
├── stochastic/       # Multiple configs
├── stochrsi/         # Multiple configs
├── tema/             # Periods 10, 20
├── trima/            # Periods 10, 20
├── trix/             # Periods 15, 20
├── williams_r/       # Periods 14, 21
├── wma/              # Periods 5, 20
├── zscore/           # Periods 20, 30
└── edge_cases/       # NaN, constants, extremes

The edge case vectors are the most informative. Constant 1.0 across 100 values. Alternating high/low. Random noise. NaN in the middle of the series. These stress the edge cases that standard fixtures never trigger.

Generating your own vectors

Golden vectors are generated from TA-Lib via a Python script. TA-Lib is the single source of truth — tflo never generates vectors from its own output, which would be circular.

From TA-Lib — requires the TA-Lib C library and Python bindings:

python3 tflo-fintech/tests/golden/scripts/generate_talib_vectors.py --all --output-dir tflo-fintech/tests/golden/fixtures

You can also generate vectors for a single indicator:

python3 tflo-fintech/tests/golden/scripts/generate_talib_vectors.py --indicator rsi --periods 7 14 21 --output-dir tflo-fintech/tests/golden/fixtures

Other reference sources — not yet automated (the README lists this as future work), but the vector format supports them. The metadata tracks provenance so you always know which reference a fixture was generated against.

Each vector records its source in the metadata.provenance field. You can tell at a glance whether a fixture is validated against TA-Lib or just a baseline snapshot.

Why this matters for streaming

In a batch context, you can visually inspect indicator output. Plot the RSI, squint, decide it looks right.

In a streaming context, you don’t get a plot. You get a number at every tick. If that number is off by 0.3 at period 22, it shifts every downstream event — crossovers fire early or late, thresholds trigger at the wrong level, and you don’t notice until someone asks why the live system disagrees with the offline replay.

Golden tests close that gap. The fixture is the contract. If your output matches the fixture at 1e-6, your operators produce the same values as the reference. Period.

Source code

The full working example is available on GitHub: tflo-examples/examples/golden-tests

— Matt