Crate Architecture

How the workspace crates compose into a temporal event-processing platform — a generic engine plus operator catalogs, scripting, persistence, connectors, and a browser binding. tflo is experimental and pre-1.0 — not yet published to crates.io.

Dependency layers

tflo-core is contract-only above the engine: connectors, state stores, and scripting integrations live in separate crates so the engine stays free of direct deps on aws-sdk-s3, rdkafka, rumqttc, or reqwest. Operators were extracted from the engine into tflo-ops (and the finance-specific subset into tflo-fintech) so that domain catalogs evolve on their own cadence.

┌──────────────────────────────────────────────────────────────────┐
                       Connectors & sinks                         
   tflo-connect-kafka   tflo-connect-mqtt                         
   tflo-sink-influx     tflo-arrow (Parquet / Polars)             
├──────────────────────────────────────────────────────────────────┤
                  Scripting & policy filters                     
          tflo-cel    tflo-rhai    tflo-rego                      
├──────────────────────────────────────────────────────────────────┤
             Pattern matching (event sequences)                  
          tflo-cep (closure-based, WASM-clean)                    
├──────────────────────────────────────────────────────────────────┤
                     Operator catalogs                            
   tflo-ops (generic catalog)   tflo-fintech (finance plugin)     
├──────────────────────────────────────────────────────────────────┤
                          Engine                                  
   tflo-core                                                      
     graph + windowing + keyed execution                          
     timers + dedup + metrics + async contracts                   
     typed Absent + Checkpointer + ShardRouter                    
├──────────────────────────────────────────────────────────────────┤
                       Persistence                                
      tflo-state-files          tflo-state-s3                     
├──────────────────────────────────────────────────────────────────┤
                         Browser                                  
      tflo-wasm (ops/fintech)    tflo-cep-wasm (patterns)         
└──────────────────────────────────────────────────────────────────┘

Engine

tflo-core

The temporal event-processing engine. Provides the .tflo() / .with() / .tflo_keyed() iterator adapters, property extraction, windowing primitives, the compiled graph runtime, the typed Computed = Result<f64, Absent> semantics, the Operator trait, timers, the Deduplicator, the Metrics trait, the async state / checkpoint / shard-router contracts, and Builder::fingerprint(). Async support via --features async.

cargo add tflo-core

Operator catalogs

tflo-ops

The generic operator catalog. Owns every windowed aggregation (SMA, EMA, STD, VAR, WMA, median, quantile, skewness, kurtosis, correlation, covariance), the stateful trackers (prev, lag, cumulative_*, pct_change, zscore, rate_of_change, momentum, peak_decline), and the event detectors (cross, glitch, runt, pulse_width, window_detector, zone). Extracted from tflo-core in the hardening pass so domain catalogs can evolve independently.

cargo add tflo-ops

tflo-fintech

Financial technical-analysis indicators (MACD, ADX, ATR, KAMA, Stochastic, CCI, OBV, VWAP, Bollinger, Ichimoku) as a tflo-ops plugin. Bit-exact golden-fixture suite validated against TA-Lib. Finance is one domain catalog among many.

cargo add tflo-fintech

Pattern matching

tflo-cep

Closure-based event-pattern matching — "A then B within T", "A then not B within T", bounded sequences. Six-method builder (when / then / not_then / within / emit plus timestamp), Match<E> with first / last / at(name) / named() / all(). The shared engine::Runtime is also reused by tflo-cep-wasm. WASM-clean by construction.

cargo add tflo-cep

tflo-cep-wasm

WebAssembly bindings for tflo-cep. Wraps engine::Runtime with JS-callback adapters (Rc-based, since wasm32 is single-threaded). ~41 KB optimized .wasm. Consumed by the @tflo/browser-events TypeScript SDK — see the browser analytics page for the end-to-end flow.

Persistence

tflo-state-files

File-based checkpoint store. Implements both the sync StateStore and (under the async feature) the AsyncStateStore trait via spawn_blocking — same backend, both interfaces.

tflo-state-s3

S3-compatible checkpoint store. Rewritten in the hardening pass as a direct AsyncStateStore impl over a pluggable S3Client trait (drop in aws-sdk-s3, MinIO, or a mock). Coordinates with Checkpointer for crash-safe (snapshot → cursor) ordering.

Connectors & sinks

tflo-connect-kafka

Kafka integration. KafkaShardRouter requires an AsyncStateStore at construction (compile-time poka-yoke against sharded execution without durable state). Optional rdkafka-backend feature wraps the production client. AssignmentEpoch fences events arriving during rebalance.

tflo-connect-mqtt

MQTT adapter for the edge. MqttCursor enforces a bounded QoS-2 dedup window (64 KiB cap) so the in-flight set cannot grow unbounded. Optional rumqttc-backend — pure Rust, cross-compiles cleanly to ARM.

tflo-sink-influx

InfluxDB write sink. LineProtocol builder plus a Batcher with soft flush and a hard max_buffer_bytes cap. Pluggable InfluxHttpClient trait so HTTP client choice is the user's.

tflo-arrow

Arrow / Parquet / Polars interop. schema_fingerprint() stamps a hash into snapshot metadata so backfills against a structurally-different schema are rejected on restore. parquet_io::{read,write}_batches helpers cover replay sources and result sinks.

Scripting & policy filters

tflo-cel

Common Expression Language filter. Implement IntoCelContext for your domain type and call .cel_filter("expr") on any iterator — no graph step cycle required.

tflo-rhai

Rhai scripting integration. Implement IntoRhaiScope and call .rhai_filter("expr") — or load full scripts at runtime without recompiling.

tflo-rego

OPA / Rego policy engine integration. Load policies, then call .rego_filter(engine, "data.<pkg>.allow") for compliance and governance use cases.

Tooling & examples

tflo-wasm

WebAssembly bindings for the operator catalogs (tflo-ops and tflo-fintech). Run streaming detectors and TA indicators live in a dashboard, extension, or service worker. For browser event-pattern matching, see the sibling tflo-cep-wasm crate above. See the WebAssembly page.

tflo-examples

Runnable examples backing the docs and blog. The iot-portal example stitches MQTT, Kafka, Influx, and Parquet into the end-to-end reference deployment.

Adding crates to your project

Pick the crates you need — you don't have to pull in everything:

# Minimal: engine + operator catalog
cargo add tflo-core tflo-ops

# Add the financial technical-analysis plugin
cargo add tflo-core tflo-ops tflo-fintech

# Engine plus async stream support (Tokio)
cargo add tflo-core --features async

Next: Explore Contracts for the four pluggable traits, or Advanced for checkpointing, async, dedup, metrics, and scripting.