Reference deployment

The iot-portal example is the canonical end-to-end pipeline using every connector tflo ships: an edge gateway conditions MQTT sensor data with tflo, publishes typed events into Kafka, a central worker shards them by sensor with KafkaShardRouter, computes derived signals, and writes results to Influx and a Parquet archive. The whole thing runs in CI with in-process mock clients — no broker, cluster, or TSDB required for a smoke test.

┌──────────────────────────────────────────────────────────────────┐
│                       iot-portal reference deployment            │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│   MQTT broker ──── edge gateway (tflo) ──── Kafka topic           │
│     (sensors)         conditioning,           (sharded events)    │
│                       hysteresis,                                 │
│                       SMA / cross                                 │
│                                                                  │
│                                                  │               │
│                                                  ▼               │
│                                         central worker (tflo)    │
│                                         keyed by sensor,         │
│                                         Checkpointer,            │
│                                         KafkaShardRouter         │
│                                                  │               │
│                          ┌───────────────────────┴────┐          │
│                          ▼                            ▼          │
│                   Influx (TSDB)             Parquet (archive)   │
│                   tflo-sink-influx           tflo-arrow           │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

The five stages

1. MQTT consume. The edge leg uses tflo-connect-mqtt (rumqttc-backend in production, an in-process mock in the example) to consume sensor readings. MqttCursor holds the bounded QoS-2 dedup window so the in-flight set can't grow unbounded.
2. Edge conditioning with tflo. An Iterator::tflo(...) closure applies SMA smoothing, hysteresis, and a threshold cross detector. Out goes a typed Signal<Mode, Payload> stream, not raw floats.
3. Kafka publish. tflo-connect-kafka publishes the signals into a sharded topic. Partition key is the sensor id, so the same sensor's events always land in the same partition.
4. Central worker (sharded). Worker processes consume from Kafka and route by sensor key with KafkaShardRouter. The Checkpointer coordinates snapshot writes with offset commits — snapshot first, cursor last, deadline per stage, circuit breaker on consecutive failures.
5. Sink and archive. Derived signals go two places: tflo-sink-influx writes line protocol to a TSDB, and tflo-arrow writes a Parquet batch for cold-storage replay. The Parquet schema fingerprint is stamped into SnapshotMetadata so a future backfill can refuse a schema-incompatible dataset at restore time.

Run it locally

The example uses in-process mocks so it runs offline:

cargo run --example iot-portal

Source lives at tflo-examples/examples/iot-portal/. The mocks implement the same KafkaConsumer / KafkaProducer / MqttConsumer / InfluxHttpClient traits as the real backends, so swapping in production clients is a feature-flag flip.

What this demonstrates

• The "tflo at the edge, mature engine in the center" pattern from positioning, with concrete code.
• End-to-end use of all four core contracts (AsyncStateStore, Cursor, ShardRouter, Operator) — see Contracts.
• The full connector ecosystem: Kafka, MQTT, Influx, Arrow / Parquet.
• Mock-friendly trait surfaces — production wiring is a feature flip, not a rewrite.