The acquisition pipeline

A technical reference for how a single byte travels from a connected device to a rendered widget, and how the frame parser and dataset transforms plug into that pipeline. If you're looking for the high-level user view, start with Data Flow. For threading-specific guarantees (what is and isn't guaranteed, why FrameReader and FrameBuilder run on a dedicated pipeline thread instead of the GUI thread), see Threading and Timing Guarantees. This page is for advanced users, plugin authors, and anyone debugging throughput, latency, or timing problems.

The acquisition pipeline is the chain of components that runs once per received frame at full data rate. Serial Studio targets sustained data rates above 256 kHz, so every stage on this path avoids allocations, copies, and cross-thread context switches.

Stages

flowchart TD
    A["Driver<br/>(driver thread or GUI thread)"] -->|CapturedDataPtr<br/>queued, chunk rate| B["FrameReader<br/>(pipeline thread)"]
    B -->|enqueue| Q["Lock-free queue<br/>65536 slots"]
    Q -->|dequeue, direct call| R["PipelineHost::routeFrames<br/>(pipeline thread)"]
    R --> E["FrameBuilder<br/>decode / parse / transforms<br/>(pipeline thread)"]
    E -->|stage rows into pooled DataBlock| S["DataBlock<br/>flush on display tick or sample cap"]
    S -->|SPSC ring, drained on<br/>Dashboard::onDisplayTick| F["Dashboard (GUI thread)"]
    S -->|clone_block_trimmed,<br/>if any sink active| G["CSV / MDF4 / API / gRPC / Sessions / MQTT"]

The driver-to-FrameReader hop is a genuine cross-thread QueuedConnection, but it fires once per received chunk, not once per frame — a chunk can carry many frames, so the cost is amortized across all of them. Every hop after that lives entirely on the pipeline thread and is either a direct in-thread call or a Qt::DirectConnection signal: a queued connection between two objects already on the same thread is avoided there, because at 10+ kHz it would fill Qt's event queue faster than the consumer can drain it, and FrameReader's bounded queue would start dropping frames.

MDF4 export, the Historian, and the MQTT bridge are Pro-only; a free build compiles those sinks out and skips them in fan-out.

Stage 1: driver

Drivers wrap one transport each (UART, TCP/UDP, BLE, Audio, Modbus, CAN Bus, MQTT, USB, HID, Process I/O). They publish data through HAL_Driver::publishReceivedData(...), which carries a shared IO::CapturedData payload:

• data (QByteArray; Qt's copy-on-write means consumers get an atomic refcount bump, not a deep copy)
• timestamp (steady-clock time of acquisition)
• frameStep (cadence in nanoseconds, when the driver knows it)
• logicalFramesHint (how many logical frames are encoded in the chunk, when known)

Drivers that already know their cadence fill frameStep so downstream stages can fan out timestamps without re-measuring. The audio driver, for example, backdates the chunk start by step * (totalFrames - 1) so each parsed sample lines up with the moment it was captured, not the moment Qt happened to deliver it.

If a driver posts to the main thread (via QMetaObject::invokeMethod or a queued connection), it must capture SteadyClock::now() before queueing and pass that into publishReceivedData. A default-constructed timestamp would otherwise be filled in on the receiving thread and report fictional timing.

Stage 2: frame reader

IO::FrameReader runs on the pipeline thread and owns a single producer / single consumer CircularBuffer. It scans the buffer for frame boundaries (start delimiter, end delimiter, both, or none) and pulls out one logical frame at a time.

• Single-delimiter modes use the KMP fast path.
• Multi-delimiter modes use CircularBuffer::findFirstOfPatterns(), a single-pass scan with a stack-allocated PatInfo array of up to eight patterns; it never touches the heap.
• Optional checksum validation (XOR-8, MOD-256, CRC-8, CRC-16, CRC-16-MODBUS, CRC-16-CCITT, Fletcher-16, CRC-32, Adler-32) runs immediately after extraction.

Each completed frame is enqueued into a lock-free moodycamel::ReaderWriterQueue<CapturedDataPtr> with 65536 slots. When that queue is full, frames are dropped and a log line is emitted. That message is the canonical signal that a downstream stage is too slow.

The frame reader is configuration-immutable. To change delimiters, decoder, or checksum mode, the live FrameReader is destroyed and a new one is created via ConnectionManager::resetFrameReader() or DeviceManager::reconfigure(). There are no mutexes anywhere on this path.

Stage 3: pipeline routing

IO::PipelineHost::routeFrames drains a FrameReader's queue on the pipeline thread and routes each dequeued frame directly into FrameBuilder::hotpathRxFrame or, for multi-source projects, hotpathRxSourceFrame(sourceId, data), as a plain same-thread call.

Stage 4: frame builder, parser, and transforms

FrameBuilder is where the project's parsing rules turn raw bytes into a populated Frame object. That Frame is a reused per-source staging buffer, not the object published downstream — once it's populated, FrameBuilder copies its values into the current DataBlock (see Stage 5 below). It runs three things in order:

1. The selected decoder (Plain Text (UTF8), Hexadecimal, Base64, or Binary (Direct)) converts the raw bytes into the form parse() expects.
2. The frame parser (parse(frame) in Lua or JavaScript) returns an array of values. See Frame Parser Scripting for the full API.
3. Per-dataset transforms (transform(value)) are called in group then dataset order. A transform can read raw values from any dataset and final values from datasets earlier in the order, plus shared variables from Variables. See Dataset Value Transforms.

In Quick Plot mode, steps 1 and 2 are replaced by a built-in line splitter that treats commas as the field separator. In Console-Only mode, the FrameBuilder stage is a no-op: bytes go straight to the terminal via DeviceManager::rawDataReceived.

When a single captured chunk expands into N logical frames, FrameBuilder timestamps them at data->timestamp + step * i. That only spreads the N frames across real time when the driver filled in a real frameStep. frameStep defaults to 1 ns, and most transports (UART, network, BLE, CAN, Modbus, MQTT) never override it, so on those a coalesced read still timestamps all of its extracted frames within nanoseconds of the arrival instant — there's no measured cadence to interpolate from. The spread is opt-in, per driver: Audio is the example that fills in a real frameStep (computed from the device sample rate) precisely so a multi-sample buffer's frames land at their true capture times instead of all landing on the buffer's arrival time.

The parser and transforms are the only points in the pipeline where user code runs. Both run under a runtime watchdog and are wrapped so that a thrown error, infinite loop, or non-finite return value falls back to the safe path: the raw value, or an empty frame. Errors do not interrupt the data stream. Transform watchdogs use a 100 ms budget; for JavaScript transforms the budget is armed once per frame in applyDatasetValues and covers all of that frame's transforms collectively, not per dataset call.

Transforms that don't declare an info parameter (function transform(value)) pay no extra cost for it: the engine inspects each transform's parameter count at compile time and skips building the info table or object when it isn't used.

Parser engine details

• One engine instance per source, never shared across sources.
• Lua uses an embedded LuaJIT 2.1 interpreter (Lua 5.1 syntax with compatibility shims) with base, table, string, math, utf8, and coroutine loaded. JavaScript uses Qt's QJSEngine with the Console and GC extensions only.
• Each parser is compiled once when the project loads or the connection opens, then called many times. Compilation cost is paid up front, not per frame.

Transform isolation

• In Lua, top-level local declarations become upvalues of the transform closure, so each dataset has its own state on the shared Lua state.
• In JavaScript, the user's code is wrapped in an IIFE at compile time so top-level var declarations are private to that dataset's closure on the shared QJSEngine.

Two datasets that copy the same EMA template will not clobber each other's state in either language.

Stage 5: fan-out

FrameBuilder doesn't publish per frame. stageFrameValues appends each parsed frame's values as one row into the current per-source DataBlock — a pooled, pre-sized slot, no allocation — and publishBlock flushes that block when it reaches its sample cap (kFrameBlockSampleCap, 64; dense/stream sources use the larger kStreamBlockSampleCap, 4096) or when the display tick has moved since the block last flushed, whichever comes first:

• the dashboard drains the pooled block straight from an SPSC ring on Dashboard::onDisplayTick — no copy;
• the asynchronous sinks below share a single detached, trimmed copy of the same block (clone_block_trimmed), made once per flush and only when at least one sink is active (cached in m_anyAsyncSink):
  • the CSV and MDF4 (Pro) export workers,
  • the Historian (Pro),
  • the API server (port 7777, MCP and legacy JSON-RPC),
  • the gRPC server (when built with ENABLE_GRPC),
  • the MQTT bridge (Pro).

The pipeline thread is the single producer for every one of those consumers. The dashboard never copies the block; the trimmed copy exists so a slow export backlog can't pin the pool slot the dashboard is reading. Export workers run on dedicated threads and consume from lock-free queues, so writing to disk or the network never blocks the dashboard.

Timestamp ownership

The driver owns time. Every component downstream of the driver propagates the timestamp attached to the captured chunk; nothing in the pipeline calls steady_clock::now() to stamp a frame after the fact.

• IO::CapturedData carries the chunk timestamp.
• FrameReader::frameTimestamp(endOffsetExclusive) walks pending chunks to assign each extracted logical frame the correct moment, advancing the per-chunk clock by frameStep.
• FrameBuilder interpolates timestamps when one chunk expands into multiple frames, but only to the resolution of the driver's frameStep — see the note in Stage 4 above.
• Export workers derive strictly-increasing offsets via FrameConsumerWorkerBase::monotonicFrameNs(frame->timestamp, baseline). That helper is a safety net against same-nanosecond collisions on coarse clocks (Windows steady_clock has about 15 ms resolution); it is not the source of truth.

If timing looks wrong on a chart or in an export, work from left to right: driver stamp, CapturedData propagation, FrameReader split, FrameBuilder fan-out, and only then the export or report. Patching a downstream stage to "fix" timing usually masks an earlier bug.

Performance characteristics

The acquisition pipeline is designed around three rules:

1. No allocations after init. FrameBuilder reuses one Frame per source as a staging buffer, and draws each DataBlock from a fixed-size slot pool (claimBlockSlot) as an aliasing shared_ptr that shares the slot's control block (no deleter, no per-block control block), with columns pre-sized once at bind; a slot is recycled once its use_count drops back to 1, detected by a probe at the next claimBlockSlot. Parser engines are compiled once, CircularBuffer and lock-free queues are pre-sized, and per-source transform engines are looked up once per source switch (not per dataset).
2. No copy on the dashboard path. The dashboard draws the pooled DataBlock directly. The only block copy is the single trimmed one made for the asynchronous export sinks (clone_block_trimmed), and only when one is active (gated on m_anyAsyncSink), so a slow sink can't pin the slot pool.
3. No queued connections between pipeline-thread objects. Direct connections turn signal emissions into ordinary function calls; the only queued hops on the frame path are the chunk-rate crossing in from the driver and the display-tick-rate crossing out to the GUI.

The cost per frame is dominated by:

• The parser call (parse), which the user controls.
• The transform calls, one per dataset that defines a transform, again user-controlled.
• The dashboard widget update path, which is rate-capped by the UI refresh rate (default 60 Hz, configurable from 1 Hz to 240 Hz). Every frame is still parsed and exported; only the visual refresh is throttled.

To measure this pipeline end-to-end on your own hardware, the Benchmark Dialog (About > Benchmark) drives the real FrameReader -> FrameBuilder -> consumer chain and reports sustained frames/second per stage and per parser language, gated against the targets above. The same engine runs headless for CI; see Command-Line Interface.

Where the parser and transforms fit

The parser and transforms are the user-visible parts of the pipeline. Everything else is fixed; the parser and transforms are where you add custom protocol logic and signal conditioning without rebuilding the application.

flowchart LR
    R["Raw bytes<br/>from FrameReader"] --> D["Decoder"]
    D --> P["parse(frame)<br/>Lua or JavaScript"]
    P --> V["Raw values array"]
    V --> T1["Dataset 1<br/>transform(value)"]
    V --> T2["Dataset 2<br/>transform(value)"]
    V --> T3["Dataset N<br/>transform(value)"]
    T1 --> F["Final frame<br/>(shared with all consumers)"]
    T2 --> F
    T3 --> F

A few things to keep in mind:

• The parser runs once per frame; transforms run once per frame per dataset that defines one. Heavy work in a transform that runs at 10 kHz across 50 datasets adds up quickly.
• Transforms can read raw values from any dataset and final values from datasets that come earlier in group / dataset order; reading a later dataset's final value returns the previous frame's result. They can also publish to computed variables in the project's shared tables, which other transforms can then read, in the same frame or in any later frame, since computed variables persist.
• Computed variables hold their last written value indefinitely. For per-dataset state isolated from other datasets, transform-local upvalues (top-level local in Lua, top-level var/let in JavaScript) are still the lightest option.
• A transform can return a string for label-style datasets. Non-finite numbers (NaN, Infinity) and errors fall back to the raw value silently.

For the full parser and transform API, see Frame Parser Scripting and Dataset Value Transforms.

Debugging the pipeline

Threading invariants

Treat these as load-bearing:

• FrameReader and FrameBuilder run on IO::PipelineHost's dedicated pipeline thread, never the GUI thread. FrameReader's configuration is immutable; recreate the reader to change delimiters, decoder, or checksum.
• CircularBuffer is single-producer / single-consumer. Never make it MPMC.
• Dashboard is GUI-thread only. It drains the same pooled DataBlocks that export workers get a trimmed copy of, from an SPSC ring on Dashboard::onDisplayTick, and reads dataset values from the block directly.
• Export workers (CSV, MDF4 (Pro), Historian (Pro), API, gRPC, MQTT (Pro)) consume from lock-free queues on worker threads. They never block the pipeline thread or the GUI thread.

If you're writing a plugin or a new driver, follow the existing drivers (see BluetoothLE.h and BluetoothLE.cpp as the canonical reference) and keep these invariants intact.

See also

• Data Flow: the user-facing version of this pipeline, with troubleshooting tips.
• Frame Parser Scripting: full Lua and JavaScript parse() reference.
• Dataset Value Transforms: per-dataset calibration, filtering, and unit conversion.
• Variables: shared constants and computed variables used by transforms.
• Operation Modes: how Project File, Quick Plot, and Console-Only modes shape the pipeline.
• Benchmark Dialog: the interactive tool that drives this exact pipeline and reports its sustained throughput against the 256 kHz gates.
• Communication Protocols: protocol comparison and per-driver setup.
• API Reference: how the API server consumes the same shared frame object.