dynalab-core/docs/Processing.md

DLUnit processing and derivatives

DLUnits can optionally request a DeriveContext to calculate time derivatives of their declared inputs. The same interface works during realtime acquisition, offline Core.reprocess(), and playback with playback_mode="reprocess". Playback with playback_mode="read" reads previously saved outputs instead.

Example: speed to acceleration

Use this as the DLUnit's unit.py source:

from dynalab_core.processing import DeriveContext
from dynalab_core.protocols.packets.data import ValueDescriptor
from dynalab_core.protocols.packets.handshake import SignalDescriptor


def process(
    speed: ValueDescriptor,
    ctx: DeriveContext,
    output: SignalDescriptor,
) -> ValueDescriptor | None:
    acceleration = ctx.derivative(speed, order=1, max_gap_s=0.5)
    if acceleration is None:
        return None
    return ValueDescriptor(output.id, acceleration, speed.timestamp)

Bind it using the existing API:

from uuid import uuid4
from dynalab_core.dlunit import DLUnit
from dynalab_core.protocols.packets.handshake import SignalDescriptor

acceleration_signal = SignalDescriptor(
    id=uuid4(), name="Acceleration (m/s²)", type="number", origin="derived"
)
unit = DLUnit()
unit.set_manifest(uuid4(), [speed_signal], acceleration_signal)
unit.set_data(source_text)  # The unit.py source above.
core.bind_derive_unit(dlunit=unit)
# Optionally persist it with unit.write(directory, "acceleration.dlunit").

speed_signal must describe the existing speed input. If its values are in m/s, the output values are in m/s². Core does not infer physical units or rename signals automatically. Derivative outputs are ordinary derived signals: they can feed another DLUnit, be recorded, and be saved in a DLPak.

Function contract

A function takes its ValueDescriptor inputs in manifest order, optionally one DeriveContext, then one final SignalDescriptor describing the output. Use the actual imported classes in annotations, without postponed/string annotations. The source must expose exactly one Python function; importing the context class is supported.

Existing functions such as process(a, b, output) -> ValueDescriptor continue to work. To skip output, explicitly annotate -> ValueDescriptor | None and return None. This also works without a context. Returning None with the original -> ValueDescriptor annotation remains an invalid result.

No placeholder is emitted when a function returns None: live values are not updated, recording gets no sample, and downstream units receive no event for that output. A previously published live output remains subject to its normal timeout; skipping an output does not explicitly clear it.

An offline unit annotated to allow None may produce an empty series. Core logs derive.offline_no_values and skips units requiring that empty output, logging derive.offline_upstream_empty. Reprocessing still succeeds, and descriptors are retained for the empty outputs. Units that do not allow None retain the DeriveError(code="no_values") behavior if they execute but emit nothing.

ctx.derivative(value, order=1, *, max_gap_s=None) -> float | None

• value is one of the function's supplied input descriptors.
• order is an integer: 1, 2, or 3. Other values raise ValueError.
• max_gap_s is an optional finite positive maximum interval between consecutive samples in the derivative window. Invalid limits raise ValueError. The default places no gap limit.
• The result uses seconds: input units/s, input units/s², or input units/s³.
• Warm-up requires 2, 3, or 4 distinct timestamps, respectively. Until then, the result is None. A window containing a gap above max_gap_s also returns None; estimates resume after enough samples arrive following that gap.
• Repeated calls are read-only and return the same estimate. You can request different orders for the same input in one invocation.

The estimator differentiates the degree-order polynomial interpolating the most recent order + 1 samples. For order 1 this is the usual backward secant: (new_value - previous_value) / elapsed_seconds. Orders 2 and 3 use the actual unequal sample intervals rather than repeatedly dividing by a nominal period. Timestamp differences are taken as integers before converting to seconds.

These are numerical estimates without smoothing. They use only present and past samples, and higher orders can magnify measurement noise. Very uneven spacing and floating-point precision also limit accuracy. A non-finite computed estimate returns None.

History and alignment

Core retains at most four samples per declared input, per context-enabled unit. The router advances history only when that input itself updates and queues a snapshot for each invocation. Later arrivals cannot alter an already queued context, and calls to derivative() do not advance history.

Context-enabled units wait until all declared inputs have a sample at or before the triggering event's timestamp. Each input retains its own original timestamp. Only the most recent four accepted samples are available, so severely late events may no longer have a companion sample in history and are skipped. Equal or older timestamps for a signal are ignored by that unit's history and do not trigger its processing function. A newer non-finite value clears that signal's history and skips the invocation; finite samples restart warm-up. These rules do not remove or alter raw recording samples.

When another input changes, a derivative of an unchanged input reuses that input's existing estimate. Its history does not gain another sample. If the output combines multiple inputs, choose its timestamp explicitly, for example max(a.timestamp, b.timestamp). Companion values are retained in the bounded history; they do not expire through the realtime live-value timeout. Apply any additional age policy in the function using the supplied timestamps.

Offline processing uses the same bounded history and causal alignment, with a fresh history on every pass. Functions without a context retain the original offline nearest-neighbor alignment, including possible future samples and timestamps assigned from the triggering event.

Online and offline estimates match for the same chronological accepted input sequence. Offline processing sorts stored samples; online out-of-order arrivals can therefore differ. Queue overload can also change the observed sequence: router drops omit samples from history, while a unit queue drop omits that invocation but its sample remains in later history snapshots. Asynchronous multi-input derived chains can observe a different event ordering online. Offline processing does not drop samples due to queue capacity.

Starting or stopping recording does not reset ongoing online history. If a recording begins midstream, its first online derivative outputs may depend on samples preceding the recording. Offline recalculation warms up from the recorded samples and can omit those early estimates.

Lifecycle

Online histories are invalidated when Core changes mode, a source connector registers/disconnects, processing or playback data is replaced, or a playback control changes its generation (play/resume, pause, seek, or restart). Connector changes conservatively invalidate all units' histories. Queued requests and results from an older revision are discarded, including results completing after the reset. New data warms up the estimator again.

Offline histories are local to the processing pass and never share online state. DLUnit archives save source and manifest, not history. DLPak and connector packet formats are unchanged. No numerical-library dependency is required.