dynalab-core/docs/Processing.md

# DLUnit processing and derivatives

DLUnits can optionally request a `DeriveContext` to calculate time derivatives
of their declared inputs. The same interface works during realtime acquisition,
offline `Core.reprocess()`, and playback with `playback_mode="reprocess"`.
Playback with `playback_mode="read"` reads previously saved outputs instead.

## Example: speed to acceleration

Use this as the DLUnit's `unit.py` source:

```python
from dynalab_core.processing import DeriveContext
from dynalab_core.protocols.packets.data import ValueDescriptor
from dynalab_core.protocols.packets.handshake import SignalDescriptor


def process(
    speed: ValueDescriptor,
    ctx: DeriveContext,
    output: SignalDescriptor,
) -> ValueDescriptor | None:
    acceleration = ctx.derivative(speed, order=1, max_gap_s=0.5)
    if acceleration is None:
        return None
    return ValueDescriptor(output.id, acceleration, speed.timestamp)
```

Bind it using the existing API:

```python
from uuid import uuid4
from dynalab_core.dlunit import DLUnit
from dynalab_core.protocols.packets.handshake import SignalDescriptor

acceleration_signal = SignalDescriptor(
    id=uuid4(), name="Acceleration (m/s²)", type="number", origin="derived"
)
unit = DLUnit()
unit.set_manifest(uuid4(), [speed_signal], acceleration_signal)
unit.set_data(source_text)  # The unit.py source above.
core.bind_derive_unit(dlunit=unit)
# Optionally persist it with unit.write(directory, "acceleration.dlunit").
```

`speed_signal` must describe the existing speed input. If its values are in m/s,
the output values are in m/s². Core does not infer physical units or rename
signals automatically. Derivative outputs are ordinary derived signals: they
can feed another DLUnit, be recorded, and be saved in a DLPak.

## Function contract

A function takes its `ValueDescriptor` inputs in manifest order, optionally one
`DeriveContext`, then one final `SignalDescriptor` describing the output. Use
the actual imported classes in annotations, without postponed/string
annotations. The source must expose exactly one Python function; importing the
context class is supported.

Existing functions such as `process(a, b, output) -> ValueDescriptor` continue
to work. To skip output, explicitly annotate `-> ValueDescriptor | None` and
return `None`. This also works without a context. Returning `None` with the
original `-> ValueDescriptor` annotation remains an invalid result.

No placeholder is emitted when a function returns `None`: live values are not
updated, recording gets no sample, and downstream units receive no event for
that output. A previously published live output remains subject to its normal
timeout; skipping an output does not explicitly clear it.

An offline unit annotated to allow `None` may produce an empty series. Core logs
`derive.offline_no_values` and skips units requiring that empty output, logging
`derive.offline_upstream_empty`. Reprocessing still succeeds, and descriptors
are retained for the empty outputs. Units that do not allow `None` retain the
`DeriveError(code="no_values")` behavior if they execute but emit nothing.

## `ctx.derivative(value, order=1, *, max_gap_s=None) -> float | None`

- `value` is one of the function's supplied input descriptors.
- `order` is an integer: 1, 2, or 3. Other values raise `ValueError`.
- `max_gap_s` is an optional finite positive maximum interval between
  consecutive samples in the derivative window. Invalid limits raise
  `ValueError`. The default places no gap limit.
- The result uses seconds: input units/s, input units/s², or input units/s³.
- Warm-up requires 2, 3, or 4 distinct timestamps, respectively. Until then,
  the result is `None`. A window containing a gap above `max_gap_s` also returns
  `None`; estimates resume after enough samples arrive following that gap.
- Repeated calls are read-only and return the same estimate. You can request
  different orders for the same input in one invocation.

The estimator differentiates the degree-`order` polynomial interpolating the
most recent `order + 1` samples. For order 1 this is the usual backward secant:
`(new_value - previous_value) / elapsed_seconds`. Orders 2 and 3 use the actual
unequal sample intervals rather than repeatedly dividing by a nominal period.
Timestamp differences are taken as integers before converting to seconds.

These are numerical estimates without smoothing. They use only present and
past samples, and higher orders can magnify measurement noise. Very uneven
spacing and floating-point precision also limit accuracy. A non-finite
computed estimate returns `None`.

## History and alignment

Core retains at most four samples per declared input, per context-enabled unit.
The router advances history only when that input itself updates and queues a
snapshot for each invocation. Later arrivals cannot alter an already queued
context, and calls to `derivative()` do not advance history.

Context-enabled units wait until all declared inputs have a sample at or before
the triggering event's timestamp. Each input retains its own original timestamp.
Only the most recent four accepted samples are available, so severely late
events may no longer have a companion sample in history and are skipped.
Equal or older timestamps for a signal are ignored by that unit's history and
do not trigger its processing function. A newer non-finite value clears that
signal's history and skips the invocation; finite samples restart warm-up.
These rules do not remove or alter raw recording samples.

When another input changes, a derivative of an unchanged input reuses that
input's existing estimate. Its history does not gain another sample. If the
output combines multiple inputs, choose its timestamp explicitly, for example
`max(a.timestamp, b.timestamp)`. Companion values are retained in the bounded
history; they do not expire through the realtime live-value timeout. Apply any
additional age policy in the function using the supplied timestamps.

Offline processing uses the same bounded history and causal alignment, with a
fresh history on every pass. Functions without a context retain the original
offline nearest-neighbor alignment, including possible future samples and
timestamps assigned from the triggering event.

Online and offline estimates match for the same chronological accepted input
sequence. Offline processing sorts stored samples; online out-of-order arrivals
can therefore differ. Queue overload can also change the observed sequence:
router drops omit samples from history, while a unit queue drop omits that
invocation but its sample remains in later history snapshots. Asynchronous
multi-input derived chains can observe a different event ordering online.
Offline processing does not drop samples due to queue capacity.

Starting or stopping recording does not reset ongoing online history. If a
recording begins midstream, its first online derivative outputs may depend on
samples preceding the recording. Offline recalculation warms up from the
recorded samples and can omit those early estimates.

## Lifecycle

Online histories are invalidated when Core changes mode, a source connector
registers/disconnects, processing or playback data is replaced, or a playback
control changes its generation (play/resume, pause, seek, or restart). Connector
changes conservatively invalidate all units' histories. Queued requests and
results from an older revision are discarded, including results completing
after the reset. New data warms up the estimator again.

Offline histories are local to the processing pass and never share online
state. DLUnit archives save source and manifest, not history. DLPak and connector
packet formats are unchanged. No numerical-library dependency is required.