dynalab-core/docs/API.md

# DynaLab Core API

`Core` runs a DynaLab Core instance: it accepts connector data, retains live
values, records data, prepares DLPak archives, plays recordings, and manages
derive units. Connector authors should also read the [protocol guide](Protocol.md).

> [!IMPORTANT]
> **Packet model change.** The default packet and signal classes are lightweight
> `@dataclass(slots=True)` objects. They intentionally do not validate or coerce
> input at construction time. Pydantic models are retained only at a JSON
> boundary and are named with the `Json` prefix, such as `JsonHeartbeat` and
> `JsonConnectorHello`. Use the `Json*` classes with JSON-over-TCP and convert
> with `to_pure()`; use the unprefixed classes inside Core, connectors, and
> derive code. See [Packet representations](Protocol.md#packet-representations).

> [!WARNING]
> DynaLab Core is in alpha. Its API and protocol can change between releases.

## Quick start

```python
from dynalab_core import Core
from dynalab_core.config import CoreConfig

core = Core(CoreConfig(host="127.0.0.1", port=58763))
try:
    core.start()
    core.wait()
finally:
    core.stop()
```

Constructing `Core` starts its internal workers. `start()` starts the TCP
server. Always call `stop()` during shutdown; a stopped core cannot be started
again.

## Configuration

`CoreConfig` is a Pydantic configuration model in `dynalab_core.config`.

| Field | Type | Default | Meaning |
| --- | --- | --- | --- |
| `host` | `str` | `"127.0.0.1"` | TCP bind address. |
| `port` | `int` | `58763` | TCP bind port; validated from 0 through 65535. |
| `thread_debug` | `bool` | `False` | Enables periodic per-thread CPU debug logging. |
| `playback_mode` | `"reprocess" \| "read"` | `"reprocess"` | Whether playback recalculates bound derived signals or reads their recorded values. Offline `reprocess()` requires `"read"`. |

`CoreConfig.bind_str() -> str` returns `"host:port"`.

## Lifecycle

### `Core(config: CoreConfig)`

Creates a core, its input, playback, and derive workers, plus the JSON/TCP
server object. It does not bind the network port until `start()`.

### `start() -> None`

Starts the configured JSON/TCP server. It may be called only once, while Core
is initialised; otherwise it raises `CoreError(code="invalid_state")`. A start
timeout or bind failure raises `JsonServerError` with code `"start_timeout"` or
`"startup_failed"`.

### `wait(timeout: float | None = None) -> None`

Blocks until `stop()` is called or the optional timeout elapses.

### `stop() -> bool`

Stops the server, connector registry, derive registry, input worker, and
playback worker. Returns `True` only when all required components stop within
their time limits; it returns `False` if shutdown is incomplete.

## Modes and buffers

`CoreMode` is `"realtime" | "playback" | "offline"`; new instances start in
`"realtime"`. `set_mode()` clears cached live values and cannot run during an
active recording (`CoreError(code="recording_active")`).

| Mode | Source | Purpose |
| --- | --- | --- |
| `"realtime"` | Connected connectors | Live values and recording. |
| `"playback"` | Loaded DLPak | Time-based replay. |
| `"offline"` | Recorded or loaded DLPak | Prepare a processing buffer without replaying it. |

Endpoints remain connected in non-realtime modes, but only realtime connector
values are routed into Core.

### `set_mode(mode: CoreMode) -> None` / `get_mode() -> CoreMode`

Sets or returns the current mode. `CoreMode` is a type contract; callers must
pass one of the three values above. Changing mode does not automatically pause
playback, so pause it before leaving playback mode.

### `load_playback_engine(input: Path | None = None) -> None`

Available only in `"playback"` mode (`CoreError(code="invalid_mode")`). Pass a
`.dlpak` path, or omit `input` to load the current processing buffer. With no
processing buffer, omitting it raises `CoreError(code="no_playback_data")`.
Loading does not start playback; use `play()`.

### `load_processing_buffer(input: Path | None = None) -> None`

Available only in `"offline"` mode (`CoreError(code="invalid_mode")`). Pass a
`.dlpak` path to load it, or omit `input` to turn the completed recording into
a processing buffer and create its manifest. Omitting `input` before a
recording is available raises `CoreError(code="no_playback_data")`.

### `reprocess() -> None`

Recalculates derived signals in the current processing buffer using the bound
derive units. Call it in `"offline"` mode after `load_processing_buffer()`, with
`CoreConfig(playback_mode="read")`. Units with source inputs run first; units
that depend on their derived outputs run in dependency order. Each unit runs
in sample order using per-signal binary searches instead of full-buffer scans.
For other inputs it selects the nearest recorded value, preferring the earlier
sample when distances tie. This retains the previous offline alignment policy,
including selection of future samples. The operation
removes previously recorded derived samples and descriptors, then merges the
new outputs into the processing buffer. Calling it again replaces those
outputs. It does not write an archive or start playback; use `write()` to save
the result, or switch to `"playback"` mode, call `load_playback_engine()`, and
then `play()`.

The mode, configuration, and buffer checks raise `CoreError` with codes
`"invalid_mode"`, `"invalid_config"`, and `"empty_buffer"`, respectively.
Unresolved derive dependencies raise `DeriveError(code="unresolved_dependencies")`;
a unit that produces no values raises `DeriveError(code="no_values")`. Errors
from a derive function propagate to the caller. Reprocessing changes the
in-memory processing buffer before all units finish, so a failed call may
leave its old derived data removed. Reload an unchanged archive before retrying
if that data is needed.

## Signals and live values

`SignalDescriptor` is the lightweight internal signal type in
`dynalab_core.protocols.packets.handshake`. Its fields are `id`, `name`,
`type` (`"number"` or `"binary"`), `min_value`, `max_value`, `unit`,
`timeout_ms` (default `2000`), and `origin` (`"source"` or `"derived"`, default
`"source"`). Use `JsonSignalDescriptor` only when serializing a descriptor.

### `get_signal_descriptor(signal_id: UUID) -> SignalDescriptor | None`

Returns a copy of the matching descriptor or `None`. Realtime descriptors come
from connectors and bound derive units. Playback descriptors come from the
loaded archive; derived descriptors are recalculated only when
`playback_mode="reprocess"`.

### `get_all_signal_descriptors() -> list[SignalDescriptor]`

Returns descriptors available in the current mode. In `"read"` playback mode,
recorded derived descriptors are included; in `"reprocess"` mode, bound derive
units supply them instead.

### `get_live_value(signal_id: UUID) -> float | None`

Returns the most recent non-stale value for a signal, or `None` when unknown,
not yet received, or past the signal's `timeout_ms`.

### `get_all_live_values() -> dict[UUID, float]`

Returns a snapshot of current non-stale values. Signals without a live value
are omitted. Ingress timestamps use `time.monotonic_ns()`; see the protocol
guide for details.

## Recording and DLPak archives

### `start_recording() -> None` / `stop_recording() -> None`

Recording is available only in `"realtime"` mode. Starting clears the prior
in-memory buffer; stopping normalizes and sorts samples and creates an
offload buffer with a manifest. Errors use `"incorrect_mode"`,
`"recording_active"`, or `"not_recording"` as appropriate.

### `get_recording_state() -> bool`

Returns whether a recording is active.

### `write(dir: str | Path, filename: str, *, source: BufferSource = "processing") -> None`

Writes the selected buffer as a `.dlpak` archive; the extension is appended
when absent. `BufferSource` is `"recording" | "processing" | "playback"`, defined
in `dynalab_core.types`. The default remains `"processing"` for compatibility.
Use `source="recording"` to save a completed recording directly, without
switching modes or loading the processing buffer. Stop recording first;
saving an active recording raises `CoreError(code="recording_active")`.
`source="playback"` saves the loaded archive, not the current playback position
or newly recalculated live values. Saving does not change mode or load data.

An unavailable buffer raises `CoreError(code="no_recording")`. Archive failures
raise `DLPakError(code="write_failed")`. Writes use a temporary file in the
destination directory and replace the destination only after the archive has
been completed successfully. The destination directory must already exist.

```python
from pathlib import Path

core.start_recording()
# Receive connector data, then finish and save it.
core.stop_recording()
core.write("recordings", "run-001", source="recording")

core.set_mode("offline")
core.load_processing_buffer(Path("recordings/run-001.dlpak"))
```

### DLPak format and standalone access

`DLPak` in `dynalab_core.dlpak` can also be used without Core:

```python
from pathlib import Path
from dynalab_core.dlpak import DLPak

archive = DLPak().read(Path("recordings/run-001.dlpak"))
series = archive.get_data_range(start_ns=0, end_ns=1_000_000_000)
archive.write("recordings", "run-001-copy")
```

Archives are ZIP files containing `manifest.json` and `data.csv`. The manifest
contains `format_version` (currently `1`), a UTC recording timestamp, signal
descriptors, and comments. Older manifests without `format_version` are read
as version 1; unsupported versions are rejected. Signal IDs in the manifest
must be unique, and every signal present in the CSV must have a descriptor.
Descriptors without samples are allowed. Invalid archives raise
`DLPakError(code="read_failed")`, with the underlying cause retained.

The CSV header is `timestamp_ns` followed by signal UUID columns. Empty cells
represent absent samples. Rows are ordered by timestamp. Repeated timestamps
preserve multiple values for the same signal; samples are not collapsed.

## Buffer queries for plotting

### `get_data_range(signal_ids: list[UUID] | None = None, start_ns: int | None = None, end_ns: int | None = None, *, source: BufferSource | None = None) -> dict[UUID, list[tuple[int, float]]]`

Returns an independent snapshot grouped by signal ID. Each list contains
`(timestamp_ns, value)` pairs in timestamp order, including duplicate
timestamps. Both bounds are inclusive. Omit a bound for an open-ended range,
or omit both for all samples. Omit `signal_ids` to select every signal; an
unknown requested ID maps to an empty list. An inverted range raises
`ValueError`. Data is returned at its original resolution without resampling
or downsampling.

By default, the selected buffer follows the current mode:

| Mode | Queried buffer |
| --- | --- |
| `"realtime"` | Active recording, or the latest completed recording when not recording. |
| `"offline"` | Loaded processing buffer. |
| `"playback"` | Loaded playback archive, including its recorded derived samples. |

Pass `source="recording"`, `"processing"`, or `"playback"` to query a specific
buffer without changing modes. An unavailable buffer raises
`CoreError(code="no_recording")`; an available empty buffer returns empty data.
No history is collected outside recordings. Playback queries access the
loaded archive, not newly recalculated live derived values.

Completed recordings use nanoseconds relative to their first sample. Active
recordings still contain ingress `time.monotonic_ns()` timestamps. Imported
archives retain their stored timestamps. Bounds and returned timestamps always
use the selected buffer's stored time base; stopping a recording normalizes it.

```python
bounds = core.get_time_range()
series = core.get_data_range(
    [rpm_signal_id, torque_signal_id],
    start_ns=1_000_000_000,
    end_ns=2_000_000_000,
)
# series[rpm_signal_id] is a list of (timestamp_ns, value) pairs.
```

### `get_time_range(*, source: BufferSource | None = None) -> tuple[int, int] | None`

Returns the minimum and maximum stored timestamps for the same buffer selection
as `get_data_range()`. An empty buffer returns `None`; a single sample returns
its timestamp for both bounds.

`DLPak` and `ValueBuffer` expose the same `get_data_range()` and
`get_time_range()` methods without the `source` argument. Buffer mutations
maintain per-signal indexes; out-of-order signal samples are sorted before
querying. Once sorted, a range query uses binary search plus copying the
selected samples, rather than scanning unrelated signals.

## Playback

### `play() -> None` / `pause() -> None`

Start or resume, and pause, a loaded playback engine. Both require
`"playback"` mode (`CoreError(code="invalid_mode")`). `play()` before loading
data raises `PlaybackError(code="no_data")`. Playback returns to timestamp zero
after it reaches the end.

### `seek(timestamp_ns: int) -> None`

Sets the playback position relative to the recording's first sample. Requires
`"playback"` mode; a negative timestamp raises
`PlaybackError(code="invalid_timestamp")`.

### `get_playback_state() -> bool`

Returns `True` while the playback engine is emitting data.

## Derived signals

### `bind_derive_unit(dlunit: DLUnit | None = None, file: Path | None = None) -> UUID`

Binds an in-memory `DLUnit` or loads one from a `.dlunit` path, then returns
its ID. Provide exactly one source. Missing or competing sources raise
`CoreError(code="bind_derive_failed")`; invalid source or function shape raises
`DeriveError` (for example `"invalid_source"` or `"invalid_signature"`).

Derive source operates on internal `ValueDescriptor` and `SignalDescriptor`
dataclasses, not their `Json*` equivalents. Source is executed with `exec()`;
only load units from trusted sources.

### `unbind_derive_unit(unit_id: UUID) -> None`

Stops and removes a bound derive unit. An unknown ID has no effect.

## Exceptions

Expected library errors inherit from `DynaLabError` in `dynalab_core.errors`
and expose a stable `code` attribute. Catch a specific subsystem class for
recovery, or `DynaLabError` at an application boundary. Operating-system and
validation causes are retained through `error.__cause__` where applicable.

| Exception | Area |
| --- | --- |
| `CoreError` | Lifecycle, modes, recording, buffer preparation, derive binding. |
| `DLPakError` | DLPak archive read/write and completeness. |
| `DLUnitError` | DLUnit archive read/write and completeness. |
| `DeriveError` | Derive source, signature, arguments, registration. |
| `PlaybackError` | Playback data, state, and timestamps. |
| `ConnectorError` | Connector queues and registration. |
| `JsonServerError` | JSON/TCP server startup and timeouts. |