dynalab-core/docs/API.md

DynaLab Core API

Core runs a DynaLab Core instance: it accepts connector data, retains live values, records data, prepares DLPak archives, plays recordings, and manages derive units. Connector authors should also read the protocol guide.

Important

Packet model change. The default packet and signal classes are lightweight @dataclass(slots=True) objects. They intentionally do not validate or coerce input at construction time. Pydantic models are retained only at a JSON boundary and are named with the Json prefix, such as JsonHeartbeat and JsonConnectorHello. Use the Json* classes with JSON-over-TCP and convert with to_pure(); use the unprefixed classes inside Core, connectors, and derive code. See Packet representations.

Warning

DynaLab Core is in alpha. Its API and protocol can change between releases.

Quick start

from dynalab_core import Core
from dynalab_core.config import CoreConfig

core = Core(CoreConfig(host="127.0.0.1", port=58763))
try:
    core.start()
    core.wait()
finally:
    core.stop()

Constructing Core starts its internal workers. start() starts the TCP server. Always call stop() during shutdown; a stopped core cannot be started again.

Configuration

CoreConfig is a Pydantic configuration model in dynalab_core.config.

Field	Type	Default	Meaning
host	str	"127.0.0.1"	TCP bind address.
port	int	58763	TCP bind port; validated from 0 through 65535.
thread_debug	bool	False	Enables periodic per-thread CPU debug logging.
derive_queue_capacity	int	16384	Maximum queued requests in the derive router and each derive unit; minimum 1.
playback_mode	"reprocess" | "read"	"reprocess"	Whether playback recalculates bound derived signals or reads their recorded values. Offline reprocess() requires "read".

CoreConfig.bind_str() -> str returns "host:port".

Lifecycle

Core(config: CoreConfig)

Creates a core, its input, playback, and derive workers, plus the JSON/TCP server object. It does not bind the network port until start().

start() -> None

Starts the configured JSON/TCP server. It may be called only once, while Core is initialised; otherwise it raises CoreError(code="invalid_state"). A start timeout or bind failure raises JsonServerError with code "start_timeout" or "startup_failed".

wait(timeout: float | None = None) -> None

Blocks until stop() is called or the optional timeout elapses.

stop() -> bool

Stops the server, connector registry, derive registry, input worker, and playback worker. It also waits up to five seconds for an active data operation; an operation still running makes shutdown incomplete. A processing operation that finishes after stopping begins cannot commit a new processing buffer. Returns True only when all required components stop within their time limits; it returns False if shutdown is incomplete. Calling it on a constructed Core before start(), or calling it again after shutdown, is supported.

Modes and buffers

CoreMode is "realtime" | "playback" | "offline"; new instances start in "realtime". set_mode() clears cached live values and cannot run during an active recording (CoreError(code="recording_active")).

Mode	Source	Purpose
"realtime"	Connected connectors	Live values and recording.
"playback"	Loaded DLPak	Time-based replay.
"offline"	Recorded or loaded DLPak	Prepare a processing buffer without replaying it.

Endpoints remain connected in non-realtime modes, but only realtime connector values are routed into Core.

set_mode(mode: CoreMode) -> None / get_mode() -> CoreMode

Sets or returns the current mode. CoreMode is a type contract; callers must pass one of the three values above. Leaving playback mode automatically pauses it at its current position. Returning to playback mode does not automatically resume it; call play() when ready.

load_playback_engine(input: Path | None = None, *, background: bool = False) -> UUID | None

Available only in "playback" mode (CoreError(code="invalid_mode")). Pass a .dlpak path, or omit input to load the current processing buffer. With no processing buffer, omitting it raises CoreError(code="no_playback_data"). Loading takes a deep copy of the archive, resets the playback position to zero, and clears cached live values. It does not start playback; use play().

load_processing_buffer(input: Path | None = None, *, background: bool = False) -> UUID | None

Available only in "offline" mode (CoreError(code="invalid_mode")). Pass a .dlpak path to load it, or omit input to turn the completed recording into a processing buffer and create its manifest. Omitting input before a recording is available raises CoreError(code="no_playback_data").

Completed recordings, processing buffers, and loaded playback archives have independent sample containers, indexes, locks, and manifest metadata. Loading a processing buffer deep-copies the completed recording. Loading playback deep-copies its input archive. Reprocessing, editing metadata, or beginning a new recording therefore does not mutate earlier snapshots. DLPak.set_data() also takes an independent data snapshot; copy() and deepcopy() of a DLPak copy both its data and its nested manifest metadata.

reprocess(*, background: bool = False) -> UUID | None

Recalculates derived signals in the current processing buffer using the bound derive units. Call it in "offline" mode after load_processing_buffer(), with CoreConfig(playback_mode="read"). Units with source inputs run first; units that depend on their derived outputs run in dependency order. Each unit runs in sample order. Legacy alignment uses per-signal binary searches; contexts advance bounded histories. For functions without a processing context, other inputs select the nearest recorded value, preferring the earlier sample when distances tie; this can select future samples. Functions accepting DeriveContext use causal history and preserve each input's timestamp instead. See processing and derivatives. The operation works on a deep copy, removes its previously recorded derived samples and descriptors, and merges the new outputs. It replaces the processing buffer only after all units finish successfully. Calling it again replaces those outputs. It does not write an archive or start playback; use write() to save the result, or switch to "playback" mode, call load_playback_engine(), and then play().

The mode, configuration, and buffer checks raise CoreError with codes "invalid_mode", "invalid_config", and "empty_buffer", respectively. Missing inputs raise DeriveError(code="missing_inputs"), cycles raise DeriveError(code="dependency_cycle"). A unit that produces no values raises DeriveError(code="no_values") unless its return annotation allows None. An optional-output unit may produce an empty series; units requiring that output are skipped with a diagnostic. Errors from a derive function propagate to the synchronous caller or are stored in the background operation status. Failure leaves the original processing buffer, including its prior derived results and metadata, unchanged.

Signals and live values

SignalDescriptor is the lightweight internal signal type in dynalab_core.protocols.packets.handshake. Its fields are id, name, type ("number" or "binary"), min_value, max_value, unit, timeout_ms (default 2000), and origin ("source" or "derived", default "source"). Use JsonSignalDescriptor only when serializing a descriptor.

get_signal_descriptor(signal_id: UUID) -> SignalDescriptor | None

Returns a copy of the matching descriptor or None. Realtime descriptors come from connectors and bound derive units. Playback descriptors come from the loaded archive; derived descriptors are recalculated only when playback_mode="reprocess".

get_all_signal_descriptors() -> list[SignalDescriptor]

Returns descriptors available in the current mode. In "read" playback mode, recorded derived descriptors are included; in "reprocess" mode, bound derive units supply them instead.

get_live_value(signal_id: UUID) -> float | None

Returns the most recent non-stale value for a signal, or None when unknown, not yet received, or past the signal's timeout_ms.

get_all_live_values() -> dict[UUID, float]

Returns a snapshot of current non-stale values. Signals without a live value are omitted. Ingress timestamps use time.monotonic_ns(); see the protocol guide for details.

Recording and DLPak archives

start_recording() -> None / stop_recording() -> None

Recording is available only in "realtime" mode. Start and stop insert markers into Core's input queue and wait for the input worker to acknowledge them. Samples queued before the start marker are excluded; samples between the start and stop markers are included. This is a boundary at Core's input queue: packets still buffered in an endpoint and derive results not yet queued when the stop marker is inserted are outside the recording. There is no fixed sleep or guess about when queued samples have finished.

Starting creates a new in-memory buffer. Stopping normalizes and sorts the completed samples and creates an isolated offload archive. Signal descriptors are retained throughout recording, including descriptors attached to incoming values. A connector can join after recording starts and disconnect before its queued samples are ingested without losing their metadata.

Errors use "incorrect_mode", "recording_active", or "not_recording" as appropriate. A failed input worker raises CoreError(code="worker_failed"); a full queue that cannot accept a control marker within five seconds raises CoreError(code="input_overload").

get_recording_state() -> bool

Returns whether a recording is active.

write(dir: str | Path, filename: str, *, source: BufferSource = "processing", background: bool = False) -> UUID | None

Writes the selected buffer as a .dlpak archive; the extension is appended when absent. BufferSource is "recording" | "processing" | "playback", defined in dynalab_core.types. The default remains "processing" for compatibility. Use source="recording" to save a completed recording directly, without switching modes or loading the processing buffer. Stop recording first; saving an active recording raises CoreError(code="recording_active"). source="playback" saves the loaded archive, not the current playback position or newly recalculated live values. Saving does not change mode or load data.

An unavailable buffer raises CoreError(code="no_recording"). Archive failures raise DLPakError(code="write_failed"). Writes use a temporary file in the destination directory and replace the destination only after the archive has been completed successfully. The destination directory must already exist.

from pathlib import Path

core.start_recording()
# Receive connector data, then finish and save it.
core.stop_recording()
core.write("recordings", "run-001", source="recording")

core.set_mode("offline")
core.load_processing_buffer(Path("recordings/run-001.dlpak"))

DLPak format and standalone access

DLPak in dynalab_core.dlpak can also be used without Core:

from pathlib import Path
from dynalab_core.dlpak import DLPak

archive = DLPak().read(Path("recordings/run-001.dlpak"))
series = archive.get_data_range(start_ns=0, end_ns=1_000_000_000)
archive.write("recordings", "run-001-copy")

Archives are ZIP files containing manifest.json and data.csv. The manifest contains format_version (currently 1), a UTC recording timestamp, signal descriptors, and comments. Older manifests without format_version are read as version 1; unsupported versions are rejected. Signal IDs in the manifest must be unique, and every signal present in the CSV must have a descriptor. Descriptors without samples are allowed. Invalid archives raise DLPakError(code="read_failed"), with the underlying cause retained.

The CSV header is timestamp_ns followed by signal UUID columns. Empty cells represent absent samples. Rows are ordered by timestamp. Repeated timestamps preserve multiple values for the same signal; samples are not collapsed.

Buffer queries for plotting

get_data_range(signal_ids: list[UUID] | None = None, start_ns: int | None = None, end_ns: int | None = None, *, source: BufferSource | None = None) -> dict[UUID, list[tuple[int, float]]]

Returns an independent snapshot grouped by signal ID. Each list contains (timestamp_ns, value) pairs in timestamp order, including duplicate timestamps. Both bounds are inclusive. Omit a bound for an open-ended range, or omit both for all samples. Omit signal_ids to select every signal; an unknown requested ID maps to an empty list. An inverted range raises ValueError. Data is returned at its original resolution without resampling or downsampling.

By default, the selected buffer follows the current mode:

Mode	Queried buffer
"realtime"	Active recording, or the latest completed recording when not recording.
"offline"	Loaded processing buffer.
"playback"	Loaded playback archive, including its recorded derived samples.

Pass source="recording", "processing", or "playback" to query a specific buffer without changing modes. An unavailable buffer raises CoreError(code="no_recording"); an available empty buffer returns empty data. No history is collected outside recordings. Playback queries access the loaded archive, not newly recalculated live derived values.

Completed recordings use nanoseconds relative to their first sample. Active recordings still contain ingress time.monotonic_ns() timestamps. Imported archives retain their stored timestamps. Bounds and returned timestamps always use the selected buffer's stored time base; stopping a recording normalizes it.

bounds = core.get_time_range()
series = core.get_data_range(
    [rpm_signal_id, torque_signal_id],
    start_ns=1_000_000_000,
    end_ns=2_000_000_000,
)
# series[rpm_signal_id] is a list of (timestamp_ns, value) pairs.

get_time_range(*, source: BufferSource | None = None) -> tuple[int, int] | None

Returns the minimum and maximum stored timestamps for the same buffer selection as get_data_range(). An empty buffer returns None; a single sample returns its timestamp for both bounds.

DLPak and ValueBuffer expose the same get_data_range() and get_time_range() methods without the source argument. Buffer mutations maintain per-signal indexes; out-of-order signal samples are sorted before querying. Once sorted, a range query uses binary search plus copying the selected samples, rather than scanning unrelated signals.

Playback

play() -> None / pause() -> None

Start or resume, and pause, a loaded playback engine. Both require "playback" mode (CoreError(code="invalid_mode")). play() before loading data raises PlaybackError(code="no_data"). Pause preserves the current position and resume continues with the next pending sample. Controls interrupt waits across long sample gaps; a full input queue does not block pause or stop. Playback returns to timestamp zero after its last playable sample, and a later play() restarts it. An empty archive completes without emitting samples.

Playback timing starts at the archive's first stored timestamp, even if it is nonzero. The loaded archive itself keeps its original timestamps for range queries and saving. In reprocess mode, recorded derived values are skipped and bound derive units calculate their replacements.

seek(timestamp_ns: int) -> None

Sets the playback position relative to the recording's first sample, in nanoseconds. Works while playing or paused and preserves that state. The next emitted sample is at or after the requested position, including all samples exactly at that timestamp. Positions past the recording's end are clamped to its duration. Seeking clears cached live values; fresh values arrive when playback runs. Queued samples and derive results from before the seek are discarded rather than entering the new timeline.

Requires "playback" mode; a negative timestamp raises PlaybackError(code="invalid_timestamp"), and seeking before loading an archive raises PlaybackError(code="no_data").

get_playback_state() -> bool

Returns True while playback is running, including waits between samples.

get_playback_position() -> int

Returns the current playback position in nanoseconds relative to the archive's first sample. The position advances while playing, remains fixed while paused, and resets to zero when playback finishes or a new archive is loaded.

Derived signals

bind_derive_unit(dlunit: DLUnit | None = None, file: Path | None = None) -> UUID

Binds an in-memory DLUnit or loads one from a .dlunit path, then returns its ID. Provide exactly one source. Missing or competing sources raise CoreError(code="bind_derive_failed"); invalid source or function shape raises DeriveError (for example "invalid_source" or "invalid_signature").

Derive source operates on internal ValueDescriptor and SignalDescriptor dataclasses, not their Json* equivalents. Source is executed with exec(); only load units from trusted sources.

Functions may optionally accept a DeriveContext between their input values and the final output descriptor. ctx.derivative(value, order=1, max_gap_s=None) estimates time derivatives of orders 1–3 using bounded history in realtime, offline processing, and recalculated playback. Annotate the return type as ValueDescriptor | None to skip outputs during warm-up. Existing functions keep their original signatures and behavior. See DLUnit processing and derivatives for examples, numerical behavior, input alignment, and reset rules.

validate_derive_units() -> None

Checks the configured graph against signals available in the current mode. Missing inputs (including archive signals with no samples when reprocessing) raise DeriveError(code="missing_inputs"); dependency cycles raise "dependency_cycle"; duplicate outputs or outputs colliding with sources raise "duplicate_output". Outputs must have origin="derived", otherwise validation raises "invalid_origin". A derive function must return a ValueDescriptor whose signal_id is its configured output, or it raises "invalid_result". Functions annotated -> ValueDescriptor | None may also return None to emit no sample for that invocation.

Structural validation runs before a unit is registered, so cycles and duplicate outputs cannot begin realtime execution. Units can be bound before their sources or upstream units connect; missing inputs are checked before execution. Realtime routing revalidates when connectors, bound units, mode, or loaded data change. An incomplete graph blocks derive routing until its dependencies become available, and reports the reason in the routing worker's status. It does not stop raw acquisition or recording. Offline reprocessing validates the entire graph and sample availability before calling any derive function.

Connector signal IDs must also be unique across connectors and derive outputs; registration rejects collisions with ConnectorError(code="duplicate_signal"). Connector descriptors must have origin="source".

Saving DLUnit archives

DLUnit.set_data(source: str) accepts the Python source text. DLUnit.write(dir, filename) writes manifest.json and the UTF-8 source as unit.py inside a ZIP archive, appending .dlunit when needed. DLUnit().read(path) restores both the manifest and source, ready to pass to bind_derive_unit(). Saving without source or a manifest raises DLUnitError with code "no_data" or "no_manifest", respectively. Writes use a temporary file followed by atomic replacement, so a failed write preserves an existing unit archive.

unbind_derive_unit(unit_id: UUID) -> None

Stops and removes a bound derive unit. An unknown ID has no effect.

Background operations and progress

load_processing_buffer(), load_playback_engine(), write(), and reprocess() remain synchronous by default: they return None on success and raise errors on failure. Pass background=True to return an operation UUID immediately. Only one data operation can run at a time. Data operations are rejected while recording; mode changes, recording controls, derive binding/unbinding, and playback controls are rejected while a data operation is active, with CoreError(code="operation_active"). Queries and status polling remain available.

get_operation(operation_id: UUID | None = None) -> OperationStatus | None

Returns an immutable snapshot from dynalab_core.operations. With no ID, it returns the most recently started operation, or None before any operation. Fields are id, name, state ("running", "completed", or "failed"), completed, total, message, error_code, and error_message. The progress property is the fraction completed / total. Load/save progress reports coarse archive stages; reprocessing reports completed derive units, rather than a predicted remaining time. Completion is determined by state, not the progress fraction alone. The latest 100 operations are retained.

wait_operation(operation_id: UUID, timeout: float | None = None) -> OperationStatus

Waits for completion or the timeout. On timeout, returns the current running status. On failure, raises the original exception; the failed status remains available through get_operation(). An unknown or expired ID raises CoreError(code="unknown_operation").

operation_id = core.reprocess(background=True)
status = core.get_operation(operation_id)  # Poll from the GUI timer.
if status.state == "failed":
    show_error(status.error_message)
# Outside the GUI event loop, wait and propagate any error:
core.wait_operation(operation_id)

Connector, derive, and worker status

get_state() -> str

Returns Core's lifecycle state: "initd", "started", "stopping", "stopped", or "stop_incomplete" after construction.

get_connectors() -> list[ConnectorStatus]

Returns immutable snapshots of registered endpoints: id, name, version, signal_ids, and state ("connected", "stopping", or "failed"). Endpoints are removed after disconnect cleanup; a GUI can compare snapshots to detect connection and disconnection.

get_derive_units() -> list[DeriveUnitStatus]

Returns each bound unit's id, input_signal_ids, output_signal_id, and worker status. Failed units retain their error until unbound; unbind and rebind a repaired unit to start a new worker.

get_worker_statuses() -> list[WorkerStatus]

Returns the input, JSON server, playback, derive router, and individual derive workers. Each status contains name, state ("running", "stopped", "failed", or "blocked"), error, queue_size, queue_capacity, and dropped_samples. A blocked routing worker reports missing dependencies. A server not started yet is stopped. Worker status describes worker health; use get_playback_state() to distinguish playing from paused. The status dataclasses are defined in dynalab_core.types.

Derive overload policy

CoreConfig.derive_queue_capacity bounds both the routing queue and every unit's input queue. Full derive queues drop the newest request, increment their dropped_samples counter, and log a warning on the first drop. They do not block raw ingestion or discard raw samples already accepted into Core's recording buffer. Unit output waits for space in Core's bounded input queue and remains interruptible during shutdown. Core's input queue has its own existing limit; connector input overload disconnects the affected endpoint and is logged. Offline processing does not use these derive queues and does not drop samples.

Exceptions

Expected library errors inherit from DynaLabError in dynalab_core.errors and expose a stable code attribute. Catch a specific subsystem class for recovery, or DynaLabError at an application boundary. Operating-system and validation causes are retained through error.__cause__ where applicable.

Exception	Area
CoreError	Lifecycle, modes, recording, buffer preparation, derive binding.
DLPakError	DLPak archive read/write and completeness.
DLUnitError	DLUnit archive read/write and completeness.
DeriveError	Derive source, signature, arguments, registration.
PlaybackError	Playback data, state, and timestamps.
ConnectorError	Connector queues and registration.
JsonServerError	JSON/TCP server startup and timeouts.