Exergie/dynalab-core
1
1 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Updated docs and updated API to fetch signal descriptors in all modes

Browse Source 
API and Protocol docs have been updated (GPT 5.6)

The API methods for fetching a sepcific signal descriptor and fetching
all signal descriptors now work in offline mode, the core_tui.py example
has been updated to fit the new tri mode architecture and loading
recording offload -> processing and then processing -> playback engine
all works
This commit is contained in:
h3cx committed 2026-09-21 13:33:55 +02:00
1 parent 47773df572
commit eab0718a72
6 files changed
+356 -596
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/API.md
+141 -366
View File
@@ -1,438 +1,213 @@
# DynaLab Core API
`Core` is the Python interface for running a DynaLab Core instance, inspecting
signals and their current values, recording incoming data, playing recordings,
and creating derived signals. Connectors provide realtime data to a running
core using the [DynaLab protocol](Protocol.md).
`Core` runs a DynaLab Core instance: it accepts connector data, retains live
values, records data, prepares DLPak archives, plays recordings, and manages
derive units. Connector authors should also read the [protocol guide](Protocol.md).
> [!IMPORTANT]
> **Packet model change.** The default packet and signal classes are lightweight
> `@dataclass(slots=True)` objects. They intentionally do not validate or coerce
> input at construction time. Pydantic models are retained only at a JSON
> boundary and are named with the `Json` prefix, such as `JsonHeartbeat` and
> `JsonConnectorHello`. Use the `Json*` classes with JSON-over-TCP and convert
> with `to_pure()`; use the unprefixed classes inside Core, connectors, and
> derive code. See [Packet representations](Protocol.md#packet-representations).
> [!WARNING]
> DynaLab Core is in alpha. Its public API may change between releases.
> DynaLab Core is in alpha. Its API and protocol can change between releases.
## Quick Start
Create a configuration, construct the core, then start it. Always stop the core
when the application exits so its server and worker threads are shut down.
## Quick start
```python
from dynalab_core import Core
from dynalab_core.config import CoreConfig
core = Core(CoreConfig(host="127.0.0.1", port=58763))
try:
core.start()
core.wait() # Blocks until stop() is called or a timeout expires.
core.wait()
finally:
core.stop()
```
`Core` does not currently implement the context-manager protocol. Use
`try`/`finally` when the core may be started.
Constructing `Core` starts its internal workers. `start()` starts the TCP
server. Always call `stop()` during shutdown; a stopped core cannot be started
again.
## Configuration
`CoreConfig` is defined in `dynalab_core.config`.
`CoreConfig` is a Pydantic configuration model in `dynalab_core.config`.
```python
from dynalab_core.config import CoreConfig
config = CoreConfig(
host="127.0.0.1", # Default
port=58763, # Default; must be in the range 0 through 65535
)
```
| Field | Type | Default | Description |
| Field | Type | Default | Meaning |
| --- | --- | --- | --- |
| `host` | `str` | `"127.0.0.1"` | Address on which the core server listens. |
| `port` | `int` | `58763` | TCP port on which the core server listens. |
| `host` | `str` | `"127.0.0.1"` | TCP bind address. |
| `port` | `int` | `58763` | TCP bind port; validated from 0 through 65535. |
| `thread_debug` | `bool` | `False` | Enables periodic per-thread CPU debug logging. |
| `playback_mode` | `"reprocess" \| "read"` | `"reprocess"` | Whether playback recalculates bound derived signals or reads their recorded values. |
`CoreConfig.bind_str() -> str` returns the configured address in
`"host:port"` form.
`CoreConfig.bind_str() -> str` returns `"host:port"`.
## Lifecycle
### `Core(config: CoreConfig)`
Creates a core and starts its internal input worker. The network server is not
started until `start()` is called.
Creates a core, its input, playback, and derive workers, plus the JSON/TCP
server object. It does not bind the network port until `start()`.
### `start() -> None`
Starts the configured server and begins accepting connector connections. This
method returns after the server start sequence completes.
Call `start()` once per `Core` instance. Calling it in any state other than the
initial state raises `CoreError` with code `"invalid_state"`. Server startup
failures, such as an address already in use, raise `JsonServerError` while
retaining the underlying exception as their cause.
Starts the configured JSON/TCP server. It may be called only once, while Core
is initialised; otherwise it raises `CoreError(code="invalid_state")`. A start
timeout or bind failure raises `JsonServerError` with code `"start_timeout"` or
`"startup_failed"`.
### `wait(timeout: float | None = None) -> None`
Blocks until `stop()` is called or until `timeout` seconds have elapsed. Omit
`timeout` to wait indefinitely. This is useful for keeping a command-line
application alive without a busy loop.
Blocks until `stop()` is called or the optional timeout elapses.
### `stop() -> bool`
Stops the server, disconnects connectors, signals the input worker to stop, and
stops the playback engine and all bound derive units. It also releases any call
currently blocked in `wait()`.
Stops the server, connector registry, derive registry, input worker, and
playback worker. Returns `True` only when all required components stop within
their time limits; it returns `False` if shutdown is incomplete.
Returns `True` when the server, connector registry, derive registry, and input
worker stop within their allowed timeouts, or `False` when shutdown is
incomplete.
## Modes and buffers
Call this during application shutdown, including when startup or runtime work
raises an exception. A stopped core cannot be started again.
`CoreMode` is `"realtime" | "playback" | "offline"`; new instances start in
`"realtime"`. `set_mode()` clears cached live values and cannot run during an
active recording (`CoreError(code="recording_active")`).
## Modes
Core operates in one of two modes:
| Mode | Signal source | Available operation |
| Mode | Source | Purpose |
| --- | --- | --- |
| `"realtime"` | Connected connectors | Recording |
| `"playback"` | A loaded recording | Playback |
| `"realtime"` | Connected connectors | Live values and recording. |
| `"playback"` | Loaded DLPak | Time-based replay. |
| `"offline"` | Recorded or loaded DLPak | Prepare a processing buffer without replaying it. |
The initial mode is `"realtime"`. Connected endpoints remain connected in
playback mode, but their incoming values are not routed into Core.
Endpoints remain connected in non-realtime modes, but only realtime connector
values are routed into Core.
`CoreMode` is defined in `dynalab_core.types` as the literal union of these two
strings.
### `set_mode(mode: CoreMode) -> None` / `get_mode() -> CoreMode`
### `set_mode(mode: CoreMode) -> None`
Changes the active mode and clears all cached live values. `mode` must be
`"realtime"` or `"playback"`; any other value raises `CoreError` with code
`"invalid_mode"`.
The mode cannot be changed while a recording is active. Attempting to do so
raises `CoreError` with code `"recording_active"`.
> [!IMPORTANT]
> Changing mode does not automatically pause active playback. Call `pause()`
> before switching from playback to realtime mode.
### `get_mode() -> CoreMode`
Returns the active mode as either `"realtime"` or `"playback"`.
## Signals
Signals are registered by connectors during their protocol handshake. A signal
is described by `SignalDescriptor` from
`dynalab_core.protocols.packets.handshake`.
```python
from uuid import UUID
from dynalab_core.protocols.packets.handshake import SignalDescriptor
signal = SignalDescriptor(
id=UUID("2d60a378-f37c-4c37-867b-c68f5116598b"),
name="Engine RPM",
type="number",
min_value=0.0,
max_value=6000.0,
unit="rpm",
timeout_ms=2_000,
)
```
| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `id` | `UUID` | Required | Stable, unique signal identifier. |
| `name` | `str` | Required | Human-readable signal name. |
| `type` | `"number"` or `"binary"` | Required | Signal value category. Values are represented as `float` by the current data API. |
| `min_value` | `float \| None` | `None` | Optional lower bound. |
| `max_value` | `float \| None` | `None` | Optional upper bound. |
| `unit` | `str \| None` | `None` | Optional display unit. |
| `timeout_ms` | `int` | `2000` | How long a received value remains live. |
| `origin` | `"source"` or `"derived"` | `"source"` | Whether a connector or a derive unit produces the signal. |
### `get_signal_descriptor(signal_id: UUID) -> SignalDescriptor | None`
Returns the descriptor for `signal_id`, including a derived-signal output, or
`None` when no matching signal is available. In realtime mode, source
descriptors come from connected connectors. In playback mode, they come from
the loaded recording. Use a UUID, not its string representation.
### `get_all_signal_descriptors() -> list[SignalDescriptor]`
Returns the source descriptors available in the current mode followed by the
descriptors from bound derive units. In realtime mode, source descriptors come
from connected connectors. In playback mode, they come from the loaded
recording; previously recorded derived descriptors are omitted because bound
derive units recalculate those values.
Connector membership is dynamic: a disconnected connector's signals are no
longer returned in realtime mode.
## Live Values
After a connector sends a value, Core retains its latest value until the
signal's `timeout_ms` has elapsed. A stale, unknown, or not-yet-received value
is represented by `None`.
### `get_live_value(signal_id: UUID) -> float | None`
Returns the latest non-stale value for one signal.
```python
rpm = core.get_live_value(signal_id)
if rpm is not None:
print(f"Engine speed: {rpm:.0f} rpm")
```
### `get_all_live_values() -> dict[UUID, float]`
Returns a snapshot mapping signal IDs to their latest non-stale values. Signals
without a current value are omitted.
```python
for signal_id, value in core.get_all_live_values().items():
print(signal_id, value)
```
Incoming data timestamps use the monotonic-nanosecond timebase. See
[Protocol timestamps](Protocol.md#timestamps) when implementing a connector.
## Recording
Recording captures incoming connector and derived values in an in-memory
buffer. Recording is only available in realtime mode.
### `start_recording() -> None`
Clears the current recording buffer, records a new UTC start timestamp, and
begins capturing subsequent values. Calling this method in playback mode raises
`CoreError` with code `"incorrect_mode"`; calling it while already recording
raises `CoreError` with code `"recording_active"`.
> [!IMPORTANT]
> Starting a recording discards any values captured by a previous recording.
### `stop_recording() -> None`
Stops capture, normalizes the recorded data, and creates a processing buffer
containing the completed recording and its signal manifest. Timestamps in the
completed recording are nanoseconds relative to its first sample.
Calling this method in playback mode raises `CoreError` with code
`"incorrect_mode"`; calling it when no recording is active raises `CoreError`
with code `"not_recording"`.
> [!IMPORTANT]
> Stopping a recording replaces the previous processing buffer. Persist or
> process it before stopping another recording.
### `get_recording_state() -> bool`
Returns `True` while a recording is active and `False` otherwise.
### `write(dir: str | Path, filename: str) -> None`
Writes the most recently completed recording to a DLPak archive. The `.dlpak`
extension is added to `filename` when it is not already present.
```python
from pathlib import Path
core.write(Path("recordings"), "dyno-run-001")
```
Complete a recording with `stop_recording()` before calling `write()`. If no
completed recording is available, the method raises `CoreError` with code
`"no_recording"`. Archive write failures raise `DLPakError` with code
`"write_failed"` and retain the underlying exception as their cause.
## Playback
Playback feeds a completed recording through the same live-value and derive
pipeline used for realtime data. Switch Core to playback mode and load data
before starting playback:
```python
from pathlib import Path
core.set_mode("playback")
core.load_playback_engine(Path("recordings/dyno-run-001.dlpak"))
core.play()
```
Sets or returns the current mode. `CoreMode` is a type contract; callers must
pass one of the three values above. Changing mode does not automatically pause
playback, so pause it before leaving playback mode.
### `load_playback_engine(input: Path | None = None) -> None`
Loads playback data. Pass a DLPak archive path to load a saved recording, or
omit `input` to load the recording most recently completed by this `Core`
instance.
Available only in `"playback"` mode (`CoreError(code="invalid_mode")`). Pass a
`.dlpak` path, or omit `input` to load the current processing buffer. With no
processing buffer, omitting it raises `CoreError(code="no_playback_data")`.
Loading does not start playback; use `play()`.
This method is only available in playback mode; otherwise it raises `CoreError`
with code `"invalid_mode"`. Omitting `input` before a recording has been
completed raises `CoreError` with code `"no_playback_data"`. A file that cannot
be read or validated raises `DLPakError` with code `"read_failed"`. Replacing
data while playback is active raises `PlaybackError` with code
`"already_playing"`.
### `load_processing_buffer(input: Path | None = None) -> None`
Loading data does not start playback.
Available only in `"offline"` mode (`CoreError(code="invalid_mode")`). Pass a
`.dlpak` path to load it, or omit `input` to turn the completed recording into
a processing buffer and create its manifest. Omitting `input` before a
recording is available raises `CoreError(code="no_playback_data")`.
### `play() -> None`
## Signals and live values
Starts or resumes playback from the current position. Playback stops
automatically at the end of the recording and resets its position to the
beginning.
`SignalDescriptor` is the lightweight internal signal type in
`dynalab_core.protocols.packets.handshake`. Its fields are `id`, `name`,
`type` (`"number"` or `"binary"`), `min_value`, `max_value`, `unit`,
`timeout_ms` (default `2000`), and `origin` (`"source"` or `"derived"`, default
`"source"`). Use `JsonSignalDescriptor` only when serializing a descriptor.
Calling this method outside playback mode raises `CoreError` with code
`"invalid_mode"`. Starting before data is loaded raises `PlaybackError` with
code `"no_data"`.
### `get_signal_descriptor(signal_id: UUID) -> SignalDescriptor | None`
### `pause() -> None`
Returns a copy of the matching descriptor or `None`. Realtime descriptors come
from connectors and bound derive units. Playback descriptors come from the
loaded archive; derived descriptors are recalculated only when
`playback_mode="reprocess"`.
Pauses playback. A subsequent `play()` resumes from the current position.
Calling this method outside playback mode raises `CoreError` with code
`"invalid_mode"`. Calling it when playback is already paused has no effect.
### `get_all_signal_descriptors() -> list[SignalDescriptor]`
Returns descriptors available in the current mode. In `"read"` playback mode,
recorded derived descriptors are included; in `"reprocess"` mode, bound derive
units supply them instead.
### `get_live_value(signal_id: UUID) -> float | None`
Returns the most recent non-stale value for a signal, or `None` when unknown,
not yet received, or past the signal's `timeout_ms`.
### `get_all_live_values() -> dict[UUID, float]`
Returns a snapshot of current non-stale values. Signals without a live value
are omitted. Ingress timestamps use `time.monotonic_ns()`; see the protocol
guide for details.
## Recording and DLPak archives
### `start_recording() -> None` / `stop_recording() -> None`
Recording is available only in `"realtime"` mode. Starting clears the prior
in-memory buffer; stopping normalizes and sorts samples and creates an
offload buffer with a manifest. Errors use `"incorrect_mode"`,
`"recording_active"`, or `"not_recording"` as appropriate.
### `get_recording_state() -> bool`
Returns whether a recording is active.
### `write(dir: str | Path, filename: str) -> None`
Writes the current **processing** buffer as a `.dlpak` archive; the extension
is appended when absent. After recording, select `"offline"` mode and call
`load_processing_buffer()` before writing. If no processing buffer exists,
this raises `CoreError(code="no_recording")`. Archive failures raise
`DLPakError` with a stable code such as `"write_failed"`.
## Playback
### `play() -> None` / `pause() -> None`
Start or resume, and pause, a loaded playback engine. Both require
`"playback"` mode (`CoreError(code="invalid_mode")`). `play()` before loading
data raises `PlaybackError(code="no_data")`. Playback returns to timestamp zero
after it reaches the end.
### `seek(timestamp_ns: int) -> None`
Sets the playback position to `timestamp_ns`, measured in nanoseconds from the
recording's first sample. Samples after that timestamp are emitted when
playback starts or resumes. A negative timestamp raises `PlaybackError` with
code `"invalid_timestamp"`.
Calling this method outside playback mode raises `CoreError` with code
`"invalid_mode"`.
Sets the playback position relative to the recording's first sample. Requires
`"playback"` mode; a negative timestamp raises
`PlaybackError(code="invalid_timestamp")`.
### `get_playback_state() -> bool`
Returns `True` while the playback engine is actively emitting data and `False`
while it is paused, has not started, or has reached the end of the recording.
Returns `True` while the playback engine is emitting data.
## Derived Signals
A derive unit calculates a new value whenever one of its input signals receives
an update and all inputs have non-stale live values. Package the calculation,
its input descriptors, and its output descriptor in a `DLUnit`, then bind it
with `bind_derive_unit()`.
## Derived signals
### `bind_derive_unit(dlunit: DLUnit | None = None, file: Path | None = None) -> UUID`
Provide exactly one of `dlunit` or `file`:
Binds an in-memory `DLUnit` or loads one from a `.dlunit` path, then returns
its ID. Provide exactly one source. Missing or competing sources raise
`CoreError(code="bind_derive_failed")`; invalid source or function shape raises
`DeriveError` (for example `"invalid_source"` or `"invalid_signature"`).
- `dlunit` binds an in-memory `DLUnit`.
- `file` loads and binds a `.dlunit` archive.
To create and bind an in-memory unit:
```python
from uuid import uuid4
from dynalab_core.dlunit import DLUnit
from dynalab_core.protocols.packets.handshake import SignalDescriptor
processing_function = """
from dynalab_core.protocols.packets.data import ValueDescriptor
from dynalab_core.protocols.packets.handshake import SignalDescriptor
def add(left: ValueDescriptor, right: ValueDescriptor, output: SignalDescriptor) -> ValueDescriptor:
return ValueDescriptor(
signal_id=output.id,
value=left.value + right.value,
timestamp=left.timestamp,
)
"""
return_signal = SignalDescriptor(
id=uuid4(),
name="Total",
type="number",
origin="derived",
)
derive_unit = DLUnit()
derive_unit.set_manifest(
uuid4(),
[left_signal, right_signal],
return_signal,
)
derive_unit.set_data(processing_function)
unit_id = core.bind_derive_unit(dlunit=derive_unit, file=None)
```
To bind a previously saved archive instead, pass `None` for `dlunit` and its
path for `file`:
```python
from pathlib import Path
unit_id = core.bind_derive_unit(
dlunit=None,
file=Path("units/total.dlunit"),
)
```
The unit's Python source must define exactly one function. Its parameters must
be:
1. Zero or more `ValueDescriptor` parameters, in the same order as
the manifest's `input_signals`.
2. One final `SignalDescriptor` parameter for the manifest's `return_signal`.
3. A return annotation of `ValueDescriptor`.
The parameter annotations must use the imported `ValueDescriptor` and
`SignalDescriptor` classes shown above. The source string is executed with
`exec()`, so only bind code from trusted sources.
The returned UUID identifies the bound unit. Invalid source or function shape
raises `DeriveError`. Its code is `"invalid_source"` when the source cannot be
loaded or does not define exactly one function, and `"invalid_signature"` when
the function annotations or parameter order are invalid.
Derive source operates on internal `ValueDescriptor` and `SignalDescriptor`
dataclasses, not their `Json*` equivalents. Source is executed with `exec()`;
only load units from trusted sources.
### `unbind_derive_unit(unit_id: UUID) -> None`
Stops and removes the derive unit identified by `unit_id`. Passing an unknown
or already unbound ID has no effect.
The returned ID is no longer valid after unbinding. Create and bind a new unit
to resume derived-value processing.
Stops and removes a bound derive unit. An unknown ID has no effect.
## Exceptions
All expected library exceptions are defined in `dynalab_core.errors` and derive
from `DynaLabError`. Exceptions are grouped by subsystem rather than by every
individual failure reason:
Expected library errors inherit from `DynaLabError` in `dynalab_core.errors`
and expose a stable `code` attribute. Catch a specific subsystem class for
recovery, or `DynaLabError` at an application boundary. Operating-system and
validation causes are retained through `error.__cause__` where applicable.
| Exception | Area | Codes |
| --- | --- | --- |
| `CoreError` | Core lifecycle, mode, recording, and derive-unit binding operations. | `invalid_state`, `invalid_mode`, `incorrect_mode`, `recording_active`, `not_recording`, `no_recording`, `no_playback_data`, `bind_derive_failed` |
| `DLPakError` | Recording archive preparation, reading, and writing. | `no_data`, `no_manifest`, `read_failed`, `write_failed` |
| `DLUnitError` | Derive-unit archive preparation, reading, and writing. | `no_data`, `no_manifest`, `read_failed`, `write_failed` |
| `DeriveError` | Derive source, signatures, arguments, and registration. | `invalid_source`, `invalid_signature`, `input_signal_mismatch`, `argument_mismatch`, `already_registered` |
| `PlaybackError` | Playback state and data. | `no_data`, `incomplete_data`, `already_playing`, `invalid_timestamp`, `stopped` |
| `ConnectorError` | Connector queues and registration. | `queue_full`, `already_registered` |
| `JsonServerError` | JSON server startup and timeouts. | `start_timeout`, `startup_failed` |
Every exception has a human-readable message and a stable, machine-readable
`code`. Catch a subsystem exception when the application can recover from that
area, or catch `DynaLabError` at the application's outer boundary:
```python
from dynalab_core.errors import CoreError, DynaLabError
try:
core.start_recording()
except CoreError as error:
if error.code == "incorrect_mode":
core.set_mode("realtime")
core.start_recording()
try:
run_application()
except DynaLabError as error:
print(f"DynaLab operation failed ({error.code}): {error}")
```
Wrapped operating-system, validation, and archive errors are available through
the standard `error.__cause__` attribute. Internal transport conditions such as
a peer disconnect or an invalid incoming frame are handled by the server and
logged instead of being raised from a background task to application code.
| Exception | Area |
| --- | --- |
| `CoreError` | Lifecycle, modes, recording, buffer preparation, derive binding. |
| `DLPakError` | DLPak archive read/write and completeness. |
| `DLUnitError` | DLUnit archive read/write and completeness. |
| `DeriveError` | Derive source, signature, arguments, registration. |
| `PlaybackError` | Playback data, state, and timestamps. |
| `ConnectorError` | Connector queues and registration. |
| `JsonServerError` | JSON/TCP server startup and timeouts. |
docs/Protocol.md
+113 -185
View File
@@ -1,261 +1,189 @@
# DynaLab Core Protocol
# DynaLab Core connector protocol
> [!IMPORTANT]
> **Packet model change.** Unprefixed packet classes (`Heartbeat`, `ConnectorHello`,
> `ValueDescriptor`, and so on) are lightweight slotted dataclasses for the
> internal data path. They do not perform Pydantic validation. Every packet that
> crosses the JSON-over-TCP boundary uses its Pydantic `Json*` counterpart:
> `JsonHeartbeat`, `JsonConnectorHello`, `JsonValueDescriptor`, etc. This
> separation avoids validating packets repeatedly under load. Convert at the
> boundary with `to_pure()` (received JSON) and `to_json()` (outgoing internal
> packet).
> [!WARNING]
> Protocol documentation is currently being written. It may contain incomplete or incorrect information. While DynaLab Core is in it's alpha stage it is the users responsibility to cross check the information provided in documentation with the codebase.
> The protocol is alpha. Validate a connector against the Core version it will
> run with.
> [!NOTE]
> All protocol documentation uses JSON packets as examples, however it applies to any form of communication protocol used for data ingress
## Transport and framing
> [!NOTE]
> Any transport dependent specificities will be noted using this format, look out for any specificities to the transport you are using when building connectors
The current transport is JSON over TCP. A frame is one UTF-8 JSON object
followed by a newline (`\n`). The maximum incoming frame is 65,536 bytes,
including its newline delimiter. A missing delimiter, oversized frame,
invalid JSON, or a JSON object that does not match a `Json*` packet closes the
connection.
## Transports
DynaLab currently only has one transport option:
- JSON over TCP
More will be added as the project develops
## Layers
Internally DynaLab Core works in three layers, it is important to understand this architecture to understand how the handshake procedure works.
The layers are as follows:
- Server - Layer 1
- Endpoint - Layer 2
- Core - Layer 3
**Layer 1** - The server, this is what any connectors connect to, it is responsible for the initial `dynalab_hello` and `connector_hello` handshake before elevating to the next layer. Once elevated the server sends incoming deserialized packets to Layer 2
**Layer 2** - The endpoint, this is a *connector mapping* of sorts, each connector gets its own endpoint, the endpoint handles packet filtering and the connector heartbeat. Once the handshake is accepted this layer send all data oriented packets to Layer 3
**Layer 3** - The core, this is the heart of DynaLab, it handles live value storage, as well as all of the recording and processing aspects of DynaLab
## Important Details
### Timestamps
DynaLab works off of one common OS timebase, this can be accessed in python via `time.monotonic()` and `time.monotonic_ns()`
DynaLab Core uses two different time units:
- Heartbeat - milliseconds
- Data - nano seconds
**Heartbeat** - Timestamps for heartbeat can be fetched using one of the following methods, both return the same value so it is up to you what you use, for the frequency at which these are called their performance does not really matter:
Use `JsonProtocolMessage` with the JSON wire helpers:
```python
round(time.monotonic() * 1_000.0)
# or
time.monotonic_ns() // 1_000_000
from dynalab_core.protocols.json.wire import read_message, write_message
from dynalab_core.protocols.packets import JsonConnectorHello
# Receive JSON/TCP data (validated once), then enter the internal data path.
json_packet = await read_message(reader)
packet = json_packet.to_pure()
# Send a JSON packet directly, or convert an internal packet first.
await write_message(writer, JsonConnectorHello(...))
# await write_message(writer, packet.to_json())
```
**Data** - Timestamps for data are fetched directly:
## Packet representations
```python
time.monotonic_ns()
```
| Purpose | Internal type | JSON/TCP type |
| --- | --- | --- |
| Signal metadata | `SignalDescriptor` | `JsonSignalDescriptor` |
| Server hello | `DynaLabHello` | `JsonDynaLabHello` |
| Connector hello | `ConnectorHello` | `JsonConnectorHello` |
| Accepted / rejected | `HandshakeAccepted` / `HandshakeRejected` | `JsonHandshakeAccepted` / `JsonHandshakeRejected` |
| Heartbeat | `Heartbeat` | `JsonHeartbeat` |
| One value | `ValueDescriptor` | `JsonValueDescriptor` |
| Value batch | `ValueBatch` | `JsonValueBatch` |
`JsonProtocolMessage` is the discriminated union used to parse JSON frames;
`ProtocolMessage` is the corresponding union for the internal packet types.
The JSON model includes the discriminator field `type`; pure packets do not.
JSON models validate and serialize UUIDs and nested fields. Pure packets accept
the annotated values as supplied, so construct them only from trusted or
already-validated data.
## Timestamps
DynaLab uses the host monotonic clock:
- Heartbeat timestamps are milliseconds: `time.monotonic_ns() // 1_000_000`.
- Data timestamps are nanoseconds: `time.monotonic_ns()`.
Do not use wall-clock (`time.time()`) timestamps. The core compares heartbeat
times and plays data against this monotonic timebase.
## Handshake
> [!TIP]
> All handshake relevant packets are located in the `dynalab_core.protocols.packets.handshake` module
### Step 1
The first step to establishing a communication with the DynaLab Core is proceeding with the initialization handshake.
All examples below are **wire JSON** and therefore correspond to `Json*`
packet classes.
With most transports upon connection DynaLab Core will send the first packet, this is know as the `dynalab_hello` packet.
1. Core sends `JsonDynaLabHello` immediately after the TCP connection opens.
2. The connector replies with `JsonConnectorHello` within five seconds.
3. Core returns `JsonHandshakeAccepted` or `JsonHandshakeRejected`.
4. On acceptance, the connector can send values and must return heartbeats.
### 1. `JsonDynaLabHello`
```json
{
"type": "dynalab_hello",
"instance_id": "2f3c7f62-0a45-4d66-9f32-7fbeb92f9d36",
"core_version": {
"type": "alpha",
"major": 0,
"minor": 1,
"patch": 0
},
"protocol_version": {
"type": "alpha",
"major": 0,
"minor": 0,
"patch": 1
},
"core_version": {"type": "alpha", "major": 0, "minor": 1, "patch": 0},
"protocol_version": {"type": "alpha", "major": 0, "minor": 0, "patch": 1},
"heartbeat_interval_ms": 1000,
"heartbeat_timeout_ms": 5000
}
```
It is recommended to store this packet for potential later use even if it is not critical for a simple connector
Store the `protocol_version`, interval, and timeout. The connector must use
the exact protocol version in its reply.
### Step 2
Once the `dynalab_hello` has been received it is time to return a `connector_hello` packet, this can be found in the `handshake` module as `ConnectorHello`
### 2. `JsonConnectorHello`
```json
{
"type": "connector_hello",
"connector_uuid": "7b3d9a25-2d54-4cd3-b8ef-9c0c44d2dd81",
"protocol_version": {
"type": "alpha",
"major": 0,
"minor": 0,
"patch": 1
},
"protocol_version": {"type": "alpha", "major": 0, "minor": 0, "patch": 1},
"connector_name": "DynaLab Test Connector",
"connector_version": "0.1.0",
"signals": [
{
"id": "2d60a378-f37c-4c37-867b-c68f5116598b",
"name": "Engine RPM",
"type": "number",
"min_value": 0.0,
"max_value": 6000.0,
"unit": "rpm",
"timeout_ms": 2000
},
{
"id": "5097265d-cdf4-4872-a0e2-b9515ae66b19",
"name": "Torque",
"type": "number",
"min_value": 0.0,
"max_value": 10.0,
"unit": "Nm",
"timeout_ms": 2000
},
{
"id": "d6e633cd-7ca3-4974-a164-44482ddb2a8e",
"name": "Emergency Stop",
"type": "binary",
"min_value": null,
"max_value": null,
"unit": null,
"timeout_ms": 500
}
]
"signals": [{
"id": "2d60a378-f37c-4c37-867b-c68f5116598b",
"name": "Engine RPM",
"type": "number",
"min_value": 0.0,
"max_value": 6000.0,
"unit": "rpm",
"timeout_ms": 2000,
"origin": "source"
}]
}
```
This packet will tell DynaLab Core information about itself and the available signals.
Each connection needs a unique connector UUID and at least one signal. Signal
UUIDs should be stable across connector restarts. `origin` defaults to
`"source"`; connectors should not advertise derived signals.
> [!IMPORTANT]
> `protocol_version` must match the one sent in `dynalab_hello` otherwise the connector will be rejected
### 3. Result
> [!IMPORTANT]
> All id's are UUID's and should be generated using the v4 UUID generation algorithm if possible, they can be stored in config files for repeatable UUID's across restarts but should never be set with human created values
Accepted:
### Step 3
Once the DynaLab Core has received this `connector_hello` packet it will elevate the connection internally to layer 2. At this point DynaLab Core will respond with one of two packets:
- `handshake_accepted`
- `handshake_declined`
**`handshake_accepted`** - This means that DynaLab Core has accepted the handshake and that it is ready for data
```json
{
"type": "handshake_accepted",
"accepted_signals": [
"2d60a378-f37c-4c37-867b-c68f5116598b",
"5097265d-cdf4-4872-a0e2-b9515ae66b19",
"d6e633cd-7ca3-4974-a164-44482ddb2a8e"
]
"accepted_signals": ["2d60a378-f37c-4c37-867b-c68f5116598b"]
}
```
> [!NOTE]
> It is recommended to check the accepted signals list as not all signals may be accepted. DynaLab Core will drop any incoming values that it doesn't recognize however this is not recommended as it can hinder performance especially when a signals frequency is very high
**`handshake_declined`** - This means that DynaLab Core has rejected the handshake for some reason, usually this will be a protocol mismatch or an empty list of signals, after this packet is sent, the connection will be closed by DynaLab Core
Rejected:
```json
{
"type": "handshake_rejected",
"reason": "The connector uses an unsupported protocol version."
}
{"type": "handshake_rejected", "reason": "Protocol versions mismatch"}
```
### Step 4
The handshake is now complete, and either accepted or rejected
Core rejects a protocol-version mismatch, a hello without signals, and a
connector UUID already registered on another active endpoint. Do not send data
until `handshake_accepted` arrives. Only send values for accepted signal IDs.
The next order of business is handling the heartbeat which is a key part of the DynaLab Core - Connector protocol
## Heartbeats
## Heartbeat
> [!TIP]
> All heartbeat relevant packets are located in the `dynalab_core.protocols.packets.heartbeat` module
Core periodically sends `JsonHeartbeat`; return the same `sequence` and
`send_timestamp`, adding a local monotonic-millisecond `return_timestamp`.
Heartbeats are sent out from DynaLab Core at the interval specified by the `dynalab_hello` packet during the handshake procedure. These heartbeats need to be returned to DynaLab Core with a return timestamp.
A typical incoming heartbeat looks like so:
```json
{
"type": "heartbeat",
"sequence": 42,
"send_timestamp": 1938475621,
"return_timestamp": null
}
{"type": "heartbeat", "sequence": 42, "send_timestamp": 1938475621, "return_timestamp": null}
```
It must then be returned with a timestamp generated by the connector, in python:
```python
return_timestamp = round(monotonic() * 1000)
```
This gives a millisecond timestamp that is OS wide to avoid any synchronization issues between DynaLab Core and connectors.
A typical returned heartbeat looks like so:
```json
{
"type": "heartbeat",
"sequence": 42,
"send_timestamp": 1938475621,
"return_timestamp": 1938475624
}
{"type": "heartbeat", "sequence": 42, "send_timestamp": 1938475621, "return_timestamp": 1938475624}
```
You may have noticed that in the `dynalab_hello` packet there was a timeout value, if DynaLab Core receives no returned heartbeats from the connector for a longer than this timeout, the connector is deemed dead and the connection is closed
Return heartbeats before `heartbeat_timeout_ms` elapses. A connector that does
not reply is disconnected.
## Data
> [!TIP]
> All heartbeat relevant packets are located in the `dynalab_core.protocols.packets.heartbeat` module
## Data packets
Data can be piped into DynaLab Core via one of two ways, either a values can be sent as individual packets, or they can be sent batched together
All timestamps use the `monotonic_ns()` timebase
### Individual Value Packets
Sending individual packets per value use the `value_descriptor` packet
Send `JsonValueDescriptor` for one sample:
```json
{
"type": "value_descriptor",
"signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b",
"value": 3247.6,
"timestamp": 1938475621
"timestamp": 1938475621000000
}
```
This method of data input is fine for lower frequency data. However in testing this method clearly has its limits. It is recommended to avoid going over 10 kPackets/s. This limitation is caused by Python's `asyncio`. Future transport options should allow for higher data rates, for the time being it is recommended to use `value_batch` packets
### Batch Value Packets
Sending batch value packets uses the `value_batch` packet
For higher rates, send `JsonValueBatch` to amortize framing and scheduling
overhead:
```json
{
"type": "value_batch",
"values": [
{
"type": "value_descriptor",
"signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b",
"value": 3247.6,
"timestamp": 1938475621
},
{
"type": "value_descriptor",
"signal_id": "5097265d-cdf4-4872-a0e2-b9515ae66b19",
"value": 6.42,
"timestamp": 1938475621
},
{
"type": "value_descriptor",
"signal_id": "d6e633cd-7ca3-4974-a164-44482ddb2a8e",
"value": 1.0,
"timestamp": 1938475622
}
{"type": "value_descriptor", "signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b", "value": 3247.6, "timestamp": 1938475621000000},
{"type": "value_descriptor", "signal_id": "5097265d-cdf4-4872-a0e2-b9515ae66b19", "value": 6.42, "timestamp": 1938475621000100}
]
}
```
This allows to achieve much higher data rates than sending individual packets as it dilutes the per packet overhead a lot, especially when batching many values together.
> [!NOTE]
> There is not hard limit on the number of values, the only limit is the packet size limit which is set to 65535 bytes
Every value timestamp is in monotonic nanoseconds. The batch has no separate
configured item limit, but its complete newline-delimited JSON frame must fit
within 65,536 bytes. Core silently ignores values for a signal that was not
declared by that connector, and connector data is routed only while Core is in
`"realtime"` mode.