Exergie/dynalab-core
1
1 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Updated docs and updated API to fetch signal descriptors in all modes

Browse Source 
API and Protocol docs have been updated (GPT 5.6)

The API methods for fetching a sepcific signal descriptor and fetching
all signal descriptors now work in offline mode, the core_tui.py example
has been updated to fit the new tri mode architecture and loading
recording offload -> processing and then processing -> playback engine
all works
This commit is contained in:
h3cx committed 2026-09-21 13:33:55 +02:00
1 parent 47773df572
commit eab0718a72
6 files changed
+356 -596
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/Protocol.md
+113 -185
View File
@@ -1,261 +1,189 @@
# DynaLab Core Protocol
# DynaLab Core connector protocol
> [!IMPORTANT]
> **Packet model change.** Unprefixed packet classes (`Heartbeat`, `ConnectorHello`,
> `ValueDescriptor`, and so on) are lightweight slotted dataclasses for the
> internal data path. They do not perform Pydantic validation. Every packet that
> crosses the JSON-over-TCP boundary uses its Pydantic `Json*` counterpart:
> `JsonHeartbeat`, `JsonConnectorHello`, `JsonValueDescriptor`, etc. This
> separation avoids validating packets repeatedly under load. Convert at the
> boundary with `to_pure()` (received JSON) and `to_json()` (outgoing internal
> packet).
> [!WARNING]
> Protocol documentation is currently being written. It may contain incomplete or incorrect information. While DynaLab Core is in it's alpha stage it is the users responsibility to cross check the information provided in documentation with the codebase.
> The protocol is alpha. Validate a connector against the Core version it will
> run with.
> [!NOTE]
> All protocol documentation uses JSON packets as examples, however it applies to any form of communication protocol used for data ingress
## Transport and framing
> [!NOTE]
> Any transport dependent specificities will be noted using this format, look out for any specificities to the transport you are using when building connectors
The current transport is JSON over TCP. A frame is one UTF-8 JSON object
followed by a newline (`\n`). The maximum incoming frame is 65,536 bytes,
including its newline delimiter. A missing delimiter, oversized frame,
invalid JSON, or a JSON object that does not match a `Json*` packet closes the
connection.
## Transports
DynaLab currently only has one transport option:
- JSON over TCP
More will be added as the project develops
## Layers
Internally DynaLab Core works in three layers, it is important to understand this architecture to understand how the handshake procedure works.
The layers are as follows:
- Server - Layer 1
- Endpoint - Layer 2
- Core - Layer 3
**Layer 1** - The server, this is what any connectors connect to, it is responsible for the initial `dynalab_hello` and `connector_hello` handshake before elevating to the next layer. Once elevated the server sends incoming deserialized packets to Layer 2
**Layer 2** - The endpoint, this is a *connector mapping* of sorts, each connector gets its own endpoint, the endpoint handles packet filtering and the connector heartbeat. Once the handshake is accepted this layer send all data oriented packets to Layer 3
**Layer 3** - The core, this is the heart of DynaLab, it handles live value storage, as well as all of the recording and processing aspects of DynaLab
## Important Details
### Timestamps
DynaLab works off of one common OS timebase, this can be accessed in python via `time.monotonic()` and `time.monotonic_ns()`
DynaLab Core uses two different time units:
- Heartbeat - milliseconds
- Data - nano seconds
**Heartbeat** - Timestamps for heartbeat can be fetched using one of the following methods, both return the same value so it is up to you what you use, for the frequency at which these are called their performance does not really matter:
Use `JsonProtocolMessage` with the JSON wire helpers:
```python
round(time.monotonic() * 1_000.0)
# or
time.monotonic_ns() // 1_000_000
from dynalab_core.protocols.json.wire import read_message, write_message
from dynalab_core.protocols.packets import JsonConnectorHello
# Receive JSON/TCP data (validated once), then enter the internal data path.
json_packet = await read_message(reader)
packet = json_packet.to_pure()
# Send a JSON packet directly, or convert an internal packet first.
await write_message(writer, JsonConnectorHello(...))
# await write_message(writer, packet.to_json())
```
**Data** - Timestamps for data are fetched directly:
## Packet representations
```python
time.monotonic_ns()
```
| Purpose | Internal type | JSON/TCP type |
| --- | --- | --- |
| Signal metadata | `SignalDescriptor` | `JsonSignalDescriptor` |
| Server hello | `DynaLabHello` | `JsonDynaLabHello` |
| Connector hello | `ConnectorHello` | `JsonConnectorHello` |
| Accepted / rejected | `HandshakeAccepted` / `HandshakeRejected` | `JsonHandshakeAccepted` / `JsonHandshakeRejected` |
| Heartbeat | `Heartbeat` | `JsonHeartbeat` |
| One value | `ValueDescriptor` | `JsonValueDescriptor` |
| Value batch | `ValueBatch` | `JsonValueBatch` |
`JsonProtocolMessage` is the discriminated union used to parse JSON frames;
`ProtocolMessage` is the corresponding union for the internal packet types.
The JSON model includes the discriminator field `type`; pure packets do not.
JSON models validate and serialize UUIDs and nested fields. Pure packets accept
the annotated values as supplied, so construct them only from trusted or
already-validated data.
## Timestamps
DynaLab uses the host monotonic clock:
- Heartbeat timestamps are milliseconds: `time.monotonic_ns() // 1_000_000`.
- Data timestamps are nanoseconds: `time.monotonic_ns()`.
Do not use wall-clock (`time.time()`) timestamps. The core compares heartbeat
times and plays data against this monotonic timebase.
## Handshake
> [!TIP]
> All handshake relevant packets are located in the `dynalab_core.protocols.packets.handshake` module
### Step 1
The first step to establishing a communication with the DynaLab Core is proceeding with the initialization handshake.
All examples below are **wire JSON** and therefore correspond to `Json*`
packet classes.
With most transports upon connection DynaLab Core will send the first packet, this is know as the `dynalab_hello` packet.
1. Core sends `JsonDynaLabHello` immediately after the TCP connection opens.
2. The connector replies with `JsonConnectorHello` within five seconds.
3. Core returns `JsonHandshakeAccepted` or `JsonHandshakeRejected`.
4. On acceptance, the connector can send values and must return heartbeats.
### 1. `JsonDynaLabHello`
```json
{
"type": "dynalab_hello",
"instance_id": "2f3c7f62-0a45-4d66-9f32-7fbeb92f9d36",
"core_version": {
"type": "alpha",
"major": 0,
"minor": 1,
"patch": 0
},
"protocol_version": {
"type": "alpha",
"major": 0,
"minor": 0,
"patch": 1
},
"core_version": {"type": "alpha", "major": 0, "minor": 1, "patch": 0},
"protocol_version": {"type": "alpha", "major": 0, "minor": 0, "patch": 1},
"heartbeat_interval_ms": 1000,
"heartbeat_timeout_ms": 5000
}
```
It is recommended to store this packet for potential later use even if it is not critical for a simple connector
Store the `protocol_version`, interval, and timeout. The connector must use
the exact protocol version in its reply.
### Step 2
Once the `dynalab_hello` has been received it is time to return a `connector_hello` packet, this can be found in the `handshake` module as `ConnectorHello`
### 2. `JsonConnectorHello`
```json
{
"type": "connector_hello",
"connector_uuid": "7b3d9a25-2d54-4cd3-b8ef-9c0c44d2dd81",
"protocol_version": {
"type": "alpha",
"major": 0,
"minor": 0,
"patch": 1
},
"protocol_version": {"type": "alpha", "major": 0, "minor": 0, "patch": 1},
"connector_name": "DynaLab Test Connector",
"connector_version": "0.1.0",
"signals": [
{
"id": "2d60a378-f37c-4c37-867b-c68f5116598b",
"name": "Engine RPM",
"type": "number",
"min_value": 0.0,
"max_value": 6000.0,
"unit": "rpm",
"timeout_ms": 2000
},
{
"id": "5097265d-cdf4-4872-a0e2-b9515ae66b19",
"name": "Torque",
"type": "number",
"min_value": 0.0,
"max_value": 10.0,
"unit": "Nm",
"timeout_ms": 2000
},
{
"id": "d6e633cd-7ca3-4974-a164-44482ddb2a8e",
"name": "Emergency Stop",
"type": "binary",
"min_value": null,
"max_value": null,
"unit": null,
"timeout_ms": 500
}
]
"signals": [{
"id": "2d60a378-f37c-4c37-867b-c68f5116598b",
"name": "Engine RPM",
"type": "number",
"min_value": 0.0,
"max_value": 6000.0,
"unit": "rpm",
"timeout_ms": 2000,
"origin": "source"
}]
}
```
This packet will tell DynaLab Core information about itself and the available signals.
Each connection needs a unique connector UUID and at least one signal. Signal
UUIDs should be stable across connector restarts. `origin` defaults to
`"source"`; connectors should not advertise derived signals.
> [!IMPORTANT]
> `protocol_version` must match the one sent in `dynalab_hello` otherwise the connector will be rejected
### 3. Result
> [!IMPORTANT]
> All id's are UUID's and should be generated using the v4 UUID generation algorithm if possible, they can be stored in config files for repeatable UUID's across restarts but should never be set with human created values
Accepted:
### Step 3
Once the DynaLab Core has received this `connector_hello` packet it will elevate the connection internally to layer 2. At this point DynaLab Core will respond with one of two packets:
- `handshake_accepted`
- `handshake_declined`
**`handshake_accepted`** - This means that DynaLab Core has accepted the handshake and that it is ready for data
```json
{
"type": "handshake_accepted",
"accepted_signals": [
"2d60a378-f37c-4c37-867b-c68f5116598b",
"5097265d-cdf4-4872-a0e2-b9515ae66b19",
"d6e633cd-7ca3-4974-a164-44482ddb2a8e"
]
"accepted_signals": ["2d60a378-f37c-4c37-867b-c68f5116598b"]
}
```
> [!NOTE]
> It is recommended to check the accepted signals list as not all signals may be accepted. DynaLab Core will drop any incoming values that it doesn't recognize however this is not recommended as it can hinder performance especially when a signals frequency is very high
**`handshake_declined`** - This means that DynaLab Core has rejected the handshake for some reason, usually this will be a protocol mismatch or an empty list of signals, after this packet is sent, the connection will be closed by DynaLab Core
Rejected:
```json
{
"type": "handshake_rejected",
"reason": "The connector uses an unsupported protocol version."
}
{"type": "handshake_rejected", "reason": "Protocol versions mismatch"}
```
### Step 4
The handshake is now complete, and either accepted or rejected
Core rejects a protocol-version mismatch, a hello without signals, and a
connector UUID already registered on another active endpoint. Do not send data
until `handshake_accepted` arrives. Only send values for accepted signal IDs.
The next order of business is handling the heartbeat which is a key part of the DynaLab Core - Connector protocol
## Heartbeats
## Heartbeat
> [!TIP]
> All heartbeat relevant packets are located in the `dynalab_core.protocols.packets.heartbeat` module
Core periodically sends `JsonHeartbeat`; return the same `sequence` and
`send_timestamp`, adding a local monotonic-millisecond `return_timestamp`.
Heartbeats are sent out from DynaLab Core at the interval specified by the `dynalab_hello` packet during the handshake procedure. These heartbeats need to be returned to DynaLab Core with a return timestamp.
A typical incoming heartbeat looks like so:
```json
{
"type": "heartbeat",
"sequence": 42,
"send_timestamp": 1938475621,
"return_timestamp": null
}
{"type": "heartbeat", "sequence": 42, "send_timestamp": 1938475621, "return_timestamp": null}
```
It must then be returned with a timestamp generated by the connector, in python:
```python
return_timestamp = round(monotonic() * 1000)
```
This gives a millisecond timestamp that is OS wide to avoid any synchronization issues between DynaLab Core and connectors.
A typical returned heartbeat looks like so:
```json
{
"type": "heartbeat",
"sequence": 42,
"send_timestamp": 1938475621,
"return_timestamp": 1938475624
}
{"type": "heartbeat", "sequence": 42, "send_timestamp": 1938475621, "return_timestamp": 1938475624}
```
You may have noticed that in the `dynalab_hello` packet there was a timeout value, if DynaLab Core receives no returned heartbeats from the connector for a longer than this timeout, the connector is deemed dead and the connection is closed
Return heartbeats before `heartbeat_timeout_ms` elapses. A connector that does
not reply is disconnected.
## Data
> [!TIP]
> All heartbeat relevant packets are located in the `dynalab_core.protocols.packets.heartbeat` module
## Data packets
Data can be piped into DynaLab Core via one of two ways, either a values can be sent as individual packets, or they can be sent batched together
All timestamps use the `monotonic_ns()` timebase
### Individual Value Packets
Sending individual packets per value use the `value_descriptor` packet
Send `JsonValueDescriptor` for one sample:
```json
{
"type": "value_descriptor",
"signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b",
"value": 3247.6,
"timestamp": 1938475621
"timestamp": 1938475621000000
}
```
This method of data input is fine for lower frequency data. However in testing this method clearly has its limits. It is recommended to avoid going over 10 kPackets/s. This limitation is caused by Python's `asyncio`. Future transport options should allow for higher data rates, for the time being it is recommended to use `value_batch` packets
### Batch Value Packets
Sending batch value packets uses the `value_batch` packet
For higher rates, send `JsonValueBatch` to amortize framing and scheduling
overhead:
```json
{
"type": "value_batch",
"values": [
{
"type": "value_descriptor",
"signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b",
"value": 3247.6,
"timestamp": 1938475621
},
{
"type": "value_descriptor",
"signal_id": "5097265d-cdf4-4872-a0e2-b9515ae66b19",
"value": 6.42,
"timestamp": 1938475621
},
{
"type": "value_descriptor",
"signal_id": "d6e633cd-7ca3-4974-a164-44482ddb2a8e",
"value": 1.0,
"timestamp": 1938475622
}
{"type": "value_descriptor", "signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b", "value": 3247.6, "timestamp": 1938475621000000},
{"type": "value_descriptor", "signal_id": "5097265d-cdf4-4872-a0e2-b9515ae66b19", "value": 6.42, "timestamp": 1938475621000100}
]
}
```
This allows to achieve much higher data rates than sending individual packets as it dilutes the per packet overhead a lot, especially when batching many values together.
> [!NOTE]
> There is not hard limit on the number of values, the only limit is the packet size limit which is set to 65535 bytes
Every value timestamp is in monotonic nanoseconds. The batch has no separate
configured item limit, but its complete newline-delimited JSON frame must fit
within 65,536 bytes. Core silently ignores values for a signal that was not
declared by that connector, and connector data is routed only while Core is in
`"realtime"` mode.