dynalab-core/docs/Protocol.md

DynaLab Core connector protocol

Important

Packet model change. Unprefixed packet classes (Heartbeat, ConnectorHello, ValueDescriptor, and so on) are lightweight slotted dataclasses for the internal data path. They do not perform Pydantic validation. Every packet that crosses the JSON-over-TCP boundary uses its Pydantic Json* counterpart: JsonHeartbeat, JsonConnectorHello, JsonValueDescriptor, etc. This separation avoids validating packets repeatedly under load. Convert at the boundary with to_pure() (received JSON) and to_json() (outgoing internal packet).

Warning

The protocol is alpha. Validate a connector against the Core version it will run with.

Transport and framing

The current transport is JSON over TCP. A frame is one UTF-8 JSON object followed by a newline (\n). The maximum incoming frame is 65,536 bytes, including its newline delimiter. A missing delimiter, oversized frame, invalid JSON, or a JSON object that does not match a Json* packet closes the connection.

Use JsonProtocolMessage with the JSON wire helpers:

from dynalab_core.protocols.json.wire import read_message, write_message
from dynalab_core.protocols.packets import JsonConnectorHello

# Receive JSON/TCP data (validated once), then enter the internal data path.
json_packet = await read_message(reader)
packet = json_packet.to_pure()

# Send a JSON packet directly, or convert an internal packet first.
await write_message(writer, JsonConnectorHello(...))
# await write_message(writer, packet.to_json())

Packet representations

Purpose	Internal type	JSON/TCP type
Signal metadata	SignalDescriptor	JsonSignalDescriptor
Server hello	DynaLabHello	JsonDynaLabHello
Connector hello	ConnectorHello	JsonConnectorHello
Accepted / rejected	HandshakeAccepted / HandshakeRejected	JsonHandshakeAccepted / JsonHandshakeRejected
Heartbeat	Heartbeat	JsonHeartbeat
One value	ValueDescriptor	JsonValueDescriptor
Value batch	ValueBatch	JsonValueBatch

JsonProtocolMessage is the discriminated union used to parse JSON frames; ProtocolMessage is the corresponding union for the internal packet types. The JSON model includes the discriminator field type; pure packets do not.

JSON models validate and serialize UUIDs and nested fields. Pure packets accept the annotated values as supplied, so construct them only from trusted or already-validated data.

Timestamps

DynaLab uses the host monotonic clock:

• Heartbeat timestamps are milliseconds: time.monotonic_ns() // 1_000_000.
• Data timestamps are nanoseconds: time.monotonic_ns().

Do not use wall-clock (time.time()) timestamps. The core compares heartbeat times and plays data against this monotonic timebase.

Handshake

All examples below are wire JSON and therefore correspond to Json* packet classes.

1. Core sends JsonDynaLabHello immediately after the TCP connection opens.
2. The connector replies with JsonConnectorHello within five seconds.
3. Core returns JsonHandshakeAccepted or JsonHandshakeRejected.
4. On acceptance, the connector can send values and must return heartbeats.

1. JsonDynaLabHello

{
  "type": "dynalab_hello",
  "instance_id": "2f3c7f62-0a45-4d66-9f32-7fbeb92f9d36",
  "core_version": {"type": "alpha", "major": 0, "minor": 2, "patch": 1},
  "protocol_version": {"type": "alpha", "major": 0, "minor": 0, "patch": 1},
  "heartbeat_interval_ms": 1000,
  "heartbeat_timeout_ms": 5000
}

Store the protocol_version, interval, and timeout. The connector must use the exact protocol version in its reply.

2. JsonConnectorHello

{
  "type": "connector_hello",
  "connector_uuid": "7b3d9a25-2d54-4cd3-b8ef-9c0c44d2dd81",
  "protocol_version": {"type": "alpha", "major": 0, "minor": 0, "patch": 1},
  "connector_name": "DynaLab Test Connector",
  "connector_version": "0.1.0",
  "signals": [{
    "id": "2d60a378-f37c-4c37-867b-c68f5116598b",
    "name": "Engine RPM",
    "type": "number",
    "min_value": 0.0,
    "max_value": 6000.0,
    "unit": "rpm",
    "timeout_ms": 2000,
    "origin": "source"
  }]
}

Each connection needs a unique connector UUID and at least one signal. Signal UUIDs should be stable across connector restarts. origin defaults to "source"; connectors should not advertise derived signals.

3. Result

Accepted:

{
  "type": "handshake_accepted",
  "accepted_signals": ["2d60a378-f37c-4c37-867b-c68f5116598b"]
}

Rejected:

{"type": "handshake_rejected", "reason": "Protocol versions mismatch"}

Core rejects a protocol-version mismatch, a hello without signals, and a connector UUID already registered on another active endpoint. Do not send data until handshake_accepted arrives. Only send values for accepted signal IDs.

Heartbeats

Core periodically sends JsonHeartbeat; return the same sequence and send_timestamp, adding a local monotonic-millisecond return_timestamp.

{"type": "heartbeat", "sequence": 42, "send_timestamp": 1938475621, "return_timestamp": null}

{"type": "heartbeat", "sequence": 42, "send_timestamp": 1938475621, "return_timestamp": 1938475624}

Return heartbeats before heartbeat_timeout_ms elapses. A connector that does not reply is disconnected.

Data packets

Send JsonValueDescriptor for one sample:

{
  "type": "value_descriptor",
  "signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b",
  "value": 3247.6,
  "timestamp": 1938475621000000
}

For higher rates, send JsonValueBatch to amortize framing and scheduling overhead:

{
  "type": "value_batch",
  "values": [
    {"type": "value_descriptor", "signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b", "value": 3247.6, "timestamp": 1938475621000000},
    {"type": "value_descriptor", "signal_id": "5097265d-cdf4-4872-a0e2-b9515ae66b19", "value": 6.42, "timestamp": 1938475621000100}
  ]
}

Every value timestamp is in monotonic nanoseconds. The batch has no separate configured item limit, but its complete newline-delimited JSON frame must fit within 65,536 bytes. Core silently ignores values for a signal that was not declared by that connector, and connector data is routed only while Core is in "realtime" mode.