dynalab-core/docs/Protocol.md

# DynaLab Core connector protocol

> [!IMPORTANT]
> **Packet model change.** Unprefixed packet classes (`Heartbeat`, `ConnectorHello`,
> `ValueDescriptor`, and so on) are lightweight slotted dataclasses for the
> internal data path. They do not perform Pydantic validation. Every packet that
> crosses the JSON-over-TCP boundary uses its Pydantic `Json*` counterpart:
> `JsonHeartbeat`, `JsonConnectorHello`, `JsonValueDescriptor`, etc. This
> separation avoids validating packets repeatedly under load. Convert at the
> boundary with `to_pure()` (received JSON) and `to_json()` (outgoing internal
> packet).

> [!WARNING]
> The protocol is alpha. Validate a connector against the Core version it will
> run with.

## Transport and framing

The current transport is JSON over TCP. A frame is one UTF-8 JSON object
followed by a newline (`\n`). The maximum incoming frame is 65,536 bytes,
including its newline delimiter. A missing delimiter, oversized frame,
invalid JSON, or a JSON object that does not match a `Json*` packet closes the
connection.

Use `JsonProtocolMessage` with the JSON wire helpers:

```python
from dynalab_core.protocols.json.wire import read_message, write_message
from dynalab_core.protocols.packets import JsonConnectorHello

# Receive JSON/TCP data (validated once), then enter the internal data path.
json_packet = await read_message(reader)
packet = json_packet.to_pure()

# Send a JSON packet directly, or convert an internal packet first.
await write_message(writer, JsonConnectorHello(...))
# await write_message(writer, packet.to_json())
```

## Packet representations

| Purpose | Internal type | JSON/TCP type |
| --- | --- | --- |
| Signal metadata | `SignalDescriptor` | `JsonSignalDescriptor` |
| Server hello | `DynaLabHello` | `JsonDynaLabHello` |
| Connector hello | `ConnectorHello` | `JsonConnectorHello` |
| Accepted / rejected | `HandshakeAccepted` / `HandshakeRejected` | `JsonHandshakeAccepted` / `JsonHandshakeRejected` |
| Heartbeat | `Heartbeat` | `JsonHeartbeat` |
| One value | `ValueDescriptor` | `JsonValueDescriptor` |
| Value batch | `ValueBatch` | `JsonValueBatch` |

`JsonProtocolMessage` is the discriminated union used to parse JSON frames;
`ProtocolMessage` is the corresponding union for the internal packet types.
The JSON model includes the discriminator field `type`; pure packets do not.

JSON models validate and serialize UUIDs and nested fields. Pure packets accept
the annotated values as supplied, so construct them only from trusted or
already-validated data.

## Timestamps

DynaLab uses the host monotonic clock:

- Heartbeat timestamps are milliseconds: `time.monotonic_ns() // 1_000_000`.
- Data timestamps are nanoseconds: `time.monotonic_ns()`.

Do not use wall-clock (`time.time()`) timestamps. The core compares heartbeat
times and plays data against this monotonic timebase.

## Handshake

All examples below are **wire JSON** and therefore correspond to `Json*`
packet classes.

1. Core sends `JsonDynaLabHello` immediately after the TCP connection opens.
2. The connector replies with `JsonConnectorHello` within five seconds.
3. Core returns `JsonHandshakeAccepted` or `JsonHandshakeRejected`.
4. On acceptance, the connector can send values and must return heartbeats.

### 1. `JsonDynaLabHello`

```json
{
  "type": "dynalab_hello",
  "instance_id": "2f3c7f62-0a45-4d66-9f32-7fbeb92f9d36",
  "core_version": {"type": "alpha", "major": 0, "minor": 1, "patch": 0},
  "protocol_version": {"type": "alpha", "major": 0, "minor": 0, "patch": 1},
  "heartbeat_interval_ms": 1000,
  "heartbeat_timeout_ms": 5000
}
```

Store the `protocol_version`, interval, and timeout. The connector must use
the exact protocol version in its reply.

### 2. `JsonConnectorHello`

```json
{
  "type": "connector_hello",
  "connector_uuid": "7b3d9a25-2d54-4cd3-b8ef-9c0c44d2dd81",
  "protocol_version": {"type": "alpha", "major": 0, "minor": 0, "patch": 1},
  "connector_name": "DynaLab Test Connector",
  "connector_version": "0.1.0",
  "signals": [{
    "id": "2d60a378-f37c-4c37-867b-c68f5116598b",
    "name": "Engine RPM",
    "type": "number",
    "min_value": 0.0,
    "max_value": 6000.0,
    "unit": "rpm",
    "timeout_ms": 2000,
    "origin": "source"
  }]
}
```

Each connection needs a unique connector UUID and at least one signal. Signal
UUIDs should be stable across connector restarts. `origin` defaults to
`"source"`; connectors should not advertise derived signals.

### 3. Result

Accepted:

```json
{
  "type": "handshake_accepted",
  "accepted_signals": ["2d60a378-f37c-4c37-867b-c68f5116598b"]
}
```

Rejected:

```json
{"type": "handshake_rejected", "reason": "Protocol versions mismatch"}
```

Core rejects a protocol-version mismatch, a hello without signals, and a
connector UUID already registered on another active endpoint. Do not send data
until `handshake_accepted` arrives. Only send values for accepted signal IDs.

## Heartbeats

Core periodically sends `JsonHeartbeat`; return the same `sequence` and
`send_timestamp`, adding a local monotonic-millisecond `return_timestamp`.

```json
{"type": "heartbeat", "sequence": 42, "send_timestamp": 1938475621, "return_timestamp": null}
```

```json
{"type": "heartbeat", "sequence": 42, "send_timestamp": 1938475621, "return_timestamp": 1938475624}
```

Return heartbeats before `heartbeat_timeout_ms` elapses. A connector that does
not reply is disconnected.

## Data packets

Send `JsonValueDescriptor` for one sample:

```json
{
  "type": "value_descriptor",
  "signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b",
  "value": 3247.6,
  "timestamp": 1938475621000000
}
```

For higher rates, send `JsonValueBatch` to amortize framing and scheduling
overhead:

```json
{
  "type": "value_batch",
  "values": [
    {"type": "value_descriptor", "signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b", "value": 3247.6, "timestamp": 1938475621000000},
    {"type": "value_descriptor", "signal_id": "5097265d-cdf4-4872-a0e2-b9515ae66b19", "value": 6.42, "timestamp": 1938475621000100}
  ]
}
```

Every value timestamp is in monotonic nanoseconds. The batch has no separate
configured item limit, but its complete newline-delimited JSON frame must fit
within 65,536 bytes. Core silently ignores values for a signal that was not
declared by that connector, and connector data is routed only while Core is in
`"realtime"` mode.