# DynaLab Protocol

> [!NOTE]
> All protocol documentation uses JSON packets as examples, however it applies to any form of communication protocol used for data ingress

> [!NOTE]
> Any transport dependent specificities will be noted using this format, look out for any specificities to the transport you are using when building connectors

## Transports

DynaLab currently only has one transport option:
- JSON over TCP

More will be added as the project develops

## Layers

Internally DynaLab Core works in three layers, it is important to understand this architecture to understand how the handshake procedure works.

The layers are as follows:
- Server - Layer 1
- Endpoint - Layer 2
- Core - Layer 3

**Layer 1** - The server, this is what any connectors connect to, it is responsible for the initial `dynalab_hello` and `connector_hello` handshake before elevating to the next layer. Once elevated the server sends incoming deserialized packets to Layer 2

**Layer 2** - The endpoint, this is a *connector mapping* of sorts, each connector gets its own endpoint, the endpoint handles packet filtering and the connector heartbeat. Once the handshake is accepted this layer send all data oriented packets to Layer 3

**Layer 3** - The core, this is the heart of DynaLab, it handles live value storage, as well as all of the recording and processing aspects of DynaLab 

## Important Details
### Timestamps
DynaLab works off of one common OS timebase, this can be accessed in python via `time.monotonic()` and `time.monotonic_ns()`

DynaLab Core uses two different time units:
- Heartbeat - milliseconds
- Data - nano seconds

**Heartbeat** - Timestamps for heartbeat can be fetched using one of the following methods, both return the same value so it is up to you what you use, for the frequency at which these are called their performance does not really matter: 

```python
round(time.monotonic() * 1_000.0)
# or
time.monotonic_ns() // 1_000_000
```

**Data** - Timestamps for data are fetched directly:

```python
time.monotonic_ns()
```

## Handshake
> [!TIP]
> All handshake relevant packets are located in the `dynalab_core.protocols.packets.handshake` module

### Step 1
The first step to establishing a communication with the DynaLab Core is proceeding with the initialization handshake.

With most transports upon connection DynaLab Core will send the first packet, this is know as the `dynalab_hello` packet.

```json
{
  "type": "dynalab_hello",
  "instance_id": "2f3c7f62-0a45-4d66-9f32-7fbeb92f9d36",
  "core_version": {
    "type": "alpha",
    "major": 0,
    "minor": 1,
    "patch": 0
  },
  "protocol_version": {
    "type": "alpha",
    "major": 0,
    "minor": 0,
    "patch": 1
  },
  "heartbeat_interval_ms": 1000,
  "heartbeat_timeout_ms": 5000
}
```

It is recommended to store this packet for potential later use even if it is not critical for a simple connector

### Step 2
Once the `dynalab_hello` has been received it is time to return a `connector_hello` packet, this can be found in the `handshake` module as `ConnectorHello`

```json
{
  "type": "connector_hello",
  "connector_uuid": "7b3d9a25-2d54-4cd3-b8ef-9c0c44d2dd81",
  "protocol_version": {
    "type": "alpha",
    "major": 0,
    "minor": 0,
    "patch": 1
  },
  "connector_name": "DynaLab Test Connector",
  "connector_version": "0.1.0",
  "signals": [
    {
      "id": "2d60a378-f37c-4c37-867b-c68f5116598b",
      "name": "Engine RPM",
      "type": "number",
      "min_value": 0.0,
      "max_value": 6000.0,
      "unit": "rpm",
      "timeout_ms": 2000
    },
    {
      "id": "5097265d-cdf4-4872-a0e2-b9515ae66b19",
      "name": "Torque",
      "type": "number",
      "min_value": 0.0,
      "max_value": 10.0,
      "unit": "Nm",
      "timeout_ms": 2000
    },
    {
      "id": "d6e633cd-7ca3-4974-a164-44482ddb2a8e",
      "name": "Emergency Stop",
      "type": "binary",
      "min_value": null,
      "max_value": null,
      "unit": null,
      "timeout_ms": 500
    }
  ]
}
```

This packet will tell DynaLab Core information about itself and the available signals.

> [!IMPORTANT]
> `protocol_version` must match the one sent in `dynalab_hello` otherwise the connector will be rejected

> [!IMPORTANT]
> All id's are UUID's and should be generated using the v4 UUID generation algorithm if possible, they can be stored in config files for repeatable UUID's across restarts but should never be set with human created values 

### Step 3
Once the DynaLab Core has received this `connector_hello` packet it will elevate the connection internally to layer 2. At this point DynaLab Core will respond with one of two packets:
- `handshake_accepted`
- `handshake_declined`

**`handshake_accepted`** - This means that DynaLab Core has accepted the handshake and that it is ready for data 
```json
{
  "type": "handshake_accepted",
  "accepted_signals": [
    "2d60a378-f37c-4c37-867b-c68f5116598b",
    "5097265d-cdf4-4872-a0e2-b9515ae66b19",
    "d6e633cd-7ca3-4974-a164-44482ddb2a8e"
  ]
}
```
> [!NOTE]
> It is recommended to check the accepted signals list as not all signals may be accepted. DynaLab Core will drop any incoming values that it doesn't recognize however this is not recommended as it can hinder performance especially when a signals frequency is very high

**`handshake_declined`** - This means that DynaLab Core has rejected the handshake for some reason, usually this will be a protocol mismatch or an empty list of signals, after this packet is sent, the connection will be closed by DynaLab Core
```json
{
  "type": "handshake_rejected",
  "reason": "The connector uses an unsupported protocol version."
}
```

### Step 4
The handshake is now complete, and either accepted or rejected

The next order of business is handling the heartbeat which is a key part of the DynaLab Core - Connector protocol

## Heartbeat
> [!TIP]
> All heartbeat relevant packets are located in the `dynalab_core.protocols.packets.heartbeat` module

Heartbeats are sent out from DynaLab Core at the interval specified by the `dynalab_hello` packet during the handshake procedure. These heartbeats need to be returned to DynaLab Core with a return timestamp.

A typical incoming heartbeat looks like so:
```json
{
  "type": "heartbeat",
  "sequence": 42,
  "send_timestamp": 1938475621,
  "return_timestamp": null
}
```
It must then be returned with a timestamp generated by the connector, in python:
```python
return_timestamp = round(monotonic() * 1000)
```
This gives a millisecond timestamp that is OS wide to avoid any synchronization issues between DynaLab Core and connectors.

A typical returned heartbeat looks like so:
```json
{
  "type": "heartbeat",
  "sequence": 42,
  "send_timestamp": 1938475621,
  "return_timestamp": 1938475624
}
```

You may have noticed that in the `dynalab_hello` packet there was a timeout value, if DynaLab Core receives no returned heartbeats from the connector for a longer than this timeout, the connector is deemed dead and the connection is closed

## Data
> [!TIP]
> All heartbeat relevant packets are located in the `dynalab_core.protocols.packets.heartbeat` module

Data can be piped into DynaLab Core via one of two ways, either a values can be sent as individual packets, or they can be sent batched together

All timestamps use the `monotonic_ns()` timebase

### Individual Value Packets
Sending individual packets per value use the `value_descriptor` packet

```json
{
  "type": "value_descriptor",
  "signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b",
  "value": 3247.6,
  "timestamp": 1938475621
}
```
This method of data input is fine for lower frequency data. However in testing this method clearly has its limits. It is recommended to avoid going over 10 kPackets/s. This limitation is caused by Python's `asyncio`. Future transport options should allow for higher data rates, for the time being it is recommended to use `value_batch` packets

### Batch Value Packets
Sending batch value packets uses the `value_batch` packet

```json
{
  "type": "value_batch",
  "values": [
    {
      "type": "value_descriptor",
      "signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b",
      "value": 3247.6,
      "timestamp": 1938475621
    },
    {
      "type": "value_descriptor",
      "signal_id": "5097265d-cdf4-4872-a0e2-b9515ae66b19",
      "value": 6.42,
      "timestamp": 1938475621
    },
    {
      "type": "value_descriptor",
      "signal_id": "d6e633cd-7ca3-4974-a164-44482ddb2a8e",
      "value": 1.0,
      "timestamp": 1938475622
    }
  ]
}
```

This allows to achieve much higher data rates than sending individual packets as it dilutes the per packet overhead a lot, especially when batching many values together.

> [!NOTE]
> There is not hard limit on the number of values, the only limit is the packet size limit which is set to 65535 bytes