8.6 KiB
DynaLab Protocol
Note
All protocol documentation uses JSON packets as examples, however it applies to any form of communication protocol used for data ingress
Note
Any transport dependent specificities will be noted using this format, look out for any specificities to the transport you are using when building connectors
Transports
DynaLab currently only has one transport option:
- JSON over TCP
More will be added as the project develops
Layers
Internally DynaLab Core works in three layers, it is important to understand this architecture to understand how the handshake procedure works.
The layers are as follows:
- Server - Layer 1
- Endpoint - Layer 2
- Core - Layer 3
Layer 1 - The server, this is what any connectors connect to, it is responsible for the initial dynalab_hello and connector_hello handshake before elevating to the next layer. Once elevated the server sends incoming deserialized packets to Layer 2
Layer 2 - The endpoint, this is a connector mapping of sorts, each connector gets its own endpoint, the endpoint handles packet filtering and the connector heartbeat. Once the handshake is accepted this layer send all data oriented packets to Layer 3
Layer 3 - The core, this is the heart of DynaLab, it handles live value storage, as well as all of the recording and processing aspects of DynaLab
Important Details
Timestamps
DynaLab works off of one common OS timebase, this can be accessed in python via time.monotonic() and time.monotonic_ns()
DynaLab Core uses two different time units:
- Heartbeat - milliseconds
- Data - nano seconds
Heartbeat - Timestamps for heartbeat can be fetched using one of the following methods, both return the same value so it is up to you what you use, for the frequency at which these are called their performance does not really matter:
round(time.monotonic() * 1_000.0)
# or
time.monotonic_ns() // 1_000_000
Data - Timestamps for data are fetched directly:
time.monotonic_ns()
Handshake
Tip
All handshake relevant packets are located in the
dynalab_core.protocols.packets.handshakemodule
Step 1
The first step to establishing a communication with the DynaLab Core is proceeding with the initialization handshake.
With most transports upon connection DynaLab Core will send the first packet, this is know as the dynalab_hello packet.
{
"type": "dynalab_hello",
"instance_id": "2f3c7f62-0a45-4d66-9f32-7fbeb92f9d36",
"core_version": {
"type": "alpha",
"major": 0,
"minor": 1,
"patch": 0
},
"protocol_version": {
"type": "alpha",
"major": 0,
"minor": 0,
"patch": 1
},
"heartbeat_interval_ms": 1000,
"heartbeat_timeout_ms": 5000
}
It is recommended to store this packet for potential later use even if it is not critical for a simple connector
Step 2
Once the dynalab_hello has been received it is time to return a connector_hello packet, this can be found in the handshake module as ConnectorHello
{
"type": "connector_hello",
"connector_uuid": "7b3d9a25-2d54-4cd3-b8ef-9c0c44d2dd81",
"protocol_version": {
"type": "alpha",
"major": 0,
"minor": 0,
"patch": 1
},
"connector_name": "DynaLab Test Connector",
"connector_version": "0.1.0",
"signals": [
{
"id": "2d60a378-f37c-4c37-867b-c68f5116598b",
"name": "Engine RPM",
"type": "number",
"min_value": 0.0,
"max_value": 6000.0,
"unit": "rpm",
"timeout_ms": 2000
},
{
"id": "5097265d-cdf4-4872-a0e2-b9515ae66b19",
"name": "Torque",
"type": "number",
"min_value": 0.0,
"max_value": 10.0,
"unit": "Nm",
"timeout_ms": 2000
},
{
"id": "d6e633cd-7ca3-4974-a164-44482ddb2a8e",
"name": "Emergency Stop",
"type": "binary",
"min_value": null,
"max_value": null,
"unit": null,
"timeout_ms": 500
}
]
}
This packet will tell DynaLab Core information about itself and the available signals.
Important
protocol_versionmust match the one sent indynalab_hellootherwise the connector will be rejected
Important
All id's are UUID's and should be generated using the v4 UUID generation algorithm if possible, they can be stored in config files for repeatable UUID's across restarts but should never be set with human created values
Step 3
Once the DynaLab Core has received this connector_hello packet it will elevate the connection internally to layer 2. At this point DynaLab Core will respond with one of two packets:
handshake_acceptedhandshake_declined
handshake_accepted - This means that DynaLab Core has accepted the handshake and that it is ready for data
{
"type": "handshake_accepted",
"accepted_signals": [
"2d60a378-f37c-4c37-867b-c68f5116598b",
"5097265d-cdf4-4872-a0e2-b9515ae66b19",
"d6e633cd-7ca3-4974-a164-44482ddb2a8e"
]
}
Note
It is recommended to check the accepted signals list as not all signals may be accepted. DynaLab Core will drop any incoming values that it doesn't recognize however this is not recommended as it can hinder performance especially when a signals frequency is very high
handshake_declined - This means that DynaLab Core has rejected the handshake for some reason, usually this will be a protocol mismatch or an empty list of signals, after this packet is sent, the connection will be closed by DynaLab Core
{
"type": "handshake_rejected",
"reason": "The connector uses an unsupported protocol version."
}
Step 4
The handshake is now complete, and either accepted or rejected
The next order of business is handling the heartbeat which is a key part of the DynaLab Core - Connector protocol
Heartbeat
Tip
All heartbeat relevant packets are located in the
dynalab_core.protocols.packets.heartbeatmodule
Heartbeats are sent out from DynaLab Core at the interval specified by the dynalab_hello packet during the handshake procedure. These heartbeats need to be returned to DynaLab Core with a return timestamp.
A typical incoming heartbeat looks like so:
{
"type": "heartbeat",
"sequence": 42,
"send_timestamp": 1938475621,
"return_timestamp": null
}
It must then be returned with a timestamp generated by the connector, in python:
return_timestamp = round(monotonic() * 1000)
This gives a millisecond timestamp that is OS wide to avoid any synchronization issues between DynaLab Core and connectors.
A typical returned heartbeat looks like so:
{
"type": "heartbeat",
"sequence": 42,
"send_timestamp": 1938475621,
"return_timestamp": 1938475624
}
You may have noticed that in the dynalab_hello packet there was a timeout value, if DynaLab Core receives no returned heartbeats from the connector for a longer than this timeout, the connector is deemed dead and the connection is closed
Data
Tip
All heartbeat relevant packets are located in the
dynalab_core.protocols.packets.heartbeatmodule
Data can be piped into DynaLab Core via one of two ways, either a values can be sent as individual packets, or they can be sent batched together
All timestamps use the monotonic_ns() timebase
Individual Value Packets
Sending individual packets per value use the value_descriptor packet
{
"type": "value_descriptor",
"signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b",
"value": 3247.6,
"timestamp": 1938475621
}
This method of data input is fine for lower frequency data. However in testing this method clearly has its limits. It is recommended to avoid going over 10 kPackets/s. This limitation is caused by Python's asyncio. Future transport options should allow for higher data rates, for the time being it is recommended to use value_batch packets
Batch Value Packets
Sending batch value packets uses the value_batch packet
{
"type": "value_batch",
"values": [
{
"type": "value_descriptor",
"signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b",
"value": 3247.6,
"timestamp": 1938475621
},
{
"type": "value_descriptor",
"signal_id": "5097265d-cdf4-4872-a0e2-b9515ae66b19",
"value": 6.42,
"timestamp": 1938475621
},
{
"type": "value_descriptor",
"signal_id": "d6e633cd-7ca3-4974-a164-44482ddb2a8e",
"value": 1.0,
"timestamp": 1938475622
}
]
}
This allows to achieve much higher data rates than sending individual packets as it dilutes the per packet overhead a lot, especially when batching many values together.
Note
There is not hard limit on the number of values, the only limit is the packet size limit which is set to 65535 bytes