Completed initial Protocol documentation

docs/Protocol.md

@@ -3,7 +3,7 @@
 > [!NOTE]
 > All protocol documentation uses JSON packets as examples, however it applies to any form of communication protocol used for data ingress
 
-> !NOTE
+> [!NOTE]
 > Any transport dependent specificities will be noted using this format, look out for any specificities to the transport you are using when building connectors
 
 ## Transports
@@ -18,9 +18,9 @@ More will be added as the project develops
 Internally DynaLab Core works in three layers, it is important to understand this architecture to understand how the handshake procedure works.
 
 The layers are as follows:
-- Server - layer 1
-- Endpoint - layer 2
-- Core - layer 3
+- Server - Layer 1
+- Endpoint - Layer 2
+- Core - Layer 3
 
 **Layer 1** - The server, this is what any connectors connect to, it is responsible for the initial `dynalab_hello` and `connector_hello` handshake before elevating to the next layer. Once elevated the server sends incoming deserialized packets to Layer 2
 
@@ -28,9 +28,27 @@ The layers are as follows:
 
 **Layer 3** - The core, this is the heart of DynaLab, it handles live value storage, as well as all of the recording and processing aspects of DynaLab 
 
+## Important Details
+### Timestamps
+DynaLab works off of one common OS timebase, this can be accessed in python via `time.monotonic()` and `time.monotonic_ns()`
 
+DynaLab Core uses two different time units:
+- Heartbeat - milliseconds
+- Data - nano seconds
 
+**Heartbeat** - Timestamps for heartbeat can be fetched using one of the following methods, both return the same value so it is up to you what you use, for the frequency at which these are called their performance does not really matter: 
 
+```python
+round(time.monotonic() * 1_000.0)
+# or
+time.monotonic_ns() // 1_000_000
+```
+
+**Data** - Timestamps for data are fetched directly:
+
+```python
+time.monotonic_ns()
+```
 
 ## Handshake
 > [!TIP]
@@ -152,6 +170,9 @@ The handshake is now complete, and either accepted or rejected
 The next order of business is handling the heartbeat which is a key part of the DynaLab Core - Connector protocol
 
 ## Heartbeat
+> [!TIP]
+> All heartbeat relevant packets are located in the `dynalab_core.protocols.packets.heartbeat` module
+
 Heartbeats are sent out from DynaLab Core at the interval specified by the `dynalab_hello` packet during the handshake procedure. These heartbeats need to be returned to DynaLab Core with a return timestamp.
 
 A typical incoming heartbeat looks like so:
@@ -182,3 +203,56 @@ A typical returned heartbeat looks like so:
 You may have noticed that in the `dynalab_hello` packet there was a timeout value, if DynaLab Core receives no returned heartbeats from the connector for a longer than this timeout, the connector is deemed dead and the connection is closed
 
 ## Data
+> [!TIP]
+> All heartbeat relevant packets are located in the `dynalab_core.protocols.packets.heartbeat` module
+
+Data can be piped into DynaLab Core via one of two ways, either a values can be sent as individual packets, or they can be sent batched together
+
+All timestamps use the `monotonic_ns()` timebase
+
+### Individual Value Packets
+Sending individual packets per value use the `value_descriptor` packet
+
+```json
+{
+  "type": "value_descriptor",
+  "signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b",
+  "value": 3247.6,
+  "timestamp": 1938475621
+}
+```
+This method of data input is fine for lower frequency data. However in testing this method clearly has its limits. It is recommended to avoid going over 10 kPackets/s. This limitation is caused by Python's `asyncio`. Future transport options should allow for higher data rates, for the time being it is recommended to use `value_batch` packets
+
+### Batch Value Packets
+Sending batch value packets uses the `value_batch` packet
+
+```json
+{
+  "type": "value_batch",
+  "values": [
+    {
+      "type": "value_descriptor",
+      "signal_id": "2d60a378-f37c-4c37-867b-c68f5116598b",
+      "value": 3247.6,
+      "timestamp": 1938475621
+    },
+    {
+      "type": "value_descriptor",
+      "signal_id": "5097265d-cdf4-4872-a0e2-b9515ae66b19",
+      "value": 6.42,
+      "timestamp": 1938475621
+    },
+    {
+      "type": "value_descriptor",
+      "signal_id": "d6e633cd-7ca3-4974-a164-44482ddb2a8e",
+      "value": 1.0,
+      "timestamp": 1938475622
+    }
+  ]
+}
+```
+
+This allows to achieve much higher data rates than sending individual packets as it dilutes the per packet overhead a lot, especially when batching many values together.
+
+> [!NOTE]
+> There is not hard limit on the number of values, the only limit is the packet size limit which is set to 65535 bytes