The initial API documentation is now complete and covers the whole Core
api ready for use
The underlying logic has been updated to correctly return derive unit
signals
Derive units are now available in the core and has their relevant input
values routed and outputs are looped back into the core router
Fixed the dlpak manifest which had no signal descriptors
This contains the initial DeriveUnit implementation, including signature
validation, this initial implentation only supports ValueDescriptors in
the signature alongside the SignalDescriptor and returns a single
ValueDescriptor
The worker thread still needs to be defined and started but the offline
processing has already been validate to work as a first prototype
SignalDescriptor now have a new origin field to distinguish "source"
from "derived" signals, this is ahead of the internal loopback system
for data processing, all external conectors must use source, derived is
targeted only for internal use
The connector registry has been adapted to now have a unique internal
loopback, this does not have an associated endpoint so it is stored
apart and all relevant functions have been updated to include it for
proper handling of signals by DLPak etc
Implemented DLPak design with zip storage, DLPak consists of two files,
a manifest.json which provides details on the signals, time of recording
and currently blank comment field and data.csv which stores all the
signals as csv with the nanosecond timestamp and indexed against the
signal id
ValueBuffer is now the default buffer object for storing data for
recording and processing purposes, a simple recording implementation has
been done, core.py example starts recording as soon as the core has
started then on exit it saves the output as a csv file through the
ValueBuffer export_csv method
Added timestamping to ValueDescriptor to be able to record on a
timebase, timebase uses monotonic_ns() which is a system wide timebase
that can be called by connectors written in a host of languages
Added timeout to signal descriptors, live values now return None when
they are timed out
Live values are stored in the core and their fetch is relatively cheap
as it only requires locking the core live values lock
Signal descriptors are a much heavier call as they query the connector
registry and trigger an O(n) search throughout the different endpoints,
this method is not meant to be used heavily as it requires locking
multiple locks and handling alot of data, this method is meant to be
called once to retreive a list of signal descriptors that the user can
then store and use to reference values against in their frontend code
Removed useless tests, will need to complete a more comprehensive test
suite however this is not the priority right now
Added a live value dict in the core whose values are populated by the
core's input thread
Implemented handshake in json server and handoff to endpoint
Endpoint handles connector handshake accept/decline then launches both
its IO threads alongside its heartbeat thread which contains simple
timeout logic, timeout calls connection handles to close, subsequently
killing the endpoint
Further isolated json specifics to their own module under protocols/json
Moved global packet classes to their own protocols/packets module
Implemented ConnectorEndpoint and ConnectorRegistry as transport
agnostic connector endpoints for universal connector handling, exposed a
ConnectorRegistry API that will need to be made available to the
JsonServer for it to be able to register and unregister endpoints as it
opens and closes connections
Implemented a top down shutdown architecture where each object is
responsible for its children, this allows for a logical and heirachical
flow, each object has its own stop event, which when its stop method is
called, it sets and then awaits the stopped event to be set by any
worker threads etc, with a mandatory timeout to avoid hanging on
shutdown
Isolated the JsonServer to server.py inside protocols/json
Added specific errors for server startup timeout and server startup
failed
Refactored errors for Core to CoreError generic and refactored
StateError to CoreStateMismatchError derived from CoreError
Completed the initial implementation of the JsonServer class, handling
the worker thread that runs the async task, as well as the connection
handler callback
The current handler awaits readline to cleanly exit when the connection
is closed, no data ingress is happening yet
Next steps include gating the start procedure to wait for everything to
start proprerly in order to avoid having start exit early and async
tasks crashing later
h3cx