CoreConfig how has a playback mode option which can be set to reprocess
or read.
Setting it to reprocess means that all derived signals are reprocessed
during playback, in this case the PlaybackEngine only plays back the
signals whose source is set to "origin"
Setting it to read means that all derived signals are playedback from
the recorded buffer, this whill be the preferred mode once offline
processing is set up which will allow for more complex data processing
offline once a recording is completed, which would require the read
playback mode as it wouldn't support realtime processing as reprocess
does
CoreConfig now have a boolean thread debug option, when this is enabled,
it prints out the CPU usage of each thread to help identify which
threads are the most CPU hungry for later optimisation
All protocol classes have now been duplicated in order to minimize the
use of pydantic validation
The "pure" classes are named what the previous validated classes were,
now each transport has its transport name prefixed on the class.
For example before we had Heartbeat, now we have Heartbeat and
JsonHeartbeat
This new architecture allows us to reduce the number of useless backend
validation steps created when copying and processing data, reducing CPU
stress by roughly 10-20% in real world testing
Derived units now use a DLUnit class similar to a DLPak to be easily
stored and loaded, the public API has been updated to either take a
DLUnit object or a Path where it reads the DLUnit from a *.dlunit file
The .dlunit file is a ZIP compressed archive containing two files, a
manifest.json which stores the units uuid, its input signals and its
output signal, and unit.py which is the python code for the unit, the
same rules apply (one python function with a specifically structured
signature) as when a string was passed in the old API
Errors have been added to have a DLUnitError similar to the DLPakError
as the DLUnit class is heavily inspired from the DLPak architecutre
Logging may still be primitive, this will be covered in the next AI
logging sweep to homogenize and clean up the logging aspect of things
All internal and public errors have been revised (GPT 5.6 Sol) to return
more detail without horrendously long error classes, this has reduced
the number of error types but each now carries a reason and a code for
differentiation if needed
PlaybackEngine has been implemented and is functional, core mode changes
what signal descriptors are returned to avoid duplicates, the playback
engine currently discards the derived signals that would be recorded in
the DLPak and derived signals are reprocessed live
Future will require the behavior listed above to be selectable so one of
two options are possible:
A - Derived signals are reprocessed live by the relevant derive units
(current behavior
B - Derived signals are played back directly from the PlaybackEngine,
this requires turning off the routing to the live derive units to avoid
duplicate values
This will also require the option to perform offline processing with
derive units to be able to create derived signals after the fact, this
will likely incur a new 'offline' mode in the core to gate things
correctly, this will likely be useful for heavier processing that cannot
be done live, filters that require a lookahead, or more precise derived
signals using interpolated signals for higher acurracy which also brings
the ability to offline process on a fixed timebase instead of following
either input signal to the derived signal
The initial API documentation is now complete and covers the whole Core
api ready for use
The underlying logic has been updated to correctly return derive unit
signals
Derive units are now available in the core and has their relevant input
values routed and outputs are looped back into the core router
Fixed the dlpak manifest which had no signal descriptors
This contains the initial DeriveUnit implementation, including signature
validation, this initial implentation only supports ValueDescriptors in
the signature alongside the SignalDescriptor and returns a single
ValueDescriptor
The worker thread still needs to be defined and started but the offline
processing has already been validate to work as a first prototype
SignalDescriptor now have a new origin field to distinguish "source"
from "derived" signals, this is ahead of the internal loopback system
for data processing, all external conectors must use source, derived is
targeted only for internal use
The connector registry has been adapted to now have a unique internal
loopback, this does not have an associated endpoint so it is stored
apart and all relevant functions have been updated to include it for
proper handling of signals by DLPak etc
ValueBuffer is now the default buffer object for storing data for
recording and processing purposes, a simple recording implementation has
been done, core.py example starts recording as soon as the core has
started then on exit it saves the output as a csv file through the
ValueBuffer export_csv method
Added timestamping to ValueDescriptor to be able to record on a
timebase, timebase uses monotonic_ns() which is a system wide timebase
that can be called by connectors written in a host of languages
Added timeout to signal descriptors, live values now return None when
they are timed out
Live values are stored in the core and their fetch is relatively cheap
as it only requires locking the core live values lock
Signal descriptors are a much heavier call as they query the connector
registry and trigger an O(n) search throughout the different endpoints,
this method is not meant to be used heavily as it requires locking
multiple locks and handling alot of data, this method is meant to be
called once to retreive a list of signal descriptors that the user can
then store and use to reference values against in their frontend code
Removed useless tests, will need to complete a more comprehensive test
suite however this is not the priority right now
Added a live value dict in the core whose values are populated by the
core's input thread
Implemented handshake in json server and handoff to endpoint
Endpoint handles connector handshake accept/decline then launches both
its IO threads alongside its heartbeat thread which contains simple
timeout logic, timeout calls connection handles to close, subsequently
killing the endpoint
Isolated the JsonServer to server.py inside protocols/json
Added specific errors for server startup timeout and server startup
failed
Refactored errors for Core to CoreError generic and refactored
StateError to CoreStateMismatchError derived from CoreError
h3cx