Added range API, fixed offline processing and DLPak saving

Saving a buffer can now speicfy the buffer used.

Offline processing uses a per signal binary search to hugely reduce
processing time that was mostly spent on an O(n) linear search

API now has range fetching API's for graphs etc
This commit is contained in:
h3cx committed 2026-10-14 21:31:22 +02:00
1 parent 966960ac03
commit 0ae0d90f72
6 files changed
+388 -160

No files matched your search

+109 -7
View File
@@ -111,7 +111,11 @@ recording is available raises `CoreError(code="no_playback_data")`.
Recalculates derived signals in the current processing buffer using the bound
derive units. Call it in `"offline"` mode after `load_processing_buffer()`, with
`CoreConfig(playback_mode="read")`. Units with source inputs run first; units
that depend on their derived outputs run in dependency order. The operation
that depend on their derived outputs run in dependency order. Each unit runs
in sample order using per-signal binary searches instead of full-buffer scans.
For other inputs it selects the nearest recorded value, preferring the earlier
sample when distances tie. This retains the previous offline alignment policy,
including selection of future samples. The operation
removes previously recorded derived samples and descriptors, then merges the
new outputs into the processing buffer. Calling it again replaces those
outputs. It does not write an archive or start playback; use `write()` to save
@@ -172,13 +176,111 @@ offload buffer with a manifest. Errors use `"incorrect_mode"`,
Returns whether a recording is active.
### `write(dir: str | Path, filename: str) -> None`
### `write(dir: str | Path, filename: str, *, source: BufferSource = "processing") -> None`
Writes the current **processing** buffer as a `.dlpak` archive; the extension
is appended when absent. After recording, select `"offline"` mode and call
`load_processing_buffer()` before writing. If no processing buffer exists,
this raises `CoreError(code="no_recording")`. Archive failures raise
`DLPakError` with a stable code such as `"write_failed"`.
Writes the selected buffer as a `.dlpak` archive; the extension is appended
when absent. `BufferSource` is `"recording" | "processing" | "playback"`, defined
in `dynalab_core.types`. The default remains `"processing"` for compatibility.
Use `source="recording"` to save a completed recording directly, without
switching modes or loading the processing buffer. Stop recording first;
saving an active recording raises `CoreError(code="recording_active")`.
`source="playback"` saves the loaded archive, not the current playback position
or newly recalculated live values. Saving does not change mode or load data.
An unavailable buffer raises `CoreError(code="no_recording")`. Archive failures
raise `DLPakError(code="write_failed")`. Writes use a temporary file in the
destination directory and replace the destination only after the archive has
been completed successfully. The destination directory must already exist.
```python
from pathlib import Path
core.start_recording()
# Receive connector data, then finish and save it.
core.stop_recording()
core.write("recordings", "run-001", source="recording")
core.set_mode("offline")
core.load_processing_buffer(Path("recordings/run-001.dlpak"))
```
### DLPak format and standalone access
`DLPak` in `dynalab_core.dlpak` can also be used without Core:
```python
from pathlib import Path
from dynalab_core.dlpak import DLPak
archive = DLPak().read(Path("recordings/run-001.dlpak"))
series = archive.get_data_range(start_ns=0, end_ns=1_000_000_000)
archive.write("recordings", "run-001-copy")
```
Archives are ZIP files containing `manifest.json` and `data.csv`. The manifest
contains `format_version` (currently `1`), a UTC recording timestamp, signal
descriptors, and comments. Older manifests without `format_version` are read
as version 1; unsupported versions are rejected. Signal IDs in the manifest
must be unique, and every signal present in the CSV must have a descriptor.
Descriptors without samples are allowed. Invalid archives raise
`DLPakError(code="read_failed")`, with the underlying cause retained.
The CSV header is `timestamp_ns` followed by signal UUID columns. Empty cells
represent absent samples. Rows are ordered by timestamp. Repeated timestamps
preserve multiple values for the same signal; samples are not collapsed.
## Buffer queries for plotting
### `get_data_range(signal_ids: list[UUID] | None = None, start_ns: int | None = None, end_ns: int | None = None, *, source: BufferSource | None = None) -> dict[UUID, list[tuple[int, float]]]`
Returns an independent snapshot grouped by signal ID. Each list contains
`(timestamp_ns, value)` pairs in timestamp order, including duplicate
timestamps. Both bounds are inclusive. Omit a bound for an open-ended range,
or omit both for all samples. Omit `signal_ids` to select every signal; an
unknown requested ID maps to an empty list. An inverted range raises
`ValueError`. Data is returned at its original resolution without resampling
or downsampling.
By default, the selected buffer follows the current mode:
| Mode | Queried buffer |
| --- | --- |
| `"realtime"` | Active recording, or the latest completed recording when not recording. |
| `"offline"` | Loaded processing buffer. |
| `"playback"` | Loaded playback archive, including its recorded derived samples. |
Pass `source="recording"`, `"processing"`, or `"playback"` to query a specific
buffer without changing modes. An unavailable buffer raises
`CoreError(code="no_recording")`; an available empty buffer returns empty data.
No history is collected outside recordings. Playback queries access the
loaded archive, not newly recalculated live derived values.
Completed recordings use nanoseconds relative to their first sample. Active
recordings still contain ingress `time.monotonic_ns()` timestamps. Imported
archives retain their stored timestamps. Bounds and returned timestamps always
use the selected buffer's stored time base; stopping a recording normalizes it.
```python
bounds = core.get_time_range()
series = core.get_data_range(
[rpm_signal_id, torque_signal_id],
start_ns=1_000_000_000,
end_ns=2_000_000_000,
)
# series[rpm_signal_id] is a list of (timestamp_ns, value) pairs.
```
### `get_time_range(*, source: BufferSource | None = None) -> tuple[int, int] | None`
Returns the minimum and maximum stored timestamps for the same buffer selection
as `get_data_range()`. An empty buffer returns `None`; a single sample returns
its timestamp for both bounds.
`DLPak` and `ValueBuffer` expose the same `get_data_range()` and
`get_time_range()` methods without the `source` argument. Buffer mutations
maintain per-signal indexes; out-of-order signal samples are sorted before
querying. Once sorted, a range query uses binary search plus copying the
selected samples, rather than scanning unrelated signals.
## Playback