Added range API, fixed offline processing and DLPak saving
Saving a buffer can now speicfy the buffer used. Offline processing uses a per signal binary search to hugely reduce processing time that was mostly spent on an O(n) linear search API now has range fetching API's for graphs etc
This commit is contained in:
6 files changed
+388
-160
No files matched your search
+109
-7
@@ -111,7 +111,11 @@ recording is available raises `CoreError(code="no_playback_data")`.
|
||||
Recalculates derived signals in the current processing buffer using the bound
|
||||
derive units. Call it in `"offline"` mode after `load_processing_buffer()`, with
|
||||
`CoreConfig(playback_mode="read")`. Units with source inputs run first; units
|
||||
that depend on their derived outputs run in dependency order. The operation
|
||||
that depend on their derived outputs run in dependency order. Each unit runs
|
||||
in sample order using per-signal binary searches instead of full-buffer scans.
|
||||
For other inputs it selects the nearest recorded value, preferring the earlier
|
||||
sample when distances tie. This retains the previous offline alignment policy,
|
||||
including selection of future samples. The operation
|
||||
removes previously recorded derived samples and descriptors, then merges the
|
||||
new outputs into the processing buffer. Calling it again replaces those
|
||||
outputs. It does not write an archive or start playback; use `write()` to save
|
||||
@@ -172,13 +176,111 @@ offload buffer with a manifest. Errors use `"incorrect_mode"`,
|
||||
|
||||
Returns whether a recording is active.
|
||||
|
||||
### `write(dir: str | Path, filename: str) -> None`
|
||||
### `write(dir: str | Path, filename: str, *, source: BufferSource = "processing") -> None`
|
||||
|
||||
Writes the current **processing** buffer as a `.dlpak` archive; the extension
|
||||
is appended when absent. After recording, select `"offline"` mode and call
|
||||
`load_processing_buffer()` before writing. If no processing buffer exists,
|
||||
this raises `CoreError(code="no_recording")`. Archive failures raise
|
||||
`DLPakError` with a stable code such as `"write_failed"`.
|
||||
Writes the selected buffer as a `.dlpak` archive; the extension is appended
|
||||
when absent. `BufferSource` is `"recording" | "processing" | "playback"`, defined
|
||||
in `dynalab_core.types`. The default remains `"processing"` for compatibility.
|
||||
Use `source="recording"` to save a completed recording directly, without
|
||||
switching modes or loading the processing buffer. Stop recording first;
|
||||
saving an active recording raises `CoreError(code="recording_active")`.
|
||||
`source="playback"` saves the loaded archive, not the current playback position
|
||||
or newly recalculated live values. Saving does not change mode or load data.
|
||||
|
||||
An unavailable buffer raises `CoreError(code="no_recording")`. Archive failures
|
||||
raise `DLPakError(code="write_failed")`. Writes use a temporary file in the
|
||||
destination directory and replace the destination only after the archive has
|
||||
been completed successfully. The destination directory must already exist.
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
core.start_recording()
|
||||
# Receive connector data, then finish and save it.
|
||||
core.stop_recording()
|
||||
core.write("recordings", "run-001", source="recording")
|
||||
|
||||
core.set_mode("offline")
|
||||
core.load_processing_buffer(Path("recordings/run-001.dlpak"))
|
||||
```
|
||||
|
||||
### DLPak format and standalone access
|
||||
|
||||
`DLPak` in `dynalab_core.dlpak` can also be used without Core:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from dynalab_core.dlpak import DLPak
|
||||
|
||||
archive = DLPak().read(Path("recordings/run-001.dlpak"))
|
||||
series = archive.get_data_range(start_ns=0, end_ns=1_000_000_000)
|
||||
archive.write("recordings", "run-001-copy")
|
||||
```
|
||||
|
||||
Archives are ZIP files containing `manifest.json` and `data.csv`. The manifest
|
||||
contains `format_version` (currently `1`), a UTC recording timestamp, signal
|
||||
descriptors, and comments. Older manifests without `format_version` are read
|
||||
as version 1; unsupported versions are rejected. Signal IDs in the manifest
|
||||
must be unique, and every signal present in the CSV must have a descriptor.
|
||||
Descriptors without samples are allowed. Invalid archives raise
|
||||
`DLPakError(code="read_failed")`, with the underlying cause retained.
|
||||
|
||||
The CSV header is `timestamp_ns` followed by signal UUID columns. Empty cells
|
||||
represent absent samples. Rows are ordered by timestamp. Repeated timestamps
|
||||
preserve multiple values for the same signal; samples are not collapsed.
|
||||
|
||||
## Buffer queries for plotting
|
||||
|
||||
### `get_data_range(signal_ids: list[UUID] | None = None, start_ns: int | None = None, end_ns: int | None = None, *, source: BufferSource | None = None) -> dict[UUID, list[tuple[int, float]]]`
|
||||
|
||||
Returns an independent snapshot grouped by signal ID. Each list contains
|
||||
`(timestamp_ns, value)` pairs in timestamp order, including duplicate
|
||||
timestamps. Both bounds are inclusive. Omit a bound for an open-ended range,
|
||||
or omit both for all samples. Omit `signal_ids` to select every signal; an
|
||||
unknown requested ID maps to an empty list. An inverted range raises
|
||||
`ValueError`. Data is returned at its original resolution without resampling
|
||||
or downsampling.
|
||||
|
||||
By default, the selected buffer follows the current mode:
|
||||
|
||||
| Mode | Queried buffer |
|
||||
| --- | --- |
|
||||
| `"realtime"` | Active recording, or the latest completed recording when not recording. |
|
||||
| `"offline"` | Loaded processing buffer. |
|
||||
| `"playback"` | Loaded playback archive, including its recorded derived samples. |
|
||||
|
||||
Pass `source="recording"`, `"processing"`, or `"playback"` to query a specific
|
||||
buffer without changing modes. An unavailable buffer raises
|
||||
`CoreError(code="no_recording")`; an available empty buffer returns empty data.
|
||||
No history is collected outside recordings. Playback queries access the
|
||||
loaded archive, not newly recalculated live derived values.
|
||||
|
||||
Completed recordings use nanoseconds relative to their first sample. Active
|
||||
recordings still contain ingress `time.monotonic_ns()` timestamps. Imported
|
||||
archives retain their stored timestamps. Bounds and returned timestamps always
|
||||
use the selected buffer's stored time base; stopping a recording normalizes it.
|
||||
|
||||
```python
|
||||
bounds = core.get_time_range()
|
||||
series = core.get_data_range(
|
||||
[rpm_signal_id, torque_signal_id],
|
||||
start_ns=1_000_000_000,
|
||||
end_ns=2_000_000_000,
|
||||
)
|
||||
# series[rpm_signal_id] is a list of (timestamp_ns, value) pairs.
|
||||
```
|
||||
|
||||
### `get_time_range(*, source: BufferSource | None = None) -> tuple[int, int] | None`
|
||||
|
||||
Returns the minimum and maximum stored timestamps for the same buffer selection
|
||||
as `get_data_range()`. An empty buffer returns `None`; a single sample returns
|
||||
its timestamp for both bounds.
|
||||
|
||||
`DLPak` and `ValueBuffer` expose the same `get_data_range()` and
|
||||
`get_time_range()` methods without the `source` argument. Buffer mutations
|
||||
maintain per-signal indexes; out-of-order signal samples are sorted before
|
||||
querying. Once sorted, a range query uses binary search plus copying the
|
||||
selected samples, rather than scanning unrelated signals.
|
||||
|
||||
## Playback
|
||||
|
||||
|
||||
Reference in new issue
Block a user