Added range API, fixed offline processing and DLPak saving

Saving a buffer can now speicfy the buffer used.

Offline processing uses a per signal binary search to hugely reduce
processing time that was mostly spent on an O(n) linear search

API now has range fetching API's for graphs etc

docs/API.md

@@ -111,7 +111,11 @@ recording is available raises `CoreError(code="no_playback_data")`.
 Recalculates derived signals in the current processing buffer using the bound
 derive units. Call it in `"offline"` mode after `load_processing_buffer()`, with
 `CoreConfig(playback_mode="read")`. Units with source inputs run first; units
-that depend on their derived outputs run in dependency order. The operation
+that depend on their derived outputs run in dependency order. Each unit runs
+in sample order using per-signal binary searches instead of full-buffer scans.
+For other inputs it selects the nearest recorded value, preferring the earlier
+sample when distances tie. This retains the previous offline alignment policy,
+including selection of future samples. The operation
 removes previously recorded derived samples and descriptors, then merges the
 new outputs into the processing buffer. Calling it again replaces those
 outputs. It does not write an archive or start playback; use `write()` to save
@@ -172,13 +176,111 @@ offload buffer with a manifest. Errors use `"incorrect_mode"`,
 
 Returns whether a recording is active.
 
-### `write(dir: str | Path, filename: str) -> None`
+### `write(dir: str | Path, filename: str, *, source: BufferSource = "processing") -> None`
 
-Writes the current **processing** buffer as a `.dlpak` archive; the extension
-is appended when absent. After recording, select `"offline"` mode and call
-`load_processing_buffer()` before writing. If no processing buffer exists,
-this raises `CoreError(code="no_recording")`. Archive failures raise
-`DLPakError` with a stable code such as `"write_failed"`.
+Writes the selected buffer as a `.dlpak` archive; the extension is appended
+when absent. `BufferSource` is `"recording" | "processing" | "playback"`, defined
+in `dynalab_core.types`. The default remains `"processing"` for compatibility.
+Use `source="recording"` to save a completed recording directly, without
+switching modes or loading the processing buffer. Stop recording first;
+saving an active recording raises `CoreError(code="recording_active")`.
+`source="playback"` saves the loaded archive, not the current playback position
+or newly recalculated live values. Saving does not change mode or load data.
+
+An unavailable buffer raises `CoreError(code="no_recording")`. Archive failures
+raise `DLPakError(code="write_failed")`. Writes use a temporary file in the
+destination directory and replace the destination only after the archive has
+been completed successfully. The destination directory must already exist.
+
+```python
+from pathlib import Path
+
+core.start_recording()
+# Receive connector data, then finish and save it.
+core.stop_recording()
+core.write("recordings", "run-001", source="recording")
+
+core.set_mode("offline")
+core.load_processing_buffer(Path("recordings/run-001.dlpak"))
+```
+
+### DLPak format and standalone access
+
+`DLPak` in `dynalab_core.dlpak` can also be used without Core:
+
+```python
+from pathlib import Path
+from dynalab_core.dlpak import DLPak
+
+archive = DLPak().read(Path("recordings/run-001.dlpak"))
+series = archive.get_data_range(start_ns=0, end_ns=1_000_000_000)
+archive.write("recordings", "run-001-copy")
+```
+
+Archives are ZIP files containing `manifest.json` and `data.csv`. The manifest
+contains `format_version` (currently `1`), a UTC recording timestamp, signal
+descriptors, and comments. Older manifests without `format_version` are read
+as version 1; unsupported versions are rejected. Signal IDs in the manifest
+must be unique, and every signal present in the CSV must have a descriptor.
+Descriptors without samples are allowed. Invalid archives raise
+`DLPakError(code="read_failed")`, with the underlying cause retained.
+
+The CSV header is `timestamp_ns` followed by signal UUID columns. Empty cells
+represent absent samples. Rows are ordered by timestamp. Repeated timestamps
+preserve multiple values for the same signal; samples are not collapsed.
+
+## Buffer queries for plotting
+
+### `get_data_range(signal_ids: list[UUID] | None = None, start_ns: int | None = None, end_ns: int | None = None, *, source: BufferSource | None = None) -> dict[UUID, list[tuple[int, float]]]`
+
+Returns an independent snapshot grouped by signal ID. Each list contains
+`(timestamp_ns, value)` pairs in timestamp order, including duplicate
+timestamps. Both bounds are inclusive. Omit a bound for an open-ended range,
+or omit both for all samples. Omit `signal_ids` to select every signal; an
+unknown requested ID maps to an empty list. An inverted range raises
+`ValueError`. Data is returned at its original resolution without resampling
+or downsampling.
+
+By default, the selected buffer follows the current mode:
+
+| Mode | Queried buffer |
+| --- | --- |
+| `"realtime"` | Active recording, or the latest completed recording when not recording. |
+| `"offline"` | Loaded processing buffer. |
+| `"playback"` | Loaded playback archive, including its recorded derived samples. |
+
+Pass `source="recording"`, `"processing"`, or `"playback"` to query a specific
+buffer without changing modes. An unavailable buffer raises
+`CoreError(code="no_recording")`; an available empty buffer returns empty data.
+No history is collected outside recordings. Playback queries access the
+loaded archive, not newly recalculated live derived values.
+
+Completed recordings use nanoseconds relative to their first sample. Active
+recordings still contain ingress `time.monotonic_ns()` timestamps. Imported
+archives retain their stored timestamps. Bounds and returned timestamps always
+use the selected buffer's stored time base; stopping a recording normalizes it.
+
+```python
+bounds = core.get_time_range()
+series = core.get_data_range(
+    [rpm_signal_id, torque_signal_id],
+    start_ns=1_000_000_000,
+    end_ns=2_000_000_000,
+)
+# series[rpm_signal_id] is a list of (timestamp_ns, value) pairs.
+```
+
+### `get_time_range(*, source: BufferSource | None = None) -> tuple[int, int] | None`
+
+Returns the minimum and maximum stored timestamps for the same buffer selection
+as `get_data_range()`. An empty buffer returns `None`; a single sample returns
+its timestamp for both bounds.
+
+`DLPak` and `ValueBuffer` expose the same `get_data_range()` and
+`get_time_range()` methods without the `source` argument. Buffer mutations
+maintain per-signal indexes; out-of-order signal samples are sorted before
+querying. Once sorted, a range query uses binary search plus copying the
+selected samples, rather than scanning unrelated signals.
 
 ## Playback
 

src/dynalab_core/__init__.py

@@ -31,7 +31,7 @@ from dynalab_core.protocols.json.server import JsonServer
 from dynalab_core.protocols.packets import ProtocolMessage
 from dynalab_core.protocols.packets.data import ValueDescriptor
 from dynalab_core.protocols.packets.handshake import SignalDescriptor
-from dynalab_core.types import CoreMode, CoreModeState
+from dynalab_core.types import BufferSource, CoreMode, CoreModeState
 from dynalab_core.values import Value
 
 
@@ -499,13 +499,64 @@ class Core:
     def unbind_derive_unit(self, unit_id: UUID) -> None:
         self._derive_registry.unregister(unit_id)
 
-    def write(self, dir: str | Path, filename: str) -> None:
-        if self._processing_buffer is None:
+    def _get_archive(self, source: BufferSource) -> DLPak:
+        if source == "recording":
+            archive = self._offload_buffer
+        elif source == "processing":
+            archive = self._processing_buffer
+        elif source == "playback":
+            archive = self._playback_engine._data
+        else:
+            raise CoreError(f"Unknown buffer source: {source!r}", code="invalid_source")
+        if archive is None:
             raise CoreError(
-                "Cannot write recording because no recording is available",
+                f"No {source} buffer is available",
                 code="no_recording",
             )
-        self._processing_buffer.write(dir, filename)
+        return archive
+
+    def _get_value_buffer(self, source: BufferSource | None) -> ValueBuffer:
+        if source is None:
+            mode_sources: dict[CoreMode, BufferSource] = {
+                "realtime": "recording",
+                "offline": "processing",
+                "playback": "playback",
+            }
+            source = mode_sources[self._mode.value]
+        if source == "recording" and self._recording.is_set():
+            return self._recording_buffer
+        archive = self._get_archive(source)
+        if archive._data is None:
+            raise CoreError(f"The {source} buffer has no data", code="no_recording")
+        return archive._data
+
+    def get_data_range(
+        self,
+        signal_ids: list[UUID] | None = None,
+        start_ns: int | None = None,
+        end_ns: int | None = None,
+        *,
+        source: BufferSource | None = None,
+    ) -> dict[UUID, list[tuple[int, float]]]:
+        """Query the current mode's buffer, or an explicitly selected source."""
+        return self._get_value_buffer(source).get_data_range(
+            signal_ids, start_ns, end_ns
+        )
+
+    def get_time_range(
+        self, *, source: BufferSource | None = None
+    ) -> tuple[int, int] | None:
+        return self._get_value_buffer(source).get_time_range()
+
+    def write(
+        self, dir: str | Path, filename: str, *, source: BufferSource = "processing"
+    ) -> None:
+        """Save processing data by default, or explicitly save another buffer."""
+        if source == "recording" and self._recording.is_set():
+            raise CoreError(
+                "Stop recording before saving its archive", code="recording_active"
+            )
+        self._get_archive(source).write(dir, filename)
 
     def play(self) -> None:
         if self._mode.value != "playback":

src/dynalab_core/buffer.py

@@ -3,11 +3,11 @@
 # SPDX-License-Identifier: GPL-3.0-or-later
 
 from __future__ import annotations
-from bisect import bisect_left
+from bisect import bisect_left, bisect_right
 from collections import defaultdict
 import csv
 import io
-from pathlib import Path
+from itertools import groupby
 from threading import Lock
 from uuid import UUID
 
@@ -15,99 +15,166 @@ from uuid import UUID
 class ValueBuffer:
     def __init__(self) -> None:
         self._samples: list[tuple[int, UUID, float]] = []
+        self._samples_by_signal: dict[UUID, list[tuple[int, float]]] = {}
+        self._unsorted_signals: set[UUID] = set()
+        self._sorted = True
         self._lock = Lock()
 
+    def __copy__(self) -> ValueBuffer:
+        buffer = ValueBuffer()
+        buffer.merge(self)
+        return buffer
+
+    def _append_no_lock(self, timestamp: int, uuid: UUID, value: float) -> None:
+        if self._samples and timestamp < self._samples[-1][0]:
+            self._sorted = False
+        self._samples.append((timestamp, uuid, value))
+
+        signal_samples = self._samples_by_signal.setdefault(uuid, [])
+        if signal_samples and timestamp < signal_samples[-1][0]:
+            self._unsorted_signals.add(uuid)
+        signal_samples.append((timestamp, value))
+
     def append(self, timestamp: int, uuid: UUID, value: float) -> None:
         with self._lock:
-            self._samples.append((timestamp, uuid, value))
+            self._append_no_lock(timestamp, uuid, value)
+
+    def get_samples(self) -> list[tuple[int, UUID, float]]:
+        """Return an independent snapshot in the current storage order."""
+        with self._lock:
+            return self._samples.copy()
 
     def merge(self, buffer: ValueBuffer) -> None:
+        samples = buffer.get_samples()
         with self._lock:
-            self._samples += buffer._samples
+            for timestamp, uuid, value in samples:
+                self._append_no_lock(timestamp, uuid, value)
 
     def clear(self) -> None:
         with self._lock:
             self._samples.clear()
+            self._samples_by_signal.clear()
+            self._unsorted_signals.clear()
+            self._sorted = True
 
     def clear_uuid(self, uuid: UUID) -> None:
         with self._lock:
             self._samples = [sample for sample in self._samples if sample[1] != uuid]
+            self._samples_by_signal.pop(uuid, None)
+            self._unsorted_signals.discard(uuid)
 
     def get_signal_ids(self) -> list[UUID]:
         with self._lock:
-            return list({signal_id for _, signal_id, _ in self._samples})
+            return list(self._samples_by_signal)
+
+    def _get_signal_samples_no_lock(self, uuid: UUID) -> list[tuple[int, float]]:
+        samples = self._samples_by_signal.get(uuid, [])
+        if uuid in self._unsorted_signals:
+            samples.sort(key=lambda sample: sample[0])
+            self._unsorted_signals.remove(uuid)
+        return samples
 
     def get_nearest_neighbor_no_lock(self, uuid: UUID, timestamp: int) -> float | None:
-        previous: tuple[int, UUID, float] | None = None
+        """Find the nearest sample, preferring the earlier one on a tie.
 
-        for sample in self._samples:
-            if sample[1] != uuid:
-                continue
+        The caller must prevent concurrent buffer mutations. Offline processing
+        uses this method after sorting its private working buffer.
+        """
+        samples = self._get_signal_samples_no_lock(uuid)
+        if not samples:
+            return None
 
-            if sample[0] >= timestamp:
-                if previous is None:
-                    return sample[2]
+        index = bisect_left(samples, timestamp, key=lambda sample: sample[0])
+        if index == 0:
+            return samples[0][1]
+        if index == len(samples):
+            return samples[-1][1]
 
-                d1 = abs(sample[0] - timestamp)
-                d2 = abs(previous[0] - timestamp)
-
-                if d1 < d2:
-                    return sample[2]
-                else:
-                    return previous[2]
-
-            previous = sample
-
-        if previous is not None:
-            return previous[2]
-
-        return None
+        previous = samples[index - 1]
+        following = samples[index]
+        if following[0] - timestamp < timestamp - previous[0]:
+            return following[1]
+        return previous[1]
 
     def get_nearest_neighbor(self, uuid: UUID, timestamp: int) -> float | None:
-        previous: tuple[int, UUID, float] | None = None
+        with self._lock:
+            return self.get_nearest_neighbor_no_lock(uuid, timestamp)
+
+    def get_data_range(
+        self,
+        signal_ids: list[UUID] | None = None,
+        start_ns: int | None = None,
+        end_ns: int | None = None,
+    ) -> dict[UUID, list[tuple[int, float]]]:
+        """Return sorted (timestamp, value) snapshots within inclusive bounds.
+
+        Omitted bounds cover all available data; omitted IDs select all signals.
+        Unknown signals are returned with an empty list. Values are not resampled.
+        """
+        if start_ns is not None and end_ns is not None and start_ns > end_ns:
+            raise ValueError("start_ns must be less than or equal to end_ns")
 
         with self._lock:
-            for sample in self._samples:
-                if sample[1] != uuid:
-                    continue
+            selected_ids = (
+                signal_ids if signal_ids is not None else self._samples_by_signal
+            )
+            result: dict[UUID, list[tuple[int, float]]] = {}
+            for uuid in selected_ids:
+                samples = self._get_signal_samples_no_lock(uuid)
+                start = (
+                    0
+                    if start_ns is None
+                    else bisect_left(samples, start_ns, key=lambda sample: sample[0])
+                )
+                end = (
+                    len(samples)
+                    if end_ns is None
+                    else bisect_right(samples, end_ns, key=lambda sample: sample[0])
+                )
+                result[uuid] = samples[start:end]
+            return result
 
-                if sample[0] >= timestamp:
-                    if previous is None:
-                        return sample[2]
-
-                    d1 = abs(sample[0] - timestamp)
-                    d2 = abs(previous[0] - timestamp)
-
-                    if d1 < d2:
-                        return sample[2]
-                    else:
-                        return previous[2]
-
-                previous = sample
-
-            if previous is not None:
-                return previous[2]
-
-            return None
+    def get_time_range(self) -> tuple[int, int] | None:
+        """Return the earliest and latest timestamps, or None for an empty buffer."""
+        with self._lock:
+            if not self._samples:
+                return None
+            first: list[int] = []
+            last: list[int] = []
+            for uuid in self._samples_by_signal:
+                samples = self._get_signal_samples_no_lock(uuid)
+                first.append(samples[0][0])
+                last.append(samples[-1][0])
+            return min(first), max(last)
 
     def __len__(self) -> int:
         with self._lock:
             return len(self._samples)
 
     def normalize(self) -> None:
-        if len(self) == 0:
-            return
         with self._lock:
+            if not self._samples:
+                return
             minimum = min(sample[0] for sample in self._samples)
-
             self._samples = [
                 (timestamp - minimum, signal_id, value)
                 for timestamp, signal_id, value in self._samples
             ]
+            self._samples_by_signal = {
+                uuid: [(timestamp - minimum, value) for timestamp, value in samples]
+                for uuid, samples in self._samples_by_signal.items()
+            }
+
+    def _sort_no_lock(self) -> None:
+        if not self._sorted:
+            self._samples.sort(key=lambda sample: sample[0])
+            self._sorted = True
+        for uuid in list(self._unsorted_signals):
+            self._get_signal_samples_no_lock(uuid)
 
     def sort(self) -> None:
         with self._lock:
-            self._samples.sort(key=lambda sample: sample[0])
+            self._sort_no_lock()
 
     def import_csv(self, input: str) -> None:
         reader = csv.reader(io.StringIO(input, newline=""))
@@ -163,49 +230,43 @@ class ValueBuffer:
         imported_samples.sort(key=lambda sample: sample[0])
 
         with self._lock:
-            self._samples = imported_samples
+            self._samples = []
+            self._samples_by_signal = {}
+            self._unsorted_signals.clear()
+            self._sorted = True
+            for timestamp, uuid, value in imported_samples:
+                self._append_no_lock(timestamp, uuid, value)
 
     def export_csv(self) -> str:
-        with self._lock:
-            samples = list(self._samples)
-
-        rows: dict[int, dict[UUID, float]] = defaultdict(dict)
-        signal_ids: set[UUID] = set()
-
-        for timestamp_ns, signal_id, value in samples:
-            rows[timestamp_ns][signal_id] = value
-            signal_ids.add(signal_id)
-
-        ordered_signal_ids = sorted(signal_ids, key=str)
-
+        samples = self.get_samples()
+        samples.sort(key=lambda sample: sample[0])
+        signal_ids = sorted({sample[1] for sample in samples}, key=str)
         output = io.StringIO(newline="")
-
         writer = csv.writer(output)
+        writer.writerow(["timestamp_ns", *(str(uuid) for uuid in signal_ids)])
 
-        writer.writerow(
-            ["timestamp_ns", *(str(signal_id) for signal_id in ordered_signal_ids)]
-        )
+        for timestamp, group in groupby(samples, key=lambda sample: sample[0]):
+            values: dict[UUID, list[float]] = defaultdict(list)
+            for _, uuid, value in group:
+                values[uuid].append(value)
 
-        for timestamp_ns in sorted(rows):
-            timestamp_values = rows[timestamp_ns]
-
-            writer.writerow(
-                [
-                    timestamp_ns,
-                    *(
-                        timestamp_values.get(signal_id, "")
-                        for signal_id in ordered_signal_ids
-                    ),
-                ]
-            )
+            # Repeat a timestamp when a signal has multiple samples there.
+            # This preserves duplicates without changing the existing CSV layout.
+            for index in range(max(len(items) for items in values.values())):
+                writer.writerow(
+                    [
+                        timestamp,
+                        *(
+                            values[uuid][index] if index < len(values[uuid]) else ""
+                            for uuid in signal_ids
+                        ),
+                    ]
+                )
 
         return output.getvalue()
 
     def get_samples_from(self, timestamp: int) -> list[tuple[int, UUID, float]]:
         with self._lock:
-            index = bisect_left(
-                self._samples,
-                timestamp,
-                key=lambda sample: sample[0],
-            )
-            return self._samples[index:].copy()
+            self._sort_no_lock()
+            index = bisect_left(self._samples, timestamp, key=lambda sample: sample[0])
+            return self._samples[index:]

src/dynalab_core/derive.py

@@ -3,8 +3,6 @@
 # SPDX-License-Identifier: GPL-3.0-or-later
 
 from collections.abc import Callable
-from concurrent.futures import ThreadPoolExecutor
-from copy import copy
 from datetime import datetime, timezone
 from enum import IntEnum
 import inspect
@@ -202,64 +200,27 @@ class DeriveUnit:
     def is_running(self) -> bool:
         return not self._stopped_event.is_set()
 
-    def _process_offline_worker(
-        self, buffer: ValueBuffer, start: int, end: int
-    ) -> ValueBuffer:
-        samples = buffer._samples.copy()
-        output_buffer: ValueBuffer = ValueBuffer()
-        for sample in samples[start:end]:
-            if sample[1] in [signal.id for signal in self._input_signals]:
-                input_args: list[ValueDescriptor] = []
-                for signal in self._input_signals:
-                    if signal.id == sample[1]:
-                        input_args.append(
-                            ValueDescriptor(sample[1], sample[2], sample[0])
-                        )
-                    else:
-                        input_args.append(
-                            ValueDescriptor(
-                                signal.id,
-                                buffer.get_nearest_neighbor_no_lock(
-                                    signal.id, sample[0]
-                                ),
-                                sample[0],
-                            )
-                        )
-                result = self._process_function(*input_args, self._return_signal)
-                output_buffer.append(
-                    result.timestamp, self._return_signal.id, result.value
-                )
-        return output_buffer
-
     def process_offline(self, buffer: ValueBuffer) -> ValueBuffer:
-        num_samples = len(buffer._samples)
-        num_chunks = 16
-        chunk_size = num_samples // num_chunks
+        # Sort once before the pass so every input lookup is a binary search.
+        buffer.sort()
+        input_ids = {signal.id for signal in self._input_signals}
+        output_buffer = ValueBuffer()
+        for timestamp, signal_id, value in buffer.get_samples():
+            if signal_id not in input_ids:
+                continue
 
-        output_buffer: ValueBuffer = ValueBuffer()
-        with ThreadPoolExecutor(max_workers=num_chunks) as executor:
-            ranges: list[tuple[int, int]] = []
-
-            for i in range(num_chunks):
-                start = i * chunk_size
-
-                if i == num_chunks - 1:
-                    end = num_samples
+            input_args: list[ValueDescriptor] = []
+            for signal in self._input_signals:
+                if signal.id == signal_id:
+                    input_value = value
                 else:
-                    end = (i + 1) * chunk_size
-
-                ranges.append((start, end))
-
-            futures = [
-                executor.submit(self._process_offline_worker, buffer, *range)
-                for range in ranges
-            ]
-
-            results = [future.result() for future in futures]
-
-            for result in results:
-                output_buffer.merge(result)
+                    input_value = buffer.get_nearest_neighbor_no_lock(
+                        signal.id, timestamp
+                    )
+                input_args.append(ValueDescriptor(signal.id, input_value, timestamp))
 
+            result = self._process_function(*input_args, self._return_signal)
+            output_buffer.append(result.timestamp, self._return_signal.id, result.value)
         return output_buffer
 
 

src/dynalab_core/dlpak.py

@@ -6,8 +6,11 @@ from __future__ import annotations
 from copy import copy
 from datetime import datetime, timezone
 import logging
+import os
 from pathlib import Path
+from tempfile import NamedTemporaryFile
 from typing import Literal, Self
+from uuid import UUID
 from zipfile import ZIP_DEFLATED, ZipFile
 
 from pydantic import BaseModel, Field
@@ -23,6 +26,7 @@ log = logging.getLogger(__name__)
 
 
 class RecordManifest(BaseModel):
+    format_version: Literal[1] = 1
     timestamp: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
     signals: list[JsonSignalDescriptor] = Field(default_factory=list)
     comments: str = ""
@@ -108,6 +112,36 @@ class DLPak:
     def get_all_signal_descriptors(self) -> list[SignalDescriptor]:
         return copy([signal.to_pure() for signal in self._manifest.signals])
 
+    def get_data_range(
+        self,
+        signal_ids: list[UUID] | None = None,
+        start_ns: int | None = None,
+        end_ns: int | None = None,
+    ) -> dict[UUID, list[tuple[int, float]]]:
+        if self._data is None:
+            raise DLPakError("No data is available", code="no_data")
+        return self._data.get_data_range(signal_ids, start_ns, end_ns)
+
+    def get_time_range(self) -> tuple[int, int] | None:
+        if self._data is None:
+            raise DLPakError("No data is available", code="no_data")
+        return self._data.get_time_range()
+
+    @staticmethod
+    def _validate_contents(data: ValueBuffer, manifest: RecordManifest) -> None:
+        signal_ids = [signal.id for signal in manifest.signals]
+        if len(signal_ids) != len(set(signal_ids)):
+            raise DLPakError(
+                "The manifest contains duplicate signal IDs", code="invalid_manifest"
+            )
+        missing = set(data.get_signal_ids()) - set(signal_ids)
+        if missing:
+            raise DLPakError(
+                "The manifest is missing descriptors for signals: "
+                + ", ".join(sorted(str(uuid) for uuid in missing)),
+                code="invalid_manifest",
+            )
+
     def read(self, file: str | Path) -> Self:
         input_path = Path(file)
 
@@ -118,6 +152,7 @@ class DLPak:
                 )
                 data = ValueBuffer()
                 data.import_csv(archive.read("data.csv").decode("utf-8"))
+                self._validate_contents(data, manifest)
         except Exception as error:
             log.exception(
                 "Failed to read DLPak archive from %s",
@@ -174,12 +209,26 @@ class DLPak:
 
         output_path = output_dir / file_path
 
+        temporary_path: Path | None = None
         try:
-            with ZipFile(output_path, mode="w", compression=ZIP_DEFLATED) as archive:
-                archive.writestr(
-                    "manifest.json", self._manifest.model_dump_json(indent=2)
-                )
-                archive.writestr("data.csv", self._data.export_csv())
+            self._validate_contents(self._data, self._manifest)
+            with NamedTemporaryFile(
+                dir=output_path.parent,
+                prefix=f".{output_path.name}.",
+                suffix=".tmp",
+                delete=False,
+            ) as temporary_file:
+                temporary_path = Path(temporary_file.name)
+                with ZipFile(
+                    temporary_file, mode="w", compression=ZIP_DEFLATED
+                ) as archive:
+                    archive.writestr(
+                        "manifest.json", self._manifest.model_dump_json(indent=2)
+                    )
+                    archive.writestr("data.csv", self._data.export_csv())
+                temporary_file.flush()
+                os.fsync(temporary_file.fileno())
+            temporary_path.replace(output_path)
         except Exception as error:
             log.exception(
                 "Failed to write DLPak archive to %s",
@@ -194,6 +243,9 @@ class DLPak:
                 f"Could not write DLPak archive {output_path}: {error}",
                 code="write_failed",
             ) from error
+        finally:
+            if temporary_path is not None:
+                temporary_path.unlink(missing_ok=True)
 
         log.info(
             "Wrote DLPak archive to %s",

src/dynalab_core/types.py

@@ -8,6 +8,7 @@ from typing import Literal
 
 
 CoreMode = Literal["realtime", "playback", "offline"]
+BufferSource = Literal["recording", "processing", "playback"]
 
 
 @dataclass