Initial commit

ARCHITECTURE.md

@@ -0,0 +1,719 @@
+# dpg-map Architecture — Locked Rebuild
+
+This document defines the internal architecture for the rebuild.
+
+The central rule is:
+
+> Public API calls update logical state or enqueue commands. The GUI thread alone owns Dear PyGui items, draw commands, textures, handlers, and frame rendering.
+
+This prevents flicker, dragging failures, random recentering, and Dear PyGui thread-safety issues when overlays are updated from telemetry threads.
+
+## High-level architecture
+
+```text
+User API
+  ↓
+Thread-safe command/logical-state layer
+  ↓
+Map registry and MapState
+  ↓
+Renderer frame pump on GUI thread
+  ↓
+Dear PyGui child_window + drawlist + textures
+```
+
+Detailed flow:
+
+```text
+Telemetry thread / app thread / GUI callback
+  ↓
+public dpg_map function
+  ↓
+validate lightweight inputs
+  ↓
+acquire MapState.lock briefly
+  ↓
+update logical model and/or enqueue command
+  ↓
+mark specific dirty flags
+  ↓
+return immediately
+  ↓
+GUI frame callback drains commands
+  ↓
+renderer updates DPG draw layers/textures
+```
+
+## Critical invariants
+
+### Threading invariants
+
+1. Dear PyGui APIs are called only on the GUI thread.
+2. Public API functions do not call Dear PyGui directly unless explicitly documented as GUI-thread-only.
+3. Tile worker threads do network and disk work only.
+4. Tile workers do not create or delete Dear PyGui textures.
+5. Texture creation and deletion are queued to the renderer and executed on the GUI thread.
+6. Runtime overlay updates are coalesced by overlay tag.
+7. Locks are held briefly and never during network, disk, image decoding, or Dear PyGui calls.
+
+### View invariants
+
+1. `MapState.center` and `MapState.zoom` are changed only by view commands or interaction.
+2. Overlay commands must never change center or zoom.
+3. Tile loading must never change center or zoom.
+4. Renderer redraw must render the current state; it must not infer a new center from overlays.
+5. Provider switching may clamp zoom, but must not recenter unless the current center is invalid for the projection.
+
+### Rendering invariants
+
+1. Overlay redraw does not clear tile draw commands.
+2. Tile redraw does not delete overlay logical state.
+3. Provider switch clears only provider-specific tile resources.
+4. Full redraw is reserved for initial render, size change, provider change, or recovery.
+5. The renderer always reads a consistent snapshot of state for each frame.
+6. DPG draw item tags are internal and not exposed as public overlay tags.
+
+### Sizing invariants
+
+1. The child window receives the requested Dear PyGui sizing intent.
+2. The drawlist receives concrete measured dimensions.
+3. Requested size and measured size are stored separately.
+4. A measured size of zero while hidden must not permanently collapse the map.
+5. The map should delay tile loading until a non-zero visible size is measured, unless explicitly configured otherwise.
+6. Resize triggers tile and overlay redraw.
+
+## Package layout
+
+```text
+src/dpg_map/
+  __init__.py
+  api.py
+  widget.py
+  state.py
+  commands.py
+  renderer.py
+  draw_layers.py
+  interaction.py
+  overlays.py
+  providers.py
+  projection.py
+  tiles.py
+  cache.py
+  sizing.py
+  diagnostics.py
+  types.py
+  exceptions.py
+examples/
+  basic_map.py
+  sizing_window.py
+  sizing_child.py
+  sizing_table.py
+  hidden_tab.py
+  markers_live_thread.py
+  trajectory_live_thread.py
+  custom_provider.py
+  cache_stress.py
+tests/
+```
+
+## Module responsibilities
+
+### `__init__.py`
+
+Exports the public API only.
+
+No heavy initialization should happen here except registering the default provider.
+
+### `api.py`
+
+Contains thin public API wrappers, especially for runtime calls.
+
+Responsibilities:
+
+- resolve map/overlay tags
+- validate simple arguments
+- enqueue commands
+- expose stable public functions
+
+The public API should call into `state.py`, `commands.py`, and `overlays.py`, but should not know renderer internals.
+
+### `commands.py`
+
+Defines commands that can be submitted from any thread.
+
+Suggested command model:
+
+```python
+class CommandKind(Enum):
+    SET_VIEW = "set_view"
+    SET_PROVIDER = "set_provider"
+    ADD_OVERLAY = "add_overlay"
+    UPDATE_OVERLAY = "update_overlay"
+    DELETE_OVERLAY = "delete_overlay"
+    SET_LAYER_VISIBILITY = "set_layer_visibility"
+    CLEAR_LAYER = "clear_layer"
+    CLEAR_MAP = "clear_map"
+    CLEAR_MEMORY_CACHE = "clear_memory_cache"
+    CLEAR_DISK_CACHE = "clear_disk_cache"
+```
+
+Command dataclass:
+
+```python
+@dataclass
+class MapCommand:
+    kind: CommandKind
+    map_tag: str | int
+    payload: dict[str, Any]
+    created_at: float
+```
+
+Coalescing rules:
+
+- `UPDATE_OVERLAY` commands coalesce by `(map_tag, overlay_tag)`.
+- repeated `SET_VIEW` commands may coalesce, keeping the newest.
+- `ADD_OVERLAY`, `DELETE_OVERLAY`, `CLEAR_LAYER`, `CLEAR_MAP`, and `SET_PROVIDER` must preserve order.
+- provider switches form a barrier; older tile results for the previous provider must be ignored.
+
+### `state.py`
+
+Owns configuration, registries, and `MapState`.
+
+Global config:
+
+```python
+@dataclass
+class DpgMapConfig:
+    user_agent: str | None = None
+    cache_dir: Path | None = None
+    default_provider: str | TileProvider = "osm"
+    memory_cache_max_tiles: int = 512
+    disk_cache_max_bytes: int | None = 2_000_000_000
+    prefetch_margin_tiles: int = 1
+    tile_worker_count: int = 4
+    overlay_update_policy: str = "coalesce"
+    debug: bool = False
+```
+
+MapState:
+
+```python
+@dataclass
+class MapState:
+    tag: str | int
+
+    # DPG implementation tags, internal only
+    child_window_tag: str | int
+    drawlist_tag: str | int
+    texture_registry_tag: str | int
+    handler_registry_tag: str | int
+
+    # sizing
+    requested_width: int
+    requested_height: int
+    requested_autosize_x: bool
+    requested_autosize_y: bool
+    measured_width: int = 0
+    measured_height: int = 0
+    last_nonzero_width: int = 0
+    last_nonzero_height: int = 0
+    is_visible: bool = False
+
+    # view
+    center: LatLon = (0.0, 0.0)
+    zoom: int = 2
+    min_zoom: int = 0
+    max_zoom: int = 19
+    provider: TileProvider = field(default_factory=get_default_provider)
+
+    # models
+    overlays: dict[str | int, Overlay] = field(default_factory=dict)
+    layers: dict[str, LayerState] = field(default_factory=default_layers)
+
+    # subsystems
+    command_queue: MapCommandQueue = field(default_factory=MapCommandQueue)
+    tile_manager: TileManager = field(default_factory=TileManager)
+    renderer: MapRenderer | None = None
+    interaction: InteractionState = field(default_factory=InteractionState)
+
+    # concurrency
+    lock: RLock = field(default_factory=RLock)
+
+    # redraw tracking
+    dirty: DirtyFlags = DirtyFlags.FULL
+    frame_scheduled: bool = False
+    generation: int = 0
+```
+
+`generation` increments on provider switch, full clear, or other operations that invalidate pending tile results.
+
+### `types.py`
+
+Contains aliases and small dataclasses:
+
+```python
+LatLon = tuple[float, float]
+RGBA = tuple[int, int, int, int]
+ScreenPoint = tuple[float, float]
+```
+
+### `exceptions.py`
+
+Contains public exceptions:
+
+```python
+class DpgMapError(Exception): ...
+class ProviderError(DpgMapError): ...
+class UnknownProviderError(ProviderError): ...
+class MapNotFoundError(DpgMapError): ...
+class OverlayNotFoundError(DpgMapError): ...
+class CoordinateError(DpgMapError): ...
+class ThreadingError(DpgMapError): ...
+class CacheError(DpgMapError): ...
+```
+
+### `providers.py`
+
+Defines the provider system.
+
+The default OpenStreetMap provider is registered at import time.
+
+Provider identity is part of `TileID`, so caches are provider-namespaced.
+
+### `projection.py`
+
+Pure Web Mercator math only.
+
+No Dear PyGui imports.
+
+No global state.
+
+Must include tests.
+
+### `overlays.py`
+
+Defines logical overlay dataclasses only.
+
+Overlay objects are not Dear PyGui draw items.
+
+```python
+@dataclass
+class Overlay:
+    tag: str | int
+    map_tag: str | int
+    layer: str
+    show: bool = True
+    user_data: Any = None
+    revision: int = 0
+```
+
+`revision` increments when an overlay changes. The renderer can use this to decide whether overlay draw data needs rebuilding.
+
+Marker, Polyline, and Trajectory inherit from `Overlay`.
+
+Path overlays should copy incoming coordinate sequences into internal tuples/lists during command processing, not store references to mutable user lists. This prevents background buffers changing while the renderer reads them.
+
+### `widget.py`
+
+Creates the Dear PyGui structure.
+
+Internal structure:
+
+```text
+child_window
+  drawlist
+```
+
+Rules:
+
+- child window receives the user-requested size.
+- drawlist uses measured concrete content size.
+- map is registered before entering the context body.
+- current map stack is pushed for context-style overlay creation.
+- frame callback is scheduled after creation to measure and render.
+- resize/visibility handlers mark `SIZE_DIRTY`.
+
+The widget must not expose `child_window_tag` or `drawlist_tag`.
+
+### `sizing.py`
+
+Dedicated sizing helper module.
+
+Responsibilities:
+
+- measure child content region
+- maintain last non-zero measured size
+- decide whether tile loading can begin
+- resize drawlist safely
+- detect hidden-to-visible transitions
+- avoid permanent collapse from zero-size measurements
+
+Suggested function:
+
+```python
+def update_measured_size(state: MapState) -> bool:
+    """Return True if concrete drawlist size changed."""
+```
+
+### `interaction.py`
+
+Owns panning and zooming.
+
+Responsibilities:
+
+- hit-test against concrete drawlist rectangle
+- track active drag state
+- update center during drag
+- update zoom during wheel
+- zoom around cursor where practical
+- never allow overlay updates to reset interaction state
+
+Interaction should use the current measured drawlist rect, not only `dpg.is_item_hovered`.
+
+### `tiles.py`
+
+Owns tile identity, tile lifecycle, and visible tile calculation.
+
+TileID:
+
+```python
+@dataclass(frozen=True)
+class TileID:
+    provider_name: str
+    z: int
+    x: int
+    y: int
+```
+
+Tile states:
+
+```python
+class TileStatus(Enum):
+    MISSING = "missing"
+    QUEUED = "queued"
+    LOADING = "loading"
+    READY = "ready"
+    FAILED = "failed"
+```
+
+Tile manager responsibilities:
+
+- compute visible tile IDs
+- queue missing tiles
+- avoid duplicate requests
+- ignore stale tile results from old generations/providers
+- manage memory cache metadata
+- request texture creation on GUI thread
+- request texture deletion on GUI thread
+- protect currently visible tiles from eviction
+
+Tile workers:
+
+- check disk cache
+- fetch network if needed
+- decode image bytes to RGBA-compatible data if practical
+- return loaded data to GUI queue
+- never call DPG APIs
+
+### `cache.py`
+
+Owns persistent disk cache and memory cache policy.
+
+Disk path:
+
+```text
+{cache_dir}/{provider_name}/{z}/{x}/{y}.{ext}
+```
+
+Metadata path:
+
+```text
+{cache_dir}/{provider_name}/{z}/{x}/{y}.json
+```
+
+Metadata should support:
+
+```json
+{
+  "url": "...",
+  "etag": "...",
+  "last_modified": "...",
+  "expires": "...",
+  "downloaded_at": 0,
+  "last_accessed_at": 0,
+  "size_bytes": 0
+}
+```
+
+MVP disk cache behaviour:
+
+- file exists and is readable: use it
+- file missing: fetch it
+- update `last_accessed_at`
+- prune by LRU when above `disk_cache_max_bytes`
+- pruning must not delete files currently loading or currently visible
+
+HTTP-aware cache headers can be added after the stable MVP.
+
+### `renderer.py`
+
+The renderer is GUI-thread-only.
+
+Responsibilities:
+
+- drain command queue
+- apply commands to logical state
+- process tile worker results
+- create/delete textures
+- measure size
+- compute visible tiles
+- redraw dirty layers
+- draw attribution/debug information
+
+The renderer must support separate draw layers. Dear PyGui drawlist does not provide retained sublayers in the same way as a scene graph, so implementation may use internal groups/tags or controlled delete/rebuild by tag prefix. The important rule is behavioural:
+
+- overlay update must not clear tile textures or tile logical state
+- tile update must not modify overlay logical state
+- full redraw must not reset view state
+
+### `draw_layers.py`
+
+Optional helper module for draw item bookkeeping.
+
+Suggested structure:
+
+```python
+@dataclass
+class DrawLayer:
+    name: str
+    item_tags: set[str | int] = field(default_factory=set)
+```
+
+Renderer can delete and rebuild only a selected layer.
+
+### `diagnostics.py`
+
+Provides debug state for examples and troubleshooting.
+
+```python
+def get_map_debug_state(map_tag) -> dict[str, Any]:
+    ...
+```
+
+Should expose:
+
+- center
+- zoom
+- measured size
+- requested size
+- visible tile count
+- queued/loading/failed tile counts
+- memory cache count
+- disk cache bytes
+- overlay count
+- dirty flags
+- active drag state
+- pending command count
+- generation
+
+## Rendering frame lifecycle
+
+Each frame callback should roughly do:
+
+```text
+for each map:
+  1. update measured size
+  2. drain pending public commands
+  3. apply interaction input
+  4. process tile results
+  5. compute visible tiles if view/size/provider changed
+  6. queue missing visible tiles
+  7. create/delete textures requested by tile manager
+  8. redraw required layers based on dirty flags
+  9. clear dirty flags
+  10. schedule next frame only if work remains or map needs continuous interaction polling
+```
+
+## Dirty flag model
+
+Use bit flags:
+
+```python
+class DirtyFlags(IntFlag):
+    NONE = 0
+    VIEW = auto()
+    TILES = auto()
+    OVERLAYS = auto()
+    SIZE = auto()
+    PROVIDER = auto()
+    ATTRIBUTION = auto()
+    DEBUG = auto()
+    FULL = VIEW | TILES | OVERLAYS | SIZE | PROVIDER | ATTRIBUTION | DEBUG
+```
+
+Expected usage:
+
+| Operation | Dirty flags |
+|---|---|
+| marker update | `OVERLAYS` |
+| trajectory update | `OVERLAYS` |
+| layer visibility | `OVERLAYS` |
+| pan | `VIEW | TILES | OVERLAYS` |
+| zoom | `VIEW | TILES | OVERLAYS` |
+| resize | `SIZE | TILES | OVERLAYS` |
+| provider switch | `PROVIDER | TILES | ATTRIBUTION` |
+| cache tile ready | `TILES` |
+| debug setting changed | `DEBUG` |
+
+## Command queue and coalescing
+
+The command queue must prevent flooding during telemetry.
+
+Recommended approach:
+
+```python
+class MapCommandQueue:
+    ordered: deque[MapCommand]
+    overlay_updates: dict[tuple[map_tag, overlay_tag], MapCommand]
+    view_update: MapCommand | None
+    lock: Lock
+```
+
+When draining:
+
+1. preserve ordered structural commands
+2. apply newest coalesced view command
+3. apply newest overlay updates
+4. mark dirty flags once
+
+Structural commands:
+
+- add overlay
+- delete overlay
+- clear layer
+- clear map
+- provider switch
+
+Coalesced commands:
+
+- update marker
+- update polyline
+- update trajectory
+- set overlay show
+- set center/zoom/view
+
+## Tile result generation safety
+
+Every tile request should include:
+
+```python
+provider_name
+generation
+tile_id
+```
+
+When result returns:
+
+- if generation no longer matches map generation: discard result
+- if provider no longer matches: discard result
+- if tile no longer requested/visible/cacheable: optionally keep disk cache but do not create texture immediately
+
+This prevents stale workers from drawing old-provider tiles after a provider switch.
+
+## Persistent disk cache pruning
+
+Disk cache pruning should run:
+
+- at startup/open map
+- after downloads push cache above the limit
+- optionally manually via `clear_disk_cache`
+
+Pruning rules:
+
+1. calculate provider or global cache size
+2. sort cached tile metadata by `last_accessed_at`
+3. delete oldest until under target
+4. never delete files currently visible/loading
+5. tolerate missing/corrupt metadata
+
+## Sizing strategy
+
+The map widget uses two sizes:
+
+```python
+requested_width / requested_height
+measured_width / measured_height
+```
+
+The child window is created with requested values:
+
+```python
+dpg.add_child_window(width=requested_width, height=requested_height, ...)
+```
+
+The drawlist is resized to measured content size:
+
+```python
+content_width = dpg.get_item_width(child_window_tag)
+content_height = dpg.get_item_height(child_window_tag)
+dpg.configure_item(drawlist_tag, width=content_width, height=content_height)
+```
+
+If measured size is zero:
+
+- keep last non-zero size
+- do not permanently configure drawlist to zero
+- if no last non-zero size exists, render placeholder only and delay tile loading
+
+## Initial setup instructions
+
+These replace the old Step 0.
+
+```bash
+mkdir dpg-map
+cd dpg-map
+git init
+uv init --package dpg-map
+uv add dearpygui pillow platformdirs requests
+uv add --dev pytest ruff pyright
+```
+
+Create:
+
+```text
+src/dpg_map/
+examples/
+tests/
+FEATURES.md
+ARCHITECTURE.md
+STEPS.md
+AGENTS.md
+README.md
+```
+
+Use `AGENTS.md` as the rolling implementation log. After every step:
+
+```bash
+uv run pytest
+uv run ruff check .
+uv run ruff format --check .
+git status
+git add .
+git commit -m "step N: short description"
+```
+
+## Development examples required
+
+The rebuild must include examples specifically designed to reproduce the first build's failure modes:
+
+```text
+examples/basic_map.py
+examples/sizing_window.py
+examples/sizing_child.py
+examples/sizing_table.py
+examples/hidden_tab.py
+examples/markers_live_thread.py
+examples/trajectory_live_thread.py
+examples/custom_provider.py
+examples/cache_stress.py
+```