dpg-map/ARCHITECTURE.md

# dpg-map Architecture — Locked Rebuild

This document defines the internal architecture for the rebuild.

The central rule is:

> Public API calls update logical state or enqueue commands. The GUI thread alone owns Dear PyGui items, draw commands, textures, handlers, and frame rendering.

This prevents flicker, dragging failures, random recentering, and Dear PyGui thread-safety issues when overlays are updated from telemetry threads.

## High-level architecture

```text
User API
  ↓
Thread-safe command/logical-state layer
  ↓
Map registry and MapState
  ↓
Renderer frame pump on GUI thread
  ↓
Dear PyGui child_window + drawlist + textures
```

Detailed flow:

```text
Telemetry thread / app thread / GUI callback
  ↓
public dpg_map function
  ↓
validate lightweight inputs
  ↓
acquire MapState.lock briefly
  ↓
update logical model and/or enqueue command
  ↓
mark specific dirty flags
  ↓
return immediately
  ↓
GUI frame callback drains commands
  ↓
renderer updates DPG draw layers/textures
```

## Critical invariants

### Threading invariants

1. Dear PyGui APIs are called only on the GUI thread.
2. Public API functions do not call Dear PyGui directly unless explicitly documented as GUI-thread-only.
3. Tile worker threads do network and disk work only.
4. Tile workers do not create or delete Dear PyGui textures.
5. Texture creation and deletion are queued to the renderer and executed on the GUI thread.
6. Runtime overlay updates are coalesced by overlay tag.
7. Locks are held briefly and never during network, disk, image decoding, or Dear PyGui calls.

### View invariants

1. `MapState.center` and `MapState.zoom` are changed only by view commands or interaction.
2. Overlay commands must never change center or zoom.
3. Tile loading must never change center or zoom.
4. Renderer redraw must render the current state; it must not infer a new center from overlays.
5. Provider switching may clamp zoom, but must not recenter unless the current center is invalid for the projection.

### Rendering invariants

1. Overlay redraw does not clear tile draw commands.
2. Tile redraw does not delete overlay logical state.
3. Provider switch clears only provider-specific tile resources.
4. Full redraw is reserved for initial render, size change, provider change, or recovery.
5. The renderer always reads a consistent snapshot of state for each frame.
6. DPG draw item tags are internal and not exposed as public overlay tags.

### Sizing invariants

1. The child window receives the requested Dear PyGui sizing intent.
2. The drawlist receives concrete measured dimensions.
3. Requested size and measured size are stored separately.
4. A measured size of zero while hidden must not permanently collapse the map.
5. The map should delay tile loading until a non-zero visible size is measured, unless explicitly configured otherwise.
6. Resize triggers tile and overlay redraw.

## Package layout

```text
src/dpg_map/
  __init__.py
  api.py
  widget.py
  state.py
  commands.py
  renderer.py
  draw_layers.py
  interaction.py
  overlays.py
  providers.py
  projection.py
  tiles.py
  cache.py
  sizing.py
  diagnostics.py
  types.py
  exceptions.py
examples/
  basic_map.py
  sizing_window.py
  sizing_child.py
  sizing_table.py
  hidden_tab.py
  markers_live_thread.py
  trajectory_live_thread.py
  custom_provider.py
  cache_stress.py
tests/
```

## Module responsibilities

### `__init__.py`

Exports the public API only.

No heavy initialization should happen here except registering the default provider.

### `api.py`

Contains thin public API wrappers, especially for runtime calls.

Responsibilities:

- resolve map/overlay tags
- validate simple arguments
- enqueue commands
- expose stable public functions

The public API should call into `state.py`, `commands.py`, and `overlays.py`, but should not know renderer internals.

### `commands.py`

Defines commands that can be submitted from any thread.

Suggested command model:

```python
class CommandKind(Enum):
    SET_VIEW = "set_view"
    SET_PROVIDER = "set_provider"
    ADD_OVERLAY = "add_overlay"
    UPDATE_OVERLAY = "update_overlay"
    DELETE_OVERLAY = "delete_overlay"
    SET_LAYER_VISIBILITY = "set_layer_visibility"
    CLEAR_LAYER = "clear_layer"
    CLEAR_MAP = "clear_map"
    CLEAR_MEMORY_CACHE = "clear_memory_cache"
    CLEAR_DISK_CACHE = "clear_disk_cache"
```

Command dataclass:

```python
@dataclass
class MapCommand:
    kind: CommandKind
    map_tag: str | int
    payload: dict[str, Any]
    created_at: float
```

Coalescing rules:

- `UPDATE_OVERLAY` commands coalesce by `(map_tag, overlay_tag)`.
- repeated `SET_VIEW` commands may coalesce, keeping the newest.
- `ADD_OVERLAY`, `DELETE_OVERLAY`, `CLEAR_LAYER`, `CLEAR_MAP`, and `SET_PROVIDER` must preserve order.
- provider switches form a barrier; older tile results for the previous provider must be ignored.

### `state.py`

Owns configuration, registries, and `MapState`.

Global config:

```python
@dataclass
class DpgMapConfig:
    user_agent: str | None = None
    cache_dir: Path | None = None
    default_provider: str | TileProvider = "osm"
    memory_cache_max_tiles: int = 512
    disk_cache_max_bytes: int | None = 2_000_000_000
    prefetch_margin_tiles: int = 1
    tile_worker_count: int = 4
    overlay_update_policy: str = "coalesce"
    debug: bool = False
```

MapState:

```python
@dataclass
class MapState:
    tag: str | int

    # DPG implementation tags, internal only
    child_window_tag: str | int
    drawlist_tag: str | int
    texture_registry_tag: str | int
    handler_registry_tag: str | int

    # sizing
    requested_width: int
    requested_height: int
    requested_autosize_x: bool
    requested_autosize_y: bool
    measured_width: int = 0
    measured_height: int = 0
    last_nonzero_width: int = 0
    last_nonzero_height: int = 0
    is_visible: bool = False

    # view
    center: LatLon = (0.0, 0.0)
    zoom: int = 2
    min_zoom: int = 0
    max_zoom: int = 19
    provider: TileProvider = field(default_factory=get_default_provider)

    # models
    overlays: dict[str | int, Overlay] = field(default_factory=dict)
    layers: dict[str, LayerState] = field(default_factory=default_layers)

    # subsystems
    command_queue: MapCommandQueue = field(default_factory=MapCommandQueue)
    tile_manager: TileManager = field(default_factory=TileManager)
    renderer: MapRenderer | None = None
    interaction: InteractionState = field(default_factory=InteractionState)

    # concurrency
    lock: RLock = field(default_factory=RLock)

    # redraw tracking
    dirty: DirtyFlags = DirtyFlags.FULL
    frame_scheduled: bool = False
    generation: int = 0
```

`generation` increments on provider switch, full clear, or other operations that invalidate pending tile results.

### `types.py`

Contains aliases and small dataclasses:

```python
LatLon = tuple[float, float]
RGBA = tuple[int, int, int, int]
ScreenPoint = tuple[float, float]
```

### `exceptions.py`

Contains public exceptions:

```python
class DpgMapError(Exception): ...
class ProviderError(DpgMapError): ...
class UnknownProviderError(ProviderError): ...
class MapNotFoundError(DpgMapError): ...
class OverlayNotFoundError(DpgMapError): ...
class CoordinateError(DpgMapError): ...
class ThreadingError(DpgMapError): ...
class CacheError(DpgMapError): ...
```

### `providers.py`

Defines the provider system.

The default OpenStreetMap provider is registered at import time.

Provider identity is part of `TileID`, so caches are provider-namespaced.

### `projection.py`

Pure Web Mercator math only.

No Dear PyGui imports.

No global state.

Must include tests.

### `overlays.py`

Defines logical overlay dataclasses only.

Overlay objects are not Dear PyGui draw items.

```python
@dataclass
class Overlay:
    tag: str | int
    map_tag: str | int
    layer: str
    show: bool = True
    user_data: Any = None
    revision: int = 0
```

`revision` increments when an overlay changes. The renderer can use this to decide whether overlay draw data needs rebuilding.

Marker, Polyline, and Trajectory inherit from `Overlay`.

Path overlays should copy incoming coordinate sequences into internal tuples/lists during command processing, not store references to mutable user lists. This prevents background buffers changing while the renderer reads them.

### `widget.py`

Creates the Dear PyGui structure.

Internal structure:

```text
child_window
  drawlist
```

Rules:

- child window receives the user-requested size.
- drawlist uses measured concrete content size.
- map is registered before entering the context body.
- current map stack is pushed for context-style overlay creation.
- frame callback is scheduled after creation to measure and render.
- resize/visibility handlers mark `SIZE_DIRTY`.

The widget must not expose `child_window_tag` or `drawlist_tag`.

### `sizing.py`

Dedicated sizing helper module.

Responsibilities:

- measure child content region
- maintain last non-zero measured size
- decide whether tile loading can begin
- resize drawlist safely
- detect hidden-to-visible transitions
- avoid permanent collapse from zero-size measurements

Suggested function:

```python
def update_measured_size(state: MapState) -> bool:
    """Return True if concrete drawlist size changed."""
```

### `interaction.py`

Owns panning and zooming.

Responsibilities:

- hit-test against concrete drawlist rectangle
- track active drag state
- update center during drag
- update zoom during wheel
- zoom around cursor where practical
- never allow overlay updates to reset interaction state

Interaction should use the current measured drawlist rect, not only `dpg.is_item_hovered`.

### `tiles.py`

Owns tile identity, tile lifecycle, and visible tile calculation.

TileID:

```python
@dataclass(frozen=True)
class TileID:
    provider_name: str
    z: int
    x: int
    y: int
```

Tile states:

```python
class TileStatus(Enum):
    MISSING = "missing"
    QUEUED = "queued"
    LOADING = "loading"
    READY = "ready"
    FAILED = "failed"
```

Tile manager responsibilities:

- compute visible tile IDs
- queue missing tiles
- avoid duplicate requests
- ignore stale tile results from old generations/providers
- manage memory cache metadata
- request texture creation on GUI thread
- request texture deletion on GUI thread
- protect currently visible tiles from eviction

Tile workers:

- check disk cache
- fetch network if needed
- decode image bytes to RGBA-compatible data if practical
- return loaded data to GUI queue
- never call DPG APIs

### `cache.py`

Owns persistent disk cache and memory cache policy.

Disk path:

```text
{cache_dir}/{provider_name}/{z}/{x}/{y}.{ext}
```

Metadata path:

```text
{cache_dir}/{provider_name}/{z}/{x}/{y}.json
```

Metadata should support:

```json
{
  "url": "...",
  "etag": "...",
  "last_modified": "...",
  "expires": "...",
  "downloaded_at": 0,
  "last_accessed_at": 0,
  "size_bytes": 0
}
```

MVP disk cache behaviour:

- file exists and is readable: use it
- file missing: fetch it
- update `last_accessed_at`
- prune by LRU when above `disk_cache_max_bytes`
- pruning must not delete files currently loading or currently visible

HTTP-aware cache headers can be added after the stable MVP.

### `renderer.py`

The renderer is GUI-thread-only.

Responsibilities:

- drain command queue
- apply commands to logical state
- process tile worker results
- create/delete textures
- measure size
- compute visible tiles
- redraw dirty layers
- draw attribution/debug information

The renderer must support separate draw layers. Dear PyGui drawlist does not provide retained sublayers in the same way as a scene graph, so implementation may use internal groups/tags or controlled delete/rebuild by tag prefix. The important rule is behavioural:

- overlay update must not clear tile textures or tile logical state
- tile update must not modify overlay logical state
- full redraw must not reset view state

### `draw_layers.py`

Optional helper module for draw item bookkeeping.

Suggested structure:

```python
@dataclass
class DrawLayer:
    name: str
    item_tags: set[str | int] = field(default_factory=set)
```

Renderer can delete and rebuild only a selected layer.

### `diagnostics.py`

Provides debug state for examples and troubleshooting.

```python
def get_map_debug_state(map_tag) -> dict[str, Any]:
    ...
```

Should expose:

- center
- zoom
- measured size
- requested size
- visible tile count
- queued/loading/failed tile counts
- memory cache count
- disk cache bytes
- overlay count
- dirty flags
- active drag state
- pending command count
- generation

## Rendering frame lifecycle

Each frame callback should roughly do:

```text
for each map:
  1. update measured size
  2. drain pending public commands
  3. apply interaction input
  4. process tile results
  5. compute visible tiles if view/size/provider changed
  6. queue missing visible tiles
  7. create/delete textures requested by tile manager
  8. redraw required layers based on dirty flags
  9. clear dirty flags
  10. schedule next frame only if work remains or map needs continuous interaction polling
```

## Dirty flag model

Use bit flags:

```python
class DirtyFlags(IntFlag):
    NONE = 0
    VIEW = auto()
    TILES = auto()
    OVERLAYS = auto()
    SIZE = auto()
    PROVIDER = auto()
    ATTRIBUTION = auto()
    DEBUG = auto()
    FULL = VIEW | TILES | OVERLAYS | SIZE | PROVIDER | ATTRIBUTION | DEBUG
```

Expected usage:

| Operation | Dirty flags |
|---|---|
| marker update | `OVERLAYS` |
| trajectory update | `OVERLAYS` |
| layer visibility | `OVERLAYS` |
| pan | `VIEW | TILES | OVERLAYS` |
| zoom | `VIEW | TILES | OVERLAYS` |
| resize | `SIZE | TILES | OVERLAYS` |
| provider switch | `PROVIDER | TILES | ATTRIBUTION` |
| cache tile ready | `TILES` |
| debug setting changed | `DEBUG` |

## Command queue and coalescing

The command queue must prevent flooding during telemetry.

Recommended approach:

```python
class MapCommandQueue:
    ordered: deque[MapCommand]
    overlay_updates: dict[tuple[map_tag, overlay_tag], MapCommand]
    view_update: MapCommand | None
    lock: Lock
```

When draining:

1. preserve ordered structural commands
2. apply newest coalesced view command
3. apply newest overlay updates
4. mark dirty flags once

Structural commands:

- add overlay
- delete overlay
- clear layer
- clear map
- provider switch

Coalesced commands:

- update marker
- update polyline
- update trajectory
- set overlay show
- set center/zoom/view

## Tile result generation safety

Every tile request should include:

```python
provider_name
generation
tile_id
```

When result returns:

- if generation no longer matches map generation: discard result
- if provider no longer matches: discard result
- if tile no longer requested/visible/cacheable: optionally keep disk cache but do not create texture immediately

This prevents stale workers from drawing old-provider tiles after a provider switch.

## Persistent disk cache pruning

Disk cache pruning should run:

- at startup/open map
- after downloads push cache above the limit
- optionally manually via `clear_disk_cache`

Pruning rules:

1. calculate provider or global cache size
2. sort cached tile metadata by `last_accessed_at`
3. delete oldest until under target
4. never delete files currently visible/loading
5. tolerate missing/corrupt metadata

## Sizing strategy

The map widget uses two sizes:

```python
requested_width / requested_height
measured_width / measured_height
```

The child window is created with requested values:

```python
dpg.add_child_window(width=requested_width, height=requested_height, ...)
```

The drawlist is resized to measured content size:

```python
content_width = dpg.get_item_width(child_window_tag)
content_height = dpg.get_item_height(child_window_tag)
dpg.configure_item(drawlist_tag, width=content_width, height=content_height)
```

If measured size is zero:

- keep last non-zero size
- do not permanently configure drawlist to zero
- if no last non-zero size exists, render placeholder only and delay tile loading

## Initial setup instructions

These replace the old Step 0.

```bash
mkdir dpg-map
cd dpg-map
git init
uv init --package dpg-map
uv add dearpygui pillow platformdirs requests
uv add --dev pytest ruff pyright
```

Create:

```text
src/dpg_map/
examples/
tests/
FEATURES.md
ARCHITECTURE.md
STEPS.md
AGENTS.md
README.md
```

Use `AGENTS.md` as the rolling implementation log. After every step:

```bash
uv run pytest
uv run ruff check .
uv run ruff format --check .
git status
git add .
git commit -m "step N: short description"
```

## Development examples required

The rebuild must include examples specifically designed to reproduce the first build's failure modes:

```text
examples/basic_map.py
examples/sizing_window.py
examples/sizing_child.py
examples/sizing_table.py
examples/hidden_tab.py
examples/markers_live_thread.py
examples/trajectory_live_thread.py
examples/custom_provider.py
examples/cache_stress.py
```