dpg-map/ARCHITECTURE.md

dpg-map Architecture — Locked Rebuild

This document defines the internal architecture for the rebuild.

The central rule is:

Public API calls update logical state or enqueue commands. The GUI thread alone owns Dear PyGui items, draw commands, textures, handlers, and frame rendering.

This prevents flicker, dragging failures, random recentering, and Dear PyGui thread-safety issues when overlays are updated from telemetry threads.

High-level architecture

User API
  ↓
Thread-safe command/logical-state layer
  ↓
Map registry and MapState
  ↓
Renderer frame pump on GUI thread
  ↓
Dear PyGui child_window + drawlist + textures

Detailed flow:

Telemetry thread / app thread / GUI callback
  ↓
public dpg_map function
  ↓
validate lightweight inputs
  ↓
acquire MapState.lock briefly
  ↓
update logical model and/or enqueue command
  ↓
mark specific dirty flags
  ↓
return immediately
  ↓
GUI frame callback drains commands
  ↓
renderer updates DPG draw layers/textures

Critical invariants

Threading invariants

1. Dear PyGui APIs are called only on the GUI thread.
2. Public API functions do not call Dear PyGui directly unless explicitly documented as GUI-thread-only.
3. Tile worker threads do network and disk work only.
4. Tile workers do not create or delete Dear PyGui textures.
5. Texture creation and deletion are queued to the renderer and executed on the GUI thread.
6. Runtime overlay updates are coalesced by overlay tag.
7. Locks are held briefly and never during network, disk, image decoding, or Dear PyGui calls.

View invariants

1. MapState.center and MapState.zoom are changed only by view commands or interaction.
2. Overlay commands must never change center or zoom.
3. Tile loading must never change center or zoom.
4. Renderer redraw must render the current state; it must not infer a new center from overlays.
5. Provider switching may clamp zoom, but must not recenter unless the current center is invalid for the projection.

Rendering invariants

1. Overlay redraw does not clear tile draw commands.
2. Tile redraw does not delete overlay logical state.
3. Provider switch clears only provider-specific tile resources.
4. Full redraw is reserved for initial render, size change, provider change, or recovery.
5. The renderer always reads a consistent snapshot of state for each frame.
6. DPG draw item tags are internal and not exposed as public overlay tags.

Sizing invariants

1. The child window receives the requested Dear PyGui sizing intent.
2. The drawlist receives concrete measured dimensions.
3. Requested size and measured size are stored separately.
4. A measured size of zero while hidden must not permanently collapse the map.
5. The map should delay tile loading until a non-zero visible size is measured, unless explicitly configured otherwise.
6. Resize triggers tile and overlay redraw.

Package layout

src/dpg_map/
  __init__.py
  api.py
  widget.py
  state.py
  commands.py
  renderer.py
  draw_layers.py
  interaction.py
  overlays.py
  providers.py
  projection.py
  tiles.py
  cache.py
  sizing.py
  diagnostics.py
  types.py
  exceptions.py
examples/
  basic_map.py
  sizing_window.py
  sizing_child.py
  sizing_table.py
  hidden_tab.py
  markers_live_thread.py
  trajectory_live_thread.py
  custom_provider.py
  cache_stress.py
tests/

Module responsibilities

__init__.py

Exports the public API only.

No heavy initialization should happen here except registering the default provider.

api.py

Contains thin public API wrappers, especially for runtime calls.

Responsibilities:

• resolve map/overlay tags
• validate simple arguments
• enqueue commands
• expose stable public functions

The public API should call into state.py, commands.py, and overlays.py, but should not know renderer internals.

commands.py

Defines commands that can be submitted from any thread.

Suggested command model:

class CommandKind(Enum):
    SET_VIEW = "set_view"
    SET_PROVIDER = "set_provider"
    ADD_OVERLAY = "add_overlay"
    UPDATE_OVERLAY = "update_overlay"
    DELETE_OVERLAY = "delete_overlay"
    SET_LAYER_VISIBILITY = "set_layer_visibility"
    CLEAR_LAYER = "clear_layer"
    CLEAR_MAP = "clear_map"
    CLEAR_MEMORY_CACHE = "clear_memory_cache"
    CLEAR_DISK_CACHE = "clear_disk_cache"

Command dataclass:

@dataclass
class MapCommand:
    kind: CommandKind
    map_tag: str | int
    payload: dict[str, Any]
    created_at: float

Coalescing rules:

• UPDATE_OVERLAY commands coalesce by (map_tag, overlay_tag).
• repeated SET_VIEW commands may coalesce, keeping the newest.
• ADD_OVERLAY, DELETE_OVERLAY, CLEAR_LAYER, CLEAR_MAP, and SET_PROVIDER must preserve order.
• provider switches form a barrier; older tile results for the previous provider must be ignored.

state.py

Owns configuration, registries, and MapState.

Global config:

@dataclass
class DpgMapConfig:
    user_agent: str | None = None
    cache_dir: Path | None = None
    default_provider: str | TileProvider = "osm"
    memory_cache_max_tiles: int = 512
    disk_cache_max_bytes: int | None = 2_000_000_000
    prefetch_margin_tiles: int = 1
    tile_worker_count: int = 4
    overlay_update_policy: str = "coalesce"
    debug: bool = False

MapState:

@dataclass
class MapState:
    tag: str | int

    # DPG implementation tags, internal only
    child_window_tag: str | int
    drawlist_tag: str | int
    texture_registry_tag: str | int
    handler_registry_tag: str | int

    # sizing
    requested_width: int
    requested_height: int
    requested_autosize_x: bool
    requested_autosize_y: bool
    measured_width: int = 0
    measured_height: int = 0
    last_nonzero_width: int = 0
    last_nonzero_height: int = 0
    is_visible: bool = False

    # view
    center: LatLon = (0.0, 0.0)
    zoom: int = 2
    min_zoom: int = 0
    max_zoom: int = 19
    provider: TileProvider = field(default_factory=get_default_provider)

    # models
    overlays: dict[str | int, Overlay] = field(default_factory=dict)
    layers: dict[str, LayerState] = field(default_factory=default_layers)

    # subsystems
    command_queue: MapCommandQueue = field(default_factory=MapCommandQueue)
    tile_manager: TileManager = field(default_factory=TileManager)
    renderer: MapRenderer | None = None
    interaction: InteractionState = field(default_factory=InteractionState)

    # concurrency
    lock: RLock = field(default_factory=RLock)

    # redraw tracking
    dirty: DirtyFlags = DirtyFlags.FULL
    frame_scheduled: bool = False
    generation: int = 0

generation increments on provider switch, full clear, or other operations that invalidate pending tile results.

types.py

Contains aliases and small dataclasses:

LatLon = tuple[float, float]
RGBA = tuple[int, int, int, int]
ScreenPoint = tuple[float, float]

exceptions.py

Contains public exceptions:

class DpgMapError(Exception): ...
class ProviderError(DpgMapError): ...
class UnknownProviderError(ProviderError): ...
class MapNotFoundError(DpgMapError): ...
class OverlayNotFoundError(DpgMapError): ...
class CoordinateError(DpgMapError): ...
class ThreadingError(DpgMapError): ...
class CacheError(DpgMapError): ...

providers.py

Defines the provider system.

The default OpenStreetMap provider is registered at import time.

Provider identity is part of TileID, so caches are provider-namespaced.

projection.py

Pure Web Mercator math only.

No Dear PyGui imports.

No global state.

Must include tests.

overlays.py

Defines logical overlay dataclasses only.

Overlay objects are not Dear PyGui draw items.

@dataclass
class Overlay:
    tag: str | int
    map_tag: str | int
    layer: str
    show: bool = True
    user_data: Any = None
    revision: int = 0

revision increments when an overlay changes. The renderer can use this to decide whether overlay draw data needs rebuilding.

Marker, Polyline, and Trajectory inherit from Overlay.

Path overlays should copy incoming coordinate sequences into internal tuples/lists during command processing, not store references to mutable user lists. This prevents background buffers changing while the renderer reads them.

widget.py

Creates the Dear PyGui structure.

Internal structure:

child_window
  drawlist

Rules:

• child window receives the user-requested size.
• drawlist uses measured concrete content size.
• map is registered before entering the context body.
• current map stack is pushed for context-style overlay creation.
• frame callback is scheduled after creation to measure and render.
• resize/visibility handlers mark SIZE_DIRTY.

The widget must not expose child_window_tag or drawlist_tag.

sizing.py

Dedicated sizing helper module.

Responsibilities:

• measure child content region
• maintain last non-zero measured size
• decide whether tile loading can begin
• resize drawlist safely
• detect hidden-to-visible transitions
• avoid permanent collapse from zero-size measurements

Suggested function:

def update_measured_size(state: MapState) -> bool:
    """Return True if concrete drawlist size changed."""

interaction.py

Owns panning and zooming.

Responsibilities:

• hit-test against concrete drawlist rectangle
• track active drag state
• update center during drag
• update zoom during wheel
• zoom around cursor where practical
• never allow overlay updates to reset interaction state

Interaction should use the current measured drawlist rect, not only dpg.is_item_hovered.

tiles.py

Owns tile identity, tile lifecycle, and visible tile calculation.

TileID:

@dataclass(frozen=True)
class TileID:
    provider_name: str
    z: int
    x: int
    y: int

Tile states:

class TileStatus(Enum):
    MISSING = "missing"
    QUEUED = "queued"
    LOADING = "loading"
    READY = "ready"
    FAILED = "failed"

Tile manager responsibilities:

• compute visible tile IDs
• queue missing tiles
• avoid duplicate requests
• ignore stale tile results from old generations/providers
• manage memory cache metadata
• request texture creation on GUI thread
• request texture deletion on GUI thread
• protect currently visible tiles from eviction

Tile workers:

• check disk cache
• fetch network if needed
• decode image bytes to RGBA-compatible data if practical
• return loaded data to GUI queue
• never call DPG APIs

cache.py

Owns persistent disk cache and memory cache policy.

Disk path:

{cache_dir}/{provider_name}/{z}/{x}/{y}.{ext}

Metadata path:

{cache_dir}/{provider_name}/{z}/{x}/{y}.json

Metadata should support:

{
  "url": "...",
  "etag": "...",
  "last_modified": "...",
  "expires": "...",
  "downloaded_at": 0,
  "last_accessed_at": 0,
  "size_bytes": 0
}

MVP disk cache behaviour:

• file exists and is readable: use it
• file missing: fetch it
• update last_accessed_at
• prune by LRU when above disk_cache_max_bytes
• pruning must not delete files currently loading or currently visible

HTTP-aware cache headers can be added after the stable MVP.

renderer.py

The renderer is GUI-thread-only.

Responsibilities:

• drain command queue
• apply commands to logical state
• process tile worker results
• create/delete textures
• measure size
• compute visible tiles
• redraw dirty layers
• draw attribution/debug information

The renderer must support separate draw layers. Dear PyGui drawlist does not provide retained sublayers in the same way as a scene graph, so implementation may use internal groups/tags or controlled delete/rebuild by tag prefix. The important rule is behavioural:

• overlay update must not clear tile textures or tile logical state
• tile update must not modify overlay logical state
• full redraw must not reset view state

draw_layers.py

Optional helper module for draw item bookkeeping.

Suggested structure:

@dataclass
class DrawLayer:
    name: str
    item_tags: set[str | int] = field(default_factory=set)

Renderer can delete and rebuild only a selected layer.

diagnostics.py

Provides debug state for examples and troubleshooting.

def get_map_debug_state(map_tag) -> dict[str, Any]:
    ...

Should expose:

• center
• zoom
• measured size
• requested size
• visible tile count
• queued/loading/failed tile counts
• memory cache count
• disk cache bytes
• overlay count
• dirty flags
• active drag state
• pending command count
• generation

Rendering frame lifecycle

Each frame callback should roughly do:

for each map:
  1. update measured size
  2. drain pending public commands
  3. apply interaction input
  4. process tile results
  5. compute visible tiles if view/size/provider changed
  6. queue missing visible tiles
  7. create/delete textures requested by tile manager
  8. redraw required layers based on dirty flags
  9. clear dirty flags
  10. schedule next frame only if work remains or map needs continuous interaction polling

Dirty flag model

Use bit flags:

class DirtyFlags(IntFlag):
    NONE = 0
    VIEW = auto()
    TILES = auto()
    OVERLAYS = auto()
    SIZE = auto()
    PROVIDER = auto()
    ATTRIBUTION = auto()
    DEBUG = auto()
    FULL = VIEW | TILES | OVERLAYS | SIZE | PROVIDER | ATTRIBUTION | DEBUG

Expected usage:

Operation	Dirty flags
marker update	OVERLAYS
trajectory update	OVERLAYS
layer visibility	OVERLAYS
pan	`VIEW
zoom	`VIEW
resize	`SIZE
provider switch	`PROVIDER
cache tile ready	TILES
debug setting changed	DEBUG

Command queue and coalescing

The command queue must prevent flooding during telemetry.

Recommended approach:

class MapCommandQueue:
    ordered: deque[MapCommand]
    overlay_updates: dict[tuple[map_tag, overlay_tag], MapCommand]
    view_update: MapCommand | None
    lock: Lock

When draining:

1. preserve ordered structural commands
2. apply newest coalesced view command
3. apply newest overlay updates
4. mark dirty flags once

Structural commands:

• add overlay
• delete overlay
• clear layer
• clear map
• provider switch

Coalesced commands:

• update marker
• update polyline
• update trajectory
• set overlay show
• set center/zoom/view

Tile result generation safety

Every tile request should include:

provider_name
generation
tile_id

When result returns:

• if generation no longer matches map generation: discard result
• if provider no longer matches: discard result
• if tile no longer requested/visible/cacheable: optionally keep disk cache but do not create texture immediately

This prevents stale workers from drawing old-provider tiles after a provider switch.

Persistent disk cache pruning

Disk cache pruning should run:

• at startup/open map
• after downloads push cache above the limit
• optionally manually via clear_disk_cache

Pruning rules:

1. calculate provider or global cache size
2. sort cached tile metadata by last_accessed_at
3. delete oldest until under target
4. never delete files currently visible/loading
5. tolerate missing/corrupt metadata

Sizing strategy

The map widget uses two sizes:

requested_width / requested_height
measured_width / measured_height

The child window is created with requested values:

dpg.add_child_window(width=requested_width, height=requested_height, ...)

The drawlist is resized to measured content size:

content_width = dpg.get_item_width(child_window_tag)
content_height = dpg.get_item_height(child_window_tag)
dpg.configure_item(drawlist_tag, width=content_width, height=content_height)

If measured size is zero:

• keep last non-zero size
• do not permanently configure drawlist to zero
• if no last non-zero size exists, render placeholder only and delay tile loading

Initial setup instructions

These replace the old Step 0.

mkdir dpg-map
cd dpg-map
git init
uv init --package dpg-map
uv add dearpygui pillow platformdirs requests
uv add --dev pytest ruff pyright

Create:

src/dpg_map/
examples/
tests/
FEATURES.md
ARCHITECTURE.md
STEPS.md
AGENTS.md
README.md

Use AGENTS.md as the rolling implementation log. After every step:

uv run pytest
uv run ruff check .
uv run ruff format --check .
git status
git add .
git commit -m "step N: short description"

Development examples required

The rebuild must include examples specifically designed to reproduce the first build's failure modes:

examples/basic_map.py
examples/sizing_window.py
examples/sizing_child.py
examples/sizing_table.py
examples/hidden_tab.py
examples/markers_live_thread.py
examples/trajectory_live_thread.py
examples/custom_provider.py
examples/cache_stress.py