dpg-map/STEPS.md

# dpg-map Rebuild Steps

This file replaces the old long step list.

There is no Step 0. Initial setup is listed separately, then implementation starts at Step 1.

## Workflow rules

1. Use `uv` for all Python package and dependency management.
2. Always read `FEATURES.md`, `ARCHITECTURE.md`, and `AGENTS.md` before making code changes.
3. Keep `AGENTS.md` as a rolling log of what has been done, what is broken, and what comes next.
4. Update `README.md` whenever public behaviour or examples change.
5. After every step:
   - update `AGENTS.md`
   - run relevant checks
   - commit to git
6. Do not casually change public API once introduced.
7. Public runtime APIs must be safe to call from background threads unless explicitly documented otherwise.
8. Dear PyGui calls must happen on the GUI thread only.
9. Prefer stable, boring implementation over clever rendering tricks.

## Initial setup instructions

Run these before Step 1 when starting from a clean repository:

```bash
mkdir dpg-map
cd dpg-map
git init
uv init --package dpg-map
uv add dearpygui pillow platformdirs requests
uv add --dev pytest ruff pyright
```

Create this structure:

```text
src/dpg_map/
  __init__.py
  api.py
  widget.py
  state.py
  commands.py
  renderer.py
  draw_layers.py
  interaction.py
  overlays.py
  providers.py
  projection.py
  tiles.py
  cache.py
  sizing.py
  diagnostics.py
  types.py
  exceptions.py
examples/
tests/
FEATURES.md
ARCHITECTURE.md
STEPS.md
AGENTS.md
README.md
```

Recommended `pyproject.toml` additions:

```toml
[tool.ruff]
line-length = 100
target-version = "py311"

[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B", "SIM"]

[tool.pyright]
typeCheckingMode = "basic"
```

Create `AGENTS.md` with:

```markdown
# AGENTS.md

## Current status

Rebuild initialized.

## Completed steps

None yet.

## Current step

Step 1 — Public API contract and pure core.

## Design decisions

- Package is managed with uv.
- Public import is `import dpg_map as dpgm`.
- Dear PyGui calls are GUI-thread-only.
- Runtime public calls enqueue commands or update logical state.
- Overlay updates must not reset center/zoom.
- The widget uses child_window + measured-size drawlist.
- Tiles use a memory cache and persistent disk cache.
- Tile providers are interchangeable.

## Known issues

None yet.

## Commands used

None yet.

## Next action

Implement Step 1.
```

Initial commit:

```bash
uv run python -c "import dpg_map; print(dpg_map.__name__)"
uv run ruff check .
uv run ruff format .
git add .
git commit -m "initial uv project setup"
```

## Step 1 — Public API contract and pure core

Goal: lock the public API surface and implement pure non-DPG components first.

Tasks:

1. Define public exports in `__init__.py`.

Required API names:

```python
configure

TileProvider
register_provider
unregister_provider
get_provider
list_providers

map_widget

set_center
get_center
set_zoom
get_zoom
set_view
fit_bounds
screen_to_latlon
latlon_to_screen

add_marker
add_polyline
add_trajectory
update_marker
update_polyline
update_trajectory
set_marker_position
set_marker_label
set_polyline_points
set_overlay_show
delete_overlay

add_layer
show_layer
hide_layer
clear_layer
clear_map

set_provider

clear_memory_cache
clear_disk_cache
get_cache_stats
get_map_debug_state
```

2. Add exceptions in `exceptions.py`.

3. Add common types in `types.py`.

4. Implement `TileProvider` and provider registry in `providers.py`.

5. Implement Web Mercator projection in `projection.py`.

6. Implement initial cache dataclasses and `CacheStats`.

7. Stub GUI-dependent public functions so imports succeed.

8. Add tests for:
   - package imports
   - provider registration
   - provider URL building
   - invalid provider templates
   - projection roundtrips
   - cache stats dataclass construction

Checks:

```bash
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run pyright
```

Acceptance criteria:

- public API imports cleanly
- provider system works
- projection tests pass
- no Dear PyGui context is required for tests
- API names are fixed before renderer work begins

Commit:

```bash
git add .
git commit -m "step 1: lock public api and pure core"
```

## Step 2 — Thread-safe state, commands, overlays, and cache model

Goal: build the logical state model before rendering anything.

Tasks:

1. Implement `DpgMapConfig` and `configure(...)`.

2. Implement `MapState`.

3. Implement global map registry and current map context stack.

4. Implement `DirtyFlags`.

5. Implement `MapCommand`, `CommandKind`, and `MapCommandQueue`.

6. Implement coalescing rules:
   - latest overlay update per overlay wins
   - latest view update wins
   - structural commands preserve order

7. Implement logical overlay dataclasses:
   - `Overlay`
   - `MarkerOverlay`
   - `PolylineOverlay`
   - `TrajectoryOverlay`
   - `LayerState`

8. Implement public overlay creation/update functions so they:
   - resolve map context or `map_tag`
   - validate inputs
   - copy coordinate sequences
   - enqueue commands or mutate logical model under lock
   - never call Dear PyGui

9. Implement cache model:
   - memory cache metadata
   - disk path generation
   - disk metadata read/write
   - disk size scanning
   - prune planning without deleting active files yet

10. Add tests for:
    - command coalescing
    - overlay update does not alter center/zoom
    - trajectory input lists are copied
    - mismatched lat/lon lengths raise
    - layer state
    - disk cache path generation
    - disk cache prune ordering

Checks:

```bash
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run pyright
```

Acceptance criteria:

- runtime overlay update functions are thread-safe at the logical level
- no public overlay function calls Dear PyGui
- center/zoom cannot be changed by overlay commands
- cache has separate memory and disk configuration
- persistent disk cache layout is decided

Commit:

```bash
git add .
git commit -m "step 2: add thread safe state commands and cache model"
```

## Step 3 — Widget shell, sizing system, and GUI-thread frame pump

Goal: create a stable Dear PyGui widget shell before loading real tiles.

Tasks:

1. Implement `map_widget(...)`.

2. Internally create:

```text
child_window
  drawlist
```

3. Child window receives requested sizing values.

4. Drawlist receives concrete measured size only.

5. Implement `sizing.py`:
   - measure child content region
   - track last non-zero size
   - resize drawlist
   - detect hidden/show-later transitions
   - avoid permanent zero-size collapse

6. Implement the renderer frame pump:
   - schedules frame callbacks
   - drains commands on GUI thread
   - updates size
   - draws background and attribution placeholder
   - exposes debug state

7. Implement map interaction hit-test rectangle calculation, but not full panning yet.

8. Add examples:
   - `examples/basic_map.py`
   - `examples/sizing_window.py`
   - `examples/sizing_child.py`
   - `examples/sizing_table.py`
   - `examples/hidden_tab.py`

9. Add tests where possible without opening a GUI:
   - sizing state transitions
   - zero-size does not overwrite last non-zero size
   - dirty flags from resize
   - command drain ordering

Manual checks:

```bash
uv run python examples/basic_map.py
uv run python examples/sizing_window.py
uv run python examples/sizing_child.py
uv run python examples/sizing_table.py
uv run python examples/hidden_tab.py
```

Automated checks:

```bash
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run pyright
```

Acceptance criteria:

- widget appears
- map area fills its child container correctly
- drawlist size matches measured child content size
- hidden tab example loads/render placeholder after becoming visible
- no map gets stuck at zero size
- public API remains unchanged

Commit:

```bash
git add .
git commit -m "step 3: add widget shell sizing and frame pump"
```

## Step 4 — Tile manager, persistent cache, and asynchronous loading

Goal: implement stable tile loading and both cache limits before advanced overlays.

Tasks:

1. Implement `TileID`, `TileStatus`, `Tile`, and `TileManager`.

2. Implement visible tile calculation.

3. Implement persistent disk cache:
   - provider-namespaced paths
   - metadata files
   - `last_accessed_at`
   - disk size scanning
   - LRU pruning to `disk_cache_max_bytes`

4. Implement memory cache:
   - LRU or approximate LRU
   - `memory_cache_max_tiles`
   - visible-tile protection
   - deferred GUI-thread texture deletion

5. Implement worker thread/tile queue:
   - read disk cache
   - fetch HTTP if missing
   - decode tile image
   - return tile result with map generation and provider name
   - never call Dear PyGui

6. Renderer GUI thread:
   - processes tile results
   - ignores stale generation/provider results
   - creates textures
   - deletes evicted textures
   - draws tile layer

7. Ensure OpenStreetMap provider uses configured User-Agent or warns/raises clearly.

8. Add `examples/cache_stress.py`.

Manual checks:

```bash
uv run python examples/basic_map.py
uv run python examples/cache_stress.py
```

Automated checks:

```bash
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run pyright
```

Acceptance criteria:

- OSM tiles display
- GUI does not freeze while tiles load
- only visible tiles plus margin are requested
- returning to recently viewed area uses memory cache
- restarting app uses disk cache
- disk cache prunes to configured limit
- stale tile results are ignored after provider/generation change
- texture deletion happens on GUI thread

Commit:

```bash
git add .
git commit -m "step 4: add async tiles and persistent cache"
```

## Step 5 — Interaction: pan, zoom, and view commands

Goal: make interaction stable before adding live overlays.

Tasks:

1. Implement left mouse drag panning.

2. Implement mouse wheel zoom.

3. Zoom around cursor where practical.

4. Implement programmatic view commands:
   - `set_center`
   - `set_zoom`
   - `set_view`
   - `fit_bounds`
   - `screen_to_latlon`
   - `latlon_to_screen`

5. Interaction must:
   - use concrete map rectangle
   - work in nested containers
   - not depend only on child-window hover state
   - not reset when overlay commands are queued
   - not recenter from tile results

6. Add debug fields for:
   - active drag
   - last mouse position
   - current center
   - current zoom
   - dirty flags

Manual checks:

```bash
uv run python examples/basic_map.py
uv run python examples/sizing_child.py
uv run python examples/sizing_table.py
```

Automated checks:

```bash
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run pyright
```

Acceptance criteria:

- dragging pans smoothly
- mouse wheel zooms
- tile grid remains aligned
- map works in nested containers
- view commands work
- no unexpected recenter occurs from tile loading

Commit:

```bash
git add .
git commit -m "step 5: add stable pan zoom and view commands"
```

## Step 6 — Overlay rendering and runtime update stress tests

Goal: implement overlays in a way that cannot break interaction.

Tasks:

1. Implement marker rendering.

2. Implement polyline rendering.

3. Implement trajectory rendering.

4. Implement overlay layer rendering separate from tile layer.

5. Overlay updates must set only `OVERLAYS` dirty unless a structural map operation requires more.

6. Overlay draw rebuild must not:
   - clear tile textures
   - clear tile draw state
   - reset center
   - reset zoom
   - reset active drag state
   - rebuild the whole map unnecessarily

7. Implement:
   - `add_marker`
   - `add_polyline`
   - `add_trajectory`
   - `update_marker`
   - `update_polyline`
   - `update_trajectory`
   - `set_marker_position`
   - `set_marker_label`
   - `set_polyline_points`
   - `set_overlay_show`
   - `delete_overlay`

8. Add examples:
   - `examples/markers_live_thread.py`
   - `examples/trajectory_live_thread.py`

The live examples must update overlays from a background thread while the user drags and zooms.

Manual stress checks:

```bash
uv run python examples/markers_live_thread.py
uv run python examples/trajectory_live_thread.py
```

While examples run:

- drag continuously for at least 30 seconds
- zoom in and out
- verify no flicker through blank map
- verify no random recentering
- verify overlay remains geographically attached
- verify update thread does not block badly

Automated checks:

```bash
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run pyright
```

Acceptance criteria:

- marker updates at 20–60 Hz do not break dragging
- trajectory updates at 10–30 Hz do not break dragging
- overlay updates are coalesced
- center/zoom remain stable unless explicitly changed
- overlay update APIs are safe from background threads
- tile layer is not destroyed by overlay updates

Commit:

```bash
git add .
git commit -m "step 6: add stable overlays and live update stress tests"
```

## Step 7 — Layers, provider switching, and clearing APIs

Goal: add structural map operations after rendering stability is proven.

Tasks:

1. Implement layer operations:
   - `add_layer`
   - `show_layer`
   - `hide_layer`
   - `clear_layer`
   - `clear_map`

2. Implement provider switching:
   - `set_provider`

3. Provider switch must:
   - keep overlays
   - keep center
   - clamp zoom if needed
   - increment map generation
   - clear old provider tile draw state
   - ignore stale tile results
   - redraw attribution
   - not perform Dear PyGui work from caller thread

4. Implement cache control:
   - `clear_memory_cache`
   - `clear_disk_cache`
   - `get_cache_stats`

5. Add examples:
   - `examples/custom_provider.py`
   - extend `examples/cache_stress.py`

Manual checks:

```bash
uv run python examples/custom_provider.py
uv run python examples/cache_stress.py
```

Automated checks:

```bash
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run pyright
```

Acceptance criteria:

- layers can be shown/hidden/cleared
- provider switch keeps overlays
- provider switch does not render stale old-provider tiles
- cache clear functions do not call Dear PyGui from non-GUI threads
- memory/disk cache stats are usable

Commit:

```bash
git add .
git commit -m "step 7: add layers provider switching and cache controls"
```

## Step 8 — Documentation, hardening, and internal release

Goal: make the package usable as a dependency in DataFlux or another project.

Tasks:

1. Update `README.md` with:
   - uv install
   - local editable dependency
   - basic map
   - sizing examples
   - live marker update
   - live trajectory update
   - custom provider
   - memory/disk cache config
   - OpenStreetMap attribution/User-Agent note
   - thread-safety contract

2. Add docstrings for all public functions.

3. Review public API exports.

4. Add tests for failure cases:
   - unknown map
   - unknown overlay
   - unknown provider
   - invalid coordinates
   - mismatched lat/lon lengths
   - empty trajectory
   - clear deleted overlay
   - provider switch while tiles loading
   - overlay update while dragging model state

5. Run all examples manually.

6. Test local editable install from another project:

```bash
cd ../some-test-project
uv add -e ../dpg-map
uv run python -c "import dpg_map as dpgm; print(dpgm.list_providers())"
```

7. Bump version.

Suggested initial rebuilt beta:

```text
0.3.0b1
```

8. Tag release:

```bash
git tag v0.3.0b1
```

Final checks:

```bash
uv sync
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run pyright
```

Acceptance criteria:

- all examples run
- local editable dependency works
- live overlay stress examples are stable
- cache survives restart
- disk cache limit works
- sizing examples work
- README accurately documents thread-safety and cache behaviour
- AGENTS.md accurately describes status

Commit:

```bash
git add .
git commit -m "step 8: harden docs and prepare rebuilt beta"
git tag v0.3.0b1
```