Amp Local Development Experience - Persistent PostgreSQL

[LNSD]

Summary

Evolve the amp local development experience to support a persistent, zero-configuration PostgreSQL metadata database that lives entirely within the project's .amp/ directory. This eliminates the need for Docker, system-installed PostgreSQL, or manual database management while providing a seamless developer experience.

Goals

1. Zero-config experience: Running ampd solo should "just work" without requiring any database setup
2. Persistence: Metadata DB state persists across amp daemon restarts
3. Self-contained: All amp data lives under <project>/.amp/ directory
4. No external dependencies: No Docker, no system PostgreSQL installation required
5. IPC communication: Use Unix sockets for database communication (easier discovery, better security)
6. Sensible defaults: Opinionated configuration that works for 90% of use cases

Current Pain Points

Issue	Impact
config.toml required	Cannot run ampd dev without setup
Directories not auto-created	Manual setup of data/, providers/, manifests/
No default metadata DB	Must configure PostgreSQL or set ALLOW_TEMP_DB=true
pgtemp is ephemeral	All state lost on daemon restart
pgtemp needs system PostgreSQL	Additional system dependency
No standard project structure	Each developer may use different paths

Requirements

1. ampup install --component postgres downloads PostgreSQL binaries successfully
2. ampd solo works after PostgreSQL component is installed (no other setup required)
3. Data persists across daemon restarts
4. No Docker or system PostgreSQL required
5. Clear error messages when PostgreSQL component is missing
6. Clear upgrade path from current Docker-based setup
7. < 30 second first-run experience (after component installation)

Proposal

Phase 0: Refactoring - Establish Foundations

• Rename dev subcommand to solo
  • chore(ampd): rename dev subcommand to solo #1458
• Create crates/services/metadata-db-postgres/ service crate
• Remove temp.rs from metadata-db
• Clean up config.rs - remove temp DB spawning logic
  • refactor(common): decouple Store creation from Config loading #1440
  • feat(config): break circular deps moving types to owner crates #1456
  • feat(tempdb): add tempdb service crate #1140
  • feat(metadata-db-postgres): extract metadata-db tempdb service #1457
• Rework config loading for sensible defaults

Phase 1: Persistent pgtemp (Quick Win)

• Configure pgtemp to store data in .amp/metadb/data/
• Use PgTempDBBuilder::persist_data(true) to prevent cleanup on shutdown
• Add reconnection logic to existing database on daemon restart
• Limitation: Still requires system PostgreSQL installation

Phase 2: PostgreSQL as an ampup Component (Full Solution)

Key Principle: PostgreSQL binary download MUST NOT happen at ampd startup. Instead:

1. ampup is responsible for downloading and managing PostgreSQL binaries
2. PostgreSQL is treated as a component alongside ampd and ampctl
3. postgresql_embedded crate is used by the tempdb service to manage PostgreSQL lifecycle (init, start, stop, connect) using pre-installed binaries
4. ampd solo expects PostgreSQL binaries to already be installed (or embedded)

Architecture:

┌──────────────────────────────────────────────────────────────────┐
│                         ampup                                    │
│  (downloads PostgreSQL binaries to ~/.ampup/components/postgres) │
└──────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────────┐
│                    tempdb service                                │
│  (uses postgresql_embedded to manage PostgreSQL lifecycle)       │
│  - Locates binaries from ampup components dir                    │
│  - Initializes data directory (.amp/metadb/data/)                │
│  - Starts/stops PostgreSQL process                               │
│  - Provides connection URL to config                             │
└──────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌──────────────────────────────────────────────────────────────────┐
│                      metadata-db                                 │
│  (connects to PostgreSQL using provided URL)                     │
└──────────────────────────────────────────────────────────────────┘

Component Management via ampup:

# Install PostgreSQL component (downloads binaries)
ampup install --component postgres

# List installed components
ampup list
# Output:
#   ampd     v0.5.0 (active)
#   ampctl   v0.5.0 (active)
#   postgres 18.x   (active)

Milestone 3: Zero-Config Experience

• Implement smart config resolution (default -> config.toml -> env vars)
• Auto-create .amp/ structure on first run
• Auto-discover existing .amp/metadb/ and reconnect

Decisions

1. PostgreSQL version: Prefer PostgreSQL 18 if available in zonky/theseus builds, otherwise use the latest available version

Open Questions

1. Binary source: Which PostgreSQL binary distribution should ampup download from?

   • theseus-rs/postgresql-binaries - Rust-focused, smaller
   • zonky/embedded-postgres-binaries - Java ecosystem, well-tested

2. Multi-project support: How do we handle users with multiple amp projects?

3. Database migrations on version upgrade: How do we handle amp upgrades that require schema migrations?

4. Disk space management: Should we provide tools to clean up old data or compact the database?

5. Component versioning: How do we handle PostgreSQL version upgrades? Should ampup update also update components?

6. Windows support: Unix sockets work differently on Windows. Do we fall back to TCP localhost?

References

• postgresql_embedded crate
• theseus-rs/postgresql-embedded
• theseus-rs/postgresql-binaries
• zonky/embedded-postgres-binaries
• pgtemp crate

[fubhy]

I like this but we should name the postgres component different if we put it into the same bin path so it wouldn't interfere with a system postgres in PATH also what are your thoughts on a multi-tenant daemon with ampd (default to fully shared). Having shared data and provider dirs will inevitably force us to discuss that

[LNSD]

I like this but we should name the postgres component different if we put it into the same bin path so it wouldn't interfere with a system postgres in PATH

Agree. The PostgreSQL binary would be distributed under a different name to avoid any interference with the system postgres. I think there is also the possibility of embedding the PostgreSQL binary into an ampd component (e.g., ampd-metadb). In any case, this seems an implementation detail that we can discuss later.

also what are your thoughts on a multi-tenant daemon with ampd (default to fully shared). Having shared data and provider dirs will inevitably force us to discuss that.

The local dev experience I proposed here is based on a .amp/ daemon state directory that lives within a project's workspace. This is not compatible with a global/shared daemon state. But if we combine this workspace daemon state with the external tables functionality, one could access data stored in a daemon external to the workspace, having the best of both worlds: local development with a fast feedback cycle, with access to large datasets (e.g., ethereum-mainnet, base-mainnet, arbitrum-mainnet, etc.) without having to re-synchronize the data.

Basically, I see value in having a shared daemon between different projects. But I wouldn't approach it as a multitenancy setup.