FootprintAI · hsinatfootprintai · Apr 24, 2026 · Apr 24, 2026 · Apr 24, 2026
diff --git a/.github/workflows/go-build.yaml b/.github/workflows/go-build.yaml
@@ -20,10 +20,10 @@ jobs:
     name: Build
     runs-on: ubuntu-22.04
     steps:
-      - name: Set up Go 1.x
+      - name: Set up Go
         uses: actions/setup-go@v5
         with:
-          go-version: "1.21.7"
+          go-version: "1.24.1"
         id: go
 
       - name: Check out code into the Go module directory
@@ -34,9 +34,15 @@ jobs:
           go mod tidy
           go mod vendor
 
+      - name: Vet
+        run: go vet ./...
+
+      - name: Build
+        run: go build ./...
+
       - name: Test
         id: test
         run: |
           export GOPATH=/home/runner/go
           export PATH=$PATH:/home/runner/go/bin
-          go test -test.v ./... --short -coverprofile coverage.out
+          go test -race -v ./... -short -coverprofile=coverage.out
diff --git a/.gitignore b/.gitignore
@@ -1,2 +1,20 @@
 vendor
 **/charts/**
+
+# Build artifacts
+/restcol
+*.out
+*.test
+coverage.*
+dist/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+.DS_Store
+
+# Local env
+.env
+.env.*
+!.env.example
diff --git a/Makefile b/Makefile
@@ -0,0 +1,48 @@
+.PHONY: build test test-race test-short vet lint tidy run-local run-postgres gen-proto clean help
+
+BINARY := restcol
+GO      ?= go
+
+## build: Compile the server binary
+build:
+	$(GO) build -o $(BINARY) ./
+
+## test: Run tests (excludes tests that require postgres via -short)
+test:
+	$(GO) test ./... -short
+
+## test-race: Run tests with the race detector
+test-race:
+	$(GO) test -race ./... -short
+
+## test-full: Run all tests including integration tests that require postgres
+test-full:
+	$(GO) test -race ./... -coverprofile=coverage.out
+
+## vet: Run go vet
+vet:
+	$(GO) vet ./...
+
+## tidy: Tidy module dependencies
+tidy:
+	$(GO) mod tidy
+
+## run-postgres: Start the local postgres container used by tests and run-local
+run-postgres:
+	./run_postgres.sh
+
+## run-local: Build and run the server against local postgres
+run-local:
+	./run_local.sh
+
+## gen-proto: Regenerate proto/OpenAPI clients (requires buf)
+gen-proto:
+	cd api && ./gen-proto-go.sh
+
+## clean: Remove build artifacts
+clean:
+	rm -f $(BINARY) coverage.out
+
+## help: Show this help message
+help:
+	@grep -E '^## ' $(MAKEFILE_LIST) | sed -e 's/## //'
diff --git a/README.md b/README.md
@@ -1,100 +1,168 @@
 # Restcol
 
-**One Single RESTful API for Collaborating, Sharing, and Streaming Data**
+**One RESTful API for collaborative, schema-free document storage.**
 
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
 [![GitHub Issues](https://img.shields.io/github/issues/FootprintAI/restcol)](https://github.com/FootprintAI/restcol/issues)
 [![GitHub Stars](https://img.shields.io/github/stars/FootprintAI/restcol)](https://github.com/FootprintAI/restcol/stargazers)
 
 ## Overview
 
-`Restcol` is a RESTful document storage solution designed for collaboration, built to work with any kind of storage backend. It organizes data into **collections** and **documents**, providing a flexible and scalable way to manage and share data. Collections group documents with similar schemas, enabling schema evolution tracking over time, while documents store client data in formats like JSON, CSV, XML, or even media files. No predefined schema is required—schemas are dynamically created or updated with each document request.
+Restcol organises data into **projects**, **collections**, and **documents**:
 
-This project aims to simplify data collaboration and streaming by offering a unified API that adapts to your application's needs.
+- **Project** — a tenant boundary. Every request is scoped to one project.
+- **Collection** — a set of documents with similar shape. Collections track schema evolution over time.
+- **Document** — the actual payload (JSON today; CSV/XML/media on the roadmap). Schemas are inferred on write; no up-front definition required.
 
-## Features
+Restcol speaks gRPC natively and exposes the same service over HTTP/JSON via `grpc-gateway`. Swagger/OpenAPI is auto-generated from the proto.
 
-- **Flexible Storage**: Works with any storage backend.
-- **Collections**: Organizes documents with similar schemas for easy management and schema change detection.
-- **Dynamic Schemas**: Automatically creates or modifies schemas based on document requests—no upfront schema definition needed.
-- **Supported Formats**: Handles JSON, CSV, XML, and media data.
-- **RESTful API**: Simple, intuitive endpoints for collaboration and data streaming.
+## Architecture
 
-## Installation
+```
+┌────────────┐    HTTP/JSON     ┌───────────────┐
+│   client   │ ───────────────▶ │ grpc-gateway  │
+└────────────┘                  │  (port 50091) │
+                                └───────┬───────┘
+                                        │ gRPC (internal)
+                                        ▼
+┌────────────┐       gRPC       ┌───────────────┐     GORM      ┌───────────┐
+│ gRPC client│ ───────────────▶ │ restcol gRPC  │ ─────────────▶│ Postgres  │
+└────────────┘   (port 50090)   │    server     │               └───────────┘
+                                └───────────────┘
+```
 
-1. Clone the repository:
-   ```bash
-   git clone https://github.com/FootprintAI/restcol.git
-   cd restcol
-   ```
+Server entrypoint: `main.go` → `pkg/server/app` wires auth middleware, storage (`pkg/storage/...`), schema inference (`pkg/schema`), and the RestColService handlers (`pkg/app`).
 
-2. Install dependencies (assuming a Go-based project; adjust if different):
-   ```bash
-   go mod tidy
-   ```
+## Quick start
 
-3. Build and run:
-   ```bash
-   go build
-   ./restcol
-   ```
+### 1. Start Postgres
+```bash
+make run-postgres   # or: ./run_postgres.sh
+```
+Spins up `library/postgres:16-alpine3.18` on `:5432` with user `postgres`, password `password`, database `unittest`.
 
-*Note*: Specific setup instructions may vary depending on your environment and storage backend. Check the source code or configuration files for additional requirements.
+### 2. Build and run the server
+```bash
+make run-local      # or: ./run_local.sh
+```
+Serves gRPC on `:50090` and HTTP/JSON on `:50091`. On first boot the default project (`1001`) is seeded so anonymous requests have a tenant.
 
-## Usage
+### 3. Try it out
+Swagger UI: <http://localhost:50091/swaggerui/>
+Per-project API doc: <http://localhost:50091/v1/projects/1001/apidoc>
 
-### Basic Example
-To create a collection and add a document via the API:
+## API examples
 
-```bash
-# Create a new collection
-curl -X POST http://localhost:8080/collections -d '{"name": "my-collection"}'
+All examples use the default project `1001`. Replace with your own project ID as needed.
 
-# Add a document to the collection
-curl -X POST http://localhost:8080/collections/my-collection/documents -d '{"data": {"id": 1, "name": "example"}}'
+### Create a collection
+```bash
+curl -X POST http://localhost:50091/v1/projects/1001/collections \
+  -H 'Content-Type: application/json' \
+  -d '{"description": "user events"}'
 ```
 
-For detailed API documentation, refer to the [API Reference](#api-reference) section (to be added).
+### Create a document (auto-inferred schema)
+```bash
+curl -X POST http://localhost:50091/v1/projects/1001/collections/<COLLECTION_ID>:newdoc \
+  -H 'Content-Type: application/json' \
+  -d '{"data": {"user": "alice", "event": "login", "ts": 1714000000}}'
+```
+Omit `collections/<id>:newdoc` to let the server auto-provision a collection:
+```bash
+curl -X POST http://localhost:50091/v1/projects/1001/newdoc \
+  -H 'Content-Type: application/json' \
+  -d '{"data": {"hello": "world"}}'
+```
 
-## Configuration
+### Get a document
+```bash
+curl http://localhost:50091/v1/projects/1001/collections/<COLLECTION_ID>/docs/<DOC_ID>
+```
+Scope mismatch (wrong project or collection for the doc) returns `404 Not Found`, not an empty body.
 
-- **Storage Backend**: Configure your preferred storage system (e.g., local filesystem, S3, etc.) in the config file or environment variables.
-- **Port**: Default is `8080`. Override with the `PORT` environment variable.
+### Query documents
+```bash
+curl 'http://localhost:50091/v1/projects/1001/collections/<COLLECTION_ID>/docs?limitCount=10'
+```
 
-Example configuration:
+### Delete a document
 ```bash
-export STORAGE_TYPE="filesystem"
-export STORAGE_PATH="/path/to/storage"
-export PORT=8080
+curl -X DELETE http://localhost:50091/v1/projects/1001/collections/<COLLECTION_ID>/docs/<DOC_ID>
 ```
 
-## Contributing
+### Delete a collection
+By default, deleting a non-empty collection returns `409 Conflict`:
+```bash
+curl -X DELETE http://localhost:50091/v1/projects/1001/collections/<COLLECTION_ID>
+# {"code":2,"message":"collection ... contains N documents; pass force=true to cascade-delete"}
+```
+Pass `force=true` to cascade-delete all documents first:
+```bash
+curl -X DELETE 'http://localhost:50091/v1/projects/1001/collections/<COLLECTION_ID>?force=true'
+```
 
-We welcome contributions! To get started:
+## Authentication
 
-1. Fork the repository.
-2. Create a feature branch: `git checkout -b my-feature`.
-3. Commit your changes: `git commit -m "Add my feature"`.
-4. Push to your fork: `git push origin my-feature`.
-5. Open a pull request.
+The server ships with `AnnonymousClaimParser` + `AllowEveryOne` authorisation — every request is accepted and mapped to the default project. This is fine for local development and demos; **configure a real JWT claim parser before exposing the service publicly**. See `pkg/server/app/app.go` for the middleware wiring.
 
-Please read our [Contributing Guidelines](CONTRIBUTING.md) for more details (to be created if not present).
+## Development
 
-## License
+### Common commands
+```bash
+make build           # compile server binary
+make test            # short tests only (no postgres required)
+make test-race       # short tests with -race
+make test-full       # full suite (requires run-postgres first)
+make vet             # go vet ./...
+make tidy            # go mod tidy
+make gen-proto       # regenerate api/pb/* from api/restcol.proto (requires buf)
+make clean           # remove build artifacts
+make help            # list all targets
+```
 
-This project is licensed under the Apache License 2.0. See the [LICENSE](LICENSE) file for details.
+### Repository layout
+
+| Path                    | Purpose                                                |
+|-------------------------|--------------------------------------------------------|
+| `main.go`               | entrypoint; parses flags and starts the server         |
+| `api/`                  | proto definitions + generated gRPC / OpenAPI clients   |
+| `pkg/app/`              | gRPC service handlers (collections, documents)         |
+| `pkg/server/app/`       | server assembly: middleware + storage + handlers       |
+| `pkg/server/`           | swagger / OpenAPI route registration                   |
+| `pkg/storage/`          | GORM-backed CRUD for projects, collections, documents  |
+| `pkg/models/`           | domain models (what the storage layer reads/writes)    |
+| `pkg/schema/`           | schema inference and field-path building               |
+| `pkg/bootstrap/`        | seeds the default project used for anonymous auth      |
+| `pkg/encoding/`         | JSON/CSV/XML payload decoders                          |
+| `pkg/runtime/js/`       | goja-based JavaScript runtime for evaluating swagger   |
+| `pkg/authn/`, `authz/`  | anonymous auth + allow-all authorisation (dev default) |
+| `integrationtest/`      | end-to-end tests against a live server                 |
+
+### Configuration
+
+The server reads Postgres connection settings from flags prefixed with `--restcol_`. See `run_local.sh` for a working example. Flags:
+
+| Flag                          | Default  |
+|-------------------------------|----------|
+| `--grpc_port`                 | `50090`  |
+| `--http_port`                 | `50091`  |
+| `--restcol_db_endpoint`       | —        |
+| `--restcol_db_name`           | —        |
+| `--restcol_db_user`           | —        |
+| `--restcol_db_password`       | —        |
+| `--restcol_auto_migrate`      | `false`  |
 
-## Roadmap
+## Contributing
 
-- Add support for additional storage backends.
-- Implement real-time streaming capabilities.
-- Enhance schema versioning and migration tools.
+1. Fork and create a feature branch.
+2. `make test-race` before pushing.
+3. Open a pull request against `main`.
 
-## Contact
+## License
 
-For questions or support, open an issue on the [GitHub Issues page](https://github.com/FootprintAI/restcol/issues) or reach out to the [FootprintAI team](https://github.com/FootprintAI).
+Apache License 2.0 — see [LICENSE](LICENSE).
 
----
+## Contact
 
-*Maintained by [FootprintAI](https://github.com/FootprintAI)*  
-*Last updated: March 02, 2025*
+Issues: <https://github.com/FootprintAI/restcol/issues>
diff --git a/api/buf.lock b/api/buf.lock
@@ -4,10 +4,10 @@ deps:
   - remote: buf.build
     owner: googleapis
     repository: googleapis
-    commit: 1f6ed065c9f04b5cb843d6e7603d6454
-    digest: shake256:7149cf5e9955c692d381e557830555d4e93f205a0f1b8e2dfdae46d029369aa3fc1980e35df0d310f7cc3b622f93e19ad276769a283a967dd3065ddfd3a40e13
+    commit: c17df5b2beca46928cc87d5656bd5343
+    digest: shake256:c62ecead9b13485a02893cd678a6c81e40879bf00ea509bbc6fd8f1b2cc33eccf6a85c259b08d1e0f052f693cbfc7dfda236e9665b1d6869b8e1132a794a61e2
   - remote: buf.build
     owner: grpc-ecosystem
     repository: grpc-gateway
-    commit: 18e8bf1967c947f5bd991710f490df5f
-    digest: shake256:67b115260e12cb2d6c5d5ce8dbbf3a095c86f0e52b84f9dbd16dec9433b218f8694bc9aadb1d45eb6fd52f5a7029977d460e2d58afb3208ab6c680e7b21c80e4
+    commit: 4836b6d552304e1bbe47e66a523f0daa
+    digest: shake256:9747ec8da8c45fe6e0c9860f5495897b28c77b985c2c65a75a54c22f2f1f168039f06925aca6cd8d856a723b60eb80d510d6db1f7d0f3bd27f8d91a8a8c6182c