Version v0.13.0 - GitHub OAuth, redesigned UI, Go code refactoring

.github/workflows/ci.yml

@@ -27,6 +27,17 @@ jobs:
       - name: Run Tests
         run: make GOBIN=$HOME/gopath/bin test
 
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.13'
+
+      - name: Create Python Virtual Environment
+        run: python -m venv .pyvenv
+
+      - name: Run API Unit Tests
+        run: make api-test
+
       - name: Run Coverage Tests
         env:
           COVERALLS_TOKEN: ${{ secrets.COVERALLS_TOKEN }}
@@ -112,12 +123,15 @@ jobs:
         env:
           # Set ROLE_ARN based on the branch
           ROLE_ARN: ${{ github.ref == 'refs/heads/master' && secrets.PROD_LAMBDA_ROLE_ARN || secrets.STAGING_LAMBDA_ROLE_ARN }}
+          # Set GitHub OAuth credentials based on the branch
+          GITHUB_OAUTH_CLIENT_ID: ${{ github.ref == 'refs/heads/master' && secrets.PROD_GITHUB_OAUTH_CLIENT_ID || secrets.STAGING_GITHUB_OAUTH_CLIENT_ID }}
+          GITHUB_OAUTH_CLIENT_SECRET: ${{ github.ref == 'refs/heads/master' && secrets.PROD_GITHUB_OAUTH_CLIENT_SECRET || secrets.STAGING_GITHUB_OAUTH_CLIENT_SECRET }}
         run: |
           cd gogen-api
           if [ "${{ github.ref }}" = "refs/heads/master" ]; then
-            # Prod deployment, ROLE_ARN is already set via env
+            # Prod deployment
             bash deploy_lambdas.sh
           else
-            # Staging deployment, ROLE_ARN is already set via env
+            # Staging deployment
             bash deploy_lambdas.sh -e staging
           fi

.github/workflows/release.yml

@@ -172,6 +172,8 @@ jobs:
       - name: Deploy Lambda Functions
         env:
           ROLE_ARN: ${{ secrets.PROD_LAMBDA_ROLE_ARN }}
+          GITHUB_OAUTH_CLIENT_ID: ${{ secrets.PROD_GITHUB_OAUTH_CLIENT_ID }}
+          GITHUB_OAUTH_CLIENT_SECRET: ${{ secrets.PROD_GITHUB_OAUTH_CLIENT_SECRET }}
         run: |
           cd gogen-api
           bash deploy_lambdas.sh

.gitignore

@@ -21,11 +21,13 @@ roveralls*
 .specstory
 .pyvenv
 gogen-api/__pycache__
+__pycache__/
 gogen-api/build
+gogen-api/env.json
 ui/node_modules/*
 ui/dist/*
 ui/build/*
 ui/coverage/*
 ui/public/gogen.wasm
 ui/.vite
-*.idea
+*.idea

AGENTS.md

@@ -0,0 +1,116 @@
+# AGENTS.md
+
+This file gives coding agents the repo-specific context needed to work effectively in `gogen`.
+
+## Project Overview
+
+Gogen is a data generator for demo and test data, especially time-series logs and metrics. The repo contains:
+
+- a Go CLI core
+- a Python AWS Lambda API backend in `gogen-api/`
+- a React/TypeScript UI in `ui/`
+
+## Common Commands
+
+### Go
+
+```bash
+make install          # Preferred install path; injects ldflags from Makefile
+make build            # Cross-compiles linux, darwin, windows, wasm
+make test             # go test -v ./...
+go test -v ./internal
+go test -v -run TestName ./internal
+```
+
+Notes:
+
+- Use `make install` instead of bare `go install`; version/build metadata and OAuth settings are injected through `-ldflags`.
+- Dependencies are vendored. After dependency changes, run `go mod vendor`.
+
+### Python API
+
+```bash
+cd gogen-api
+./start_dev.sh
+./setup_local_db.sh
+./deploy_lambdas.sh
+```
+
+Repo-standard Python environment:
+
+```bash
+source /home/clint/local/src/gogen/.pyvenv/bin/activate
+```
+
+Focused API unit tests:
+
+```bash
+make api-test
+```
+
+### UI
+
+```bash
+cd ui
+npm run dev
+npm run build
+npm test
+```
+
+## Architecture
+
+### Go Package Layout
+
+- `main.go`: CLI entry point using `urfave/cli.v1`; maps flags to `GOGEN_*` env vars
+- `internal/`: core config, sample, token, API/share logic
+- `generator/`: generation workers
+- `outputter/`: output workers and destinations
+- `run/`: pipeline orchestration
+- `timer/`: one timer goroutine per sample
+- `rater/`: event-rate control
+- `template/`: output formatting
+- `logger/`: log wrapper
+
+### Data Flow
+
+```text
+YAML/JSON config -> internal.Config singleton
+  -> timer goroutines
+  -> generator worker pool
+  -> outputter worker pool
+  -> output destination
+```
+
+### Config System
+
+- Config is a singleton guarded by `sync.Once`
+- Remote configs default to `https://api.gogen.io` and can be overridden by `GOGEN_APIURL`
+- In Go tests, reset config state with `config.ResetConfig()` before `config.NewConfig()`
+- Tests often use `config.SetupFromString(...)` for inline YAML
+
+### Python API
+
+- Lambda handlers live as separate files in `gogen-api/api/`
+- Backed by DynamoDB + S3
+- Local development uses Docker Compose plus SAM
+- Use `.pyvenv` rather than system Python when running repo Python commands
+
+### UI
+
+- Vite + React 18 + TypeScript + Tailwind
+- Components live in `ui/src/components/`
+- Pages live in `ui/src/pages/`
+- Tests are colocated as `.test.tsx`
+
+## CI/CD
+
+- `.github/workflows/ci.yml` runs Go tests on pushes to `master`/`dev` and on PRs
+- CI also runs `make api-test`
+- Branch builds/deploys happen on `master` and `dev`
+- Release workflow is handled separately in `.github/workflows/release.yml`
+
+## Practical Notes
+
+- Prefer minimal, targeted edits; this repo spans Go, Python, and frontend code in one tree
+- For Python work, prefer adding tests that avoid external AWS dependencies unless the task explicitly needs integration coverage
+- For UI tests, keep them aligned with the current design system rather than hardcoding old color classes

CLAUDE.md

@@ -0,0 +1,102 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Gogen is an open source data generator for generating demo and test data, especially time series log and metric data. It's a Go CLI tool with an embedded Lua scripting engine, a Python AWS Lambda API backend, and a React/TypeScript UI.
+
+## Common Commands
+
+### Go (core CLI)
+
+```bash
+make install          # Build and install to $GOPATH/bin (default target)
+make build            # Cross-compile for linux, darwin, windows, wasm
+make test             # Run all Go tests: go test -v ./...
+go test -v ./internal # Run tests for a single package
+go test -v -run TestName ./internal  # Run a single test
+```
+
+Version, git summary, build date, and GitHub OAuth credentials are injected via `-ldflags` in the Makefile. Always use `make install` rather than bare `go install`.
+
+Dependencies are vendored in `vendor/`. After adding deps, run `go mod vendor`.
+
+### Python API (`gogen-api/`)
+
+```bash
+cd gogen-api
+./start_dev.sh        # Starts DynamoDB Local + MinIO via docker-compose, then SAM local API on port 4000
+./setup_local_db.sh   # Seeds local DynamoDB schema
+sam build && sam local start-api --port 4000 --docker-network lambda-local
+./deploy_lambdas.sh   # Deploy to AWS (requires credentials)
+```
+
+### UI (`ui/`)
+
+```bash
+cd ui
+npm run dev           # Vite dev server (copies wasm from build/wasm/ first)
+npm run build         # Production build
+npm test              # Jest tests
+```
+
+## Architecture
+
+### Go Package Layout
+
+All packages are at the top level (no `cmd/` or `pkg/` convention):
+
+- **`main.go`** — CLI entry point using `urfave/cli.v1`. Maps CLI flags to `GOGEN_*` env vars.
+- **`internal/`** — Core package. Config singleton, `Sample` struct, `Token` processing, API client, sharing. Imported as `config` throughout (`config "github.com/coccyx/gogen/internal"`).
+- **`generator/`** — Reads `GenQueueItem` from channel, dispatches to sample-based or Lua generators.
+- **`outputter/`** — Reads `OutQueueItem` from channel, dispatches to output destinations (stdout, file, HTTP, Kafka, network, devnull, buf).
+- **`run/`** — Orchestrates the pipeline: timers -> generator worker pool -> outputter worker pool.
+- **`timer/`** — One timer goroutine per Sample; handles backfill and realtime intervals.
+- **`rater/`** — Controls event rate (config-based, time-of-day/weekday, kbps, Lua script).
+- **`template/`** — Output formatting (raw, JSON, CSV, splunkhec, syslog, elasticsearch).
+- **`logger/`** — Thin logrus wrapper with file/func/line context hook.
+
+### Data Flow
+
+```
+YAML/JSON Config -> internal.Config singleton (sync.Once)
+    -> [Timer goroutine per Sample]
+    -> GenQueueItem channel -> [Generator worker pool]
+    -> OutQueueItem channel -> [Outputter worker pool]
+    -> output destination
+```
+
+Concurrency is channel + goroutine worker pools. Worker counts set by `GeneratorWorkers` and `OutputWorkers` config fields.
+
+### Key Interfaces
+
+- `internal.Generator` — `Gen(item *GenQueueItem) error`
+- `internal.Outputter` — `Send(events []map[string]string, sample *Sample, outputTemplate string) error`
+- `internal.Rater` — `EventsPerInterval(s *Sample) int`
+
+### Config System
+
+Config is a **singleton** via `sync.Once`. Controlled by environment variables:
+- `GOGEN_HOME`, `GOGEN_FULLCONFIG`, `GOGEN_CONFIG_DIR`, `GOGEN_SAMPLES_DIR`
+- Remote configs fetched from `https://api.gogen.io` (override with `GOGEN_APIURL`)
+
+In tests, call `config.ResetConfig()` before `config.NewConfig()` to get a fresh instance. Tests commonly use `config.SetupFromString(yamlStr)` to inject inline YAML config.
+
+### gogen-api (Python Lambda)
+
+Each Lambda function is a separate `.py` file in `gogen-api/`. Backed by DynamoDB + S3. Originally Python 2.7, being updated to Python 3. AWS SAM template at `gogen-api/template.yaml`.
+
+### UI (React/TypeScript)
+
+Vite + React 18 + TypeScript + Tailwind CSS. Components in `src/components/`, pages in `src/pages/`, API clients in `src/api/`, types in `src/types/`. Tests use Jest + React Testing Library, placed adjacent to source as `.test.tsx`.
+
+## CI/CD
+
+GitHub Actions (`.github/workflows/ci.yml`):
+- Push to `master`/`dev` or any PR: runs `make test`, then on `master`/`dev` cross-compiles, builds Docker, pushes artifacts to S3, deploys UI and Lambdas.
+- Tag pushes (`v*.*.*`): full release workflow via `release.yml` — builds, creates GitHub release, pushes Docker images, deploys to production.
+
+## Lua Scripting
+
+Generators (`generator/lua.go`) and raters (`rater/script.go`) support embedded Lua via `gopher-lua` + `gopher-luar`. Lua state persists across calls within a run.

Makefile

@@ -7,7 +7,7 @@ SUMMARY = $(shell git describe --tags --always --dirty)
 DATE = $(shell date --rfc-3339=date)
 
 
-.PHONY: all build deps install test docker splunkapp embed
+.PHONY: all build deps install test api-test docker splunkapp embed
 
 ifeq ($(OS),Windows_NT)
 	dockercmd := docker run -e TERM -e HOME=/go/src/github.com/coccyx/gogen --rm -it -v $(CURDIR):/go/src/github.com/coccyx/gogen -v $(HOME)/.ssh:/root/.ssh clintsharp/gogen bash
@@ -33,6 +33,8 @@ install:
 test:
 	go test -v ./...
 
+api-test:
+	./.pyvenv/bin/python -m unittest gogen-api/test_auth_utils.py gogen-api/test_upsert_auth.py
+
 docker:
 	$(dockercmd)
-

VERSION

@@ -1 +1 @@
-0.12.1
+0.13.0