rots - Remote OTS Commander

Service orchestration CLI for OneTimeSecret infrastructure.

Dual-purpose management tool:

• Container orchestration: Containerized OTS deployments via Podman Quadlets (systemd integration)
• Service management: Native systemd services for dependencies (Valkey, Redis)

Installation

With pipx (Recommended)

pipx installs CLI tools in isolated environments, preventing dependency conflicts and enabling clean upgrades.

# Install pipx if needed
pip install pipx
pipx ensurepath

# Install rots
pipx install rots

# Or from git
pipx install git+https://github.com/onetimesecret/ots-containers.git

Migrating from pip to pipx

If you previously installed with pip:

pip uninstall rots
pipx install rots

With pip

Not recommended for production. Use pipx instead.

pip install rots

From source

git clone https://github.com/onetimesecret/ots-containers.git
cd ots-containers
pipx install .

From a git branch

Install a specific branch, tag, or commit for testing:

# Install from branch
pipx install --force git+https://github.com/onetimesecret/rots.git@feature/sidecar

# Install from tag
pipx install --force git+https://github.com/onetimesecret/rots.git@v0.6.0

# Install from commit
pipx install --force git+https://github.com/onetimesecret/rots.git@abc123f

Upgrading

# Check for updates
rots self check

# Upgrade to latest
rots self upgrade

# Upgrade to specific version
rots self upgrade --version 0.24.0

The rots self upgrade command wraps pipx and is safe to invoke remotely via the sidecar.

Usage

rots --help
rots --version

Instance Types

Three container types with explicit systemd unit naming:

Type	Unit Name	Identifier	Use
--web	onetime-web@{port}	Port number	HTTP servers
--worker	onetime-worker@{id}	Name/number	Background jobs
--scheduler	onetime-scheduler@{id}	Name/number	Scheduled tasks

Managing OTS Containers

# List all instances
rots instances
rots instances --json

# List by type
rots instances --web
rots instances --worker
rots instances --scheduler

# Deploy instances
rots instances deploy --web 7043 7044
rots instances deploy --worker billing emails
rots instances deploy --scheduler main

# Redeploy (regenerate quadlet and restart)
rots instances redeploy                    # all running
rots instances redeploy --web 7043         # specific

# Start/stop/restart
rots instances start --web 7043
rots instances stop --scheduler main
rots instances restart                     # all running

# Status and logs
rots instances status
rots instances logs --web 7043 -f
rots instances logs --scheduler main -f

# Enable/disable at boot
rots instances enable --web 7043
rots instances disable --scheduler main -y

# Interactive shell
rots instances exec --web 7043

Managing systemd Services (Valkey, Redis)

# Initialize new service instance
rots service init valkey 6379
rots service init redis 6380 --bind 0.0.0.0

# Start/stop/restart
rots service start valkey 6379
rots service stop redis 6380
rots service restart valkey 6379

# Status and logs
rots service status valkey 6379
rots service logs valkey 6379 --follow

# Enable/disable at boot
rots service enable valkey 6379
rots service disable redis 6380

# List available service packages
rots service

Generating Cloud-Init Configurations

# Generate basic cloud-init config
rots cloudinit generate > user-data.yaml

# Include PostgreSQL repository
rots cloudinit generate --include-postgresql --postgresql-key /path/to/pgdg.asc

# Include Valkey repository
rots cloudinit generate --include-valkey --valkey-key /path/to/valkey.gpg

# Validate configuration
rots cloudinit validate user-data.yaml

Sidecar Daemon

The sidecar daemon enables remote control of OTS instances via RabbitMQ or local control via Unix socket.

# Install and start the sidecar
rots sidecar install           # Write systemd unit (auto-detects rots path)
rots sidecar start             # Start daemon
rots sidecar status            # Check daemon status
rots sidecar logs --follow     # View logs

# Send commands via Unix socket (local, default)
rots sidecar send health --socket
rots sidecar send status --socket
rots sidecar send restart.web identifier=7043 --socket

# Send commands via RabbitMQ (remote)
rots sidecar send health --rabbitmq
rots sidecar send status --rabbitmq
rots sidecar send rots.proxy.reload --rabbitmq

# Trigger remote self-upgrade from git branch
rots sidecar send rots.self.upgrade \
  args=--source \
  args=git+https://github.com/onetimesecret/rots.git@main \
  --rabbitmq

Remote Control via SSH Tunnel

To send RabbitMQ commands from a local machine:

# Forward RabbitMQ port through SSH
ssh -L 5672:maindb:5672 user@server

# Send command through the tunnel
RABBITMQ_URL=amqp://user:pass@localhost:5672/ots_production \
  rots sidecar send health --rabbitmq

Configuration via .otsinfra.env

The sidecar send --rabbitmq command reads RABBITMQ_URL from:

1. Environment variable (RABBITMQ_URL=...)
2. Walk-up discovery of .otsinfra.env files
3. /etc/default/onetimesecret (on the server, for the daemon)

This enables per-jurisdiction targeting. Place .otsinfra.env files in your ops directory:

ops-jurisdictions/
  eu/.otsinfra.env     # OTS_HOST=eu-prod RABBITMQ_URL=amqp://...
  ca/.otsinfra.env     # OTS_HOST=ca-prod RABBITMQ_URL=amqp://...
  us/.otsinfra.env     # OTS_HOST=us-prod RABBITMQ_URL=amqp://...

When you cd into a jurisdiction and run sidecar commands, the walk-up resolver finds the appropriate .otsinfra.env automatically:

cd ops-jurisdictions/eu
rots sidecar send health --rabbitmq  # Uses eu/.otsinfra.env

Environment Variables

# Use a specific image tag
TAG=v0.23.0 rots instances redeploy --web 7043

# Use a different image
IMAGE=ghcr.io/onetimesecret/onetimesecret TAG=latest rots instances deploy --web 7044

Prerequisites

• Linux with systemd
• Podman installed and configured
• Python 3.11+

Server Setup

FHS-compliant directory structure:

OTS Container Configuration

/etc/onetimesecret/              # System configuration
├── config.yaml                  # Application configuration
├── auth.yaml                    # Authentication config
└── logging.yaml                 # Logging config

/etc/default/onetimesecret       # Environment file (shared by all instances)

/etc/containers/systemd/         # Quadlet templates (managed by tool)
├── onetime-web@.container
├── onetime-worker@.container
└── onetime-scheduler@.container

/var/lib/onetimesecret/          # Runtime data
└── deployments.db               # Deployment timeline (SQLite)

Service Configuration (Valkey/Redis)

/etc/valkey/                     # Valkey system configuration
├── valkey.conf                  # Default config template
└── instances/                   # Instance configs (created by tool)
    ├── 6379.conf
    └── 6379-secrets.conf        # Secrets file (mode 0640)

/var/lib/valkey/                 # Runtime data
└── 6379/
    └── dump.rdb

How It Works

Container Management

1. Quadlet templates: Writes systemd unit templates to /etc/containers/systemd/
2. Environment: Reads from /etc/default/onetimesecret
3. Secrets: Uses Podman secrets for sensitive values
4. Timeline: Records deployments to SQLite for audit and rollback

Service Management

1. Config files: Copies package defaults to instance-specific configs
2. Secrets: Creates separate secrets files with restricted permissions
3. Data directories: Creates per-instance data directories with correct ownership
4. systemd: Manages services using package-provided templates

Troubleshooting

# Check instance status
rots instances status
systemctl status onetime-web@7043

# View logs
rots instances logs --web 7043 -f
journalctl -u onetime-web@7043 -f

# Unified log filtering (all instance types)
journalctl -t onetime -f

# List all onetime systemd units
systemctl list-units 'onetime-*'

# Verify Quadlet templates
cat /etc/containers/systemd/onetime-web@.container

# Reload systemd after manual changes
systemctl daemon-reload

Sidecar Missing Dependencies

If the sidecar fails with No module named 'pika', inject it into the pipx environment:

pipx inject rots pika
systemctl restart onetime-sidecar

Shell Command Not Found After pipx Install

If you get "command not found" after installing with pipx while a venv is active, clear the shell's command cache:

hash -r
# or deactivate the venv first
deactivate

Development

# Editable install
git clone https://github.com/onetimesecret/ots-containers.git
cd ots-containers
pip install -e ".[dev,test]"

# Run tests
pytest tests/

# Run with coverage (CI threshold: 70%)
pytest tests/ --cov=ots_containers --cov-fail-under=70

# Pre-commit hooks
pre-commit install

Running as root

# Use full path
sudo /home/youruser/.local/bin/rots instances status

# Or create symlink
sudo ln -s /home/youruser/.local/bin/rots /usr/local/bin/rots

License

MIT