Duckfile

Universal remote templating for DevOps tools

Duckfile lets you keep your Makefiles, Taskfiles, Helm values, and other config as remote templates, render them locally with variables, and run the tool seamlessly.

Features

• Git-sourced templates: repo + ref + path
• Variable tags: !env, !cmd, !file, and literals
• Automatic .env file loading for environment variable management
• Target descriptions + duck list for discoverability
• Go templates with Sprig functions
• Custom delimiters to avoid collisions (e.g., Taskfile)
• Deterministic caching with stable symlinks
• Checksum validation of remote templates
• Commit hash tracking and validation for reproducible builds
• Host allow/deny lists for supply-chain security
• Enterprise-grade security with digital signatures and access control
• Simple CLI that forwards args to your tool (make, task, helm, …)
• Render-only workflow via duck sync when you don't want duck to execute your tools

Security Features

Duck includes comprehensive security features to protect your DevOps workflows:

• 🔐 Digital Signatures: Ed25519 cryptographic signatures for configuration integrity
• 🛡️ Host Access Control: Allow/deny lists to prevent supply chain attacks
• 📁 File Permissions: Secure configuration file handling with validation
• ⚡ Precedence System: 5-tier security hierarchy with signed configs taking priority
• 🔧 Security CLI: Complete security management command suite
• 🌍 Environment Integration: Secure CI/CD and automation support

Quick security setup:

# Create security configuration
mkdir -p .duckfile
echo "version: 1
allowedHosts: [github.com, gitlab.com]
strictMode: true" > .duckfile/security.yaml

# Check security status
duck security status

# Optional: Add digital signatures
duck security generate-keys
duck security sign .duckfile/security.yaml

📖 Complete Security User Guide - Comprehensive guide covering all security features

Install

go install github.com/CyberDuck79/duckfile/cmd/duck@latest

Go 1.24.0+ recommended.

Quick start

1. Create duck.yaml at the repo root:

version: 1

default: build

targets:
  build:
    binary: make
    fileFlag: -f
    template:
      repo: https://github.com/CyberDuck79/duckfile-test-templates.git
      ref: main
      path: Makefile.tpl
      checksum: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
      trackCommitHash: true        # Track commit changes
      autoUpdateOnChange: true     # Auto-update when changed
    variables:
      PROJECT: my-service
      DATE: !cmd date +%Y-%m-%d
    renderedPath: Makefile

  test:
    binary: task
    fileFlag: --taskfile
    template:
      repo: https://github.com/CyberDuck79/duckfile-test-templates.git
      ref: v2.3.3
      path: task/Taskfile.yml.tpl
      delims: { left: "[[", right: "]]" }  # avoid Task's {{ }}
      allowMissing: true                   # missing vars => ""
      trackCommitHash: true                # Track tags for updates
    variables:
      GO_VERSION: !env GO_VERSION
      PLATFORM: linux/amd64
    args: ["--silent"]

  docs:
    # No binary: this target is sync-only. Use `duck sync docs` to render.
    template:
      repo: https://github.com/CyberDuck79/duckfile-test-docs-templates.git
      ref: main
      path: index.md.tpl
    variables:
      AUTHOR: Cyberduck

2. Run

# print version
duck version

# run default target (renders Makefile and calls make -f Makefile)
duck

# run a named target and pass additional args after --
duck test --

# explicitly run a target (useful when target names conflict with subcommands)
duck exec build
duck exec sync    # runs 'sync' target, not sync subcommand
duck exec test -- --verbose

# list targets (names, binaries, descriptions)
duck list
# include remote info / variable kinds / execution line
duck list -rve

# render-only workflows (no binary execution)
# sync all targets into cache and update symlinks
duck sync
# force re-render ignoring cache
duck sync -f
# clean cache for all or a single target
duck clean
duck clean test

# verbosity / debugging
# show high-level steps (render, cache hits, pruning, exec line)
duck --log-level=info
# extremely detailed (includes variable values, paths, clone steps)
duck --log-level=debug
duck sync --log-level=debug   # set log level for subcommands

# commit hash tracking and validation
# enable tracking with manual updates (warns about changes)
duck build --track-commit-hash
# enable tracking with automatic updates (transparent updates)  
duck sync --track-commit-hash --auto-update-on-change
# disable tracking entirely (backward compatible)
duck --no-track-commit-hash build

# security management commands
# check security configuration status
duck security status
# verify security configuration and signatures
duck security verify
# generate cryptographic key pairs
duck security generate-keys
# sign security configurations
duck security sign .duckfile/security.yaml
# check and fix file permissions
duck security check-permissions
duck security fix-permissions --all

# host access control via CLI flags
# restrict to specific hosts
duck --allowed-hosts github.com,gitlab.com build
# block specific hosts
duck --denied-hosts malicious-host.com sync  
# enforce strict host validation
duck --strict-hosts build

Security Features

Duckfile includes comprehensive supply-chain security features to prevent malicious template injection and enforce organizational policies:

Phase 1-4: Comprehensive Security System

✅ Implemented Phases:

• Phase 1: Enhanced SecurityConfig with comprehensive validation
• Phase 2: Ed25519 digital signatures for configuration integrity
• Phase 3: File permission validation and security checks
• Phase 4: Policy Enforcement Points - Comprehensive policy validation with violation reporting

Host Allow/Deny Lists

Control which Git hosts can be accessed for templates. Security configurations are kept separate from duck.yaml to prevent attackers from modifying both targets and security policies together.

Configuration Precedence (highest to lowest):

1. CLI Flags - Override everything else
2. Environment Variables - System-level defaults
3. Security Config File - Centralized policy management
4. No Restrictions - Allow all hosts (default)

# CLI flags (highest precedence)
duck build --allowed-hosts github.com,gitlab.internal.com
duck sync --denied-hosts malicious-host.com --strict-hosts

# Environment variables (system-level)
export DUCK_ALLOWED_HOSTS="github.com,gitlab.internal.com"
export DUCK_DENIED_HOSTS="malicious-host.com" 
export DUCK_STRICT_MODE="true"  # Fail if no restrictions configured

# Security config file (recommended for organizations)
# Create ~/.duck/security.yaml or use --security-config flag
duck build --security-config /etc/duck/production-security.yaml

Policy Enforcement Points (Phase 4)

Comprehensive policy validation ensures compliance with organizational security requirements:

Policy Types:

• Checksum Validation: Force all templates to have checksums
• Commit Tracking: Require commit hash tracking for reproducibility
• Auto-Update Control: Override auto-update settings for stability
• File Permissions: Validate file ownership and permissions
• Template Validation: Ensure secure template configurations
• Host Access Control: Enforce repository access policies

Policy Violation Reporting:

# Policy violations are reported with detailed messages
duck sync --security-config production-security.yaml

# Example output:
# ❌ Policy Violations Found:
# • Template 'config-template' missing required checksum (forceChecksumValidation)
# • Template 'app-config' has trackCommitHash=false (forceCommitTracking)  
# • File '/etc/app/config.yaml' has insecure permissions 0644 (enforceFilePermissions)
# 
# 🔧 Policy Overrides Applied:
# • Template 'dynamic-config': autoUpdateOnChange disabled (disableAutoUpdate)
# 
# 📊 Policy Enforcement Summary: 3 violations, 1 override, operation failed

Security Rules:

• Deny beats allow: Denied hosts are blocked even if in allow list
• Strict mode: Fail if no restrictions are configured
• Fast validation: Host checking happens before git operations
• Case insensitive: GitHub.COM matches github.com
• Policy enforcement: Mandatory overrides ensure compliance
• Detailed reporting: Clear violation messages for remediation

See the security schema, example configurations, and full specification for complete details.

Commit Hash Validation

Duckfile can track and validate commit hashes to detect when remote templates change, ensuring reproducible builds and providing early warning of template updates.

Basic Usage

Configuration in duck.yaml:

targets:
  build:
    template:
      repo: https://github.com/example/templates.git
      ref: main  # Use branch/tag names, not commit hashes
      path: Makefile.tpl
      trackCommitHash: true        # Enable tracking
      autoUpdateOnChange: true     # Auto-update when commits change

# Or set globally in settings
settings:
  trackCommitHash: true
  autoUpdateOnChange: false  # Manual updates by default

CLI flags override configuration:

# Enable tracking with manual updates
duck build --track-commit-hash

# Enable tracking with automatic updates
duck sync --track-commit-hash --auto-update-on-change

# Disable tracking entirely
duck --no-track-commit-hash build

Environment variables for system-wide defaults:

export DUCK_TRACK_COMMIT_HASH="true"
export DUCK_AUTO_UPDATE_ON_CHANGE="true"
duck build  # Uses environment settings

How It Works

First Run (tracking enabled):

1. Fetches template from repo@ref
2. Resolves branch/tag to actual commit hash
3. Stores commit hash alongside cached template
4. Renders and executes normally

Subsequent Runs:

1. Checks if remote commit hash has changed
2. Without auto-update: Warns about changes, uses cached template
3. With auto-update: Automatically re-fetches and updates cache

Configuration Precedence

Settings are resolved in this order (highest to lowest priority):

1. CLI flags: --track-commit-hash, --no-track-commit-hash, --auto-update-on-change, --no-auto-update-on-change
2. Environment variables: DUCK_TRACK_COMMIT_HASH, DUCK_AUTO_UPDATE_ON_CHANGE
3. Template config: template.trackCommitHash, template.autoUpdateOnChange
4. Global config: settings.trackCommitHash, settings.autoUpdateOnChange
5. Default: false (disabled)

Example Workflows

Development (auto-update enabled):

# Templates update automatically as upstream changes
duck build --track-commit-hash --auto-update-on-change
# Output: "📦 Updating template cache: repo@main" (if changed)

Production (manual updates):

# Get warned about changes but continue with cached template
duck build --track-commit-hash --no-auto-update-on-change  
# Output: "🔄 commit hash changed for repo@main: abc123 -> def456"

Strict reproducibility (tracking disabled):

# Use exact commit hashes in config, disable tracking
duck build --no-track-commit-hash
# Template ref: "a1b2c3d4e5f6..." (40-char commit hash)

Validation Rules

• Branch/tag only: Commit hash tracking requires ref to be a branch or tag name, not a commit hash
• Auto-update dependency: autoUpdateOnChange requires trackCommitHash to be enabled
• Network resilience: Network failures during validation result in warnings, not errors
• Fast feedback: Remote checking happens before expensive git operations

Configuration File Discovery

Duck automatically discovers your configuration file or you can specify a custom path using the --config flag or DUCK_CONFIG environment variable.

Discovery Precedence (highest to lowest):

1. CLI flag: --config custom.yaml or -c custom.yaml
2. Environment variable: DUCK_CONFIG=custom.yaml
3. Auto-discovery: duck.yaml, duck.yml, .duck.yaml, .duck.yml

Examples:

# Use custom config file (highest precedence)
duck --config prod.yaml build
duck -c staging.yaml deploy

# Use environment variable
export DUCK_CONFIG="configs/dev.yaml"
duck build
duck sync

# Auto-discovery (default behavior)
duck build  # Uses duck.yaml, duck.yml, .duck.yaml, or .duck.yml

# Multi-environment workflows
duck --config configs/dev.yaml sync
duck --config configs/staging.yaml deploy --dry-run
duck --config configs/prod.yaml deploy

# CI/CD integration
DUCK_CONFIG="ci/pipeline.yaml" duck test
DUCK_CONFIG="ci/deploy.yaml" duck deploy

Use Cases:

• Environment-specific configs: Separate files for dev/staging/prod
• Team customization: Personal config variants
• CI/CD pipelines: Different configs per build stage
• Multi-project support: Multiple duck files in the same workspace

Logging Configuration

Control verbosity with log levels, configured via CLI flag, environment variable, or config file.

Precedence (highest to lowest):

1. CLI flag: --log-level=debug
2. Environment variable: DUCK_LOG_LEVEL=info
3. Config file: settings.logLevel: warn
4. Default: info

Log Levels (from least to most verbose):

• error: Only critical errors
• warn: Warnings and errors
• info: High-level steps (default)
• debug: Detailed debugging information

Examples:

# Set via CLI flag (highest precedence)
duck --log-level=debug build
duck sync --log-level=warn

# Set via environment variable
export DUCK_LOG_LEVEL=debug
duck build

# Set via config file (duck.yaml)
settings:
  logLevel: info

Target Name Conflicts and Explicit Execution

When target names conflict with Duck's built-in subcommands (like sync, list, clean), Duck shows warnings and provides the exec command for explicit target execution:

Example conflict:

targets:
  sync:  # Conflicts with 'duck sync' subcommand
    binary: rsync
    # ... template config

Behavior:

# This runs the subcommand, not the target
duck sync

# Warning is shown for subcommands when conflicts exist
duck list
# ⚠️  Target 'sync' conflicts with subcommand. Use 'duck exec sync' to run the target.

# Use exec to explicitly run the target
duck exec sync          # Runs 'sync' target
duck exec sync -- -v    # Runs 'sync' target with args

Available commands that can conflict:

• sync, list, clean, add, init, security, version

Environment variable management with .env files

Duckfile automatically loads environment variables from .env files to streamline configuration management:

Create a .env file:

# Project defaults
PROJECT_NAME=myapp
GO_VERSION=1.21
DOCKER_REGISTRY=ghcr.io/myorg
DEBUG=true

Use in duck.yaml:

targets:
  build:
    variables:
      PROJECT: !env PROJECT_NAME      # Uses value from .env
      GO_VERSION: !env GO_VERSION     # Uses value from .env
      REGISTRY: !env DOCKER_REGISTRY  # Uses value from .env

File discovery order:

1. .env (current directory) - highest priority
2. .duck/.env (duck cache directory)
3. .env.duck (duck-specific variant) - lowest priority

Precedence: CLI environment variables > .env file > defaults

Environment Variables for Advanced Templates

Duckfile automatically exposes repository and template paths as environment variables during execution, enabling advanced template workflows:

# Available in your executed commands
echo "Repository: ${DUCK_REPO_PATH}"
echo "Template: ${DUCK_TEMPLATE_PATH}"
echo "Target: ${DUCK_TARGET_NAME}"

# Copy repository assets
cp -r "${DUCK_REPO_PATH}/assets" ./

# Execute repository scripts  
"${DUCK_REPO_PATH}/scripts/setup.sh"

Available Variables:

Variable	Description	Example
DUCK_REPO_PATH	Path to cloned repository	.duck/objects/remote/abc123
DUCK_REPO_URL	Repository URL	https://github.com/org/templates.git
DUCK_REPO_REF	Git reference used	main
DUCK_TEMPLATE_PATH	Source template file path	.duck/objects/template/ghi789/raw.tpl
DUCK_RENDERED_PATH	Rendered template file path	.duck/objects/rendered/def456/Makefile
DUCK_SYMLINK_PATH	Symlink path (what user sees)	.duck/build/Makefile
DUCK_TARGET_NAME	Target name being executed	build
DUCK_CACHE_DIR	Per-target cache directory	.duck/build

Example Use Cases:

# Template that copies assets alongside rendering
targets:
  frontend:
    binary: npm
    fileFlag: --config
    template:
      repo: https://github.com/company/frontend-templates.git
      path: package.json.tpl
    args: ["run", "build"]
    # Script can use: cp -r "${DUCK_REPO_PATH}/public/*" ./public/
    
  deploy:
    binary: bash
    fileFlag: -c
    template:
      repo: https://github.com/company/deploy-scripts.git  
      path: deploy.sh.tpl
    # Template renders to a script that references other files in the repo
    # Script content: source "${DUCK_REPO_PATH}/common.sh"

See the specification for complete documentation.

How it works

• Load .env files automatically before any operation
• Resolve variables:
  • !env NAME → os.Getenv(NAME)
  • !cmd SHELL → /bin/sh -c SHELL (trimmed)
  • !file PATH → file contents
  • literal scalars (string/number/bool)
• Two‑tier deterministic caching:
  • Remote layer: key = SHA1(repo + ref + path)
    • Directory: .duck/objects/remote/<remoteKey>/
    • Contains: raw template file (<basename>), commit.hash (always written on fetch), optional checksum.sha256
  • Rendered layer: key = SHA1(sorted resolvedVarsJSON)
    • Directory: .duck/objects/rendered/<renderedKey>/
    • Contains: rendered file (<basename>), remote.key (points back to remote layer key)
• Remote fetch happens only when the remote layer is missing, explicitly forced (-f), checksum mismatch, or commit hash validation (when tracking enabled) says it changed and auto‑update is on.
• Variable changes never trigger a remote fetch; they only produce a new rendered layer directory and prune the old one for that target.
• Commit hash behavior:
  • The actual commit hash is always captured and stored in commit.hash on every remote fetch (even if tracking disabled later).
  • When trackCommitHash is enabled, Duck validates the stored hash against the current remote; if changed:
    • With autoUpdateOnChange: remote layer is refreshed, then re-rendered.
    • Without auto-update: warning logged; existing remote layer kept.
• Rendering uses Go text/template + Sprig with optional custom delimiters.
• A stable symlink at renderedPath (or .duck/<target>/<basename>) points to the current rendered object.
• Execute the tool: binary fileFlag renderedPath [args …] (unless target has no binary).
• Use duck sync for render-only workflows (no binary required).

Templating tips

• Use Sprig to transform values: {{ .PROJECT | upper }}
• Add now/env helpers: {{ now }} and {{ env "HOME" }}
• When the generated file itself uses Go templates (e.g., Taskfile), set delims so our engine renders only your placeholders and leaves the downstream engine’s {{ ... }} intact.
• If you want missing variables to become empty strings, set allowMissing: true. Default is strict.

Project layout

Path	Purpose
cmd/	Entry point (main.go)
cmd/duck/	Cobra command (root.go)
internal/config/	Parser for duck.yaml
internal/git/	Git wrapper for clone/fetch/checkout
internal/run/	Render + cache + exec

Troubleshooting

• git exit status 128: usually wrong ref or network; error message includes git’s stderr.
• “map has no entry …” during rendering: you are missing a variable and allowMissing is false, or your delimiters collide with the target tool (set delims).
• On macOS, if a symlink isn’t resolving, remove it and re-run; Duck recreates it.

Spec

See the full specification: docs/spec.md

Documentation

• Configuration Specification - Complete duck.yaml configuration reference
• Security User Guide - Comprehensive security features guide
• Security Schema - JSON schema for security configurations

Using Duckfile without executing tools

If you prefer Duckfile to only manage templates and never launch external binaries, omit binary from your targets. You can then:

• duck sync [target] [-f] to render and refresh symlinks
• duck clean [target] to purge caches

If a target has no binary, attempting to execute it via the root command will error with guidance to use sync instead.