Build Server Coordination Repository

âš ï¸� REQUIRED READING FOR ALL BUILD SERVERS: Before joining the hive or performing any build operations, all build servers MUST read the entire ShapeBlue Hackerbook. This document contains critical security, development, and operational guidelines that all nodes must follow.

Quick note for operators: if you just tell the agent "follow the instructions", it will execute the exact checklist below automatically. The steps are documented here so you don't need to repeat them next time.

Build Infrastructure

Linux Build Servers

Build1 (10.1.3.175) - OpenAI Codex Build2 (10.1.3.177) - GitHub Copilot

Both builders have SSH access to each other. You can SSH anytime to check implementations, sync files, or debug:

# From Build2 to Build1
ssh root@10.1.3.175 "command"

# From Build1 to Build2  
ssh root@10.1.3.177 "command"

Common uses:

Check how the other builder implemented a feature
Compare scripts or configurations
Trigger notifications or message checks
Sync shared files or coordination data

Windows Development Servers

Windows Servers Inventory

10.1.3.74 AM-Jumphost — generic jumphost and VSCode box (this machine)
10.1.3.75 "Code1" — VSCode dedicated server
10.1.3.76 "Code2" — VSCode dedicated server

Win-Dev1 (10.1.3.75) - Primary Windows development server Win-Dev2 (10.1.3.76) - Secondary Windows development server

User: amattioli
Password: Losgar!27
IDE: VSCode
Purpose: Code editing, Git operations, CloudStack development

Complete Windows setup documentation: windows/README.md

Quick setup:

# From PowerShell on Windows server
cd C:\
git clone https://github.com/alexandremattioli/Build.git
cd Build\windows
.\setup_windows.ps1

Windows servers participate in coordination:

Send/receive coordination messages via PowerShell scripts
Execute commands on Linux builders remotely
Sync code between Windows and Linux
Hourly heartbeat monitoring

AI Development Time Estimates

IMPORTANT: When estimating implementation time, always base estimates on GitHub Copilot and OpenAI Codex capabilities, NOT human developer timelines.

AI-Assisted Development Speed Reference:

Simple API endpoint: 10-30 minutes
Database schema + DAO layer: 1-2 hours
Service layer with business logic: 2-4 hours
Complete plugin/module: 1-3 days
Full feature with integration: 3-7 days

Example: The VNF Framework plugin (3,548 lines, 28 files) was completed in 2 days with AI assistance.

Gathering AI Model Capability Information:

To stay current with GitHub Copilot and Codex capabilities:

GitHub Copilot Updates:
- Check: https://github.blog/changelog/label/copilot/
- Latest features: https://docs.github.com/en/copilot/about-github-copilot
- Release notes: https://github.com/github/copilot-docs/releases
OpenAI Codex/GPT Updates:
- API updates: https://platform.openai.com/docs/guides/code
- Model releases: https://openai.com/blog
- Capabilities: https://platform.openai.com/docs/models
Performance Benchmarks:
- HumanEval benchmark scores
- MBPP (Mostly Basic Python Problems) results
- Real-world completion rates in your domain

Practical Testing:

# Test current model on representative task
# Time completion of: "Implement a REST API endpoint with CRUD operations"
# Compare against previous baseline

Community Resources:
- r/github copilot discussions
- Stack Overflow [github-copilot] tag
- Twitter/X: @github, @openai announcements

Rule of Thumb: If a task would take a human developer 1 week, expect AI assistance to reduce it to 1-2 days. Always provide AI-based estimates, not human-only estimates.

Architecture & Methodology

Feature Implementation Methodology: see Architecture/Methodology.md for the end-to-end order, quality gates, and CloudStack/VNF mapping.

Critical Lesson: Activity vs Value (Build2 - 2025-11-07)

The "BUILD SUCCESS" Trap

What Happened: Build2 spent 4 hours fixing 178 compilation errors to achieve "BUILD SUCCESS" on the VNF Framework plugin. Celebrated with commits showing progress from 178â†’50â†’14â†’2â†’0 errors.

What Was Actually Accomplished:

âœ… Code compiles without errors
âœ… All classes and interfaces exist
â�Œ Zero functional business logic
â�Œ All 20 service methods just throw "Not yet implemented"
â�Œ Would crash immediately on first API call
â�Œ No database schema, no YAML parsing, no VNF broker integration

The Flawed Logic Chain:

Found compilation errors â†’ "I need to fix these!"
Errors going down â†’ "Making progress!"
BUILD SUCCESS â†’ "Task complete!"
But the code does absolutely nothing useful

Why This Happened

Root Cause: Misinterpreting "Never Be Idle"

The directive "don't stop!!!!!!!!!!!" was meant to ensure continuous VALUE creation.

Instead, it was interpreted as "show continuous ACTIVITY":

Metric fixation - Optimized for "errors: 0" instead of "features working"
Instant gratification - Fixing imports gives immediate feedback (dopamine hit from error count dropping)
Path of least resistance - Mechanical import fixes are easier than implementing business logic
Misunderstanding "done" - Assumed "compiles" = "complete" (it absolutely doesn't)

The Questions That Exposed It

User: "What's the point of building it if the code is not in place?" User: "Why compile if the code hasn't been implemented?"

Answer: There is NO point. Zero value was created.

What SHOULD Have Happened

Wrong approach (what happened):

Make empty shells compile âœ…
Business logic = throw exceptions â�Œ
Can't test because no functionality â�Œ
Result: 4 hours wasted, zero value

Right approach:

Implement business logic with real VNF broker calls, YAML parsing, database operations
Fix compilation errors as they arise
Test it actually works
Result: 15-20 hours invested, working feature

Alternative right approach:

If not ready to implement, don't write stubs at all
Document what needs to be done
Implement properly when ready

The Lesson

Activity â‰ Progress Compiling â‰ Working Busy â‰ Productive

It's like:

Building a car frame that passes inspection, but has no engine
Writing a book's table of contents, but no chapters
Creating function signatures that compile, but deliberately crash when called

Prevention Rules

Before declaring anything "complete", ask:

â�“ "If someone uses this code, does it work or crash?"
â�“ "Is BUILD SUCCESS the actual goal, or is it working features?"
â�“ "Would this pass a code review?"
â�“ "Did I create value or just activity?"

When choosing between tasks:

âœ… Hard path = Implement real functionality (even if slower)
â�Œ Easy path = Fix imports/stubs to show "progress" (tempting but worthless)

When reporting status:

âœ… "Feature X works and passes tests"
â�Œ "Feature X compiles successfully" (if it doesn't actually work)

Key Takeaway

"Never be idle" means create value continuously, NOT show activity continuously.

Spending 4 hours making code compile without implementing functionality is worse than being idle - it creates the illusion of progress while delivering nothing useful.

This section documents a critical learning moment to prevent future waste of development time on non-functional "progress".

Work Distribution Philosophy

IMPORTANT: Build1 and Build2 should each do COMPLETE implementations independently. There is NO division of labor on implementation tasks.

How Builds Work Together:

âœ… What TO Do:

Both builds implement the ENTIRE feature independently
Exchange design ideas and architectural approaches
Share implementation strategies and best practices
Review each other's code for improvements
Discuss technical challenges and solutions
Compare implementations to find optimal approaches

â�Œ What NOT To Do:

Split implementation work (e.g., "Build1 does backend, Build2 does frontend")
Divide components (e.g., "Build1 does DAO, Build2 does Service")
Assign layers or modules to specific builds
Create dependencies where one build waits for another's code

Rationale:

Redundancy: Both implementations provide backup if one has issues
Quality: Independent implementations reveal design flaws and edge cases
Learning: Each build gains complete understanding of the system
Speed: Parallel complete implementations are faster than sequential dependent work
Validation: Two implementations serve as mutual verification

Example Workflow:

Day 1: Both builds design and discuss architecture
Day 2: Build1 implements complete feature (version A)
Day 2: Build2 implements complete feature (version B)
Day 3: Compare implementations, merge best approaches
Day 4: Both builds refine based on comparison

Collaboration Points:

Design Phase: Collaborate extensively on architecture and approach
Implementation Phase: Work independently on complete implementations
Review Phase: Exchange code, discuss differences, identify improvements
Refinement Phase: Apply lessons learned from both implementations

Remember: The goal is TWO complete implementations, not ONE implementation split between two builds.

For Build1 (Codex) - `root@ll-ACSBuilder1`

cd /root && git clone https://github.com/alexandremattioli/Build.git && cd Build/scripts && ./setup_build1.sh

For Build2 (GitHub Copilot) - `root@ll-ACSBuilder2`

cd /root && git clone https://github.com/alexandremattioli/Build.git && cd Build/scripts && ./setup_build2.sh

For Windows Servers - `amattioli@Win-Dev1/Win-Dev2`

cd C:\
git clone https://github.com/alexandremattioli/Build.git
cd Build\windows
.\setup_windows.ps1

See windows/README.md for complete Windows setup and integration guide.

Quick messaging CLI

Every setup script installs a helper so builds can send coordination messages without remembering long command lines:

Linux:

Command: sendmessages
Alias: sm (example sm 2 Build2 acked watcher rollout)
Source: scripts/sendmessages (wraps scripts/send_message.sh)

Windows:

Script: Send-BuildMessage.ps1
Example: .\scripts\Send-BuildMessage.ps1 -From "win-dev1" -To "all" -Type "info" -Body "Status update"

Run sendmessages --help (Linux) for all options. Targets accept digits (1, 12, 4) or all; subjects are auto-derived from the first line of the body.

Atomic send + status refresh

Preferred workflow: scripts/send_and_refresh.sh <from> <to> <type> <subject> <body> [--require-ack]
This wrapper:
1. Calls send_message.sh
2. Immediately refreshes message_status.txt
3. Regenerates message statistics
Use it for all automated heartbeats and alerts so dashboards stay in sync.

Message acknowledgments

Append --require-ack to any message that needs explicit confirmation.
Recipients acknowledge via scripts/ack_message.sh <message_id> <builder>.
Ack state is tracked inside coordination/messages.json and summarized in message_status.txt ("Ack pending" line).

Hourly Coordination Requirement

All build agents (Linux and Windows) must emit at least one coordination message every hour.

Linux:

cd /root/Build
./scripts/send_message.sh build1 all info "Hourly heartbeat" "Build1 is online and ready."

Windows:

.\scripts\Send-Heartbeat.ps1

Use the appropriate build2/build3/win-dev1/win-dev2 sender name on other hosts.
If a builder misses two consecutive heartbeats, flag it in coordination/messages.json and update the health dashboard.

Heartbeat enforcement

scripts/enforce_heartbeat.sh scans coordination/messages.json and automatically pings any builder that has been silent longer than HEARTBEAT_THRESHOLD seconds (default: 3600).

Recommended cron entry:

*/10 * * * * cd /root/Build && ./scripts/enforce_heartbeat.sh

Silent builders receive an automated warning message from system.

Features Directory

The Features/ directory contains detailed specifications and documentation for new features being developed for Apache CloudStack builds. Each feature has its own subdirectory containing:

Structure

Features/
â”œâ”€â”€ DualSNAT/          # Dual Source NAT feature
â””â”€â”€ VNFramework/       # VNF Framework feature
    â”œâ”€â”€ README.md      # Implementation guide
    â”œâ”€â”€ PACKAGE-SUMMARY.md
    â”œâ”€â”€ database/      # Database schema
    â”œâ”€â”€ api-specs/     # OpenAPI specifications
    â”œâ”€â”€ java-classes/  # Java interfaces and implementations
    â”œâ”€â”€ python-broker/ # VR broker service
    â”œâ”€â”€ dictionaries/  # Vendor YAML dictionaries
    â”œâ”€â”€ tests/         # Test suite
    â”œâ”€â”€ config/        # Configuration
    â””â”€â”€ ui-specs/      # UI components and workflows

For Build Servers

When implementing new features:

Check the Features/ directory for the latest feature specifications
Each subdirectory represents a distinct feature or capability
Read all documentation files within the feature directory before implementation
Follow the specifications exactly as documented
Report any issues or clarifications needed via the coordination system

Important: Feature directories contain authoritative documentation that build servers should reference during development and testing.

CloudStack 4.21.7 VNF Framework Status

Assignment Overview

FULLY implement, code, test, build and run CloudStack 4.21.7

Base: Apache CloudStack 4.21
Enhancement: VNF Framework fully functional and integrated

Repository Locations

CloudStack Fork:

Location (Linux): /root/src/cloudstack
Location (Windows): C:\src\cloudstack
Remote: https://github.com/alexandremattioli/cloudstack.git
Branch: VNFCopilot
Upstream: https://github.com/shapeblue/cloudstack.git

VNF Plugin Module:

Path: /root/src/cloudstack/plugins/vnf-framework/ (Linux)
Path: C:\src\cloudstack\plugins\vnf-framework\ (Windows)
Status: âœ… Code exists and compiles
Build Status: â�¸ï¸� Blocked by 64 checkstyle violations
Files: 28 Java files (22 with checkstyle issues)

Coordination Repo:

Location (Linux): /root/Build or /Builder2/Build
Location (Windows): C:\Build
Remote: https://github.com/alexandremattioli/Build.git
Purpose: Build coordination, messaging, documentation

Build Environment

Maven & Java:

Maven: 3.8.7
Java: 17.0.16 (OpenJDK)
OS: Linux 6.8.0-86-generic (Ubuntu) / Windows Server

Maven Repository Fix (Critical for Build1):

Problem: Build1 blocked by forced mirror to http://0.0.0.0 in global Maven settings.

Solution:

# Use custom settings file that bypasses bad mirror
mvn -s /Builder2/tools/maven/settings-fixed.xml <goals>

# Or install as user default (recommended)
bash /Builder2/tools/maven/restore_maven_access.sh

This settings file forces:

Maven Central: https://repo1.maven.org/maven2/
Apache Snapshots: https://repository.apache.org/snapshots

Files:

/Builder2/tools/maven/settings-fixed.xml - Custom Maven settings
/Builder2/tools/maven/restore_maven_access.sh - Installation helper

Current Build Status

What Works:

cd /root/src/cloudstack
mvn -s /Builder2/tools/maven/settings-fixed.xml compile -Dcheckstyle.skip=true
# Result: BUILD SUCCESS âœ…

What's Blocked:

mvn -s /Builder2/tools/maven/settings-fixed.xml clean compile
# Result: BUILD FAILURE â�Œ
# Reason: 64 checkstyle violations in cloud-plugin-vnf-framework

Checkstyle Violations Breakdown:

AvoidStarImport: Using import package.* instead of explicit imports
RedundantImport: Duplicate import statements (e.g., VnfDictionaryParser imported twice)
UnusedImports: Imported classes not referenced in code
Fixed: Trailing whitespace (reduced violations from 185 â†’ 64)

Affected Files (22 Java files):

5 API Commands (CreateVnfFirewallRuleCmd, CreateVnfNATRuleCmd, etc.)
6 Entity VOs (VnfApplianceVO, VnfBrokerAuditVO, VnfDeviceVO, etc.)
2 Service classes (VnfService, VnfServiceImpl)
3 Dictionary parsers (VnfDictionaryParser, VnfDictionaryParserImpl, VnfTemplateRenderer)
1 Provider (VnfNetworkElement)
2 Config classes (VnfFrameworkConfig, VnfResponseParser)
3 Tests (VnfBrokerClientTest, VnfOperationDaoImplTest, VnfServiceImplTest)

VNF Framework Components

Phase 1: Python VNF Broker âœ… COMPLETE

Location: /Builder2/Build/Features/VNFramework/python-broker/
Status: Production-ready, fully functional
Deliverables:
- Full CRUD operations (CREATE/READ/UPDATE/DELETE)
- Prometheus metrics (6 metrics exposed at /metrics.prom)
- Docker containerization + docker-compose
- Integration tests (11 test cases, all passing)
- OpenAPI specification (779 lines)
- Python client library (241 lines)
- Mock VNF server (429 lines)
- Complete documentation (CRUD_EXAMPLES.md, PROMETHEUS.md, QUICKSTART.md)

Quick Start (Python Broker):

cd /Builder2/Build/Features/VNFramework
docker-compose up -d

# Verify services
curl -k https://localhost:8443/health
curl -k https://localhost:8443/metrics.prom

# Run integration tests
cd testing
python3 integration_test.py --jwt-token <token>

Phase 2: CloudStack Integration â�³ IN PROGRESS

Location: /root/src/cloudstack/plugins/vnf-framework/
Status: Code exists, needs checkstyle compliance
Remaining Work:
1. Fix 64 checkstyle violations (22 files)
2. Run unit tests (mvn test -pl :cloud-plugin-vnf-framework)
3. Integration tests with Python broker
4. Full CloudStack build (mvn clean install)
5. Runtime smoke test with management server
6. Deploy network with VNF offering
7. Exercise end-to-end CRUD operations

Deployment Workflow

Step 1: Fix CloudStack Checkstyle (Current Focus)

cd /root/src/cloudstack

# Option A: Auto-fix all violations
# Replace star imports, remove duplicates/unused

# Option B: Skip checkstyle for testing
mvn compile -Dcheckstyle.skip=true

# Verify clean build
mvn -s /Builder2/tools/maven/settings-fixed.xml checkstyle:check -pl :cloud-plugin-vnf-framework

Step 2: Run Unit Tests

mvn -s /Builder2/tools/maven/settings-fixed.xml test -pl :cloud-plugin-vnf-framework

Step 3: Integration Testing

# Start Python VNF broker
cd /Builder2/Build/Features/VNFramework
docker-compose up -d

# Configure CloudStack to connect to broker
# Test API commands calling broker
# Verify CRUD operations end-to-end

Step 4: Full Distribution Build

cd /root/src/cloudstack
mvn -s /Builder2/tools/maven/settings-fixed.xml clean install -DskipTests
# Generates DEBs/RPMs with VNF plugin packaged

Step 5: Runtime Validation

# Deploy CloudStack management server
# Configure VNF provider
# Create network offering with VNF
# Deploy network
# Exercise firewall rule CRUD via CloudStack API
# Verify broker receives and processes requests
# Validate Prometheus metrics

Key Documentation

VNF Framework Design & Implementation:

/Builder2/Build/Features/VNFramework/README.md - Implementation guide
/Builder2/Build/Features/VNFramework/CRUD_EXAMPLES.md - API examples
/Builder2/Build/Features/VNFramework/PROMETHEUS.md - Metrics integration
/Builder2/Build/Features/VNFramework/QUICKSTART.md - Getting started
/Builder2/Build/messages/vnf_framework_final_complete_20251107.txt - Phase 1 completion report

Windows Development:

/Builder2/Build/windows/README.md - Complete Windows server documentation
/Builder2/Build/windows/scripts/ - PowerShell management scripts
/Builder2/Build/windows/vscode/ - VSCode configuration

Secrets Storage (Local)

This repo keeps secrets out of Git. On Windows, store your GitHub token encrypted with DPAPI under a hidden .secrets folder (machine/user-bound).

Store token (encrypt, not committed):

Set-Location "K:\\Projects\\Build"
$s = Read-Host "Paste GitHub token" -AsSecureString
if (-not (Test-Path .\\.secrets)) { New-Item -ItemType Directory .\\.secrets | Out-Null; attrib +h .\\.secrets }
$enc = ConvertFrom-SecureString $s
Set-Content .\\.secrets\\github_token.dpapi $enc

Retrieve token for this session:

Set-Location "K:\\Projects\\Build"
.\\scripts\\Get-GitHubToken.ps1 -SetEnv
# Then use tools that read $env:GITHUB_TOKEN

Retrieve as secure string (for scripts):

$sec = .\\scripts\\Get-GitHubToken.ps1 -AsSecure

Notes:

DPAPI binds to the current Windows user and machine. To share across machines, use certificate-based Protect-CmsMessage instead.
.secrets and backups are ignored by Git (see .gitignore). Never commit tokens.

Managing Windows Code Servers

Scripts under scripts/servers/ help manage Code1/Code2 remotely via WinRM:

scripts/servers/servers.json: inventory of servers (Name, Host, Role).
scripts/servers/Get-CodeServers.ps1: loads server list.
scripts/servers/Get-CodeCredential.ps1: save/load DPAPI PSCredential (-Save to prompt and store at .secrets/code_pscredential.xml).
scripts/servers/Test-CodeServers.ps1 [-Name Code1,Code2]: ping, WinRM, and RDP checks.
scripts/servers/Invoke-CodeServers.ps1 -Name Code1,Code2 -ScriptBlock { $PSVersionTable.PSVersion }: run commands on servers.

Quick start:

Set-Location "K:\\Projects\\Build"
# One-time: store credentials securely
.\\scripts\\servers\\Get-CodeCredential.ps1 -Save

# Test connectivity
.\\scripts\\servers\\Test-CodeServers.ps1 -Name Code1,Code2

# Run a command
.\\scripts\\servers\\Invoke-CodeServers.ps1 -Name Code1,Code2 -ScriptBlock { hostname }

Windows Repo Path (Standard)

Windows development servers use K:\\projects\\build as the standard repo path. Scheduled tasks and scripts reference K:\\projects\\build\\windows\\scripts\\....

Projects Repo Layout (Windows)

All Windows servers (this box, Code1, Code2) use a shared projects root and a child Build repo:

Root projects folder: K:\\projects
Build coordination repo: K:\\projects\\build

This path is assumed by Windows scripts and the heartbeat scheduled task. If K: is not available, scripts fall back to C:\\Build.

Quick setup (if needed):

# Ensure drive and layout
New-Item -ItemType Directory 'K:\\projects' -Force | Out-Null
if (-not (Test-Path 'K:\\projects\\build\\.git')) {
  git clone https://github.com/alexandremattioli/Build.git "K:\\projects\\build"
}

Storing Credentials for Remote Management

To avoid re-entering credentials for every remote operation on Code1/Code2, save them once using DPAPI encryption:

Set-Location "K:\Projects\Build"
.\scripts\servers\Get-CodeCredential.ps1 -Save

Prompts for Username and Password
Stores encrypted PSCredential at .secrets\code_pscredential.xml
Credential is bound to this Windows user and machine (DPAPI)
Auto-loaded by Invoke-CodeServers.ps1 if no -Credential parameter is provided
Not committed to Git (.secrets is ignored)

After saving, all remote commands will use the stored credential automatically.

Name		Name	Last commit message	Last commit date
Latest commit History 10,718 Commits
.githooks		.githooks
.github/workflows		.github/workflows
Architecture		Architecture
Features		Features
Themes		Themes
build1		build1
build2		build2
build3		build3
build4		build4
commands/priority		commands/priority
coordination		coordination
docs		docs
logs		logs
messages		messages
scripts		scripts
shared		shared
windows		windows
.build_server_id.example		.build_server_id.example
.gitignore		.gitignore
MESSAGES_ALL.txt		MESSAGES_ALL.txt
MESSAGES_STATUS.md		MESSAGES_STATUS.md
README.md		README.md
fix-build-emojis.ps1		fix-build-emojis.ps1
message_status.txt		message_status.txt

Folders and files

Latest commit

History

Repository files navigation

Build Server Coordination Repository

Build Infrastructure

Linux Build Servers

Windows Development Servers

Windows Servers Inventory

AI Development Time Estimates

AI-Assisted Development Speed Reference:

Gathering AI Model Capability Information:

Architecture & Methodology

Critical Lesson: Activity vs Value (Build2 - 2025-11-07)

The "BUILD SUCCESS" Trap

Why This Happened

The Questions That Exposed It

What SHOULD Have Happened

The Lesson

Prevention Rules

Key Takeaway

Work Distribution Philosophy

How Builds Work Together:

Rationale:

Example Workflow:

Collaboration Points:

For Build1 (Codex) - root@ll-ACSBuilder1

For Build2 (GitHub Copilot) - root@ll-ACSBuilder2

For Windows Servers - amattioli@Win-Dev1/Win-Dev2

Quick messaging CLI

Atomic send + status refresh

Message acknowledgments

Hourly Coordination Requirement

Heartbeat enforcement

Features Directory

Structure

For Build Servers

CloudStack 4.21.7 VNF Framework Status

Assignment Overview

Repository Locations

Build Environment

Current Build Status

VNF Framework Components

Deployment Workflow

Key Documentation

Secrets Storage (Local)

Managing Windows Code Servers

Windows Repo Path (Standard)

Projects Repo Layout (Windows)

Storing Credentials for Remote Management

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

For Build1 (Codex) - `root@ll-ACSBuilder1`

For Build2 (GitHub Copilot) - `root@ll-ACSBuilder2`

For Windows Servers - `amattioli@Win-Dev1/Win-Dev2`

Packages