Releases: colonyos/colonies
Releases · colonyos/colonies
v1.9.7
ColonyOS Colonies v1.9.7
Highlights
- Significantly improved test coverage with comprehensive handler unit tests
- Database performance optimizations with connection pool configuration
- API cleanup by removing deprecated fields
New Features
- Configurable Stale Executor Duration: New COLONIES_STALE_EXECUTOR_DURATION environment variable to control when idle executors are cleaned up
- Database Connection Pool Configuration: Tunable connection pool settings for better performance under load
Bug Fixes
- Fix double HTTP error response bug in process handlers
- Fix subscription race condition causing test flakiness
- Fix missing workflow statistics in HandleColonyStatistics
- Fix duplicate database indexes (PROCESSES_INDEX5, PROCESSES_INDEX11)
- Fix bugs in filesystem client
API Changes (Breaking)
- Removed deprecated FunctionSpec.Reconciliation field
- Removed Blueprint.Handler and FunctionSpec.Blueprint fields
Performance & Reliability
- Refactored handlers to use direct database calls (reduced overhead)
- Improved process removal protections
- Fixed test flakiness in CI
Testing & CI
- Added Codecov integration for coverage reporting
- Added comprehensive unit tests for all server handlers
- Added integration tests for improved handler coverage
- Test coverage now at 70%+
Internal
- Refactored handler architecture for better testability
- Added mock infrastructure for unit testing
v1.9.6
This release introduces a channels/streaming system for real-time communication, delivers a 10-13x performance improvement in process assignment, simplifies the backend architecture, and enhances executor registration and blueprint management.
Performance Optimization
Distributed Assign with SELECT FOR UPDATE SKIP LOCKED
| Metric | Before | After | Improvement |
|---|---|---|---|
| Avg Latency | 345ms | 25ms | 13x faster |
| Failure Rate | 73% | 0% | 100% success |
| Scaling | No benefit | Horizontal | True distribution |
- Bypass single-threaded command queue for concurrent assignment
- Database connection pooling prevents port exhaustion
- New indexes: GIN for executor names, partial B-tree for assign hot path
- Reduced DB operations per assign (5 → 4)
Channels System
- Add new channel router with pub/sub messaging support
- Implement channel architecture for distributed deployments
- WebSocket/realtime backend support for streaming
- New RPC messages for channel operations (append, read, subscribe)
- Lazy channel creation when clients connect
Executor Improvements
- Enhanced executor registration with detailed SW/HW metadata
- Each reconciler now has its own executor type for better isolation
- Improved executor table display in CLI with auto-detect terminal width
- Add UpdateExecutor RPC support
- Location support for executors
Blueprint Enhancements
- Improved blueprint handlers with better error handling
- Add blueprint status update support
- Enhanced CLI commands for blueprint management
- Blueprint doctor command improvements
- Add blueprintName to reconcile kwargs
Architecture Simplification
- Removed gRPC, CoAP, and libp2p backends
- Removed backend factory pattern, kept interfaces
- Simplified channel architecture to single-server model
- -559,350 lines of legacy code removed
Security
- Removed S3 credentials from debug logs
- Removed unused keychain code
v1.9.4
Improved blueprint support. Blueprint is now cron-based for improved scalability and resilience.
v1.9.3
What's Changed
- Reconcilation, blueprint and node support by @johankristianss in #160
- Improved reconil by @johankristianss in #168
Full Changelog: v1.9.0...v1.9.3
Contributors
johankristianss
Assets 11
v1.9.0
Refactor server architecture with multi-backend support, LibP2P integration, and resource reconciliation
Major architectural changes:
- Backend abstraction layer supporting HTTP, gRPC, CoAP, and LibP2P transports
- Client refactoring with automatic backend fallback and selection
- Server handler registry pattern for better modularity
- LibP2P P2P networking with relay, DHT, and NAT traversal
- Kubernetes-style resource reconciliation with drift detection
- Subscription improvements with defensive nil checking
Resource reconciliation system:
- Resource and ResourceDefinition types for declarative infrastructure
- Automatic drift detection and correction mechanism
- JSON Schema-based resource validation
- Namespaced resource management with metadata (labels, annotations)
- Resource status tracking with generation counters
- Reconciliation actions: create, update, delete, noop
- Field-level change tracking for spec, status, and metadata
- Handler specification for executor-based reconciliation
Backend improvements:
- Unified backend interface for HTTP (Gin), gRPC, CoAP, and LibP2P
- Multi-backend client with automatic failover
- DHT-based peer discovery for LibP2P
- Circuit relay v2 for NAT traversal
- Server manager for coordinating multiple backend servers
Client enhancements:
- Modular client architecture with per-domain clients
- Automatic backend selection and fallback
- LibP2P client with DHT caching
- Support for HTTP, gRPC, CoAP, and LibP2P transports
Server refactoring:
- Handler registry pattern for better organization
- Moved handlers to domain-specific packages
- Resource handler for managing ResourceDefinitions and Resources
- Improved controller architecture
- Better test isolation and mocking
Database layer:
- ResourceDatabase interface with PostgreSQL implementation
- Resource and ResourceDefinition CRUD operations
- Namespace and kind-based queries
- JSON/JSONB column support for flexible schemas
- Database migration support for resource tables
LibP2P/P2P features:
- P2P relay server with CLI integration (colonies p2p relay)
- Identity generation (colonies p2p generate)
- DHT peer discovery with caching
- AutoNAT and hole punching support
- Bootstrap peer management
Resource management features:
- Declarative resource specifications with desired state
- Custom Resource Definition (CRD) support
- Schema validation with required fields, types, and enums
- Nested object and array validation
- Helper methods for GetAPIVersion, Diff, ValidateAgainstRD
- Resource arrays and JSON conversion utilities
Bug fixes:
- Fixed nil pointer panics in subscription cleanup (Gin and LibP2P)
- Fixed DHT cache to only store active peers
- Fixed database test isolation with unique table prefixes
- Fixed TestSearchLogs timing boundary issue with PostgreSQL NOW()
- Removed APIVersion field from Resource struct (moved to ResourceDefinition)
- Updated hardcoded peer IDs to current server identity
Testing improvements:
- Fixed controller test database conflicts
- Added DatabaseMock resource interface methods
- Improved test utilities and mocking
- Better test isolation with unique database prefixes
- Added comprehensive backend tests
- Added resource validation test coverage
- Added reconciliation and diff testing
Core domain improvements:
- Resource type with Kind, Metadata, Spec, Status fields
- ResourceDefinition with Group, Version, Names, Scope, Handler
- ValidationSchema for JSON Schema validation
- Reconciliation type for tracking old/new state with diffs
- ResourceDiff with field-level change tracking
- FunctionSpec integration for reconciliation workflows
CLI enhancements:
- New 'colonies p2p' subcommand for relay and identity management
- Improved server startup with backend selection
- Better configuration management
Documentation:
- Added Container Building Guide with multi-platform instructions
- Docker buildx setup and QEMU configuration
- Multi-architecture build examples (amd64, arm64)
- CI/CD integration examples for GitHub Actions and GitLab
- Troubleshooting guide for buildx issues