feat: Elasticsearch Connector with Framework SDK Integration

[devin-ai-integration]

Implement comprehensive Elasticsearch Connector with advanced architecture

Summary

This PR implements a full-featured Elasticsearch connector for the coco-server project, leveraging the framework project's Elastic SDK capabilities. The implementation includes sophisticated features for different deployment modes, performance optimization, and robust error handling.

Key Components Added:

• Core Implementation: 5 new files (plugin.go, client.go, config.go, reader.go, types.go)
• Comprehensive Testing: 4 test files with unit and integration tests (*_test.go)
• Advanced Features: Slice scroll for parallel reading, health monitoring, retry mechanisms, and performance metrics
• Multi-Mode Support: Single node, cluster, and HA cluster deployment strategies

Architecture Highlights:

• Deployment Mode Adaptation: Optimized reading strategies based on ES cluster configuration
• Health Monitoring: Cluster status tracking with automatic failover capabilities
• Performance Optimization: Dynamic batch sizing, connection pooling, and parallel slice scroll
• Incremental Sync: Support for _seq_no and _primary_term based incremental updates
• Robust Error Handling: Exponential backoff retry logic and comprehensive error recovery

Review & Testing Checklist for Human

⚠️ Critical Risk: Tests were created but not executed due to environment limitations - they may not pass.

• Run all tests locally - Verify go test -v ./plugins/connectors/elasticsearch/... passes without errors
• Test with real Elasticsearch cluster - Validate connector works with actual ES instances in single node, cluster, and HA modes
• Verify configuration parsing - Test that complex nested configs (deployment modes, retry settings, health checks) parse correctly
• Test error handling - Verify retry mechanisms and failover work under connection failures and timeouts
• Review slice scroll implementation - Ensure parallel reading logic works correctly with different cluster sizes

Recommended Test Plan:

1. Set up test ES clusters (single node + multi-node)
2. Run unit tests and fix any failures
3. Test full workflow: configure connector → scan documents → verify queue population
4. Test failure scenarios: disconnect ES, invalid configs, network timeouts
5. Verify health monitoring and retry mechanisms work as expected

---

Diagram

%%{ init : { "theme" : "default" }}%%
graph TB
    subgraph "Elasticsearch Connector"
        plugin["plugins/connectors/<br/>elasticsearch/<br/>plugin.go"]:::major-edit
        client["plugins/connectors/<br/>elasticsearch/<br/>client.go"]:::major-edit
        config["plugins/connectors/<br/>elasticsearch/<br/>config.go"]:::major-edit
        reader["plugins/connectors/<br/>elasticsearch/<br/>reader.go"]:::major-edit
        types["plugins/connectors/<br/>elasticsearch/<br/>types.go"]:::major-edit
    end
    
    subgraph "Test Files"
        pluginTest["plugin_test.go"]:::major-edit
        clientTest["client_test.go"]:::major-edit
        readerTest["reader_test.go"]:::major-edit
        integTest["integration_test.go"]:::major-edit
    end
    
    
    subgraph "Framework Dependencies"
        elasticSDK["framework/core/<br/>elastic/api.go"]:::context
        connectorHelper["plugins/connectors/<br/>helper.go"]:::context
    end
    
    subgraph "Core System"
        cocoYml["coco.yml"]:::minor-edit
        queue["framework/core/<br/>queue"]:::context
    end
    
    plugin --> client
    plugin --> reader
    plugin --> config
    reader --> types
    client --> elasticSDK
    plugin --> connectorHelper
    plugin --> queue
    
    pluginTest --> plugin
    clientTest --> client
    readerTest --> reader
    integTest --> plugin
    
    subgraph Legend
        L1["Major Edit"]:::major-edit
        L2["Minor Edit"]:::minor-edit  
        L3["Context/No Edit"]:::context
    end
    
    classDef major-edit fill:#90EE90
    classDef minor-edit fill:#87CEEB
    classDef context fill:#FFFFFF

Loading

Notes

• Session Details: Requested by windWheel (@kaori-seasons) - https://app.devin.ai/sessions/7f237766467c435e82f35c515e7fa257
• Framework Integration: Leverages existing framework project's comprehensive Elastic SDK with support for multiple ES versions
• Test Coverage: Follows existing connector patterns (RSS connector) with proper mocking infrastructure
• Configuration: Added new connector config section to coco.yml with example deployment mode settings
• Build Tags: Integration tests use //go:build integration tags for selective CI execution
• Risk Assessment: 🟡 Medium risk due to complexity and inability to verify tests pass locally

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring