Add layered MergePipeline for multi-source entity discovery

docs/config/discovery-options.rst

@@ -87,6 +87,98 @@ The ``min_topics_for_component`` parameter (default: 1) sets the minimum number
 of topics required before creating a component. This can filter out namespaces
 with only a few stray topics.
 
+Merge Pipeline Options (Hybrid Mode)
+-------------------------------------
+
+When using ``hybrid`` mode, the merge pipeline controls how entities from
+different discovery layers are combined. The ``merge_pipeline`` section
+configures gap-fill behavior for runtime-discovered entities.
+
+Gap-Fill Configuration
+^^^^^^^^^^^^^^^^^^^^^^
+
+In hybrid mode, the manifest is the source of truth. Runtime (heuristic) discovery
+fills gaps - entities that exist at runtime but aren't in the manifest. Gap-fill
+controls restrict what the runtime layer is allowed to create:
+
+.. code-block:: yaml
+
+   discovery:
+     merge_pipeline:
+       gap_fill:
+         allow_heuristic_areas: true
+         allow_heuristic_components: true
+         allow_heuristic_apps: true
+         allow_heuristic_functions: false
+         namespace_whitelist: []
+         namespace_blacklist: []
+
+.. list-table:: Gap-Fill Options
+   :header-rows: 1
+   :widths: 35 15 50
+
+   * - Parameter
+     - Default
+     - Description
+   * - ``allow_heuristic_areas``
+     - ``true``
+     - Allow runtime layer to create Area entities not in the manifest
+   * - ``allow_heuristic_components``
+     - ``true``
+     - Allow runtime layer to create Component entities not in the manifest
+   * - ``allow_heuristic_apps``
+     - ``true``
+     - Allow runtime layer to create App entities not in the manifest
+   * - ``allow_heuristic_functions``
+     - ``false``
+     - Allow runtime layer to create Function entities (rarely useful at runtime)
+   * - ``namespace_whitelist``
+     - ``[]``
+     - If non-empty, only allow gap-fill from these ROS 2 namespaces (Areas and Components only)
+   * - ``namespace_blacklist``
+     - ``[]``
+     - Exclude gap-fill from these ROS 2 namespaces (Areas and Components only)
+
+When both whitelist and blacklist are empty, all namespaces are eligible for gap-fill.
+If whitelist is non-empty, only listed namespaces pass. Blacklist is always applied.
+
+Namespace matching uses path-segment boundaries: ``/nav`` matches ``/nav`` and ``/nav/sub``
+but NOT ``/navigation``. Both lists only filter Areas and Components (which carry
+``namespace_path``). Apps and Functions are not namespace-filtered.
+
+
+Merge Policies
+^^^^^^^^^^^^^^
+
+Each discovery layer declares a ``MergePolicy`` per field group. When two layers
+provide the same entity (matched by ID), policies determine which values win:
+
+.. list-table:: Merge Policies
+   :header-rows: 1
+   :widths: 25 75
+
+   * - Policy
+     - Description
+   * - ``AUTHORITATIVE``
+     - This layer's value wins over lower-priority layers.
+       If two AUTHORITATIVE layers conflict, a warning is logged and the
+       higher-priority (earlier) layer wins.
+   * - ``ENRICHMENT``
+     - Fill empty fields from this layer. Non-empty target values are preserved.
+       Two ENRICHMENT layers merge additively (collections are unioned).
+   * - ``FALLBACK``
+     - Use only if no other layer provides the value. Two FALLBACK layers
+       merge additively (first non-empty fills gaps).
+
+**Built-in layer policies:**
+
+- **ManifestLayer** (priority 1): IDENTITY, HIERARCHY, METADATA are AUTHORITATIVE.
+  LIVE_DATA is ENRICHMENT (runtime topics/services take precedence).
+  STATUS is FALLBACK (manifest cannot know online state).
+- **RuntimeLayer** (priority 2): LIVE_DATA and STATUS are AUTHORITATIVE.
+  METADATA is ENRICHMENT. IDENTITY and HIERARCHY are FALLBACK.
+- **PluginLayer** (priority 3+): All field groups ENRICHMENT
+
 Configuration Example
 ---------------------
 
@@ -112,6 +204,16 @@ Complete YAML configuration for runtime discovery:
            topic_only_policy: "create_component"
            min_topics_for_component: 2  # Require at least 2 topics
 
+         # Note: merge_pipeline settings only apply when mode is "hybrid"
+         merge_pipeline:
+           gap_fill:
+             allow_heuristic_areas: true
+             allow_heuristic_components: true
+             allow_heuristic_apps: true
+             allow_heuristic_functions: false
+             namespace_whitelist: []
+             namespace_blacklist: ["/rosout", "/parameter_events"]
+
 Command Line Override
 ---------------------
 
@@ -128,3 +230,5 @@ See Also
 
 - :doc:`manifest-schema` - Manifest-based configuration
 - :doc:`/tutorials/heuristic-apps` - Tutorial on runtime discovery
+- :doc:`/tutorials/manifest-discovery` - Hybrid mode tutorial
+- :doc:`/tutorials/plugin-system` - Plugin layer integration

docs/tutorials/manifest-discovery.rst

@@ -207,16 +207,67 @@ List functions:
 
    curl http://localhost:8080/api/v1/functions
 
-Understanding Runtime Linking
------------------------------
+Understanding Hybrid Mode
+-------------------------
 
-In hybrid mode, manifest apps are automatically linked to running ROS 2 nodes.
-The linking process:
+In hybrid mode, discovery uses a **merge pipeline** that combines entities from
+multiple discovery layers:
 
-1. **Discovery**: Gateway discovers running ROS 2 nodes
-2. **Matching**: For each manifest app, checks ``ros_binding`` configuration
-3. **Linking**: If match found, copies runtime resources (topics, services, actions)
-4. **Status**: Apps with matched nodes are marked ``is_online: true``
+1. **ManifestLayer** (highest priority) - entities from the YAML manifest
+2. **RuntimeLayer** - entities discovered via ROS 2 graph introspection
+3. **PluginLayers** (optional) - entities from gateway plugins
+
+The pipeline merges entities by ID. When the same entity appears in multiple layers,
+per-field-group merge policies determine which values win. See
+:doc:`/config/discovery-options` for details on merge policies and gap-fill configuration.
+
+After merging, the **RuntimeLinker** binds manifest apps to running ROS 2 nodes:
+
+1. **Discovery**: All layers produce entities
+2. **Merging**: Pipeline merges entities by ID, applying field-group policies
+3. **Linking**: For each manifest app, checks ``ros_binding`` configuration
+4. **Binding**: If match found, copies runtime resources (topics, services, actions)
+5. **Status**: Apps with matched nodes are marked ``is_online: true``
+
+Merge Report
+~~~~~~~~~~~~
+
+After each pipeline execution, the gateway produces a ``MergeReport`` available
+via the health endpoint (``GET /health``). The report includes:
+
+- Layer names and ordering
+- Total entity count, enrichment count
+- Conflict details (which layers disagreed on which field groups)
+- Cross-type ID collision warnings
+- Gap-fill filtering statistics
+
+In hybrid mode, the ``GET /health`` response includes full discovery diagnostics:
+
+.. code-block:: json
+
+   {
+     "discovery": {
+       "mode": "hybrid",
+       "strategy": "hybrid",
+       "pipeline": {
+         "layers": ["manifest", "runtime"],
+         "total_entities": 12,
+         "enriched_count": 8,
+         "conflict_count": 0,
+         "id_collisions": 0
+       },
+       "linking": {
+         "linked_count": 5,
+         "orphan_count": 1,
+         "binding_conflicts": 0,
+         "warnings": ["Orphan node: /unmanifested_node"]
+       }
+     }
+   }
+
+
+Runtime Linking
+~~~~~~~~~~~~~~~
 
 ROS Binding Configuration
 ~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -360,6 +411,12 @@ in the manifest. The ``config.unmanifested_nodes`` setting controls this:
 - ``error``: Fail startup if orphan nodes detected
 - ``include_as_orphan``: Include with ``source: "orphan"``
 
+.. note::
+   In hybrid mode with gap-fill configuration (see :doc:`/config/discovery-options`),
+   namespace filtering controls which runtime entities enter the pipeline.
+   ``unmanifested_nodes`` controls how runtime nodes that passed gap-fill
+   but did not match any manifest app are handled by the RuntimeLinker.
+
 Hot Reloading
 -------------
 

docs/tutorials/plugin-system.rst

@@ -10,8 +10,9 @@ Overview
 Plugins implement the ``GatewayPlugin`` C++ base class plus one or more typed provider interfaces:
 
 - **UpdateProvider** - software update backend (CRUD, prepare/execute, automated, status)
-- **IntrospectionProvider** *(preview)* - enriches discovered entities with platform-specific metadata.
-  This interface is defined and can be implemented, but is not yet wired into the discovery cycle.
+- **IntrospectionProvider** - enriches discovered entities with platform-specific metadata
+  via the merge pipeline. In hybrid mode, each IntrospectionProvider is wrapped as a
+  ``PluginLayer`` and added to the pipeline with ENRICHMENT merge policy.
 
 A single plugin can implement multiple provider interfaces. For example, a "systemd" plugin
 could provide both introspection (discover systemd units) and updates (manage service restarts).
@@ -300,7 +301,12 @@ Multiple Plugins
 Multiple plugins can be loaded simultaneously:
 
 - **UpdateProvider**: Only one plugin's UpdateProvider is used (first in config order)
-- **IntrospectionProvider**: All plugins' results are merged *(preview - not yet wired)*
+- **IntrospectionProvider**: All plugins are added as PluginLayers to the merge pipeline.
+  Each plugin's entities are merged with ENRICHMENT policy - they fill empty fields but
+  never override manifest or runtime values. Plugins are added after all built-in layers,
+  and the pipeline is refreshed once after all plugins are registered (batch registration).
+  The ``introspect()`` method receives an ``IntrospectionInput`` populated with all entities
+  from previous layers (manifest + runtime), enabling context-aware metadata and discovery.
 - **Custom routes**: All plugins can register endpoints (use unique path prefixes)
 
 Error Handling

src/ros2_medkit_fault_manager/CMakeLists.txt

@@ -192,12 +192,14 @@ if(BUILD_TESTING)
     TARGET test_integration
     TIMEOUT 60
   )
+  set_tests_properties(test_integration PROPERTIES LABELS "integration")
 
   add_launch_test(
     test/test_rosbag_integration.test.py
     TARGET test_rosbag_integration
     TIMEOUT 90
   )
+  set_tests_properties(test_rosbag_integration PROPERTIES LABELS "integration")
 endif()
 
 ament_package()

src/ros2_medkit_gateway/CMakeLists.txt

@@ -92,6 +92,10 @@ add_library(gateway_lib STATIC
   src/discovery/discovery_manager.cpp
   src/discovery/runtime_discovery.cpp
   src/discovery/hybrid_discovery.cpp
+  src/discovery/merge_pipeline.cpp
+  src/discovery/layers/manifest_layer.cpp
+  src/discovery/layers/runtime_layer.cpp
+  src/discovery/layers/plugin_layer.cpp
   # Discovery models (with .cpp serialization)
   src/discovery/models/app.cpp
   src/discovery/models/function.cpp
@@ -184,6 +188,7 @@ target_precompile_headers(gateway_lib PRIVATE
   <httplib.h>
   <tl/expected.hpp>
 )
+set_target_properties(gateway_lib PROPERTIES POSITION_INDEPENDENT_CODE ON)
 
 # Gateway node executable
 add_executable(gateway_node src/main.cpp)
@@ -325,6 +330,10 @@ if(BUILD_TESTING)
   ament_add_gtest(test_runtime_linker test/test_runtime_linker.cpp)
   target_link_libraries(test_runtime_linker gateway_lib)
 
+  # Add merge pipeline tests
+  ament_add_gtest(test_merge_pipeline test/test_merge_pipeline.cpp)
+  target_link_libraries(test_merge_pipeline gateway_lib)
+
   # Add capability builder tests
   ament_add_gtest(test_capability_builder test/test_capability_builder.cpp)
   target_link_libraries(test_capability_builder gateway_lib)
@@ -488,6 +497,8 @@ if(BUILD_TESTING)
       test_plugin_manager
       test_log_manager
       test_log_handlers
+      test_merge_pipeline
+      test_runtime_linker
     )
     foreach(_target ${_test_targets})
       target_compile_options(${_target} PRIVATE --coverage -O0 -g)

src/ros2_medkit_gateway/config/gateway_params.yaml

@@ -172,6 +172,26 @@ ros2_medkit_gateway:
         # Default: 1 (create component for any namespace with topics)
         min_topics_for_component: 1
 
+      # Merge pipeline configuration (only used in hybrid mode)
+      # Controls how manifest, runtime, and plugin layers merge entities
+      merge_pipeline:
+        # Gap-fill: what runtime discovery can create when manifest is incomplete
+        gap_fill:
+          allow_heuristic_areas: true
+          allow_heuristic_components: true
+          allow_heuristic_apps: true
+          allow_heuristic_functions: false
+          # namespace_blacklist: ["/rosout"]
+          # namespace_whitelist: []
+
+        # Per-layer policy overrides (NOT YET IMPLEMENTED - planned for future release)
+        # Defaults: manifest=AUTH for identity/hierarchy/metadata, runtime=AUTH for live_data/status
+        # layers:
+        #   manifest:
+        #     live_data: "authoritative"     # override: manifest topics win
+        #   runtime:
+        #     identity: "authoritative"      # override: trust runtime names
+
     # Authentication Configuration (REQ_INTEROP_086, REQ_INTEROP_087)
     # JWT-based authentication with Role-Based Access Control (RBAC)
     auth:

src/ros2_medkit_gateway/design/architecture.puml

@@ -73,8 +73,29 @@ package "ros2_medkit_gateway" {
     class EntityCache {
         + areas: vector<Area>
         + components: vector<Component>
+        + apps: vector<App>
         + last_update: time_point
     }
+
+    class MergePipeline {
+        + add_layer(): void
+        + execute(): MergeResult
+        + set_linker(): void
+    }
+
+    interface DiscoveryLayer <<interface>> {
+        + name(): string
+        + discover(): LayerOutput
+        + policy_for(FieldGroup): MergePolicy
+    }
+
+    class ManifestLayer
+    class RuntimeLayer
+    class PluginLayer
+
+    class RuntimeLinker {
+        + link(): LinkingResult
+    }
 }
 
 package "External Libraries" {
@@ -115,6 +136,13 @@ EntityCache o-right-> Component : contains many
 DiscoveryManager ..> Area : creates
 DiscoveryManager ..> Component : creates
 
+' MergePipeline layer architecture
+MergePipeline o--> DiscoveryLayer : ordered layers
+MergePipeline --> RuntimeLinker : post-merge linking
+ManifestLayer .up.|> DiscoveryLayer : implements
+RuntimeLayer .up.|> DiscoveryLayer : implements
+PluginLayer .up.|> DiscoveryLayer : implements
+
 ' REST Server uses HTTP library
 RESTServer *--> HTTPLibServer : owns
 
@@ -123,4 +151,3 @@ Area ..> JSON : serializes to
 Component ..> JSON : serializes to
 
 @enduml
-