Breaking: Remove legacy hayhooks pipeline deploy ; introduce hayhooks pipeline deploy-yaml to deploy YAML pipelines with required inputs/outputs fields

README.md

@@ -32,14 +32,17 @@ With Hayhooks, you can:
   - [Async Run API Method](#run_api_async)
   - [PipelineWrapper development with `overwrite` option](#pipelinewrapper-development-with-overwrite-option)
   - [Additional Dependencies](#additional-dependencies)
+- [Deploy a YAML Pipeline](#deploy-a-yaml-pipeline)
 - [Deploy an Agent](#deploy-an-agent)
+- [Load pipelines or agents at startup](#load-pipelines-or-agents-at-startup)
 - [Support file uploads](#support-file-uploads)
 - [Run pipelines from the CLI](#run-pipelines-from-the-cli)
   - [Run a pipeline from the CLI JSON-compatible parameters](#run-a-pipeline-from-the-cli-json-compatible-parameters)
   - [Run a pipeline from the CLI uploading files](#run-a-pipeline-from-the-cli-uploading-files)
 - [MCP support](#mcp-support)
   - [MCP Server](#mcp-server)
   - [Create a PipelineWrapper for exposing a Haystack pipeline as a MCP Tool](#create-a-pipelinewrapper-for-exposing-a-haystack-pipeline-as-a-mcp-tool)
+  - [Expose a YAML pipeline as a MCP Tool](#expose-a-yaml-pipeline-as-a-mcp-tool)
   - [Using Hayhooks MCP Server with Claude Desktop](#using-hayhooks-mcp-server-with-claude-desktop)
   - [Using Hayhooks Core MCP Tools in IDEs like Cursor](#using-hayhooks-core-mcp-tools-in-ides-like-cursor)
   - [Development and deployment of Haystack pipelines directly from Cursor](#development-and-deployment-of-haystack-pipelines-directly-from-cursor)
@@ -61,15 +64,13 @@ With Hayhooks, you can:
   - [Run Hayhooks Programmatically](#run-hayhooks-programmatically)
   - [Sharing code between pipeline wrappers](#sharing-code-between-pipeline-wrappers)
 - [Deployment Guidelines](#deployment-guidelines)
-- [Legacy Features](#legacy-features)
-  - [Deploy Pipeline Using YAML](#deploy-a-pipeline-using-only-its-yaml-definition)
 - [License](#license)
 
 ## Quick start with Docker Compose
 
 To quickly get started with Hayhooks, we provide a ready-to-use Docker Compose 🐳 setup with pre-configured integration with [open-webui](https://openwebui.com/).
 
-It's available [here](https://github.com/deepset-ai/hayhooks-open-webui-docker-compose).
+It's available in the [Hayhooks + Open WebUI Docker Compose repository](https://github.com/deepset-ai/hayhooks-open-webui-docker-compose).
 
 ## Quick start
 
@@ -162,8 +163,8 @@ CLI commands are basically wrappers around the HTTP API of the server. The full
 hayhooks run     # Start the server
 hayhooks status  # Check the status of the server and show deployed pipelines
 
-hayhooks pipeline deploy-files <path_to_dir>   # Deploy a pipeline using PipelineWrapper
-hayhooks pipeline deploy <pipeline_name>       # Deploy a pipeline from a YAML file
+hayhooks pipeline deploy-files <path_to_dir>   # Deploy a pipeline using PipelineWrapper files (preferred)
+hayhooks pipeline deploy-yaml <path_to_yaml>   # Deploy a pipeline from a YAML file
 hayhooks pipeline undeploy <pipeline_name>     # Undeploy a pipeline
 hayhooks pipeline run <pipeline_name>          # Run a pipeline
 ```
@@ -195,7 +196,7 @@ The pipeline wrapper provides a flexible foundation for deploying Haystack pipel
 - Define custom execution logic with configurable inputs and outputs
 - Optionally expose OpenAI-compatible chat endpoints with streaming support for integration with interfaces like [open-webui](https://openwebui.com/)
 
-The `pipeline_wrapper.py` file must contain an implementation of the `BasePipelineWrapper` class (see [here](src/hayhooks/server/utils/base_pipeline_wrapper.py) for more details).
+The `pipeline_wrapper.py` file must contain an implementation of the `BasePipelineWrapper` class (see [BasePipelineWrapper source](src/hayhooks/server/utils/base_pipeline_wrapper.py) for more details).
 
 A minimal `PipelineWrapper` looks like this:
 
@@ -274,6 +275,8 @@ hayhooks pipeline deploy-files -n chat_with_website examples/pipeline_wrappers/c
 
 This will deploy the pipeline with the name `chat_with_website`. Any error encountered during development will be printed to the console and show in the server logs.
 
+Alternatively, you can deploy via HTTP: `POST /deploy_files`.
+
 #### PipelineWrapper development with `overwrite` option
 
 During development, you can use the `--overwrite` flag to redeploy your pipeline without restarting the Hayhooks server:
@@ -318,6 +321,64 @@ Then, assuming you've installed the Hayhooks package in a virtual environment, y
 pip install trafilatura
 ```
 
+## Deploy a YAML Pipeline
+
+You can deploy a Haystack pipeline directly from its YAML definition using the `/deploy-yaml` endpoint. This mode builds request/response schemas from the YAML-declared `inputs` and `outputs`.
+
+Note: You can also deploy YAML pipelines from the CLI with `hayhooks pipeline deploy-yaml`. Wrapper-based deployments continue to use `/deploy_files`.
+
+Tip: You can obtain a pipeline's YAML from an existing `Pipeline` instance using `pipeline.dumps()`. See the [Haystack serialization docs](https://docs.haystack.deepset.ai/docs/serialization) for details.
+
+Requirements:
+
+- The YAML must declare both `inputs` and `outputs` fields so the API request/response schemas can be generated. If you have generated the YAML from a `Pipeline` using `pipeline.dumps()`, you will need to add the `inputs` and `outputs` fields _manually_.
+- `inputs`/`outputs` entries map friendly names to pipeline component fields (e.g. `fetcher.urls`, `prompt.query`).
+
+Minimal example:
+
+```yaml
+# ... pipeline definition ...
+
+inputs:
+  urls:
+    - fetcher.urls
+  query:
+    - prompt.query
+outputs:
+  replies: llm.replies
+```
+
+CLI:
+
+```shell
+hayhooks pipeline deploy-yaml -n inputs_outputs_pipeline --description "My pipeline" pipelines/inputs_outputs_pipeline.yml
+```
+
+Alternatively, you can deploy via HTTP: `POST /deploy-yaml`.
+
+If successful, the server exposes a run endpoint at `/{name}/run` with a request/response schema derived from the YAML IO. For example:
+
+```shell
+curl -X POST \
+  http://HAYHOOKS_HOST:HAYHOOKS_PORT/inputs_outputs_pipeline/run \
+  -H 'Content-Type: application/json' \
+  -d '{"urls": ["https://haystack.deepset.ai"], "query": "What is Haystack?"}'
+```
+
+Note: when deploying a YAML pipeline, Hayhooks will create an `AsyncPipeline` instance from the YAML source code. This is because we are in an async context, so we should avoid running sync methods using e.g. `run_in_threadpool`. With AsyncPipeline, we can await `run_async` directly, so we make use of the current event loop.
+
+Limitations:
+
+- YAML-deployed pipelines do not support OpenAI-compatible chat completion endpoints, so they cannot be used with Open WebUI. If you need chat completion/streaming, use a `PipelineWrapper` and implement `run_chat_completion` or `run_chat_completion_async` (see the OpenAI compatibility section below).
+
+Available CLI options for `hayhooks pipeline deploy-yaml`:
+
+- `--name, -n`: override the pipeline name (default: YAML file stem)
+- `--description`: optional human-readable description (used in MCP tool listing)
+- `--overwrite, -o`: overwrite if the pipeline already exists
+- `--skip-mcp`: skip exposing this pipeline as an MCP Tool
+- `--save-file/--no-save-file`: save the YAML under `pipelines/{name}.yml` on the server (default: `--save-file`)
+
 ## Deploy an Agent
 
 Deploying a [Haystack Agent](https://docs.haystack.deepset.ai/docs/agents) is very similar to deploying a pipeline.
@@ -358,6 +419,41 @@ As you can see, the `run_chat_completion_async` method is the one that will be u
 
 The `async_streaming_generator` function is a utility function that [will handle the streaming of the agent's responses](#async_streaming_generator).
 
+## Load pipelines or agents at startup
+
+Hayhooks can automatically deploy pipelines or agents on startup by scanning a pipelines directory.
+
+- Set `HAYHOOKS_PIPELINES_DIR` (defaults to `./pipelines`).
+- On startup, Hayhooks will:
+  - Deploy every YAML file at the directory root (`*.yml`/`*.yaml`) using the file name as the pipeline name.
+  - Deploy every immediate subfolder as a wrapper-based pipeline/agent if it contains a `pipeline_wrapper.py`.
+
+Example layout:
+
+```text
+my-project/
+├── .env
+└── pipelines/
+    ├── inputs_outputs_pipeline.yml        # YAML-only pipeline -> POST /inputs_outputs_pipeline/run
+    ├── chat_with_website/                 # Wrapper-based pipeline -> POST /chat_with_website/run (+ chat endpoints if implemented)
+    │   ├── pipeline_wrapper.py
+    │   └── chat_with_website.yml
+    └── agent_streaming/
+        └── pipeline_wrapper.py
+```
+
+Configure via environment or `.env`:
+
+```shell
+# .env
+HAYHOOKS_PIPELINES_DIR=./pipelines
+```
+
+Notes:
+
+- YAML-deployed pipelines require `inputs` and `outputs` in the YAML and do not expose OpenAI-compatible chat endpoints. For chat/streaming, use a `PipelineWrapper` and implement `run_chat_completion`/`run_chat_completion_async`.
+- If your wrappers import shared code, set `HAYHOOKS_ADDITIONAL_PYTHON_PATH` (see “Sharing code between pipeline wrappers”).
+
 ## Support file uploads
 
 Hayhooks can easily handle uploaded files in your pipeline wrapper `run_api` method by adding `files: Optional[List[UploadFile]] = None` as an argument.
@@ -445,6 +541,52 @@ hayhooks mcp run
 
 This will start the Hayhooks MCP Server on `HAYHOOKS_MCP_HOST:HAYHOOKS_MCP_PORT`.
 
+### Expose a YAML pipeline as a MCP Tool
+
+Hayhooks can expose YAML-deployed pipelines as MCP Tools. When you deploy a pipeline via `/deploy-yaml` (or the CLI `hayhooks pipeline deploy-yaml`), Hayhooks:
+
+- Builds flat request/response models from YAML-declared `inputs` and `outputs`.
+- Registers the pipeline as an `AsyncPipeline` and adds it to the registry with metadata required for MCP Tools.
+- Lists it in MCP `list_tools()` with:
+  - `name`: the pipeline name (YAML file stem or provided `--name`)
+  - `description`: the optional description you pass during deployment (defaults to the pipeline name)
+  - `inputSchema`: JSON schema derived from YAML `inputs`
+
+Calling a YAML pipeline via MCP `call_tool` executes the pipeline asynchronously and returns the pipeline result as a JSON string in `TextContent`.
+
+Sample YAML for a simple `sum` pipeline using only the `haystack.testing.sample_components.sum.Sum` component:
+
+```yaml
+components:
+  sum:
+    init_parameters: {}
+    type: haystack.testing.sample_components.sum.Sum
+
+connections: []
+
+metadata: {}
+
+inputs:
+  values: sum.values
+
+outputs:
+  total: sum.total
+```
+
+Example (Streamable HTTP via MCP client):
+
+```python
+tools = await client.list_tools()
+# Find YAML tool by name, e.g., "sum" (the pipeline name)
+result = await client.call_tool("sum", {"values": [1, 2, 3]})
+assert result.content[0].text == '{"total": 6}'
+```
+
+Notes and limitations:
+
+- YAML pipelines must declare `inputs` and `outputs`.
+- YAML pipelines are run-only via MCP and return JSON text; if you need OpenAI-compatible chat endpoints or streaming, use a `PipelineWrapper` and implement `run_chat_completion`/`run_chat_completion_async`.
+
 ### Create a PipelineWrapper for exposing a Haystack pipeline as a MCP Tool
 
 A [MCP Tool](https://modelcontextprotocol.io/docs/concepts/tools) requires the following properties:
@@ -971,24 +1113,6 @@ We have some dedicated documentation for deployment:
 
 We also have some additional deployment guidelines, see [deployment_guidelines.md](docs/deployment_guidelines.md).
 
-### Legacy Features
-
-#### Deploy a pipeline using only its YAML definition
-
-**⚠️ This way of deployment is not maintained anymore and will be deprecated in the future**.
-
-We're still supporting the Hayhooks _former_ way to deploy a pipeline.
-
-The former command `hayhooks deploy` is now changed to `hayhooks pipeline deploy` and can be used to deploy a pipeline only from a YAML definition file.
-
-For example:
-
-```shell
-hayhooks pipeline deploy -n chat_with_website examples/pipeline_wrappers/chat_with_website/chat_with_website.yml
-```
-
-This will deploy the pipeline with the name `chat_with_website` from the YAML definition file `examples/pipeline_wrappers/chat_with_website/chat_with_website.yml`. You then can check the generated docs at `http://HAYHOOKS_HOST:HAYHOOKS_PORT/docs` or `http://HAYHOOKS_HOST:HAYHOOKS_PORT/redoc`, looking at the `POST /chat_with_website` endpoint.
-
 ### License
 
 This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.

src/hayhooks/cli/pipeline.py

@@ -38,23 +38,49 @@ def _deploy_with_progress(ctx: typer.Context, name: str, endpoint: str, payload:
         show_error_and_abort(f"Pipeline '[bold]{name}[/bold]' already exists! ⚠️")
 
 
-@pipeline.command()
-def deploy(
+@pipeline.command(name="deploy-yaml")
+def deploy_yaml(  # noqa: PLR0913
     ctx: typer.Context,
-    name: Annotated[Optional[str], typer.Option("--name", "-n", help="The name of the pipeline to deploy.")],
     pipeline_file: Path = typer.Argument(  # noqa: B008
-        help="The path to the pipeline file to deploy."
+        help="The path to the YAML pipeline file to deploy."
     ),
+    name: Annotated[Optional[str], typer.Option("--name", "-n", help="The name of the pipeline to deploy.")] = None,
+    overwrite: Annotated[
+        bool, typer.Option("--overwrite", "-o", help="Whether to overwrite the pipeline if it already exists.")
+    ] = False,
+    description: Annotated[
+        Optional[str], typer.Option("--description", help="Optional description for the pipeline.")
+    ] = None,
+    skip_mcp: Annotated[
+        bool, typer.Option("--skip-mcp", help="If set, skip MCP integration for this pipeline.")
+    ] = False,
+    save_file: Annotated[
+        bool,
+        typer.Option(
+            "--save-file/--no-save-file",
+            help="Whether to save the YAML under pipelines/{name}.yml on the server.",
+        ),
+    ] = True,
 ) -> None:
-    """Deploy a pipeline to the Hayhooks server."""
+    """Deploy a YAML pipeline using the /deploy-yaml endpoint."""
     if not pipeline_file.exists():
         show_error_and_abort("Pipeline file does not exist.", str(pipeline_file))
 
     if name is None:
         name = pipeline_file.stem
 
-    payload = {"name": name, "source_code": pipeline_file.read_text()}
-    _deploy_with_progress(ctx=ctx, name=name, endpoint="deploy", payload=payload)
+    payload = {
+        "name": name,
+        "source_code": pipeline_file.read_text(),
+        "overwrite": overwrite,
+        "save_file": save_file,
+        "skip_mcp": skip_mcp,
+    }
+
+    if description is not None:
+        payload["description"] = description
+
+    _deploy_with_progress(ctx=ctx, name=name, endpoint="deploy-yaml", payload=payload)
 
 
 @pipeline.command()
@@ -91,6 +117,17 @@ def deploy_files(
     _deploy_with_progress(ctx=ctx, name=name, endpoint="deploy_files", payload=payload)
 
 
+@pipeline.command(name="deploy", context_settings={"ignore_unknown_options": True, "allow_extra_args": True})
+def deploy(_ctx: typer.Context) -> None:
+    """Removed command; use 'deploy-yaml' or 'deploy-files' instead."""
+    show_warning_panel(
+        "[bold yellow]`hayhooks pipeline deploy` has been removed.[/bold yellow]\n"
+        "Use: \n"
+        "`hayhooks pipeline deploy-yaml <pipeline.yml>` for YAML-based deployments or\n"
+        "`hayhooks pipeline deploy-files <pipeline_dir>` for PipelineWrapper-based deployments."
+    )
+
+
 @pipeline.command()
 def undeploy(
     ctx: typer.Context,

src/hayhooks/server/app.py

@@ -11,12 +11,7 @@
 
 from hayhooks.server.logger import log
 from hayhooks.server.routers import deploy_router, draw_router, openai_router, status_router, undeploy_router
-from hayhooks.server.utils.deploy_utils import (
-    PipelineDefinition,
-    deploy_pipeline_def,
-    deploy_pipeline_files,
-    read_pipeline_files_from_dir,
-)
+from hayhooks.server.utils.deploy_utils import deploy_pipeline_files, deploy_pipeline_yaml, read_pipeline_files_from_dir
 from hayhooks.settings import APP_DESCRIPTION, APP_TITLE, check_cors_settings, settings
 
 
@@ -35,8 +30,7 @@ def deploy_yaml_pipeline(app: FastAPI, pipeline_file_path: Path) -> dict:
     with open(pipeline_file_path) as pipeline_file:
         source_code = pipeline_file.read()
 
-    pipeline_definition = PipelineDefinition(name=name, source_code=source_code)
-    deployed_pipeline = deploy_pipeline_def(app, pipeline_definition)
+    deployed_pipeline = deploy_pipeline_yaml(pipeline_name=name, source_code=source_code, app=app)
     log.info(f"Deployed pipeline from yaml: {deployed_pipeline['name']}")
 
     return deployed_pipeline

src/hayhooks/server/exceptions.py

@@ -10,6 +10,12 @@ class PipelineWrapperError(Exception):
     pass
 
 
+class PipelineYamlError(Exception):
+    """Exception for errors loading pipeline YAML."""
+
+    pass
+
+
 class PipelineModuleLoadError(Exception):
     """Exception for errors loading pipeline module."""
 
@@ -24,3 +30,9 @@ class PipelineNotFoundError(Exception):
     """Exception for errors when a pipeline is not found."""
 
     pass
+
+
+class InvalidYamlIOError(Exception):
+    """Exception for invalid or missing YAML inputs/outputs declarations."""
+
+    pass