Support for Structured Output Between Agents

[brisacoder]

I have several agents that exchange Structured Outputs (SO) messages between them. The current spec only support textual inputs and outputs, which creates several impediments when these agents communicate. For example, JSON objects and their schemas need to be passed as serialized strings that, in turn, create a whole range of problems: proprietary messages, sends and receiver need to known the schema off-path, versioning of schemas across agents and so-forth.

Supporting SO would also get the semantics of refusal since it is baked into the SO protocol. Changes needed:

ChatCompletions API

We need a response_format object at the same level of messages

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
  model="gpt-4o",
  messages=[
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What is Barack Obama basic information?"
        }
      ]
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": "{\n  \"full_name\": \"Barack Hussein Obama II\",\n  \"date_of_birth\": \"1961-08-04\"\n}"
        }
      ]
    }
  ],
  response_format={
    "type": "json_schema",
    "json_schema": {
      "name": "person",
      "strict": True,
      "schema": {
        "type": "object",
        "properties": {
          "full_name": {
            "type": "string",
            "description": "The full name of the person."
          },
          "date_of_birth": {
            "type": "string",
            "description": "The date of birth of the person in YYYY-MM-DD format."
          }
        },
        "required": [
          "full_name",
          "date_of_birth"
        ],
        "additionalProperties": False
      }
    }
  },
  temperature=1,
  max_completion_tokens=2048,
  top_p=1,
  frequency_penalty=0,
  presence_penalty=0
)

Responses API

• In order to support to support SO, we need a text object following OpenAI format at the same level as input. https://platform.openai.com/docs/api-reference/responses/create#responses-create-text

• On the output we need similar changes. https://platform.openai.com/docs/api-reference/responses/object#responses/object-text

This would make the exchange of JSON objects between agents self-describable. Example of input below.

import json
from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="gpt-4o-2024-08-06",
    input=[
        {"role": "system", "content": "You are an expert at structured data extraction. You will be given unstructured text from a research paper and should convert it into the given structure."},
        {"role": "user", "content": "..."}
    ],
    text={
        "format": {
              "type": "json_schema",
              "name": "research_paper_extraction",
              "schema": {
                "type": "object",
                "properties": {
                    "title": { "type": "string" },
                    "authors": { 
                        "type": "array",
                        "items": { "type": "string" }
                    },
                    "abstract": { "type": "string" },
                    "keywords": { 
                        "type": "array", 
                        "items": { "type": "string" }
                    }
                },
                "required": ["title", "authors", "abstract", "keywords"],
                "additionalProperties": False
            },
            "strict": True
        },
    },
)

research_paper = json.loads(response.output_text)

[hinthornw]

Could you please cite where you are reading that the protocol only supports text interops? Communication is through arbitrary objects (whose schemas are communicated via the assistants methods), e.g.:

https://langchain-ai.github.io/agent-protocol/api.html#tag/runs/POST/threads/%7Bthread_id%7D/runs

The protocol is in most ways more flexible than openai's, but if we are miscommunicating this somewhere I'd like to improve the documentation.

[brisacoder]

William FH (@hinthornw) It is not about flexibility, it is about interoperability and standardization. We are facing a problem where we need interoperability across agents from multiple frameworks, enterprises and developers. It goes without saying that if every enterprise, framework and developer adds its own version of SO, it would be a problem.

It is perfectly okay to leave an arbitrary object there for proprietary extensions, but something like SO deserves an unified approach and Agent Protocol is well positioned to drive it.