Invalid type for parameter inputText, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

[pipelineburst]

hi all,
i have a test plan with about 30 tests. each test, when run in isolation, works fine. when the test plan contains all 30, the test run often errors out with either:

botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter inputText, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

or

pydantic_core._pydantic_core.ValidationError: 1 validation error for TestResult
reasoning
Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
For further information visit https://errors.pydantic.dev/2.10/v/string_type

the traces files generate ok, even with the run errors out. the summary file however is not written out. re-running the eval several times would eventually complete successfully once, then errors out again.

here is an example of the type of tests in the test plan:

question_30:
steps:
- Ask the agent to build a table with the region in rows and vendors in the columns showing data traffic for 17 of January 2025.
expected_results:
- The agent returns the matrix table with region in the rows and vendor in the columns, showing the intersected data traffic
Region | Vendor1 | Vendor2 | Vendor3 | Vendor4
Central | 7229821 | 2459730 | 0 | 0
East | 0 | 4123003 | 148 | 0
North | 0 | 0 | 304 | 0
South | 0 | 2796411 | 1811676 | 0
West | 2510043 | 1356714 | 3917543 | 0
Empty | 0 | 59270 | 209 | 2019

[sharonxiaohanli]

Thank you so much for reporting this issue. I couldn't reproduce the issue but I think there might be a bug in handling concurrency. I wonder if lowering the num of threads can help

agent-evaluation/src/agenteval/cli.py

Line 69 in cfdb083

"--num-threads",

?

[pipelineburst]

Thank you so much for reporting this issue. I couldn't reproduce the issue but I think there might be a bug in handling concurrency. I wonder if lowering the num of threads can help

agent-evaluation/src/agenteval/cli.py

Line 69 in cfdb083

 "--num-threads", 

?

thank you so much for the quick reply... just gave the "--num-threads" a try and set it to 1 on the eval run. all the test cases complete and the traces are written out. but the cli errors out again and the summary results are not produced.

pydantic_core._pydantic_core.ValidationError: 1 validation error for TestResult
reasoning
Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]
For further information visit https://errors.pydantic.dev/2.10/v/string_type

can I provide any other info for you to see what we do next ?

[athusv]

The error persists even with --num-threads 1, but it occurs intermittently

Hi @sharonxiaohanli,

I'm experiencing the same issue as @pipelineburst, but with some additional behavior.

• Even when setting --num-threads 1, the error still occurs intermittently.
• Sometimes the test runs successfully, but when I re-run it, the error appears again.
• The error is recurring but not deterministic, making it harder to pinpoint the exact cause.

Here’s the full error trace:

(venv) ➜  agent-evaluation git:(main) ✗ agenteval run --verbose --num-threads 1           
[13:47:36] INFO     Starting 3 test(s)                                                                                         
           INFO     Number of threads: 1                                                                                      
Traceback (most recent call last):
  File ".../venv/bin/agenteval", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File ".../site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File ".../site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/agenteval/cli.py", line 90, in run
    plan.run(
  File ".../site-packages/agenteval/plan/plan.py", line 139, in run
    self._run_concurrent()
  File ".../site-packages/agenteval/plan/plan.py", line 191, in _run_concurrent
    raise e
  File ".../site-packages/agenteval/plan/plan.py", line 189, in _run_concurrent
    future.result()
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/agenteval/plan/plan.py", line 201, in _run_test
    result = evaluator.run()
             ^^^^^^^^^^^^^^^
  File ".../site-packages/agenteval/evaluators/base_evaluator.py", line 137, in run
    self.test_result = self.evaluate()
                       ^^^^^^^^^^^^^^^
  File ".../site-packages/agenteval/evaluators/claude_3/evaluator.py", line 229, in evaluate
    self.conversation.add_turn(user_input, self._invoke_target(user_input))
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/agenteval/evaluators/claude_3/evaluator.py", line 202, in _invoke_target
    target_response = self.target.invoke(user_input)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/agenteval/targets/bedrock_agent/target.py", line 61, in invoke
    response = self.boto3_client.invoke_agent(**args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/botocore/client.py", line 980, in _make_api_call
    request_dict = self._convert_to_request_dict(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/botocore/client.py", line 1047, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/botocore/validate.py", line 381, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter inputText, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

Possible Cause of the Error

As seen in the trace, the error likely occurs because user_input is None, causing invoke_agent to fail at the time of the call.

Do you think this could be related to concurrency or maybe a race condition when handling input values?

Let me know if you can help debug this or if you need more information!

Thanks! 🚀

[manankalra]

facing similar issue
if you look at the traces, you'll find that the last trace doesn't contain a reasoning string, which is null. I encounter this only when I set the maximum turns > number of steps.

thus validation fails for this: https://github.com/awslabs/agent-evaluation/blob/main/src/agenteval/test/test_result.py#L25
maybe changing the field type can help here as a workaround

[EtiGoz]

Hello everyone,

I recently ran into the same error. It only happens to me when I use Sonnet 3.7. It may be because of the way this fondation model returns its answer. Thus, in src/agenteval/evaluators/canonical/evaluator.py, in _extract_content_from_xml(), I believe sometimes the return Tuple can be wrong after reaching the turn limit.