Skip to content

Error for the mcts strategy #37

@pss0204

Description

@pss0204

Hi I ran human eval from no chages of code, and face these kind of error could you help me? for the reflexion it runs well

===========used script=====

export CUDA_VISIBLE_DEVICES=2,3

TEMP_DATASET="$(mktemp /tmp/humaneval_subset.XXXXXX.jsonl)"
trap 'rm -f "$TEMP_DATASET"' EXIT
head -n 10 benchmarks/humaneval-py.jsonl > "$TEMP_DATASET"

python main.py
--run_name "test_gpt3"
--root_dir "root"
--dataset_path "$TEMP_DATASET"
--strategy "mcts"
--language "py"
--model "codellama-7b"
--pass_at_k "1"
--max_iters "4"
--expansion_factor "3"
--number_of_tests "4"
--verbose

Tests failed:
assert encrypt("HELLO") == "LM" # output: substring not found
assert encrypt("world") == "zx" # output: hzcwo
[reflection on previous impl 1]:
The implementation of the encrypt function is incorrect because it only rotates the alphabet by 2 places and does not handle the case where the input string contains uppercase letters. The function assumes that the input string is all lowercase, but this is not guaranteed. Therefore, the function fails to encrypt strings that contain uppercase letters, such as "HELLO" and "world". To fix this issue, the function should handle uppercase letters by rotating them by 2 places as well.

[improved impl]:
def encrypt(s: str) -> str:
"""Create a function encrypt that takes a string as an argument and
returns a string encrypted with the alphabet being rotated.
The alphabet should be rotated in a manner such that the letters
shift down by two multiplied to two places.
For example:
>>> encrypt('hi')
'lm'
>>> encrypt('asdfghjkl')
'ewhjklnop'
>>> encrypt('gf')
'kj'
>>> encrypt('et')
'ix'
"""


Traceback (most recent call last):
File "/home/pss/LanguageAgentTreeSearch/programming/main.py", line 131, in
main(args)
File "/home/pss/LanguageAgentTreeSearch/programming/main.py", line 113, in main
run_strategy(
File "/home/pss/LanguageAgentTreeSearch/programming/main.py", line 51, in kwargs_wrapper
return func(**kwargs)
File "/home/pss/LanguageAgentTreeSearch/programming/mcts.py", line 175, in run_mcts
new_solution = gen.func_impl(
File "/home/pss/LanguageAgentTreeSearch/programming/generators/py_generate.py", line 272, in func_impl
return generate_with_accumulated_context(
File "/home/pss/LanguageAgentTreeSearch/programming/generators/generator_utils.py", line 150, in generate_with_accumulated_context
func_bodies = model.generate_chat(messages=messages, num_comps=num_comps, temperature=temperature)
File "/home/pss/LanguageAgentTreeSearch/programming/generators/model.py", line 137, in generate_chat
prompt = self.prepare_prompt(messages)
File "/home/pss/LanguageAgentTreeSearch/programming/generators/model.py", line 234, in prepare_prompt
assert all([msg.role == "user" for msg in messages[::2]]) and all(
AssertionError: model only supports 'system', 'user' and 'assistant' roles, starting with 'system', then 'user' and alternating (u/a/u/a/u...)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions