Questions for the Main Performance

[ZimaBlue307]

1. I checked the SWE-Bench Leaderboard, TRAE + Doubao-Seed-Code achieved the best performance on SWE-Bench Verified (09/28/2025), which is 78.80%. However, the results in figure 1 is 70.60%. Is it becasue the test-time scaling setup, or other reason to cause the difference?

2. The best perfromance from the paper achieved by Live-SWE-Agent (77.40%) is based on Gemini 3 Pro Preview (2025-11-18), which could be one of the latest and the strongest model. And it seems only two performance from the leaderborad is from Gemini 3 Pro Preview (2025-11-18), another is from mini-SWE-Agent (74.20%). It do make sense that Live-SWE-Agent will perform better than mini-SWE-Agent with Gemini 3 Pro Preview. But what the results look like for SWE-Agent or OpenHands with the same base model? As mini-SWE-Agent + Gemini 3 Pro Preview already achieved 74.20%, chaning mini-SWE-Agent to SWE-Agent or OpenHands has a great probablity for a performance higher than 74.20% (with the same base model). In this case, if these static agent pipelines perform good or even better than Live-SWE-Agent, does that mean: "the freedom" of some agent pipeline provided is already sufficient to let strong model to process the task, while tool-self-evolving on the fly may provide extra burdden to the model and hurt the performance?

Thanks for your valuable insights!

[brutalsavage]

Hi

TRAE + Doubao-Seed-Code achieved the best performance on SWE-Bench Verified (09/28/2025), which is 78.80%. However, the results in figure 1 is 70.60%. Is it becasue the test-time scaling setup, or other reason to cause the difference?

Yes, we only include the results without test-time scaling

chaning mini-SWE-Agent to SWE-Agent or OpenHands has a great probablity for a performance higher than 74.20% (with the same base model).

This is possible, although here are some examples:
For openhands it seems that they have evaluated their framework on Opus 4.5 and their results and our results are:

• OpenHands: 77.6%
• Live-SWE-agent: 79.2%

For SWE-agent, we actually compare against it in the SWE-bench Pro dataset in the paper, the results are both using Claude Sonnet 4.5:

• SWE-agent: 43.7%
• Live-SWE-agent: 45.8%

I did not see any evaluation on Gemini-3 Pro using SWE agent and OpenHands so I cannot comment on that directly, but looking at some other results, there are still benefits to let the agent self-evolve on the fly compared with static scaffolds.

[zhewang2001]

Hi, regarding your first question, we consistently use the scores reported without test-time scaling, i.e., the single attempt scores.
You can check their report here: https://github.com/SWE-bench/experiments/tree/main/evaluation/verified/20250928_trae_doubao_seed_code — their single attempt score is 70.6%, while the parallel test-time compute score is 78.8%.

[ZimaBlue307]

Thanks for the detailed reply! That really helps a lot!

I want to study a bit more about the SWE-Bench Verified trajectories to see what type of tasks were failed by Live-SWE-Agent. I found only the results of Live-SWE-Agent on Gemini 3 is reported on the leaderboard. But I downloaded the swebench_verified, it only contains the trajectories of GPT-5, GPT-5-mini and Claude-Sonnet-4.5, where Gemini 3 trajectories are missing. Is it possible to release the trajectories of Gemini 3 and report what are the failed tasks for each model?

Thanks!

[brutalsavage]

Hi, you can find the trajectories here: https://huggingface.co/livesweagent