Token Safeguards and Read Tool Modernization

[ericproulx]

Summary

This PR introduces token-based safeguards for file reading to prevent AI agents from accidentally consuming their entire context window with large files. By enforcing a strict token limit, we force agents to use pagination (reading in subsets) or search tools when dealing with large files, ensuring they maintain enough context for reasoning and other tasks.

Key Changes

🛡️ Token Safeguards

• Context Preservation: Replaced read_line_limit (2000 lines) and line_character_limit (2000 chars/line) with read_max_tokens (default 25,000 tokens).
• Proactive Validation: The Read tool now checks token count before returning content. If the content exceeds the limit, it rejects the request with a helpful error guiding the agent to:
  • Use offset and limit parameters to read file in chunks
  • Use Grep tool to search for specific content

📖 Read Tool Overhaul

• Raw Content Output: Removed line numbers from Read tool output. Now returns raw file content, simplifying editing and copy-pasting.
• Malicious File Warning: Added system reminder warning agents to check for malicious code.
• Binary File Handling: Improved detection and clearer error messages for unsupported binary formats.
• Consistent Binary Handling: ReadTracker now reads files the same way as Read tool (UTF-8 first, fall back to binary).

🧠 Memory & Skills

• MemoryRead: Removed line numbers from output (previously used separator). Now returns raw content.
• LoadSkill: Removed line numbers from output. Now returns raw skill content.
• ScratchpadRead: Removed line numbers from output. Now returns raw content.

🔧 Tool Detection & Descriptions

• Context Manager: Updated tool detection to recognize Read output by system reminder instead of line numbers (which are now removed).
• Edit Tools: Updated Edit and MultiEdit descriptions to remove references to line number prefixes.

Configuration Updates

• Added: SwarmSDK.config.read_max_tokens (default: 25,000 tokens)
• Removed: SwarmSDK.config.read_line_limit (was 2000 lines)
• Removed: SwarmSDK.config.line_character_limit (was 2000 chars)

Documentation Updates

• Updated configuration reference with new read_max_tokens parameter
• Removed line number formatting from memory technical details

Test Coverage

• Read tool tests updated for raw output and token safeguards
• MemoryRead tests updated for raw output format
• LoadSkill tests updated for raw output format
• ScratchpadRead tests updated (implicit via integration)
• Config tests updated for new parameter
• ContextManager tests updated for new tool detection
• ReadTracker functionality preserved with consistent file reading
• Virtual entries and skills integration tests updated