Message::Send with non-RPC type is silently ignored and never reaches user's ScriptHandler

[SpenserCai]

Bug Description

In frida-rust 0.16.10, there's a critical bug in the message handling system where Message::Send messages with non-RPC types are completely ignored and never forwarded to the user's ScriptHandler::on_message() method. This breaks the fundamental communication channel between JavaScript and Rust for custom send() calls.

Root Cause

The issue is in src/script.rs in the call_on_message function (lines 112-118):

match formatted_msg {
    Message::Send(msg) => {
        if msg.payload.r#type == "frida:rpc" {
            let callback_handler: *mut CallbackHandler = user_data as _;
            on_message(callback_handler.as_mut().unwrap(), Message::Send(msg));
        }
        // ❌ MISSING: No else branch! Non-RPC Send messages are completely ignored
    }
    _ => {
        // Only non-Send messages reach the user's ScriptHandler
        let handler: &mut I = &mut *(user_data as *mut I);
        handler.on_message(&formatted_msg, data_vec);
    }
}

The problem: When a JavaScript script calls send() with a custom message type (not "frida:rpc"), the message is parsed correctly as Message::Send, but since msg.payload.r#type != "frida:rpc", it falls through without any handling and never reaches the user's ScriptHandler.

Code Example

JavaScript side:

// This message is completely lost!
send({
    type: "custom_data",
    id: 1,
    result: "success", 
    returns: {
        dll_base: "0x12345678",
        dll_name: "example.dll"
    }
});

Rust side:

struct MyHandler;

impl ScriptHandler for MyHandler {
    fn on_message(&mut self, message: &Message, _data: Option<Vec<u8>>) {
        match message {
            Message::Send(send_msg) => {
                // ❌ This will NEVER be called for non-RPC messages
                println!("Received send message: {:?}", send_msg);
            }
            Message::Log(log_msg) => {
                println!("Log: {}", log_msg.payload);
            }
            _ => {}
        }
    }
}

Expected Behavior

All Message::Send messages should be forwarded to the user's ScriptHandler::on_message() method, regardless of their type. The user should be able to handle both RPC and custom send messages.

Actual Behavior

• RPC messages (type: "frida:rpc") are sent to an internal channel and never reach the user
• Non-RPC send messages are silently ignored
• Only Message::Log, Message::Error, etc. reach the user's handler

Impact

This bug breaks:

• Custom communication protocols between JavaScript and Rust
• Any use case requiring structured data transfer via send()
• Backward compatibility with standard Frida usage patterns

Proposed Solution

Option 1: Forward all Send messages to user handler (Recommended)

match formatted_msg {
    Message::Send(msg) => {
        // First, always forward to user's handler
        let handler: &mut I = &mut *(user_data as *mut I);
        
        // Retrieve extra message data, if any
        let data_vec = if data.is_null() {
            None
        } else {
            // ... data processing code ...
        };
        
        handler.on_message(&formatted_msg, data_vec);
        
        // Then, if it's RPC, also handle internally
        if msg.payload.r#type == "frida:rpc" {
            let callback_handler: *mut CallbackHandler = user_data as _;
            on_message(callback_handler.as_mut().unwrap(), Message::Send(msg));
        }
    }
    _ => {
        // Handle other message types as before
        let handler: &mut I = &mut *(user_data as *mut I);
        // ... existing code ...
    }
}

Option 2: Add configuration flag

Allow users to choose whether they want to receive Send messages in their handler.

Workaround

Currently, the only workaround is to avoid send() entirely and use console.log() with structured strings:

// Workaround: use console.log instead of send()
console.log("CUSTOM_DATA:" + JSON.stringify({
    dll_base: "0x12345678",
    dll_name: "example.dll"
}));

Environment

• frida-rust version: 0.16.10
• Rust version: 1.70+
• Platform: All platforms (Windows, macOS, Linux)

Additional Notes

This appears to be a regression or design oversight. The current behavior makes frida-rust incompatible with standard Frida usage patterns where send() is the primary communication mechanism.

The fix should maintain backward compatibility while restoring the expected behavior that users can receive and handle their own Send messages.

---

Thank you for your time reviewing this issue. This bug significantly impacts the usability of frida-rust for custom instrumentation scenarios.

[s1341]

1. please don't use AI to create issues.
2. PRs welcome. (but not AI-generated ones)

[SpenserCai]

1. please don't use AI to create issues.
2. PRs welcome. (but not AI-generated ones)

I understand, I did encounter this problem. I just used AI to polish the issue.

[s1341]

Yeah. I prefer less polished issues to ones that are clearly AI generated.