The lag of data channel recv data is so high!

[qingwa2009]

I use esp32s3 with ov2640 sending VGA size jpg data every 50ms to computer in local network, the speed is ok!
But i also want to send back some msg to esp32 to control the servo, and the lag is so high, almost >5s
it seems the esp32 have to receive so many ack packet for its sending, not time to handle the receive data.
When i set the sending rate to 3s, the receive rate is ok.
Do you have the same problem?

[vikramdattu]

Hi @qingwa2009 did you happen to find the solution? What do you mean by When i set the sending rate to 3s, the receive rate is ok.?

[qingwa2009]

I had found my solution!

in peer_connection.c

I change the "if" to "while" to make it handle sctp_incoming_data more quickly.
The default code is not good at handle recv data in high speed, it miss so many sctp packet.

int peer_connection_loop(PeerConnection* pc) {
  ...
  switch (pc->state) {
    ...
    case PEER_CONNECTION_COMPLETED:
       ...
      // if ((pc->agent_ret = agent_recv(&pc->agent, pc->agent_buf, sizeof(pc->agent_buf))) > 0) {
      **while ((pc->agent_ret = agent_recv(&pc->agent, pc->agent_buf, sizeof(pc->agent_buf))) > 0) {**
        if (rtcp_probe(pc->agent_buf, pc->agent_ret)) {
          ...
        } 
        else if (dtls_srtp_probe(pc->agent_buf)) {
          ...
          if (ret > 0) {
            sctp_incoming_data(&pc->sctp, (char*)pc->temp_buf, ret);
            **continue;**
          }
        } 
        else if (rtp_packet_validate(pc->agent_buf, pc->agent_ret)) {
          ...
        } 
        **break;**
      }
      ...
      break;
    case PEER_CONNECTION_FAILED:
      break;
    case PEER_CONNECTION_DISCONNECTED:
      break;
    case PEER_CONNECTION_CLOSED:
      break;
    default:
      break;
  }

  return 0;
}

[thebhulawat]

@qingwa2009 It still did not work for me after applying the changes you suggested. Did you make any additional change?

[vikramdattu]

@thebhulawat if you're still interested, I enabled usrsctp for the openai SDK fork in my repo: https://github.com/vikramdattu/openai-realtime-embedded-sdk

@sepfy the data channel is instantaneous with this. I used usrsctp port from here, flattened the project and added component port for ESP-IDF. Not clear the best way to add this to libpeer.

[thebhulawat]

@vikramdattu this is very useful! I tried running your fork but it did not run. But when I added this esp_usrsctp as one of the dependencies on this line , I was able to build and data channel worked really well.

Is this change which I did expected on your repo or am I missing something?

[vikramdattu]

@thebhulawat Great to hear that you solved the issue! Cheers..!
I missed adding the idf_component.yml dependency file. I have pushed the commit to fix it just now. But what you did should also work just fine!

[thebhulawat]

thanks @vikramdattu! have you noticed any stutter on speaker? Also noticing that the the events come some time and some time they don't come at all

[vikramdattu]

@thebhulawat in my close project, I use a work_queue and whenever a message/audio packet is arrived, I push it to the work queue. The receive thread is thus unblocked and the processing, then can happen from the work_queue thread.

On top of this, I have split the receive and send tasks into two: #193
Request you to try these combinations and the experience should be much smoother.

[thebhulawat]

thanks a ton @vikramdattu! let me try