Build PeerCC for HoloLens using IL2CPP

[anders-lundgren]

I'm looking to build a P2P RTC feature into our HoloLens app that uses the IL2CPP toolchain, and want to start by getting the PeerCC-sample running on HoloLens using IL2CPP. Is there a version that is already working for this scenario, or if not, what branch would be the best starting point?

[jamescadd]

Yes, this branch uses the IL2CPP toolchain with Unity 2018.2.13 tooling, see instructions at the bottom of the readme: https://github.com/ortclib/ortclib-sdk/tree/Mosa/m62-peercc-unity

[anders-lundgren]

Thanks! This looks very promising. I was able to make some progress using the release at https://github.com/ortclib/ortclib-sdk/tree/releases/m62, built as an IL2CPP project using VS2017 and Unity 2018.2.20f1 (couldn't find the above branch but it seems to have been merged into m62).

I can now initiate a call from my PC to the HoloLens running the IL2CPP version of the app, with video and audio (although not in sync, audio is good but video is lagging behind several seconds).
I am however unable to initiate a call from HoloLens:

| PC->Holo   | OK (using h.264)                     |
| Holo->Holo | Not working, unable to initiate call |
| Holo->PC   | Not working, unable to initiate call |

When trying to initiate the call from the HoloLens the connection fails, the first time with the exception:

Exception thrown at 0x772FF2B2 (KernelBase.dll) in PeerCCUnity.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.

Will investigate whether there is a problem getting the correct stream number or if the problem lies elsewhere.

If I try again during the same session the app crashes with the following exception:

`VideoEffect Added
Exception thrown at 0x772FF2B2 (KernelBase.dll) in PeerCCUnity.exe: WinRT originate error - 0xC00DABE4 : 'RecordState'.
Exception thrown at 0x772FF2B2 in PeerCCUnity.exe: Microsoft C++ exception: Platform::COMException ^ at memory location 0x05E7F998. HRESULT:0xC00DABE4 The op is invalid.
WinRT information: RecordState

Stack trace:

combase.dll!7775eece()
Windows.Media.dll!67d4326a()
Windows.Media.dll!67d40833()
Windows.Media.dll!67d007c5()
Org.WebRtc.dll!Windows::Media::Capture::IMediaCapture::StartRecordToCustomSinkAsync(Windows::Media::MediaProperties::MediaEncodingProfile ^ __param0, Windows::Media::IMediaExtension ^ __param1)
Org.WebRtc.dll!webrtc::videocapturemodule::CaptureDevice::StartCapture::__l2::Concurrency::task (Windows::Media::IMediaExtension ^)::__l2::() Line 441
[Inline Frame] Org.WebRtc.dll!std::_Invoker_functor::_Call(webrtc::videocapturemodule::CaptureDevice::StartCapture::__l2::Concurrency::task (Windows::Media::IMediaExtension ^)::__l2::Concurrency::task (void) &)
[Inline Frame] Org.WebRtc.dll!std::invoke(webrtc::videocapturemodule::CaptureDevice::StartCapture::__l2::Concurrency::task (Windows::Media::IMediaExtension ^)::__l2::Concurrency::task (void) &)
[Inline Frame] Org.WebRtc.dll!std::_Invoker_ret<Concurrency::task,0>::_Call(webrtc::videocapturemodule::CaptureDevice::StartCapture::__l2::Concurrency::task (Windows::Media::IMediaExtension ^)::__l2::Concurrency::task (void) &)
Org.WebRtc.dll!std::_Func_impl_no_alloc<Concurrency::task (void),Concurrency::task >::_Do_call()
Org.WebRtc.dll!std::_Func_class<Concurrency::task >::operator()()
[Inline Frame] Org.WebRtc.dll!Concurrency::details::_MakeUnitToTFunc::__l2::<lambda_b75dc542f91c0e4c60552ed298ee5a55>::operator()(unsigned char) Line 2354
[Inline Frame] Org.WebRtc.dll!std::_Invoker_functor::_Call(Concurrency::details::_MakeUnitToTFunc::__l2::<lambda_b75dc542f91c0e4c60552ed298ee5a55> &)
[Inline Frame] Org.WebRtc.dll!std::invoke(Concurrency::details::_MakeUnitToTFunc::__l2::<lambda_b75dc542f91c0e4c60552ed298ee5a55> &)
[Inline Frame] Org.WebRtc.dll!std::_Invoker_ret<Concurrency::task,0>::_Call(Concurrency::details::_MakeUnitToTFunc::__l2::<lambda_b75dc542f91c0e4c60552ed298ee5a55> &)
Org.WebRtc.dll!std::_Func_impl_no_alloc<<lambda_b75dc542f91c0e4c60552ed298ee5a55>,Concurrency::task,unsigned char>::_Do_call(unsigned char && <_Args_0>)
Org.WebRtc.dll!std::_Func_class<Concurrency::task,unsigned char>::operator()(unsigned char <_Args_0>)
Org.WebRtc.dll!Concurrency::task::_ContinuationTaskHandle<void,void,std::function<Concurrency::task __cdecl(void)>,std::integral_constant<bool,0>,Concurrency::details::_TypeSelectorAsyncTask>::_LogWorkItemAndInvokeUserLambda<std::function<Concurrency::task __cdecl(unsigned char)>,unsigned char>(std::function<Concurrency::task __cdecl(unsigned char)> && _func, unsigned char && _value) Line 3627
Org.WebRtc.dll!Concurrency::task::_ContinuationTaskHandle<void,void,std::function<Concurrency::task __cdecl(void)>,std::integral_constant<bool,0>,Concurrency::details::_TypeSelectorAsyncTask>::_Continue(std::integral_constant<bool,0> __formal, Concurrency::details::_TypeSelectorAsyncOperationOrTask __formal) Line 3674
[Inline Frame] Org.WebRtc.dll!Concurrency::task::_ContinuationTaskHandle<void,void,std::function<Concurrency::task __cdecl(void)>,std::integral_constant<bool,0>,Concurrency::details::_TypeSelectorAsyncTask>::_Perform() Line 3632
Org.WebRtc.dll!Concurrency::details::_PPLTaskHandle<unsigned char,Concurrency::task::_ContinuationTaskHandle<void,void,std::function<Concurrency::task __cdecl(void)>,std::integral_constant<bool,0>,Concurrency::details::_TypeSelectorAsyncTask>,Concurrency::details::_ContinuationTaskHandleBase>::invoke() Line 1473
Org.WebRtc.dll!Concurrency::details::_TaskProcHandle::_RunChoreBridge(void * _Parameter) Line 160
Org.WebRtc.dll!Concurrency::details::_DefaultPPLTaskScheduler::_PPLTaskChore::_Callback(void * _Args) Line 52
msvcp140_app.dll!611cab96()
ntdll.dll!77a03e08()
ntdll.dll!77a04d99()
ntdll.dll!77a327d6()
ntdll.dll!77a3279b()`

I think this second error is due to the video stream not being properly closed in the previous attempt. Will keep investigating.

[jamescadd]

Hello, we yesterday discovered an issue with H.264 video freezing that's currently being investigated. You're correct that releases/m62 is the best branch currently. Thanks for reporting those bugs - I have been unable to initiate a call from HL as well on this branch. This, and getting m66+ running on HL are currently being worked on, will update the thread when we have more info.

[anders-lundgren]

      Hello, we yesterday discovered an issue with H.264 video freezing that's currently being investigated. You're correct that releases/m62 is the best branch currently. Thanks for reporting those bugs - I have been unable to initiate a call from HL as well on this branch. This, and getting m66+ running on HL are currently being worked on, will update the thread when we have more info.

Sounds great! I'll keep investigating on my end - let me know if I can be of assistance, would be happy to contribute any changes - just having a bit of a struggle overseeing the whole solution atm.

[anders-lundgren]

Update: I think the issues we've been seeing are due to some limitation in the IL2CPP toolchain in older versions of Unity. When using the latest beta (Unity 2019.1) to build the HL app, I am able to initiate a call from HL to a PC, and also get a bidirectional setup working with audio and video between two HoloLenses. Framerate is initially quite good (15+) in both devices, but there is significant delay (approximately 1 second) and video tends to freeze in one or both streams after a couple of seconds. Using default video and audio settings on the HoloLens, and 640x480x15 (H.264 and Opus 48k) on the PC.

[anders-lundgren]

Update 2: Still seeing multiple issues with the IL2CPP build. We can initiate calls from the HL to other peers, but video is soon lagging severely. Also, endpoint enumeration fails, so we can only initiate a call to the first device registered at the signaling server.
Periodically, I get this error in the visual studio output window when debugging:

Invalid parameter passed to C runtime function.

Seems similar to issues described here: https://github.com/webrtc-uwp/webrtc-uwp-sdk/issues/90

Any ideas where to start looking?

[jamescadd]

Small update, the latest branch for IL2CPP is now releases/m62. The lag you see sounds like it's related to an issue with the decoder. Does it happen on a PC to PC call? We believe we have a fix for this and should have a PR ready early next week.

[anders-lundgren]

Hi @jamescadd, thanks, very much appreciate your help looking into this. I agree, this does seem to be codec related. I tried deploying the UWP sample app to an old tablet with a "Cherry Trail" Atom CPU, and yes, when using H.264 I see the same behaviour. Video starts off with a nice framerate in both directions, but as soon as the image changes too much, I perceive a frame drop and a delay that the stream then never recovers from. I tried switching to the VP8 codec and performance was flawless, even on the limited Cherry Trail tablet.

[jamescadd]

Good feedback, thank you. I'll let the people testing the newer codec know that it's a potentially problematic setup and see if they can test there as well. Hoping the updated MF pipeline resolves the issue.

[anders-lundgren]

That sounds, great, looking forward to trying it out! As a side note, I tried switching to VP8 when calling the HoloLens, but it only resulted in a black feed. I see in earlier comments that this has worked previously, is there a change in how VP8 is handled on HoloLens when building with IL2CPP?

[anders-lundgren]

...continuing my investigation, I found 3 potentially interesting things:

1. If I switch to the lowest resolution and framerate on the PC (H.264, 640x360x15), fewer frame drops appear to occur and the call can sometimes continue for a minute or so with low (sub-second) latency on the PC->Holo video stream. The Holo->PC stream is fixed at 896x504x15fps and still lags severely in this case.
2. If I set the PC app to use VP8, but then use the HoloLens to initiate the call, it seems that the HoloLens proposes H.264 and so the call succeeds, with both parties using H.264, and the same performance as described above.
3. If I on the other hand use the PC to initiate the call, it will initiate a VP8 stream that the HoloLens app is unable to decode (using the m62/release branch, built using IL2CPP on Unity 2018.3.4f1 and Windows 10.0.17134.0 SDK). However, in this case, since we're only dealing with one video stream, the 896x504x15 stream from the HoloLens is displayed on the PC side with very little lag.

All of this indicates that we're hitting performance/throughput issues when using H.264 on the HoloLens, both for compression and decompression.
The CPU usage on the HoloLens is steady at approximately 65% CPU during a bidirectional call. Not too bad, indicating that hardware acceleration is working and that the frame drops are not related to CPU constraints.
On a bidirectional call, both streams are in the order of ~1.5 MBps. When running only one video stream, the stream from HoloLens to PC reaches ~2.5 MBps while the audio and signalling data from PC to Holo drops to approximately 80 kbps. This indicates a bottleneck somewhere in the pipeline limiting total data to approximately 3 Mbps. Our other apps can stream other types of data at speeds in excess of 150 Mbps to and from the HoloLens, so something appears to be wrong here.

Bottom line, it seems that the main problem is with how the handling of large/dropped frames is done, since this leads to a lag that the stream never recovers from. There also seems to be some kind of bottleneck in the (hardware accelerated) processing of frames, both when encoding and decoding H.264 streams on the HoloLens.

Possibly related, there are still a lot of errors in the output:

Invalid parameter passed to C runtime function.

(also mentioned in this thread: ortclib/ortclib-sdk#23)

Have tested in Unity 2018.3.4f1, 2019.1.0b1, 2019.1.0b2 and 2019.2.0a1 with the same result.

[jamescadd]

Appreciate the details! There's definitely an issue with network latency in the current pipeline. Another team internally has done work to replace the decode pipeline and the initial results are promising. This is likely the best option to solve this class of problems, will update the thread when ready.

[markgrossnickle]

Great thread. Wish I had found this information earlier. Thanks for sharing all of your findings and updates.

[anders-lundgren]

Update after using the most recent (?) branch (https://github.com/webrtc-uwp/webrtc-uwp-sdk/tree/releases/m71). The project builds and I can deploy to HoloLens using IL2CPP and Unity 2018.3.8, however, I still see a low framerate and a significant delay in the video streams after a couple of seconds when connecting a call between two HoloLens devices. Still using h.264.

I've checked and everything from the relevant pull request (webrtc-uwp/webrtc-windows#50) seems to be in the build, still the behaviour is very similar to what I saw before switching to the new "forward" decoder. Do I need to change anything in the sample project (PeerCC-Unity) before building for the new pipeline to take effect?

[jamescadd]

@anders-lundgren I'm checking that all fixes have made it into releases/m71 but that still does not resolve the issue when both encoding and decoding happen on HoloLens. I would recommend you try the WebRTC functionality in the MRTK and see if this works better for encode & decode - would be good to hear your feedback if you do.