Video over the network

[josuah]

This is a thread to gather information and keep track of the progress of video over the network for anyone interested in implementing it.

Topics include:

• Control protocol implementation (i.e. ONVIF, WebRTC, SIP)
• Streaming protocol implementation (i.e. SRT, RTP)
• Video stream container formats (i.e. MPEG-TS)
• Video pipeline control framework (i.e. Gstreamer, FFmpeg)
• Video compression support (i.e. H.264)
• Other initiatives (OpenHD, OpenIPC)
• Collaboration with other RTOS and ecosystems.
• Integration with other areas (i.e. NPU for detecting events on the feed, displays)

[josuah]

General discussion

When a topic does not really fit other categories, or simply to say hi.

[kartben]

what's the status of https://docs.zephyrproject.org/latest/samples/drivers/video/tcpserversink/README.html -- has someone tested it recently?

[josuah]

I tried it with native_sim about a month ago, and am testing now again:

Full details here.

P.S. I should remember to subscribe to my own discussion threads so that I get notifications.

[josuah]

Maybe some day we can have a Zephyr <-> Zephyr video streaming served on a static webpage: the USB Video Webcam sample for the local camera, SRTP (WebRTC) for the remote camera.

[josuah]

Streaming/control protocol (SRT, RTP, WebRTC, SIP, ONVIF...)

• https://en.wikipedia.org/wiki/Real-time_Transport_Protocol

[josuah]

Thanks for the investigation @sayednasermoravej! That's great to have a high-level overview.

Through this it seems like RTP/SRTP are building blocks for about everything. Sometimes with long chains like SIP -> SDP -> RTSP -> RTP. Sometimes as only one supported protocol like WebRTC (SRTP = RTP + AES).

That could be some interesting upgrade path to the tcpserversink sample so that the user does not have to specify the format/resolution etc.

So if I understand, no camera control (PTZ, exposure) and no format negotiation (which resolution/FPS/bandwidth is supported on each end) yet with only RTP (a streaming protocol). This would need an extra control protocol to be supported, which would become a lot easier to implement once RTP is there... and be helped by some framework to deal with media (audio/video/metadata) like Gstreamer.

[ghost]

You're welcome!

You're right — RTP/SRTP are indeed the main building blocks.

RTP typically runs on top of UDP.

And yes, to support features like PTZ control, an additional control protocol is needed.

[beriberikix]

Summoning @Sean-Der, who is the WebRTC expert and has been doing more with embedded lately.

[Sean-Der]

Hey @josuah sorry I didn't respond sooner. I would love to help with this!

WebRTC's nice because it works in the browser. Also available in GStreamer, FFmpeg, OBS etc... As a protocol I don't think it has anything special over others, it just works everywhere :)

I have been mostly working with people doing security cameras + building hardware AI assistants an espressif devices. If you have a clear project I would LOVE to go build it.

I would love to have a 'Build a Open Source security camera for $15 and it works on any hardware Zephyr supports' project. I love a project that I can lose myself in.

[josuah]

Thrilled to heard that! Let's build it then.

As far as I understand the situation [1], libsrtp might be a good candidate to get a video stream going with a standard protocol.

@thecapn32 is preparing a pull request to get H.264 frames canned into flash, and sent in loop to make a virtual H.264 video source, then it's possible to work in a simulation environment like native_sim.

There is a Zephyr example sending raw uncompressed video from Zephyr to Gstreamer on Linux. This can be used as a starting point to build libsrtp. I think @seyednasermoravej got started with it.

This is the current progress. We can discuss about it on this thread, on Discord, or any other messaging system you prefer.

'Build a Open Source security camera for $15 and it works on any hardware Zephyr supports' project

I'd love to see that happen! This was done for this CAN bus to USB adapter project: https://cannectivity.org/
It works on several affordable Aliexpress boards https://github.com/CANnectivity/cannectivity/tree/main/app/boards
We can do the same for WebRTC cameras. Espressif already uses libsrtp so the hardware should be a good match.

[josuah]

Video stream container formats (MPEG-TS...)

• https://en.wikipedia.org/wiki/Comparison_of_video_container_formats

[josuah]

Video pipeline framework (like Gstreamer)

On Linux there is Gstreamer, FFmpeg, PipeWire, libcamera...

Together they provide a fairly complete abstraction of the video <-> network connection and are bedrock for supporting video streaming protocols.

[josuah]

Presented by @ngphibang at OSS-EU 2025:

https://static.sched.com/hosted_files/osseu2025/7d/Video4Zephyr_EOSS_2025_Final.pdf#page=27

https://osseu2025.sched.com/event/25VpN/video4zephyr-from-basic-apis-towards-a-full-fledged-video-subsystem-phi-bang-nguyen-nxp-semiconductors

[josuah]

Seeing this, it becomes more clear that some easy way forward is to have a srtp or libsrtp sinkpad that makes video data go over the network.

[josuah]

Hardware compression support (MJPEG, H.264, H.265...)

[josuah]

H.264 decoding, test data

I ordered this camera and captured a raw stream with this command:

v4l2-ctl --stream-mmap=100 --device /dev/video6 --stream-to=h264_raw_data_from_arducam_b0261.bin

h264_raw_data_from_arducam_b0261.bin.zip

ffprobe /tmp/h264_raw_data_from_arducam_b0261.bin:

Input #0, h264, from '/tmp/h264_raw_data_from_arducam_b0261.bin':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuv420p(progressive), 1280x720, 25 fps, 1200k tbr, 1200k tbn

file /tmp/h264_raw_data_from_arducam_b0261.bin:

/tmp/h264_raw_data_from_arducam_b0261.bin: JVT NAL sequence, H.264 video, main @ L 41

hexdump -C /tmp/h264_raw_data_from_arducam_b0261.bin | head

00000000  00 00 00 01 67 4d 40 29  96 54 02 80 2d c8 00 00  |....gM@).T..-...|
00000010  00 01 68 ee 38 80 00 00  00 01 65 88 80 40 00 49  |..h.8.....e..@.I|
00000020  f5 00 f1 01 28 e0 06 e9  8a 07 7d f3 00 bc d9 f5  |....(.....}.....|
00000030  b7 19 ab e3 7b 16 16 ae  f3 95 7e db 27 d4 5b fd  |....{.....~.'.[.|
00000040  2f 0a 6d 09 5a e5 de 78  cb 6e ce b7 4a 50 98 65  |/.m.Z..x.n..JP.e|
00000050  a5 c4 74 0e 73 56 71 3f  d2 5a f4 19 61 a6 cf 94  |..t.sVq?.Z..a...|
00000060  43 6f 83 fb ba 13 68 3f  bb ca 15 f7 6a f8 35 a0  |Co....h?....j.5.|
00000070  42 fa 3f 8f e0 b2 f5 10  b2 40 b9 bc 47 e1 6e 66  |B.?......@..G.nf|
00000080  39 34 d2 dd 0e f1 9f 89  2d 5d 6a 45 a0 0b bc 60  |94......-]jE...`|
00000090  e5 71 97 b3 61 f9 ae 11  6f be 7f 76 b1 ff b3 62  |.q..a...o..v...b|

h264_analyze -o /dev/stdout /tmp/h264_raw_data_from_arducam_b0261.bin | head

!! Found NAL at offset 4 (0x0004), size 10 (0x000A)
0.8: forbidden_zero_bit: 0
0.7: nal->nal_ref_idc: 3
0.5: nal->nal_unit_type: 7
1.8: sps->profile_idc: 77
2.8: sps->constraint_set0_flag: 0
2.7: sps->constraint_set1_flag: 1
2.6: sps->constraint_set2_flag: 0
2.5: sps->constraint_set3_flag: 0
2.4: sps->constraint_set4_flag: 0

• h264_analyze can be obtained by compiling the https://github.com/aizvorski/h264bitstream library (no dependency, compiles easily).

• Online software: https://mradionov.github.io/h264-bitstream-viewer/

• Guide for decoding H.264 using ffmpeg and other tools: https://developer.ridgerun.com/wiki/index.php/H264_Analysis_Tools

• The raw "H.264 bitstream" (#1) is documented in the ITU document annex B: https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-H.264-200305-S!!PDF-E&type=items (PDF of annex B only)

[thecapn32]

Here is a brief H.264 introduction for anyone who might be interested to start working on this topic.

H.264

H.264 is a compression standard that converts raw video frames into compressed
binary bitstream for storage and transmission.

NAL(Network Abstraction Layer)

NAL(Network Abstraction Layer) Units are the fundamental building blocks of
every H.264 bitstream. Think of them as "packets" or "containers" that hold
different types of video data.

NAL Unit Structure

  ┌──────────────┬────────────────────────┐
  │  NAL Header  │     NAL Payload        │
  │   (1 byte)   │   (variable length)    │
  └──────────────┴────────────────────────┘

NAL Unit Header

Bit:  7   6   5   4   3   2   1   0
    ┌───┬───┬───┬───┬───┬───┬───┬───┐
    │ F │NRI│NRI│ T │ T │ T │ T │ T │
    └───┴───┴───┴───┴───┴───┴───┴───┘
     │   └─────┘   └─────────────┘
     │      │             │
     │      │             └── NAL Unit Type (5 bits)
     │      └─────────────────── NRI - Reference Indicator (2 bits)
     └────────────────────────── F - Forbidden Zero Bit (1 bit)

NAL Unit Types

Type	Value	Meaning
1	0x01	Non-IDR Slice (P or B frame)
5	0x05	IDR Slice (I-frame)
6	0x06	SEI (Supplemental Info)
7	0x07	SPS (Sequence Param Set)
8	0x08	PPS (Picture Param Set)
9	0x09	Access Unit Delimiter
0	0x00	Unspecified
10+	10+	Reserved or for extensions

Bitstream Organization Patterns

SPS → PPS → IDR → B → B → P → B → B → P → ...
 │     │     │    │           │
 │     │     │    └─ Depends on IDR and P
 │     │     └ Keyframe       └─ Depends on previous P and next P
 │     └ Picture params
 └─ Sequence params

NAL Unit Framing

Annex B Format

In annex b format NAL units are seperated by start codes. Some example usecase
would be in broadcasting, realtime encoding(V4L2) and raw h264 files, etc.

Hex: 00 00 00 01 [NAL Unit 1] 00 00 00 01 [NAL Unit 2] ...
    └Start Code┘             └Start Code┘

AVCC Format

In AVCC format there is no start code and each NAL unit is prefixed by its
length (usually 4 bytes). Some example usecases would be in ISO BMFF(MP4)
container, WebRTC and FLV, etc.

Hex: 00 00 01 23 [291 bytes of NAL data] 00 00 00 45 [69 bytes of NAL data]
     └Length=291┘                       └Length=69─┘

[josuah]

Very neat! It's immediately possible to understand the raw data captured from the webcam (like you said in another chat):

00000000  00 00 00 01 67 4d 40 29  96 54 02 80 2d c8 00 00  |....gM@).T..-...|
00000010  00 01 68 ee 38 80 00 00  00 01 65 88 80 40 00 49  |..h.8.....e..@.I|

We recognize the 00 00 00 01 start code there.

This means some conversion to do while getting the data from the hardware (Annex B framing) to the network (AVCC framing)

[josuah]

Other initiatives (OpenHD, OpenIPC, other RTOS)

• https://github.com/OpenIPC
• https://github.com/OpenHD

[josuah]

Gstreamer also uses libsrtp:
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-bad/ext/srtp/gstsrtpenc.c#L61

[babrath]

I've been looking into audio streaming using RTP. Any chance progress has been made in add libsrtp as a zephyr module, or do I give it a try myself?

[seyednasermoravej]

Yes, I'm working to add libsrtp as an external module in Zephyr RTOS.

[babrath]

That's great to hear! I'm always willing to help test if there's anything publicly available.

[seyednasermoravej]

Sure, once it gets ready I'll ask you for test.
Thanks for your help. 🙏🏻