From a6d750fd109457fa4de1d154da6cf02c0a35090a Mon Sep 17 00:00:00 2001 From: HuHeng Date: Thu, 9 Nov 2023 18:35:28 +0800 Subject: [PATCH 1/3] Update _index.md add python backend api doc --- .../multiple_features/data_backend/_index.md | 95 ++++++++++++++++++- 1 file changed, 92 insertions(+), 3 deletions(-) diff --git a/content/en/docs/bmf/multiple_features/data_backend/_index.md b/content/en/docs/bmf/multiple_features/data_backend/_index.md index 655b4947..186c3565 100644 --- a/content/en/docs/bmf/multiple_features/data_backend/_index.md +++ b/content/en/docs/bmf/multiple_features/data_backend/_index.md @@ -115,9 +115,98 @@ some example code snippet： ``` - ### Python Interface -The unified `bmf_convert` as C++ will be designed and implemented later. + +```python + +## python backend api +## arg src_vf: source videoframe +## arg src_md: source videoframe mediadescription +## arg dst_md: dst videoframe mediadescription +## return vf: convert result videoframe + +vf = bmf_convert(src_vf, src_md, dst_md) +``` +The `bmf_convert` interface in Python is bound to the corresponding interface in C++ using `pybind11`. The `MediaDesc` class follows the same pattern, corresponding to the `MediaDesc` class in C++. The usage of the interface is also similar. + + +#### VideoFrame scale, colorspace conversion, device memory transfer + +use `bmf_convert` do scale, csc, and device memory transfer + +```python + H420 = mp.PixelInfo(mp.kPF_YUV420P, mp.kCS_BT709) + vf = VideoFrame(640, 360, pix_info=H420) + md = MediaDesc() + md.width(1920).height(1080).pixel_format(mp.kPF_RGB24).device( + mp.Device("cuda:0")) + dst_vf = bmf_convert(vf, MediaDesc(), md) + assert (dst_vf.width == 1920) + assert (dst_vf.height == 1080) + assert (dst_vf.frame().format() == mp.kPF_RGB24) + assert (dst_vf.frame().device().type() == mp.kCUDA) + assert (dst_vf.frame().device().index() == 0) + +``` + + +#### Conversion between VideoFrame and third-party data structure in python + +BMF support the following types of third-party python structure conversion with VideoFrame + +1. numpy.ndarray +2. torch.Tensor + +Here, numpy.ndarray is used as an example to illustrate the conversion. Other types of conversions can be referred to in the `test_module_sdk.py`. + +##### VideoFrame to numpy.ndarray + +1. dst_dp set media_type with value MediaType::kTensor +3. do `bmf_convert`, check if the return VideoFrame is a valid VideoFrame +4. use `private_get(numpy.ndarray)` to get the numpy.ndarray from the return VideoFrame + +Some detailed explanations: + +We use a `MediaType` type called `MediaType.kTensor` as the destination type, which corresponds to the `hmp::Tensor` structure in C++. When we call the Python interface `bmf_convert`, the actual conversion happens in the core C++ layer of `bmf`. Similar to the C++ interface, this interface converts a `VideoFrame` into an `hmp::Tensor` and stores it in the private data of the `VideoFrame`. Then, we use the Python interface `private_get` to retrieve a `numpy.ndarray`. `private_get` is also a binding to the C++ interface, which converts the `hmp::Tensor` to a `numpy.ndarray` using `pybind11`. + +some example code snippet： + +```python + H420 = mp.PixelInfo(mp.kPF_YUV420P, mp.kCS_BT709) + vf = VideoFrame(640, 360, pix_info=H420) + + md = MediaDesc() + md.width(1920).height(1080).pixel_format(mp.kPF_RGB24).media_type( + MediaType.kTensor) + + dst_vf = bmf_convert(vf, MediaDesc(), md) + assert (dst_vf.width == 1920) + assert (dst_vf.height == 1080) + assert (dst_vf.frame().format() == mp.kPF_RGB24) + assert (bool(dst_vf) == True) + + np_array = dst_vf.private_get(np.ndarray) + print(np_array.shape) + assert (np_array.shape[0] == 1080) + assert (np_array.shape[1] == 1920) + assert (np_array.shape[2] == 3) + + src_vf = VideoFrame() + src_vf.private_attach(np_array) + + src_md = MediaDesc() + src_md.pixel_format(mp.kPF_RGB24).media_type(MediaType.kTensor) + + dst_md = MediaDesc() + dst_md.pixel_format(mp.kPF_YUV420P).color_space( + mp.kCS_BT709).width(320).height(180) + + new_vf = bmf_convert(src_vf, src_md, dst_md) + + assert new_vf.width == 320 + assert new_vf.height == 180 + assert new_vf.frame().format() == mp.kPF_YUV420P +``` #### Scale and colorspace conversion @@ -215,4 +304,4 @@ Sample code: frame = mp.Frame(mp.from_torch(torch_vf.contiguous()), RGB) bmf_vframe = VideoFrame(frame) #... -``` \ No newline at end of file +``` From 7137be33488d87533222d9342f4176c9f270f76b Mon Sep 17 00:00:00 2001 From: HuHeng Date: Thu, 9 Nov 2023 18:45:39 +0800 Subject: [PATCH 2/3] Update _index.md update backend doc --- .../en/docs/bmf/multiple_features/data_backend/_index.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/content/en/docs/bmf/multiple_features/data_backend/_index.md b/content/en/docs/bmf/multiple_features/data_backend/_index.md index 186c3565..8e077600 100644 --- a/content/en/docs/bmf/multiple_features/data_backend/_index.md +++ b/content/en/docs/bmf/multiple_features/data_backend/_index.md @@ -208,7 +208,8 @@ some example code snippet： assert new_vf.frame().format() == mp.kPF_YUV420P ``` -#### Scale and colorspace conversion +## BMF Data Convert(old style) +### Scale and colorspace conversion `bmf.hml.hmp.img.rgb_to_yuv` @@ -234,7 +235,7 @@ Sample code: mp.img.rgb_to_yuv(out_vf.frame().data(), src_vf.frame().plane(0), NV12, mp.kNHWC) ``` -#### Device memory transfer +### Device memory transfer Interface used: `VideoFrame.frame().device()` gets the Device property @@ -252,7 +253,7 @@ Sample code: vf = vf.cuda() #... ``` -#### Conversion between VideoFrame and third-party data structure +### Conversion between VideoFrame and third-party data structure In python API, those types of third-party data type are supported: - VideoFrame, which is the general class of video frame in BMF. And `VideoFrame` includes `Frame` as member - numpy From f5bbd87b53f6bff789fb859c146061a822ab6886 Mon Sep 17 00:00:00 2001 From: HuHeng Date: Fri, 10 Nov 2023 10:54:19 +0800 Subject: [PATCH 3/3] fix index of backend doc --- content/en/docs/bmf/multiple_features/data_backend/_index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/bmf/multiple_features/data_backend/_index.md b/content/en/docs/bmf/multiple_features/data_backend/_index.md index 8e077600..25c45c1f 100644 --- a/content/en/docs/bmf/multiple_features/data_backend/_index.md +++ b/content/en/docs/bmf/multiple_features/data_backend/_index.md @@ -162,8 +162,8 @@ Here, numpy.ndarray is used as an example to illustrate the conversion. Other ty ##### VideoFrame to numpy.ndarray 1. dst_dp set media_type with value MediaType::kTensor -3. do `bmf_convert`, check if the return VideoFrame is a valid VideoFrame -4. use `private_get(numpy.ndarray)` to get the numpy.ndarray from the return VideoFrame +2. do `bmf_convert`, check if the return VideoFrame is a valid VideoFrame +3. use `private_get(numpy.ndarray)` to get the numpy.ndarray from the return VideoFrame Some detailed explanations: