Modify source file to load curl library at runtime #80

kanjoe24 · 2025-05-28T11:26:19Z

Modify ut_kvp.c file to load curl library at runtime. This reduces the size of libut_control.so to upto 4.8M from 15M.
Also, didn't notice any change in functionality.

kanjoe24 · 2025-05-28T11:48:43Z

Adding before and after log snippets on one of the platforms:

Before:

2025-05-28-11:36:37, LOG   , ut_console.c,   639 :      Running Test : 'kvp single include url'
2025-05-28-11:36:37, ERROR , ut_kvp.c,   979 : Error: curl_easy_perform() failed: Couldn't resolve host name

2025-05-28-11:36:37, ERROR , ut_kvp.c,   979 : Error: curl_easy_perform() failed: Couldn't resolve host name

2025-05-28-11:36:37, ERROR , ut_kvp.c,   979 : Error: curl_easy_perform() failed: Couldn't resolve host name
---
"1":
  value: true


2025-05-28-11:36:37, LOG   , ut_console.c,   661 :      Test Complete : 'kvp single include url'
2025-05-28-11:36:37, LOG   , ut_console.c,   633 :

After:

2025-05-28-11:41:08, LOG   , ut_console.c,   639 :      Running Test : 'kvp single include url'
2025-05-28-11:41:08, DEBUG , ut_kvp.c,    85 : Initializing curl symbols from /usr/lib/libcurl.so.4
2025-05-28-11:41:08, ERROR , ut_kvp.c,  1041 : Error: curl_easy_perform() failed: Couldn't resolve host name

2025-05-28-11:41:08, ERROR , ut_kvp.c,  1041 : Error: curl_easy_perform() failed: Couldn't resolve host name

2025-05-28-11:41:08, ERROR , ut_kvp.c,  1041 : Error: curl_easy_perform() failed: Couldn't resolve host name
---
"1":
  value: true


2025-05-28-11:41:08, LOG   , ut_console.c,   661 :      Test Complete : 'kvp single include url'

Also, adding the logs here:
ut-control-test-full-curl-dlopen-mtk.txt
ut-control-test-full-develop-mtk.txt
e.

Size dfference:
Before:

ls -lah libut_control.so 
-rwxr-xr-x    1 root     root       19.6M May 16 06:24 libut_control.so

After:

 ls -lah libut_control.so 
-rwxr-xr-x    1 root     root        4.8M May 28 11:40 libut_control.so

kanjoe24 · 2025-05-28T12:09:27Z

Adding before and after log snippets on linux:

Before:

2025-05-28-11:53:22, LOG   , ut_console.c,   639 :      Running Test : 'kvp single include url'---
"1":
  value: true
"2":
  value: true
"3":
  value: true
"4":
  value: true


2025-05-28-11:53:23, LOG   , ut_console.c,   661 :      Test Complete : 'kvp single include url'

After:

2025-05-28-11:30:55, LOG   , ut_console.c,   639 :      Running Test : 'kvp single include url'
2025-05-28-11:30:55, DEBUG , ut_kvp.c,    85 : Initializing curl symbols from /usr/lib/x86_64-linux-gnu/libcurl.so.4---
"1":
  value: true
"2":
  value: true
"3":
  value: true
"4":
  value: true


2025-05-28-11:30:56, LOG   , ut_console.c,   661 :      Test Complete : 'kvp single include url'

Also attaching full logs here:
ut-control-test-full-develop-linux.txt
ut-control-test-curl-dlopen-linux.txt

kanjoe24 · 2025-05-28T14:03:39Z

logs from release test script:

./release-test-script-ut-control.sh -t feature/gh79-ut-control-reduce-size
==========================================================
RESULTS for ubuntu 
On the branch feature/gh79-ut-control-reduce-size. PASS
build/linux/curl/lib/libcurl.a exists. PASS
Openssl static lib does not exist. PASS 
CMake host binary does not exist. PASS 
==========================================================
==========================================================
RESULTS for VM-SYNC 
On the branch feature/gh79-ut-control-reduce-size. PASS
build/linux/curl/lib/libcurl.a exists. PASS
build/linux/openssl/lib/libssl.a exists. PASS 
host-tools/CMake-3.30.0/build/bin/cmake exists. PASS 
==========================================================
==========================================================
RESULTS for dunfell_arm 
On the branch feature/gh79-ut-control-reduce-size. PASS
build/arm/curl/lib/libcurl.a exists. PASS
build/arm/openssl/lib/libssl.a exists. PASS 
CMake host binary does not exist. PASS 
==========================================================
==========================================================
RESULTS for dunfell_linux 
On the branch feature/gh79-ut-control-reduce-size. PASS
CURL static lib does not exist. PASS
Openssl static lib does not exist. PASS 
CMake host binary exists. PASS 
==========================================================
==========================================================
RESULTS for kirkstone_arm 
On the branch feature/gh79-ut-control-reduce-size. PASS
build/arm/curl/lib/libcurl.a exists. PASS
build/arm/openssl/lib/libssl.a exists. PASS 
CMake host binary does not exist. PASS 
==========================================================
==========================================================
RESULTS for kirkstone_linux 
On the branch feature/gh79-ut-control-reduce-size. PASS
build/linux/curl/lib/libcurl.a exists. PASS
Openssl static lib does not exist. PASS 
CMake host binary does not exist. PASS 
==========================================================

Ulrond · 2025-05-29T07:55:21Z

Makefile

+LIBCURL_PATH="/usr/lib/libcurl.so.4"
 else
 #linux case
+LIBCURL_PATH="/usr/lib/x86_64-linux-gnu/libcurl.so.4"


.4 i don't think is right, the main .so should be a symlink to the right version do an ls -la on the directory. The .4 version of the library will depend on what version of curl is installed and it may not exist but the .so should always. Google the rules for the .so linkage etc.

Response from chatgpt:

Yes, you can pass the path /usr/lib/x86_64-linux-gnu/libcurl.so.4 to your dlopen code and it should work correctly.
Here’s why:
🔹 libcurl.so.4 is a symlink
Your ls -la output confirms:
libcurl.so.4 -> libcurl.so.4.7.0
This means that libcurl.so.4 is a symbolic link to the actual shared library libcurl.so.4.7.0.
When using dlopen() in your code, you can pass:
dlopen("/usr/lib/x86_64-linux-gnu/libcurl.so.4", RTLD_NOW);
This will resolve correctly because the symlink points to a valid ELF shared object file (libcurl.so.4.7.0), which contains the symbols you need.

ls -la /usr/lib/x86_64-linux-gnu/libcurl.* lrwxrwxrwx 1 root root 16 Dec 11 17:26 /usr/lib/x86_64-linux-gnu/libcurl.so.4 -> libcurl.so.4.7.0 -rw-r--r-- 1 root root 677656 Dec 11 17:26 /usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0

yup you've made my point.

libcurl.so is a sym link to the latest installed library

libcurl.so.4 is version 4 of the curl library which may or may not be installed, your checking for a min version of the library, not a max version of it.

so libcurl.so is correct.. libcurl.so.4 is not.

My problem here is different servers are giving different output .For ex:

jpn323@ceres ~/workspace/ut-core-1April/scripts (feature/gh#208-print-version)$ find /usr/ -type l -iname "libcurl.so*" /usr/lib/x86_64-linux-gnu/libcurl.so.4 /usr/lib/x86_64-linux-gnu/libcurl.so

jpn323@janus ~/workspace/ut-control-reduce-size/tests/build/bin (feature/gh79-ut-control-reduce-size)$ find /usr/ -type l -iname "libcurl.so*" /usr/lib/x86_64-linux-gnu/libcurl.so.4

I have hence come up with a common solution, that will provide me the first symlink when I try to find the library.

src/ut_kvp.c

Ulrond · 2025-06-05T11:04:02Z

Makefile

+LIBCURL_PATH="/usr/lib/libcurl.so.4"
 else
 #linux case
+LIBCURL_PATH="/usr/lib/x86_64-linux-gnu/libcurl.so.4"


yup you've made my point.

libcurl.so is a sym link to the latest installed library

libcurl.so.4 is version 4 of the curl library which may or may not be installed, your checking for a min version of the library, not a max version of it.

so libcurl.so is correct.. libcurl.so.4 is not.

kanjoe24 · 2025-06-12T15:35:31Z

Attaching the debug logs when run with LD_DEBUG=libs
LD_DEBUG-ARM.txt

Ulrond · 2025-06-12T16:57:44Z

tests/Makefile

 	$(ROOT_DIR)/build.sh
 	make -C ./ut-core test
 	@cp $(LIB_DIR)/lib* ${BIN_DIR}
+	@if [ -d $(LIB_DIR)/../curl ]; then \


this makes no sense.

the test builds test, it has no-idea what ut-control builds, wants to build or what it has internally to it.

the test calls the main ut-control makefile, which builds whatever libraries it requires ( which are copied to the output directory ), because that's what the ut-control makefile knows about.

Ulrond · 2025-06-12T16:59:58Z

configure.sh

        # For linux
        if [ "$OPENSSL_IS_SYSTEM_INSTALLED" -eq 1 ]; then
-            ./configure --prefix=${CURL_BUILD_DIR} --with-ssl --without-zlib --without-libpsl --without-libidn2 --disable-docs --disable-libcurl-option --disable-alt-svc --disable-headers-api --disable-hsts --without-libgsasl
+            ./configure --prefix=${CURL_BUILD_DIR} --with-ssl --without-zlib --without-libpsl --without-libidn2 --disable-docs --disable-libcurl-option --disable-alt-svc --disable-headers-api --disable-hsts --without-libgsasl  --enable-shared --enable-static


Do I understand this correctly

shared - because you load the .so in the dlopen right?
static - because for external consumption you take ut-control.so & include the .a into that .so?

you may need to also consider stripped the symbols for the external libraries from ut-control.so, because you may not get what you expect..

say ut-control.so is loaded by you, which your expecting the libcurl.a also to be loaded.

But in the same process, someone else already loaded their libcurl.a, the one that's already present will then be used not your one.

that's the disadvantages of not having static libraries which you control embedded into your .so's

Ulrond · 2025-06-13T10:15:06Z

this approach isn't going to work... It will lead to you using system libraries that could be in the rootfs, causing incompatibility with the testing suites ( which will break ut-controls requirements)
ut_control.so key requirement is to be self contained and contain all the requirements / libraries to run the tests, so you can't dynamically load libcurl for instance, that will break the system, because you'll get a version from the rootfs that's not expected. libcurl in ut-control, is frozen... libcurl in the system path is not, and will move on to later versions.

Modify gh #79 : Modify source file to load curl library at runtime

6c36615

kanjoe24 requested a review from a team as a code owner May 28, 2025 11:26

kanjoe24 requested review from Anbukannadhasan, Ulrond, bhanucbp, hari22yuva and srinivasgtl May 28, 2025 12:10

kanjoe24 self-assigned this May 28, 2025

kanjoe24 linked an issue May 28, 2025 that may be closed by this pull request

Task:Reduce size of ut-control to ensure minimal foot print #79

Open

kanjoe24 added the enhancement New feature or request label May 28, 2025

Ulrond reviewed May 29, 2025

View reviewed changes

Fix gh #79 : Fix review comments

2d509c6

kanjoe24 requested a review from Ulrond May 29, 2025 13:34

kanjoe24 added this to the ut-control : 1.6.4 : Dynamic Library Optimization milestone Jun 5, 2025

Ulrond requested changes Jun 5, 2025

View reviewed changes

Address gh#79 : Addressing review comments

f5c9b0e

kanjoe24 requested a review from Ulrond June 5, 2025 15:47

Fix gh #79 : Fixing further review comments

70f3db9

kanjoe24 mentioned this pull request Jun 11, 2025

Adding changes to copy openssl and curl libraries to build/bin rdkcentral/ut-core#211

Open

kanjoe24 added 2 commits June 12, 2025 14:34

Add gh #79 : Adding few more improvements

0035a79

Add gh #79 : Adding few more improvements

0f86b9a

Ulrond reviewed Jun 12, 2025

View reviewed changes

Ulrond added this to UT-Core Roadmap Jun 13, 2025

github-project-automation bot moved this to Todo in UT-Core Roadmap Jun 13, 2025

Ulrond moved this from Todo to Blocked in UT-Core Roadmap Jun 13, 2025

kanjoe24 removed this from the ut-control : 1.6.4 : Support for yaml anchors, aliases and merge keys milestone Jul 10, 2025

kanjoe24 closed this Jan 21, 2026

github-project-automation bot moved this from Blocked to Resolved in UT-Core Roadmap Jan 21, 2026

github-actions bot locked and limited conversation to collaborators Jan 21, 2026

Modify source file to load curl library at runtime #80

Modify source file to load curl library at runtime #80

Uh oh!

Conversation

kanjoe24 commented May 28, 2025

Uh oh!

kanjoe24 commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kanjoe24 commented May 28, 2025

Uh oh!

kanjoe24 commented May 28, 2025

Uh oh!

Ulrond May 29, 2025

Choose a reason for hiding this comment

Uh oh!

kanjoe24 May 29, 2025

Choose a reason for hiding this comment

Uh oh!

Ulrond Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

kanjoe24 Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Ulrond Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

kanjoe24 commented Jun 12, 2025

Uh oh!

Ulrond Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ulrond Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ulrond commented Jun 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kanjoe24 commented May 28, 2025 •

edited

Loading

Ulrond Jun 12, 2025 •

edited

Loading

Ulrond Jun 12, 2025 •

edited

Loading