Replace manual process handling by local_cmd() #359

guzu · 2025-10-17T08:03:13Z

No description provided.

guzu · 2025-10-20T09:19:21Z

I didn't manage to test everything because I don't know yet how to setup some tests.

stormi · 2025-10-20T12:59:54Z

I didn't manage to test everything because I don't know yet how to setup some tests.

Can you list the ones you could test and the ones you couldn't? We could likely either help you run the remaining ones, or have them run on the CI hosts from your branch.

guzu · 2025-10-20T13:19:59Z

Can you list the ones you could test and the ones you couldn't? We could likely either help you run the remaining ones, or have them run on the CI hosts from your branch.

Sure, I did:

tests/network/test_vif_allowed_ip.py
tests/storage/ext/test_ext_sr.py : mostly to test reboot()
tests/xapi_plugins/plugin_netdata/test_netdata.py

I think we need to test the following:

scripts/install_xcpng.py: which is not a test but is using lots of lib functions used also by the tests
tests/xo/test_xo_connection.py : I didn't manage to register xo-cli register
tests/fs_diff/test_fs_diff.py

stormi · 2025-10-20T14:50:57Z

Ok, so the first one and the third one are not covered by the main CI runs, they're in installation/upgrade tests that @ydirson is much knowledgeable about.

The test_xo_connection.py one, it would be good to make it work. It can be useful to have a local xo-cli and to connect to a XO instance.

guzu · 2025-10-21T12:19:27Z

Update:

test_xo_connection.py is working fine with @stormi VM, not mine for unknown reason yet, at least the test is not broken
no news for other tests

ydirson · 2025-10-28T14:23:05Z

I think we need to test the following:

* `scripts/install_xcpng.py`: which is not a test but is using lots of lib functions used also by the tests

You should be able to use it to launch an automated install of a nested host

* `tests/fs_diff/test_fs_diff.py`

This one requires 2 hosts to be compared, it should be possible to run using any 2 nested hosts (eg. the one you installed by hand, and one installed with scripts/install_xcpng.py

ydirson · 2025-10-28T14:27:07Z

conftest.py

+            # TODO: put that in lib/netutils ?
+            wait_for(lambda: commands.local_cmd(['nc', '-zw5', host_vm.ip, '22'], check=False),


yes, a reusable func in netutils for this, and one for ping, look like a good idea

Done.
I did have an issue with a kind of recursive import with wrap_ip, so move the import inside calling function.

lib/xo.py

scripts/install_xcpng.py

tests/fs_diff/test_fs_diff.py

tests/packages/netdata/test_netdata.py

And change a bit local_cmd() interface to behave like ssh(); introdice simple_output argument. Signed-off-by: Emmanuel Varagnat <emmanuel.varagnat@vates.tech>

Specific error messages are no longer available with this refactoring. Signed-off-by: Emmanuel Varagnat <emmanuel.varagnat@vates.tech>

Signed-off-by: Emmanuel Varagnat <emmanuel.varagnat@vates.tech>

stormi · 2025-11-21T15:55:34Z

lib/commands.py

    assert False, "unexpected type"

 def scp(hostname_or_ip, src, dest, check=True, suppress_fingerprint_warnings=True, local_dest=False):
+    from lib.netutil import wrap_ip


I suppose there's a reason for not importing at the top level, but such exception should be documented with a comment.

This is to avoid cycling import. Or I should have put the new function in something different than netutils.
Or maybe you prefer that the local import be done in netutils ? It should not make any diffrence.
I will add a comment anyway.

Comment added

stormi · 2025-11-21T15:58:58Z

lib/efi.py

        Returns path to signed image.
        """
        assert self._owner_cert is not None
+        assert self._owner_cert.key is not None


I'm not sure how this change relates to the rest of the commit

This is here to please type checking since I added type checking information for local_cmd()

Thanks. Probably something to mention in the commit message, as it's not obvious.

ydirson

There is also an "introdice" typo in the first commit message

ydirson · 2025-11-21T15:33:28Z

lib/host.py

-            wait_for(lambda: not os.system(f"ping -c1 {self.hostname_or_ip} > /dev/null 2>&1"),
+            wait_for(lambda: commands.local_cmd(['ping', '-c1', self.hostname_or_ip], check=False),
                     "Wait for host up", timeout_secs=10 * 60, retry_delay_secs=10)
-            wait_for(lambda: not os.system(f"nc -zw5 {self.hostname_or_ip} 22"),
+            wait_for(lambda: commands.local_cmd(['nc', '-zw5', self.hostname_or_ip, '22'], check=False),


That part wrongly interprets LocalCommandResult as a bool, and gets fixed later in the port-knock patch.

ydirson · 2025-11-21T15:47:50Z

tests/fs_diff/test_fs_diff.py

-    process = subprocess.Popen(
-        [fsdiff, "--reference-host", f"{hosts[0]}", "--test-host", f"{hosts[1]}", "--json-output"],
-        stdout=subprocess.PIPE, stderr=subprocess.PIPE
-    )
-    stdout, _ = process.communicate()
-
-    if process.returncode != 0:
-        print(stdout.decode())
-
-    assert process.returncode == 0
+    res = local_cmd([fsdiff, "--reference-host", f"{hosts[0]}",
+                             "--test-host", f"{hosts[1]}",
+                             "--json-output"], simple_output=False)
+    assert res.returncode == 0


Doesn't this change the behavior, when we were only showing stdout when a non-expected diff occurs?

Absolutely !
Fixed

ydirson · 2025-11-21T15:50:16Z

lib/commands.py

+    if simple_output:
+        return output.strip()
    return LocalCommandResult(res.returncode, output)


Making the default simple_output=True actually changes the default behavior of the function. A separate commit for this change would make it easier for review.

ydirson · 2025-11-21T15:56:39Z

lib/netutil.py

+def wait_for_port_knock(host, port, ping=True):
+    if ping:
+        wait_for(lambda: local_cmd(['ping', '-c1', host], check=False, simple_output=False).returncode == 0,
+                 "Wait for host up", timeout_secs=10 * 60, retry_delay_secs=10)
+    wait_for(lambda: local_cmd(['nc', '-zw5', host, str(port)], check=False, simple_output=False).returncode == 0,
+             "Wait for ssh up on host", timeout_secs=10 * 60, retry_delay_secs=5)
+


That's not what's usually been known as "port knocking" (ie. sequence of packets to different ports to cause a firewall to open a port).
Also the fact we only ever use it for ssh is still reflected in the "Wait for ssh up on host" string, but we're losing information from places (eg. "Wait for ssh up on nested host").
Maybe just call it wait_for_ssh, and pass a descriptive string as parameter for the the logs?

It was initially called wait_for_ssh() 😉

ydirson · 2025-11-21T15:59:35Z

lib/commands.py


 def scp(hostname_or_ip, src, dest, check=True, suppress_fingerprint_warnings=True, local_dest=False):
+    from lib.netutil import wrap_ip
+


Why do we need this import inside a function? This seems unrelated, and is usually frowned upon.

To avoid cyclic import. Same comment here #359 (comment)

Comment added

ydirson · 2025-11-21T16:03:15Z

conftest.py

-from pkgfixtures import formatted_and_mounted_ext4_disk, sr_disk_wiped
+from pkgfixtures import formatted_and_mounted_ext4_disk, sr_disk_wiped  # noqa


Why is this "noqa" for? Might be good to bundle with a comment explaining it.

I did not understand the big comment which looks like a big warning so I left the code.
But the reason for the noqa is that a linter is telling that these imports are not used.

And actually they're used, but in an automagical way by pytest, so indeed the linter is confused. I agree with Yann: let's add a comment to explain why "noqa" so that we don't ask the question again in the future.

ydirson · 2025-11-21T16:03:20Z

conftest.py

-    if is_uuid(ref):
+    if is_uuid(ref):  # noqa


Why is this "noqa" for? Might be good to bundle with a comment explaining it.

To please a linter, because it tells that that double if is no needed because we return refin both cases, and that could be expressed with a or and a single return.

Fixed. noqa was removed and test changed

guzu requested a review from ydirson October 17, 2025 08:03

guzu force-pushed the eva/local-cmd-generalize branch 4 times, most recently from 31da723 to 33bd187 Compare October 17, 2025 08:40

ydirson reviewed Oct 28, 2025

View reviewed changes

lib/xo.py Outdated Show resolved Hide resolved

stormi reviewed Nov 19, 2025

View reviewed changes

lib/xo.py Outdated Show resolved Hide resolved

stormi reviewed Nov 19, 2025

View reviewed changes

scripts/install_xcpng.py Outdated Show resolved Hide resolved

stormi reviewed Nov 19, 2025

View reviewed changes

tests/fs_diff/test_fs_diff.py Outdated Show resolved Hide resolved

stormi reviewed Nov 19, 2025

View reviewed changes

tests/packages/netdata/test_netdata.py Show resolved Hide resolved

guzu force-pushed the eva/local-cmd-generalize branch from 9d42bbb to 9bfd3cc Compare November 19, 2025 17:07

guzu requested a review from stormi November 19, 2025 17:08

guzu force-pushed the eva/local-cmd-generalize branch from 9bfd3cc to 49fbbca Compare November 19, 2025 17:19

guzu mentioned this pull request Nov 20, 2025

Add support for VM serial/UART console logging #364

Open

guzu added 2 commits November 21, 2025 10:12

Replace manual process handling by local_cmd()

b6c59af

And change a bit local_cmd() interface to behave like ssh(); introdice simple_output argument. Signed-off-by: Emmanuel Varagnat <emmanuel.varagnat@vates.tech>

Refactor port knocking with netcat into netutil

e92408f

Specific error messages are no longer available with this refactoring. Signed-off-by: Emmanuel Varagnat <emmanuel.varagnat@vates.tech>

guzu force-pushed the eva/local-cmd-generalize branch from 49fbbca to 27b9248 Compare November 21, 2025 09:16

conftest: fix hide some linter issues

233a249

Signed-off-by: Emmanuel Varagnat <emmanuel.varagnat@vates.tech>

guzu force-pushed the eva/local-cmd-generalize branch from 27b9248 to 233a249 Compare November 21, 2025 09:16

guzu requested a review from ydirson November 21, 2025 09:20

stormi reviewed Nov 21, 2025

View reviewed changes

ydirson requested changes Nov 21, 2025

View reviewed changes

		# TODO: put that in lib/netutils ?
		wait_for(lambda: commands.local_cmd(['nc', '-zw5', host_vm.ip, '22'], check=False),


		def scp(hostname_or_ip, src, dest, check=True, suppress_fingerprint_warnings=True, local_dest=False):
		from lib.netutil import wrap_ip

		from pkgfixtures import formatted_and_mounted_ext4_disk, sr_disk_wiped
		from pkgfixtures import formatted_and_mounted_ext4_disk, sr_disk_wiped # noqa

Replace manual process handling by local_cmd() #359

Are you sure you want to change the base?

Replace manual process handling by local_cmd() #359

Conversation

guzu commented Oct 17, 2025

Uh oh!

guzu commented Oct 20, 2025

Uh oh!

stormi commented Oct 20, 2025

Uh oh!

guzu commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stormi commented Oct 20, 2025

Uh oh!

guzu commented Oct 21, 2025

Uh oh!

ydirson commented Oct 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ydirson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

guzu commented Oct 20, 2025 •

edited

Loading