Slow Performance Query (Ubuntu Host)

[JamesMatchett]

Hi all,

I hope you are doing well, I recently had my first go with Piserver on an Ubuntu virtual machine today to set up for our monthly Raspberry Pi Jam.

We've had a brilliant start with PiServer and definitely loved the flexibility and functionality, but I have a quick query about a potential performance issue we're currently seeing.

Essentially, we connect and power Pi 3s on the same LAN network, and there doesn't seem to me much if any network traffic (nload reporting near 0 traffic), until about 2-3 minutes later where all 3 Pis will begin to pull down the image at the same time and eventually boot at the same time. Tailing Syslog seems to report some output about the Dnsmasq service being stopped and then started again during this "idle" time.

Since we'd be running about 30+ Pis and some restarts/reboots are inevitable, I was wondering if there would be any tips for speeding up this waiting period before the Pi begins booting or if anyone has seen this issue before?

My initial suspects are DHCP/DNSMasq but I well could be wrong!

More than happy to give more information about our network setup and image configuration and I'll try to export syslog for investigation.

Best, and thank you for your help,

James M. (Northern Ireland Raspberry Pi Jam)

[maxnet]

If you are using a managed Ethernet switch make sure STP is disabled.
(There is normally some code in Piserver to check for this and warn you if it sees STP packets on the network link, but our sniffing code does NOT work when you run Piserver under a virtual machine.)

[gbaman]

Is there any ways to skip the proxydhcp stage, with the IP address of the server on the SD card? Simply disabling spanning tree I am not sure is the answer, especially given the PiNet server that was previously in place with the same equipment had the same Pis booting to login screen in a consistent 30 seconds, as opposed to not far off 2 mins with PiServer in best case (and closer to 10 mins in worst case).
We are keen to improve reliability/consistency, even if it requires SD cards for each Pi.

[maxnet]

Is there any ways to skip the proxydhcp stage, with the IP address of the server on the SD card?

You can simply toss the contents of /var/lib/piserver/os/name-of-os/boot to a SD card.

And that by itself should indeed cut the waiting time in half if STP is used.
Setting IP-address on card with cmdline.txt parameters should also be possible (see kernel documentation for ip= parameter), but should not make a difference in speed.

If STP is enabled and the traditional variant is used (Cisco's portfast is not set, nor is a more modern STP flavor like rapid STP used), the Ethernet switch will be sending test packets to test for network loops for 30 seconds or so before forwarding normal traffic.
It will do that again each time the network link carrier goes up and down.
On PXE boot there will be a 30 second delay on the initial network boot, and when Linux takes over it will reset the network link causing that there will be another 30 second delay.
So if a Piserver boot would normally take say 1 minute on a properly configured network, it then takes 2 minutes.

If you let it boot from SD card, you save one 30 second delay.
But there will still be an unnecessary 30 second delay before network is usable, regardless whether you are using DHCP to get Linux an IP or have configured a static IP...

If it takes 10 minutes instead of 2, there may also be other things wrong than STP.
If it are older models Pi 3 (not 3+) there are a number of bugs in the firmware. You can try if it works better if you toss just bootcode.bin on SD card, and let it PXE boot with that.

Also you mention you are using virtualization, but do not mention your virtualization software.
Recall another user reported slowness symptoms consistent with packet loss when using Virtualbox on Apple gear, that went away after the user got rid of the virtualization.
See: #42