From de94384a04888a94e3bf86d721c671ccbfe35859 Mon Sep 17 00:00:00 2001 From: giles knap Date: Sat, 20 Dec 2025 22:21:24 +0000 Subject: [PATCH 1/4] add initial sdcard setup docs --- docs/how-to/new_raspi.md | 109 +++++++++++++++++++++++++++++++++------ 1 file changed, 94 insertions(+), 15 deletions(-) diff --git a/docs/how-to/new_raspi.md b/docs/how-to/new_raspi.md index c712f77..43cedbc 100644 --- a/docs/how-to/new_raspi.md +++ b/docs/how-to/new_raspi.md @@ -4,7 +4,7 @@ IMPORTANT: normal operation for setting up a new Raspberry Pi is to use the provided pre-built image sdcards. See [Commissioning a new Raspberry Pi Server](../tutorials/commissioning_raspi.md). -This page is only useful if you need to create a completely new image, perhaps because a new version of Raspberry Pi OS has been released. +This page is only useful if you need to create a completely new sdcard with a new image, perhaps because a new version of Raspberry Pi OS has been released. If you just want to update an existing image, e.g. to update the version of usb-remote it uses see [Updating an Existing Raspberry Pi Boot Image](updating_raspi_image.md). @@ -19,28 +19,24 @@ You will need: ## Step 1 Image the microSD Card with Raspberry Pi OS -1. Download the latest Raspberry Pi OS Full image from the [Raspberry Pi website](https://www.raspberrypi.com/software/operating-systems/). - - Note you only need full version if you plan to use the Raspberry Pi's GUI capabilities. Otherwise the "Raspberry Pi OS Lite" version is sufficient. Use full if you want to connect to the Government WiFi network which requires a GUI for setup. +1. Download the latest Raspberry Pi OS Lite image from the [Raspberry Pi website](https://www.raspberrypi.com/software/operating-systems/). + - If you plan to use the Raspberry Pi's GUI capabilities, use "Raspberry Pi OS Full" version. Use full if you want to connect to the GovWiFi network which requires a GUI for setup. 1. Use `lsblk` to get a list of block devices on your system before inserting the microSD card. 1. Insert the microSD card into your card reader and connect it to your computer. -1. Use `lsblk` again to identify the device name of the microSD card (e.g. `/dev/sdX`). +1. Use `lsblk` again to identify the device name of the microSD card (e.g. `/dev/sdb`). 1. uncompress the downloaded Raspberry Pi OS image. ```bash cd ~/Downloads - unxz ./2025-12-04-raspios-trixie-arm64-full.img.xz + unxz ./2025-12-04-raspios-trixie-arm64.img.xz ``` -1. Use `dd` to write the Raspberry Pi OS image to the microSD card. Replace `/dev/sdX` with the actual device name of your microSD card. +1. Use `dd` to write the Raspberry Pi OS image to the microSD card. Replace `/dev/sdX` with the actual device name of your microSD card. Be very careful with this command as it will overwrite the specified device. ```bash - sudo dd if=./2025-12-04-raspios-trixie-arm64-full.img of=/dev/sdX bs=4M status=progress conv=fsync - ``` -1. Once the `dd` command has completed, run `sync` to ensure all data has been written to the microSD card. - ```bash - sync + sudo dd if=./2025-12-04-raspios-trixie-arm64.img of=/dev/sdX bs=4M status=progress conv=fsync ``` ## Step 2 Configure the Raspberry Pi OS Image -1. Mount the microSD card boot partition. +1. Mount the microSD card boot partition. Replace `/dev/sdX1` with the actual device name of the boot partition of your microSD card. ```bash mkdir sdcard-bootfs sudo mount /dev/sdX1 sdcard-bootfs @@ -56,14 +52,15 @@ You will need: ``` 1. If you need a static IP address for wired ethernet, edit `firmware/cmdline.txt`. ```bash - sudo vim firmware/cmdline.txt - # or sudo vim cmdline.txt if `firmware` directory does not exist - # add ip= at the end of the single line in the file, replacing with your desired static IP address + sudo vim /boot/cmdline.txt + # add " ip=" at the end of the single line in the file. ``` 1. Finally unmount the boot partition. ```bash cd .. sudo umount sdcard-bootfs + # there may also be a second mount point: + sudo umount /dev/sdX rmdir sdcard-bootfs ``` 1. NOTE: the static IP configuration above must be undone before making a production image to be used on multiple Raspberry Pis. Instead we want the image to use DHCP only and be isolated from the internet. @@ -85,6 +82,11 @@ You will need: sudo apt update sudo apt upgrade -y ``` +1. Take this opportunity to write down the Raspberry Pi's MAC address for future reference. + ```bash + ip link show eth0 + # look for "link/ether xx:xx:xx:xx:xx:xx" + ``` ## Step 4 Install and Configure usbip 1. Add the kernel modules to `/etc/modules` so they load at boot. @@ -129,3 +131,80 @@ You will need: sudo systemctl enable --now usb-remote sudo systemctl status usb-remote ``` + +## Step 6 image-backup + +1. Clone the image-backup repository. + ```bash + cd ~ + git clone https://github.com/seamusdemora/RonR-RPi-image-utils.git + ``` +1. Install image-backup. + ```bash + sudo install --mode=755 ~/RonR-RPi-image-utils/image-* /usr/local/sbin + ``` + +## Step 7 Clean Up The Settings for Production + +The master sdcard image wants to use DHCP only and be isolated from the internet so we need to undo any temporary configuration changes made earlier. This allows us to re-use the image on multiple Raspberry Pis which will each get their own IP address via DHCP. + +1. Disable the static IP address if you set one up. + ```bash + sudo apt install vim + sudo vim /boot/firmware/cmdline.txt + # remove " ip=" that you added earlier + ``` +1. Disable Wifi if you enabled it. + ```bash + sudo vim /boot/firmware/config.txt + # add the following lines at the end of the file + dtoverlay=disable-wifi + dtoverlay=disable-bt + ``` + +## Step 8 Create a Backup Image of the microSD Card + +Before backing up the image we put the SD card into read-only mode. This avoids wearing out the SD card and makes the Pi reset to a clean state on each boot. + +1. Set the SD card to read-only mode with an overlay filesystem in RAM. + ```bash + sudo raspi-config nonint enable_overlayfs + sudo raspi-config nonint enable_bootro + # DO NOT reboot until the backup image has been created + # ALSO: ignore warnings about fstab has been modifiedsyn + ``` + +1. Insert a USB stick into the Raspberry Pi to store the backup image. + +1. use `lsblk` to identify the device name of the USB stick 1st partition (e.g. `/dev/sda1`). It should already be mounted under `/media/local/xxxx`. If there is no mount then create a mount point and mount it manually: + ```bash + sudo mkdir -p /media/local/usb + sudo mount /dev/sda1 /media/local/usb + ``` + +1. Run the image-backup script to create a backup image of the microSD card to the USB stick. Replace `/media/local/xxxx` with the actual mount point of your USB stick and adjust the output file path as needed. + ```bash + sudo image-backup + # when promted for output file, use something like: + /media/local/usb/raspi-usb-remote-2025.12.20.img + # choose the defaults for the other prompts and y to confirm file creation. + ``` +1. sync and unmount the USB stick. + ```bash + sync + sudo umount /media/local/xxxx + ``` +1. You can now power off the Raspberry Pi and remove the USB stick. + ```bash + sudo poweroff + ``` + +## Conclusion + +You now have a backup image of your configured Raspberry Pi microSD card that you can use to create additional Raspberry Pi servers as needed. To restore the image to another microSD card, use the `dd` command as described in Step 1, replacing the input file with your backup image file. + +The Raspberry Pi you have just been working with is now ready to be used as a production server. Simply power it back on and it will boot into read-only mode with usb-remote installed and ready to use. + +You should configure your router or `infoblox` to assign it a DHCP reservation so it always gets the same IP address. To do this requires knowing the Raspberry Pi's MAC address. + +See [Commissioning Additional Raspberry Pi Servers](../tutorials/commissioning_raspi.md) for instructions on deploying the backup image to new Raspberry Pis. From ec66baaf040311cfd71c9c2106bdc376a83b3c80 Mon Sep 17 00:00:00 2001 From: giles knap Date: Sun, 21 Dec 2025 11:58:38 +0000 Subject: [PATCH 2/4] add raspi commissioning docs --- docs/how-to/new_raspi.md | 22 ++++++++- docs/tutorials/commissioning_raspi.md | 68 ++++++++++++++++++++++++++- 2 files changed, 86 insertions(+), 4 deletions(-) diff --git a/docs/how-to/new_raspi.md b/docs/how-to/new_raspi.md index 43cedbc..17bb3ee 100644 --- a/docs/how-to/new_raspi.md +++ b/docs/how-to/new_raspi.md @@ -168,8 +168,7 @@ Before backing up the image we put the SD card into read-only mode. This avoids 1. Set the SD card to read-only mode with an overlay filesystem in RAM. ```bash - sudo raspi-config nonint enable_overlayfs - sudo raspi-config nonint enable_bootro + sudo raspi-config nonint do_overlayfs 0 # DO NOT reboot until the backup image has been created # ALSO: ignore warnings about fstab has been modifiedsyn ``` @@ -199,6 +198,25 @@ Before backing up the image we put the SD card into read-only mode. This avoids sudo poweroff ``` +## Step 9 Prepare the Backup Image for Distribution + +The image file you have created has removed the empty space in the root partition to make it nice and small. The contents of boot partition must be updated to tell the Raspberry Pi to expand the root partition to fill the microSD card on first boot. The `pi-shrink` script does this for you. + +1. Get the `pi-shrink` script. + ```bash + curl -fsSO https://raw.githubusercontent.com/Drewsif/PiShrink/master/pishrink.sh + chmod +x pishrink.sh + ``` +1. Run `pi-shrink` on the backup image you created in Step 8. + ```bash + sudo ./pishrink.sh /path/to/your/raspi-usb-remote-2025.12.20.img + ``` + +You can ignore the warning "Filesystem already shrunk to smallest size. Skipping filesystem shrinking". + +That's it. Your img file can now be used to create additional Raspberry Pi usb-remote servers as needed. + + ## Conclusion You now have a backup image of your configured Raspberry Pi microSD card that you can use to create additional Raspberry Pi servers as needed. To restore the image to another microSD card, use the `dd` command as described in Step 1, replacing the input file with your backup image file. diff --git a/docs/tutorials/commissioning_raspi.md b/docs/tutorials/commissioning_raspi.md index 1865af6..df47768 100644 --- a/docs/tutorials/commissioning_raspi.md +++ b/docs/tutorials/commissioning_raspi.md @@ -1,3 +1,67 @@ -# Commissioning a Raspberry new Raspberry Pi Server +# Commissioning a new Raspberry Pi as a usb-remote Server -TODO: Write this page. +## Introduction + +To add remote USB device support requires a usb-remote server at the location of the USB devices. e.g. In a beamline's experimental hutch. This tutorial describes how to commission a new Raspberry Pi to act as a usb-remote server. + +In summary the steps are: +1. Purchase and assemble the recommended hardware. +1. Flash a Raspberry Pi usb-remote server image to a microSD card. +1. Extract the Raspberry Pi MAC address and label the Raspberry Pi with it. +1. Configure an IP address on the network using the MAC address. +1. Connect the Raspberry Pi to the network and power it on. +1. Verify the new server is visible to the usb-remote client. + + + +## Step 1: Obtain and Assemble Recommended Hardware + +See [Recommended Hardware](../reference/recommended_hardware.md) for the list of recommended hardware to use for a Raspberry Pi usb-remote server. + +Any Raspberry Pi 4 or 5 with at least 4GB RAM is suitable. + +TODO: some notes on assembly go here. + +## Step 2a: Obtain Raspberry Pi usb-remote Server microSD Card + +If you have a supply of pre-built Raspberry Pi usb-remote server sdcard images you can skip this step and go to [Step 2 Extract the Raspberry Pi MAC Address](#step-2-extract-the-raspberry-pi-mac-address). + +At DLS these can be obtained from TODO: where? + +## Step 2b: Flash the Raspberry Pi usb-remote Server Image + +Alternatively, flash your own card. + +1. Obtain the latest Raspberry Pi usb-remote server image. + - At DLS this is available at /dls_sw/apps TODO: add path here. + - giles' latest can be downloaded from here: https://drive.google.com/file/d/1pvCIkpnDC90Z086w6k2WjTKtdAcx9RPU/view?usp=sharing + - Alternatively, build your own image using the instructions at [Building a Custom Raspberry Pi Image](building_raspi_image.md). +1. Insert a microSD card of at least 16GB capacity into a card reader connected to your computer. +1. Use `lsblk` to identify the device name of the microSD card (e.g. `/dev/sdb`). +1. Flash the image to a microSD card. **CAREFUL** - replace `/dev/sdX` with the correct device name for your microSD card and remember that this will overwrite the specified device. + ```bash + sudo dd if=./raspberry_pi_usb-remote_server.img of=/dev/sdX bs=4M status=progress conv=fsync + ``` + +## Step 3: Extract the Raspberry Pi MAC Address + +- TODO: this will involve booting the Pi with a printer plugged in. +- Alternatively, if you have access to a monitor and keyboard you can boot the Pi and run `ip link show eth0` to get the MAC address of the `eth0` interface. + + +## Step 4: Configure an IP Address for the Raspberry Pi + +- TODO: infoblox instructions go here. + +## Step 5: Connect the Raspberry Pi to the Network and Power it On + +- Connect the Raspberry Pi to the network using a wired ethernet connection. +- Power on the Raspberry Pi using the USB-C power supply. + +## Step 6: Verify the New Server is Visible to the usb-remote Client +On any linux machine that can route to the new Raspberry Pi server IP, run: + +```bash +uvx usb-remote config add-server +uvx usb-remote list +``` From 231487d8a747329ef7096c8725c5c8a566673940 Mon Sep 17 00:00:00 2001 From: giles knap Date: Sun, 21 Dec 2025 13:31:57 +0000 Subject: [PATCH 3/4] update commissioning docs --- docs/tutorials/commissioning_raspi.md | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/docs/tutorials/commissioning_raspi.md b/docs/tutorials/commissioning_raspi.md index df47768..7fe5111 100644 --- a/docs/tutorials/commissioning_raspi.md +++ b/docs/tutorials/commissioning_raspi.md @@ -4,21 +4,12 @@ To add remote USB device support requires a usb-remote server at the location of the USB devices. e.g. In a beamline's experimental hutch. This tutorial describes how to commission a new Raspberry Pi to act as a usb-remote server. -In summary the steps are: -1. Purchase and assemble the recommended hardware. -1. Flash a Raspberry Pi usb-remote server image to a microSD card. -1. Extract the Raspberry Pi MAC address and label the Raspberry Pi with it. -1. Configure an IP address on the network using the MAC address. -1. Connect the Raspberry Pi to the network and power it on. -1. Verify the new server is visible to the usb-remote client. - - ## Step 1: Obtain and Assemble Recommended Hardware See [Recommended Hardware](../reference/recommended_hardware.md) for the list of recommended hardware to use for a Raspberry Pi usb-remote server. -Any Raspberry Pi 4 or 5 with at least 4GB RAM is suitable. +Any Raspberry Pi 4 or 5 with at least 4GB RAM and at least 16GB microSD card is suitable. TODO: some notes on assembly go here. @@ -65,3 +56,5 @@ On any linux machine that can route to the new Raspberry Pi server IP, run: uvx usb-remote config add-server uvx usb-remote list ``` + +You should see the new server listed without errors. From 4d65304358927a6477ab1610214a04764448d6ab Mon Sep 17 00:00:00 2001 From: giles knap Date: Sun, 21 Dec 2025 15:26:24 +0000 Subject: [PATCH 4/4] fix links in docs --- docs/tutorials/commissioning_raspi.md | 6 +++--- src/usb_remote/client.py | 4 +++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/tutorials/commissioning_raspi.md b/docs/tutorials/commissioning_raspi.md index 7fe5111..47180cb 100644 --- a/docs/tutorials/commissioning_raspi.md +++ b/docs/tutorials/commissioning_raspi.md @@ -15,7 +15,7 @@ TODO: some notes on assembly go here. ## Step 2a: Obtain Raspberry Pi usb-remote Server microSD Card -If you have a supply of pre-built Raspberry Pi usb-remote server sdcard images you can skip this step and go to [Step 2 Extract the Raspberry Pi MAC Address](#step-2-extract-the-raspberry-pi-mac-address). +If you have a supply of pre-built Raspberry Pi usb-remote server sdcard images you can skip this step and go to `Step 3: Extract the Raspberry Pi MAC Address`. At DLS these can be obtained from TODO: where? @@ -25,8 +25,8 @@ Alternatively, flash your own card. 1. Obtain the latest Raspberry Pi usb-remote server image. - At DLS this is available at /dls_sw/apps TODO: add path here. - - giles' latest can be downloaded from here: https://drive.google.com/file/d/1pvCIkpnDC90Z086w6k2WjTKtdAcx9RPU/view?usp=sharing - - Alternatively, build your own image using the instructions at [Building a Custom Raspberry Pi Image](building_raspi_image.md). + - giles' latest image can be downloaded from here: https://drive.google.com/file/d/1pvCIkpnDC90Z086w6k2WjTKtdAcx9RPU/view?usp=sharing + - Alternatively, build your own image using the instructions at [Create a New Raspberry Pi Boot Image From Scratch](../how-to/new_raspi.md). 1. Insert a microSD card of at least 16GB capacity into a card reader connected to your computer. 1. Use `lsblk` to identify the device name of the microSD card (e.g. `/dev/sdb`). 1. Flash the image to a microSD card. **CAREFUL** - replace `/dev/sdX` with the correct device name for your microSD card and remember that this will overwrite the specified device. diff --git a/src/usb_remote/client.py b/src/usb_remote/client.py index 0e13f8c..68bd2db 100644 --- a/src/usb_remote/client.py +++ b/src/usb_remote/client.py @@ -63,7 +63,7 @@ def send_request( sock.sendall(request.model_dump_json().encode("utf-8")) response = sock.recv(4096).decode("utf-8") - logger.debug("Received response from server") + logger.debug(f"Received response from server: {response}") # Parse response using TypeAdapter to handle union types response_adapter = TypeAdapter( ListResponse | DeviceResponse | ErrorResponse @@ -119,6 +119,8 @@ def list_devices( assert isinstance(response, ListResponse) results[server] = response.data logger.debug(f"Server {server}: {len(response.data)} devices") + except DeviceNotFoundError: + pass # expect not to find the device on all servers except Exception as e: logger.warning(f"Failed to query server {server}: {e}") results[server] = []