Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 41 additions & 10 deletions docs/src/hil/nixos-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,32 @@
Eventually, we will support fully automating the setup process. But *for now*,
one first needs to do some manual bootstrapping.

## Setting up nix flake

This step is a common way to create a nix flake for HILs.

- Choose a hostname for you HIL. The convention we use is incremental `worldcoin-hil-[location]-[number]` (see `orb-software/nix/machines`)
- Inside `nix/machines` Copy an existing HIL flake with your hostname

```sh
cp -r worldcoin-hil-munich-0 worldcoin-hil-munich-[your-number]
```

- Update `nix/machines/flake-outputs.nix` with the new hostname (make sure formatting is correct otherwise CI will fail on the PR)

```nix
...
nixosConfigurations."worldcoin-hil-munich-8" = hilConfig {
hostname = "worldcoin-hil-munich-8";
};
# Your new configuration HERE

nixosConfigurations."worldcoin-hil-munich-[your-number]" = hilConfig {
hostname = "worldcoin-hil-munich-[your-number]";
};
```

- Open a PR in `orb-software` with your changes and merge before the next steps

## Installing NixOS to a liveusb

Expand All @@ -11,8 +37,8 @@ inexplicable reason the official NixOS installer *only* exists as a MBR partitio
disk. This means we need to build our own GPT/UEFI based NixOS live usb ;(

To work around this limitation of the official installer, we provide a liveusb
image that has NixOS on it, via [disko][disko]. The easiest way to get this liveusb image
is from the CI artifacts, it is built by the [Nix CI][nix ci] job.
image that has NixOS on it, via [disko]. The easiest way to get this liveusb image
is from the CI artifacts, it is built by the [Nix CI] job.

Once you download it, unzip it, and `gzip --decompress liveusb.raw.gz` it. You will now
have a `liveusb.raw` file. Plug your flashdrive in, identity the *disk* (not partition) of
Expand All @@ -24,8 +50,8 @@ Run the following:
```bash
sudo cp liveusb.raw /dev/<your-usb-disk>
```
This loads the liveusb onto the flashdrive.

This loads the liveusb onto the flashdrive.

## Use the liveusb to install NixOS

Expand Down Expand Up @@ -71,9 +97,11 @@ there.

Run the following in the HIL. It will print a url to the console when its the first time on this
machine.

```bash
sudo tailscale up
```

Go to the URL on your laptop, log in with your tfh google account. This will connect
the device to the tailscale network. From this point forward, any other computer that
is connected to tailscale will also be able to ssh into the HIL, even without teleport.
Expand All @@ -89,13 +117,16 @@ cloudflare warp is turned off, as it can conflict with the tailscale VPN.
1. Request teleport token for a HIL in slack. You will receive a bash one-liner.

**DO NOT RUN THE BASH, THIS IS AN EXAMPLE:**

```bash
sudo bash -c "$(curl -fsSL https://teleport-cluster.orb.internal-tools.worldcoin.dev/scripts/ffffffffffffffffffffffffffffffff/install-node.sh)"
```

The command you received on slack should look like something of the above.

Instead of running the command, delete everything except the `curl` command and then
redirect that to a file called `teleport-install.sh`, for example:

```bash
curl -fsSL https://teleport-cluster.orb.internal-tools.worldcoin.dev/scripts/ffffffffffffffffffffffffffffffff/install-node.sh > teleport-install.sh

Expand All @@ -105,6 +136,7 @@ Be sure that `teleport-install.sh` is put on the HIL, you can put it in the home
for now. Again, *DO NOT RUN THIS SCRIPT*.

2. Place the following content on the HIL at `/etc/teleport.yaml`:

```yaml
version: v3
teleport:
Expand Down Expand Up @@ -136,20 +168,22 @@ proxy_service:
```

3. run the following from the same directory that `teleport-install.sh` is at on the
HIL:
HIL:

```bash
TELEPORT_TOKEN="$(cat teleport-install.sh | grep -m1 -oP "^JOIN_TOKEN='\K[^']+")" && [ -n "${TELEPORT_TOKEN}" ] || echo "error: token not found"
TELEPORT_HOSTNAME="$(hostname)"
sudo sed -i "s/SED_TOKEN/${TELEPORT_TOKEN}/" /etc/teleport.yaml
sudo sed -i "s/SED_HOSTNAME/${TELEPORT_HOSTNAME}/" /etc/teleport.yaml
````
```

This will edit the contents of `/etc/teleport.yaml` to replace the `SED_*` strings with
your hostname and the token.

You can `sudo cat /etc/teleport.yaml` and inspect the file to see the new contents.

4. Run
4. Run

```bash
sudo rm -rf /var/lib/teleport
sudo systemctl restart teleport.service && sudo journalctl -fu teleport.service
Expand All @@ -161,9 +195,6 @@ is normal. Teleport should now be set up.
You will also need to make sure your machine's hostname matches the regex in our
terraform config [here][tf hil].


[nix config]: https://github.com/TheButlah/nix
[remote build]: https://nix.dev/manual/nix/2.18/advanced-topics/distributed-builds
[disko]: https://github.com/nix-community/disko
[tf hil]: https://github.com/worldcoin/infrastructure/blob/345bc7db0c47e369ce6529d0febed9535a0970f7/teleport/orb/orb-sw-dev-tools-teleport.tf
[nix ci]: https://github.com/worldcoin/orb-software/actions/workflows/nix-ci.yaml
[tf hil]: https://github.com/worldcoin/infrastructure/blob/345bc7db0c47e369ce6529d0febed9535a0970f7/teleport/orb/orb-sw-dev-tools-teleport.tf