From c3047241a4990c0f4daf023b4af34411c98d00aa Mon Sep 17 00:00:00 2001 From: Krzysztof Nazarewski Date: Mon, 10 Mar 2025 17:56:02 +0100 Subject: [PATCH 1/4] improve client troubleshooting docs --- src/pages/how-to/troubleshooting-client.mdx | 395 ++++++++++++++++++-- 1 file changed, 364 insertions(+), 31 deletions(-) diff --git a/src/pages/how-to/troubleshooting-client.mdx b/src/pages/how-to/troubleshooting-client.mdx index 7b35c2633..521c070e1 100644 --- a/src/pages/how-to/troubleshooting-client.mdx +++ b/src/pages/how-to/troubleshooting-client.mdx @@ -1,13 +1,17 @@ # Troubleshooting client issues -This document offers practical tips and insights to help you debug various problems, ensuring a seamless user experience. +This document offers practical tips and insights to help you debug various problems, ensuring a seamless user +experience. ## NetBird agent status -The netbird agent is a daemon service that runs in the background; it provides information about peers connected and about the NetBird control services. You can check the status of the agent with the following command: + +The netbird agent is a daemon service that runs in the background; it provides information about peers connected and +about the NetBird control services. You can check the status of the agent with the following command: ```shell netbird status --detail ```` + This will output the following information: ```shell @@ -61,29 +65,42 @@ Quantum resistance: false Routes: - Peers count: 2/2 Connected ``` -As you can see, the output shows the peers connected, the NetBird IP address, the public key, the connection status, and the connection type. -The status will also report if there is an issue connecting to the relay servers, the management server, or the signal server. + +As you can see, the output shows the peers connected, the NetBird IP address, the public key, the connection status, and +the connection type. The status will also report if there is an issue connecting to the relay servers, +the management server, or the signal server. As for Peers, the status will show the following information: -* `Connection type`: P2P, Relayed, where relayed connections indicate a limitation in the network that prevents a direct connection between the peers. -* `Direct`: true/false, where true indicates a direct connection between the peers without a local proxy. This case is common when the local peer is allocating the relay connection. -* `ICE candidate (Local/Remote)`: relay/host, where relay indicates that the local peer is using a relay connection and host indicates that the remote peer is using a direct connection. -* `Last Wireguard handshake`: Indicating the last time the Wireguard handshake was performed. Usually, this is performed every 2 minutes, and if you don't see an update here or if the value is empty, that indicates that the connection wasn't possible yet. -* `Transfer status (received/sent)`: Indicating the amount of data received and sent by the peer. This is useful to check if the connection is being used. + +* `Connection type`: P2P, Relayed, where relayed connections indicate a limitation in the network that prevents a direct + connection between the peers. +* `Direct`: true/false, where true indicates a direct connection between the peers without a local proxy. This case is + common when the local peer is allocating the relay connection. +* `ICE candidate (Local/Remote)`: relay/host, where relay indicates that the local peer is using a relay connection and + host indicates that the remote peer is using a direct connection. +* `Last Wireguard handshake`: Indicating the last time the Wireguard handshake was performed. Usually, this is performed + every 2 minutes, and if you don't see an update here or if the value is empty, that indicates that the connection + wasn't possible yet. +* `Transfer status (received/sent)`: Indicating the amount of data received and sent by the peer. This is useful to + check if the connection is being used. See more details about the status command [here](/how-to/cli#status). ## Getting client logs -By default, client logs are located in the `/var/log/netbird/client.log` file on macOS and Linux and in the `C:\ProgramData\netbird\client.log` file on Windows. -You can analyze the logs to identify the root cause of the problem. If you need help, open a [github issue](https://github.com/netbirdio/netbird/issues/new/choose) and attach the logs. +By default, client logs are located in the `/var/log/netbird/client.log` file on macOS and Linux and in the +`C:\ProgramData\netbird\client.log` file on Windows. + +You can analyze the logs to identify the root cause of the problem. If you need help, open +a [github issue](https://github.com/netbirdio/netbird/issues/new/choose) and attach the logs. ### Debug bundle -A debug archive containing the recent logs and the status at the time of execution can be generated with the following command. +A debug archive containing the recent logs and the status at the time of execution can be generated with the following +command. -Adding the `-A` flag will anonymize the logs, removing sensitive information such as public IP addresses and domain names. -Adding the `-S` flag will add system information like network routes and interfaces +Adding the `-A` flag will anonymize the logs, removing sensitive information such as public IP addresses and domain +names. Adding the `-S` flag will add system information like network routes and interfaces ```shell netbird debug bundle -AS @@ -93,17 +110,17 @@ This will output the path of the generated file, which can be accessed with admi ### Debug for a specific time -To capture logs for a specific time period, you can use the `debug for` command. This will generate a debug bundle after the specified time has elapsed. +To capture logs for a specific time period, you can use the `debug for` command. This will generate a debug bundle after +the specified time has elapsed. ```shell netbird debug for 5m -AS ``` -To capture any issues arising during the `up` and `down` processes, this will set the log level to `TRACE` and bring netbird `up` and `down` up to a few times. +To capture any issues arising during the `up` and `down` processes, this will set the log level to `TRACE` and bring +netbird `up` and `down` up to a few times. After 5 minutes the netbird status will be restored to the previous state and the debug bundle will be generated. - - ## Enabling debug logs on agent Logs can be temporarily set using the following command. @@ -125,6 +142,7 @@ Using `netbird down` and `netbird up` will not reset the log level. To permanently set the log level, see the following sections. ### On Linux with systemd + The default systemd unit file reads a set of environment variables from the path `/etc/sysconfig/netbird`. You can add the following line to the file to enable debug logs: @@ -135,6 +153,7 @@ sudo systemctl restart netbird ``` ### On Other Linux and MacOS + ```shell sudo netbird service stop sudo netbird service uninstall @@ -143,7 +162,8 @@ sudo netbird service start ``` ### On Windows -You need to run the following commands with an elevated Powershell window. + +You need to run the following commands with an elevated PowerShell or `cmd.exe` window. ```shell netbird service stop @@ -153,6 +173,7 @@ netbird service start ``` ### On Docker + You can set the environment variable `NB_LOG_LEVEL` to `debug` to enable debug logs. ```shell @@ -161,51 +182,65 @@ docker run --rm --name PEER_NAME --hostname PEER_NAME --cap-add=NET_ADMIN --cap- ``` ### On Android -Enable the ADB in the developer menu on the Android device. + +Enable the ADB in the developer menu on the Android device. In the app set the the Trace log level setting - it is a checkbox in the advanced menu. -With the ADB tool, you can get the logs from your device. The ADB is part of the SDK platform tools pack (zip file). -You can download it from [here](https://developer.android.com/tools/releases/platform-tools). -Please extract it and run the next command in the case of Linux: +With the ADB tool, you can get the logs from your device. The ADB is part of the SDK platform tools pack (zip file). +You can download it from [here](https://developer.android.com/tools/releases/platform-tools). +Please extract it and run the next command in the case of Linux: + ```shell sudo adb logcat -v time | grep GoLog ``` ## Running the agent in foreground mode -You can run the agent in foreground mode to see the logs in the terminal. This is useful to debugging issues with the agent. + +You can run the agent in foreground mode to see the logs in the terminal. This is useful to debugging issues with the +agent. ### Linux and MacOS + ```shell sudo netbird service stop sudo netbird up -F ``` ### Windows + On Windows, the agent depends on the Wireguard's `wintun.dll` and can only be executed as a system account. -To run the agent in foreground mode, you need to use a tool called [PSExec](https://learn.microsoft.com/en-us/sysinternals/downloads/psexec). +To run the agent in foreground mode, you need to use a tool +called [PSExec](https://learn.microsoft.com/en-us/sysinternals/downloads/psexec). Once you have downloaded and extracted `psexec` open an elevated Powershell window: + ```shell netbird service stop .\PsExec64.exe -s cmd.exe /c "netbird up -F --log-level debug > c:\windows\temp\netbird.out.log 2>&1" ``` -In case you need to configure environment variables, you need to add them as system variables so they get picked up by the agent on the next psexec run: +In case you need to configure environment variables, you need to add them as system variables so they get picked up by +the agent on the next psexec run: + ```shell [Environment]::SetEnvironmentVariable("PIONS_LOG_DEBUG", "all", "Machine") ```` ## Enabling WireGuard in user space -Sometimes, you want to test NetBird running on userspace mode instead of a kernel module. That can be a check to see if there is a problem with NetBird's firewall management in kernel mode. + +Sometimes, you want to test NetBird running on userspace mode instead of a kernel module. That can be a check to see if +there is a problem with NetBird's firewall management in kernel mode. You must run the agent in foreground mode and set the environment variable `NB_WG_KERNEL_DISABLED` to `true`. + ```shell sudo netbird service stop sudo bash -c 'NB_WG_KERNEL_DISABLED=true netbird up -F' > /tmp/netbird.log ``` -## Debugging GRPC +## Debugging GRPC -The NetBird agent communicates with the Management and Signal servers using the GRPC framework. With these parameters, you can +The NetBird agent communicates with the Management and Signal servers using the GRPC framework. With these parameters, +you can set verbose logging for this service. ```shell @@ -215,8 +250,10 @@ sudo bash -c 'GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info net ## Debugging ICE connections -The Netbird agent communicates with other peers through the Interactive Connectivity Establishment (ICE) protocol described in the [RFC 8445](https://datatracker.ietf.org/doc/html/rfc8445). To debug the connection procedure, -set verbose logging for the the [Pion/ICE](https://github.com/pion/ice) library with the `PIONS_LOG_DEBUG` or `PIONS_LOG_TRACE` variable. +The Netbird agent communicates with other peers through the Interactive Connectivity Establishment (ICE) protocol +described in the [RFC 8445](https://datatracker.ietf.org/doc/html/rfc8445). To debug the connection procedure, +set verbose logging for the the [Pion/ICE](https://github.com/pion/ice) library with the `PIONS_LOG_DEBUG` or +`PIONS_LOG_TRACE` variable. ```shell {{ title: 'Environment variable' }} PIONS_LOG_DEBUG=all @@ -227,3 +264,299 @@ NB_LOG_LEVEL=debug sudo netbird service stop sudo bash -c 'PIONS_LOG_DEBUG=all NB_LOG_LEVEL=debug netbird up -F' > /tmp/netbird.log ``` + +## Debugging access to network resources + +In this section we will be presenting methodology of troubleshooting access issues involving Netbird. + +We will start by presenting a glossary of all machines and services involved. +A sub-section will describe a specific use case. +Each will start with a concise summary of usual troubleshooting steps then expand into more detailed step-by-step +guides. + +### Glossary + +We will be using the following names for resources outside the Netbird network: + +- `int-net1`: an internal network `10.123.45.0/24`, +- `srv-c`: an internal HTTP server running at `10.123.45.17`, +- `int-dns1`: an internal DNS server running at `10.123.45.6`, +- `int-dns2`: an internal DNS server nunning at `10.7.8.9`, +- `cf-dns`: an Internet-accessible CloudFlare DNS server at `1.1.1.1` and `1.0.0.1`, + +and following Netbird network resources: + +- `peer-a`: end user's device running Netbird Client, +- `peer-b`: a linux server inside the internal network running Netbird Client, + - it has direct access to the whole `int-net1` IP range, +- `users:employees`: a Netbird Group containing `peer-a`, +- `routers:int-net1`: a Netbird Group containing `peer-b`, +- `access:srv-c`: a Netbird Groups used as a target of ACL rules for `srv-c` only, +- `access:int-net1`: a Netbird Groups used as a target of ACL rules for the whole subnet, +- `net-a`: a Netbird Network + - `net-a:srv-c`: a Network Resource handling traffic to `10.123.45.17/32` (`srv-c`), + - `net-a:int-net1`: a Network Resource handling traffic to `10.123.45.0/24` (`int-net1`), +- `route:int-net1`: a Netbird Network Route handling traffic to `10.123.45.0/24` (`int-net1`), +- `route:srv-c`: a Netbird Network Route handling traffic to `10.123.45.17/32` (`srv-c`), + +### Access from `peer-a` to `srv-c` + +In short: + +1. Does `peer-b` have direct access to `srv-c`'s port `80`? +2. Are Netbird's network routing resources configured? +3. Do Netbird's Access Control rules allow access from `peer-a` to `peer-b`? +4. Do Netbird's Access Control rules allow access from `peer-a` to the target's ACL Group? + +#### Does `peer-b` have direct access to `srv-c`'s port `80`? + +After logging in to `peer-b` you can confirm/troubleshoot the HTTP port `80` connection by issuing any of the following +commands: + +```shell +curl -v "http://10.123.45.17" +curl --fail -v --max-time=2 "http://10.123.45.17:80" +wget -O - --timeout=2 "http://10.123.45.17:80" +nc -nvz -w 2 10.123.45.17 80 +``` + +You can also try `ping` (an ICMP packet), but the firewall might have a different configuration for ICMP and TCP ports: + +```shell +ping --numeric --count=1 --timeout=2 10.123.45.17 +``` + +#### Are Netbird's network routing resources configured? + +For Netbird network routing resources configurations you can use either (new) _Networks_ or (old) _Network Routes_. + +A Network `net-a` should have at minimum: + +- _Network Resource_: `net-a:srv-c` with either of: + - an _Address_ set to `10.123.45.17/32` to configure route to `srv-c` exclusively and nothing else, + - _Assigned Groups_ set to `access:srv-c` +- _Routing Peer Group_ assigned to `routers:int-net1` + +A _Network Route_ `route:srv-c` should have at least: + +- a _Network Range_ set to `10.123.45.17/32` to configure route to `srv-c` exclusively and nothing else, +- _Routing Peer Group_ assigned to `routers:int-net1`, +- _Distribution Group_ assigned to `users:employees`, +- (optional) _Access Control Groups_ assigned to `access:srv-c`, + +You can loosen the rules and replace following to grant access to the whole `int-net1` network range: + +- _Address_: `10.123.45.17/32` -> `10.123.45.0/24`, +- _Assigned Groups_ / _Access Control Groups_: `access:srv-c` -> `access:int-net1` + +#### Do Netbird's Access Control rules allow access from `peer-a` to `peer-b`? + +There should be an _Access Control Policy_ present allowing traffic from `users:employees` Group to `routers:int-net1` +Group. + +You can confirm the _Policy_ is working by: + +1. logging in to `peer-a`, +2. issuing `netbird status -d` command, +3. finding `peer-b.netbird.cloud` under `Peers detail`, + +In the most specific setup it should have at: + +- have `TCP` protocol selected, +- a blue arrow should point from left to right and a second right-to-left arrow should be greyed out, +- a _Source group_ set to `users:employees`, +- a _Destination group_ set to `routers:int-net1`, +- have `80` in the Ports section, + +You can loosen above example by: + +- allowing `ALL` protocol, _Ports_ will become greyed out because all traffic will be allowed, +- creating a bidirectional rule (both arrows should be green), always true for the protocol `ALL`, +- selecting a different source group from the pool assigned to `peer-a`, + - it could be built-in `All` group, but it is discouraged, +- selecting a different destination group from the pool assigned to `peer-b`, + - it could be built-in `All` group, but it is discouraged, + +#### Do Netbird's Access Control rules allow access from `peer-a` to the target's ACL Group? + +You can skip this check, when you are using (old) Network Route feature without filling in _Access Control Groups ( +optional)_ section. + +Otherwise, there should be an _Access Control Policy_ present allowing traffic from one of `peer-a`'s Groups to: + +- _Networks Resource_'s _Assigned Groups_: `access:srv-c` or `access:int-net1`, +- _Network Route_'s _Access Control Groups_: `access:srv-c` or `access:int-net1`, + +You can confirm the _Policy_ is working by: + +1. logging in to `peer-a`, +2. issuing `netbird status -d` command, +3. finding `peer-b.netbird.cloud` under `Peers detail`, +4. finding `10.123.45.0/24` or `10.123.45.17/32` under `peer-b.netbird.cloud`'s _Networks_ field, + +In the most specific setup it should have at: + +- have `TCP` protocol selected, +- a blue arrow should point from left to right and a second right-to-left arrow should be greyed out, +- a _Source group_ set to `users:employees`, +- a _Destination group_ set to `access:srv-c`, +- have `80` in the Ports section, + +Just like with the previous section you can loosen the above example by: + +- replacing `access:srv-c` _Group_ with `access:int-net1` _Group_, +- allowing `ALL` protocol, _Ports_ will become greyed out because all traffic will be allowed, +- creating a bidirectional rule (both arrows should be green), always true for the protocol `ALL`, +- selecting a different source group from the pool assigned to `peer-a`, + - it could be built-in `All` group, but it is discouraged, +- selecting a different destination group from the pool assigned to `peer-b`, + - it could be built-in `All` group, but it is discouraged, + +### Public nameservers + +When you configure a _Nameserver_ accessible from the Internet without a VPN, the Netbird client acts as a proxy +to the public nameserver. + +There are really just two things you can check: + +1. Confirm Netbird client picked up the nameserver, +2. Confirm the operating system is configured to use Netbird client's proxy nameserver, + +You can check the first one in operating system independent manner by: + +1. running `netbird status -d`, +2. locating the _Nameserver_'s IP address +3. confirming it _is Available_ (it could also be timed out or in other state) + +``` +... +Nameservers: + [1.1.1.1:53, 1.0.0.1:53] for [.] is Available +... +``` + +#### Confirming the DNS names resolve properly in practice + +Here is a short summary of commands querying nameservers for `name.at.example.com` in different operating systems. +The `.` at the end makes sure you are querying a fully-qualified names independent of your local network's configuration +(specifically search domains): + +```shell +# MacOS +dscacheutil -q host -a name name.at.example.com. +# Windows PowerShell +Resolve-DnsName -Name name.at.example.com. +# Linux/UNIX +dig name.at.example.com. +nslookup name.at.example.com. +# Linux with systemd-resolved +resolvectl query name.at.example.com. + +``` + +#### Confirming the nameservers are properly registered in Windows operating system + +To confirm the nameservers are properly registered in Windows operating system using PowerShell: + +```shell +PS C:\Users\user> Get-DnsClientNrptRule +Name : NetBird-Match +Version : 2 +Namespace : {.netbird.cloud, .83.100.in-addr.arpa} +... +NameServers : 100.83.255.254 +... +PS C:\Users\kdn> Get-DnsClientNrptPolicy + + +Namespace : .83.100.in-addr.arpa +... +NameServers : 100.83.255.254 +.. + +Namespace : .netbird.cloud +... +NameServers : 100.83.255.254 +... +``` + +You should be searching for following in the outputs of above commands: + +- the `100.XXX.255.254` under _Nameservers_ (a local proxy address of the Netbird client) +- `.netbird.cloud` and `.XXX.100.in-addr.arpa` under matching _Namespace_ for built-in entries, +- `.your.custom.domain.example.com` under matching _Namespace_ for your custom domains, + +#### Confirming the nameservers are properly registered in MacOS operating system + +To confirm the nameservers are properly registered in MacOS operating system using terminal: + +```shell +> scutil --dns +... +resolver #2 + domain : netbird.cloud + nameserver[0] : 100.83.255.254 + port : 53 + flags : Supplemental, Request A records, Request AAAA records + reach : 0x00000002 (Reachable) + order : 101200 +... +resolver #8 + domain : 83.100.in-addr.arpa + nameserver[0] : 100.83.255.254 + port : 53 + flags : Supplemental, Request A records, Request AAAA records + reach : 0x00000002 (Reachable) + order : 102402 +... +``` + +You should be searching for following in the outputs of above commands: + +- the `100.XXX.255.254` under _nameserver[N]_ (a local proxy address of the Netbird client) +- `netbird.cloud` and `.XXX.100.in-addr.arpa` under matching _domain_ for built-in entries, +- `.your.custom.domain.example.com` under matching _domain_ for your custom domains, +- `Reachable` under `reach` field, + +#### Confirming the nameservers are properly registered in Linux operating system + +Nameserver can be configured in different ways depending on your specific distribution's configuration: + +For `systemd-resolved`, you can see the config with `resolvectl status`, + +For other configuration backends, you should see additional entries in `/etc/resolv.conf`: + +- `127.0.0.1` - default address for Netbird DNS proxy listener +- `127.0.0.153` - fallback address for Netbird DNS proxy listener +- value of `$NB_DNS_RESOLVER_ADDRESS` - a custom override for the Netbird DNS proxy listener + +You can find the address Netbird client is listening by issuing one of following commands: + +```shell +sudo ss -nlptu 'sport = 53' | grep netbird +sudo netstat -ltnup | grep ':53' | grep netbird +``` + +### Internal nameservers + +When you configure an internal _Nameserver_, not accessible from the Internet in addition to steps +described in the previous section _Public nameservers_ +you should make sure the _Nameserver_'s IP addresses are properly routed and accessible. + +Please refer to _Access from `peer-a` to `srv-c`_ section above. + +To configure `int-dns1`, while following _Access from `peer-a` to `srv-c`_ section you should: + +- substitute port `80` for port `53` +- substitute ip address `10.123.45.17` for `10.123.45.6`, + +To configure `int-dns2`, while following _Access from `peer-a` to `srv-c`_ section you should: + +- substitute port `80` for port `53` +- completely ignore the `10.123.45.0/24` network instructions, +- substitute ip address `10.123.45.17` for `10.7.8.9`, +- create a respective _Network_ (along with _Resources_ and _Routing Peers_) or _Network Route_ for the `10.7.8.9/32` IP + address range, + +To test the configuration in practice please refer to previous section _Public nameservers_. +usb-storage,drive=raw-30F4989D-AE95B60F797D,serial=30F4989D-AE95B60F797D \ No newline at end of file From c96042ac05107fda78d9ea7820e6ed709b72bf49 Mon Sep 17 00:00:00 2001 From: Krzysztof Nazarewski Date: Wed, 12 Mar 2025 20:57:44 +0100 Subject: [PATCH 2/4] feat: add information on verifying system's routing rules --- src/pages/how-to/troubleshooting-client.mdx | 99 +++++++++++++++++++++ 1 file changed, 99 insertions(+) diff --git a/src/pages/how-to/troubleshooting-client.mdx b/src/pages/how-to/troubleshooting-client.mdx index 521c070e1..8406d5478 100644 --- a/src/pages/how-to/troubleshooting-client.mdx +++ b/src/pages/how-to/troubleshooting-client.mdx @@ -307,6 +307,7 @@ In short: 2. Are Netbird's network routing resources configured? 3. Do Netbird's Access Control rules allow access from `peer-a` to `peer-b`? 4. Do Netbird's Access Control rules allow access from `peer-a` to the target's ACL Group? +5. Is `peer-a`'s operating system configured to use the route? #### Does `peer-b` have direct access to `srv-c`'s port `80`? @@ -412,6 +413,104 @@ Just like with the previous section you can loosen the above example by: - selecting a different destination group from the pool assigned to `peer-b`, - it could be built-in `All` group, but it is discouraged, +#### Is `peer-a`'s operating system configured to use the route? + +After all resources are configured in the Netbird management you should check whether they are +properly registered with your operating system. + +You can start by checking Netbird client's configuration with `netbird status -d` command: + +```shell +% netbird status -d +Peers detail: + brys-vm-nbt-ubuntu-isolated-01.netbird.cloud: +... + Status: Connected + -- detail -- + Connection type: P2P +... + Networks: 10.123.45.0/24 +... +Peers count: 1/1 Connected +``` + +You should be primarily looking for _Networks_ section under each _Peers detail_, but you can also check: + +- _Peer_'s name, +- _Peer_'s _Status_: it should be `Connected`, +- _Peer_'s _Connection type_: it can be either `P2P` (direct) or `Relayed` (over the Internet), +- _Peers count_ near the end of the output, + +##### Verifying routing configuration on the Windows operating system + +Below commands assume running a PowerShell prompt with administrator's privileges. + +The easiest way is to read output of `Get-NetRoute` command: + +```shell +PS C:\Users\user> Get-NetRoute + +ifIndex DestinationPrefix NextHop RouteMetric ifMetric PolicyStore +------- ----------------- ------- ----------- -------- ----------- +... +17 10.123.45.255/32 0.0.0.0 256 5 ActiveStore +17 10.123.45.0/24 0.0.0.0 1 5 ActiveStore +... +17 100.83.255.255/32 0.0.0.0 256 5 ActiveStore +17 100.83.183.133/32 0.0.0.0 256 5 ActiveStore +17 100.83.0.0/16 0.0.0.0 256 5 ActiveStore +... +``` + +You should be looking for your specific subnet's IP ranges (`10.123.45.0/24` in case of `int-net1`) and anything from +`100.*.0.0/16` range. + +Some other alternatives are `route print` & `Get-NetIPConfiguration`. + +##### Verifying routing configuration on the MacOS operating system + +The easiest way to verify system configuration is `netstat -nr` command: + +```shell +% netstat -nr + +Routing tables + +Internet: +Destination Gateway Flags Netif Expire +... +100.83/16 utun100 USc utun100 +100.83.19.63 100.83.19.63 UH utun100 +... +10.123.45 utun100 USc utun100 +... + +Internet6: +Destination Gateway Flags Netif Expire +... +``` + +You should be looking for `utun*` interface in 4th column and searching the rows for +your specific subnet's clamped IP ranges (`10.123.45` in case of `int-net1`) and anything from `100.*/16` range. + +##### Verifying routing configuration on the Linux operating system + +Depending on specifics of your Linux distribution (or even your configuration of it) you should be able to use either +`iproute2` or `net-tools` family of network commands. + +Netbird client stores it's custom routes in the routing table `7120` (or `0x1BD0`) when it's available (through +`iproute2` interface). + +For `iproute2` (`ip`, `ss` tools): + +- `ip route` to find built-in `100.*.0.0/16` route, +- `ip route show table 7120` or `ip route show table all` to find the specific routed networks, + +For `net-tools` (`ifconfig`, `route`, `netstat` tools): + +- `route -n` to find built-in `100.*.0.0/16` route, +- neither `route` nor `netstat` support viewing content of custom routing tables, + ### Public nameservers When you configure a _Nameserver_ accessible from the Internet without a VPN, the Netbird client acts as a proxy From 86928a67d027a0b660b563fb4e5759b313a2b68a Mon Sep 17 00:00:00 2001 From: Krzysztof Nazarewski Date: Thu, 13 Mar 2025 12:15:22 +0100 Subject: [PATCH 3/4] chore: unify terms used --- src/pages/how-to/troubleshooting-client.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/pages/how-to/troubleshooting-client.mdx b/src/pages/how-to/troubleshooting-client.mdx index 8406d5478..bbf869d52 100644 --- a/src/pages/how-to/troubleshooting-client.mdx +++ b/src/pages/how-to/troubleshooting-client.mdx @@ -534,7 +534,7 @@ Nameservers: ... ``` -#### Confirming the DNS names resolve properly in practice +#### Verifying the DNS names resolve properly in practice Here is a short summary of commands querying nameservers for `name.at.example.com` in different operating systems. The `.` at the end makes sure you are querying a fully-qualified names independent of your local network's configuration @@ -553,7 +553,7 @@ resolvectl query name.at.example.com. ``` -#### Confirming the nameservers are properly registered in Windows operating system +#### Verifying the nameservers are properly registered in Windows operating system To confirm the nameservers are properly registered in Windows operating system using PowerShell: @@ -585,7 +585,7 @@ You should be searching for following in the outputs of above commands: - `.netbird.cloud` and `.XXX.100.in-addr.arpa` under matching _Namespace_ for built-in entries, - `.your.custom.domain.example.com` under matching _Namespace_ for your custom domains, -#### Confirming the nameservers are properly registered in MacOS operating system +#### Verifying the nameservers are properly registered in MacOS operating system To confirm the nameservers are properly registered in MacOS operating system using terminal: @@ -617,7 +617,7 @@ You should be searching for following in the outputs of above commands: - `.your.custom.domain.example.com` under matching _domain_ for your custom domains, - `Reachable` under `reach` field, -#### Confirming the nameservers are properly registered in Linux operating system +#### Verifying the nameservers are properly registered in Linux operating system Nameserver can be configured in different ways depending on your specific distribution's configuration: From 01094306a17b5fcc6d25f5c3cdb9639a560fe4ca Mon Sep 17 00:00:00 2001 From: Krzysztof Nazarewski Date: Thu, 13 Mar 2025 15:31:48 +0100 Subject: [PATCH 4/4] feat: add Can a routing peer `peer-b` forward traffic to `srv-c`? --- src/pages/how-to/troubleshooting-client.mdx | 32 ++++++++++++++++++--- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/src/pages/how-to/troubleshooting-client.mdx b/src/pages/how-to/troubleshooting-client.mdx index bbf869d52..164406c66 100644 --- a/src/pages/how-to/troubleshooting-client.mdx +++ b/src/pages/how-to/troubleshooting-client.mdx @@ -304,10 +304,11 @@ and following Netbird network resources: In short: 1. Does `peer-b` have direct access to `srv-c`'s port `80`? -2. Are Netbird's network routing resources configured? -3. Do Netbird's Access Control rules allow access from `peer-a` to `peer-b`? -4. Do Netbird's Access Control rules allow access from `peer-a` to the target's ACL Group? -5. Is `peer-a`'s operating system configured to use the route? +2. Can a routing peer `peer-b` forward traffic to `srv-c`? +3. Are Netbird's network routing resources configured? +4. Do Netbird's Access Control rules allow access from `peer-a` to `peer-b`? +5. Do Netbird's Access Control rules allow access from `peer-a` to the target's ACL Group? +6. Is `peer-a`'s operating system configured to use the route? #### Does `peer-b` have direct access to `srv-c`'s port `80`? @@ -327,6 +328,29 @@ You can also try `ping` (an ICMP packet), but the firewall might have a differen ping --numeric --count=1 --timeout=2 10.123.45.17 ``` +#### Can a routing peer `peer-b` forward traffic to `srv-c`? + + + Only Linux OS machines can be assigned as routing peers. + + +This is more complicated to test, but usually boils down to confirming `net.ipv4.ip_forward` is set to `1` on `peer-b`'s +Linux operating system: + +```shell +> sysctl net.ipv4.ip_forward +net.ipv4.ip_forward = 1 +``` + +For setting up the value persistently (across reboots) please consult your operating system's documentation. +It is often handled by either `/etc/sysctl.conf` or `/etc/sysctl.d/*.conf` files. + +Testing the functionality in practice involves: + +- connecting to another machine with direct access to `peer-b`, +- adding a routing table entry to route `int-net1` (`10.123.45.0/24`) traffic through it, +- trying to at least `ping 10.123.45.17` (`srv-c`) + #### Are Netbird's network routing resources configured? For Netbird network routing resources configurations you can use either (new) _Networks_ or (old) _Network Routes_.