Goroutine leak

[Nikita2305]

Integrity requirements

• I confirm that I have read the documentation, understand the meaning of all the configuration items I wrote, and did not pile up seemingly useful options or default values.
• I provided the complete config and logs, rather than just providing the truncated parts based on my own judgment.
• I searched issues and did not find any similar issues.
• The problem can be successfully reproduced in the latest Release

Description

I have a few xray servers running with a similar load, release 25.3.6 (though it is not latest release)

Healthy server:

broken server (two snapshots with an hour-break):

This number of goroutines leads to RAM growth, and eventually to OOM. What are possible reasons for it?

dynamics of RAM:

This happens a lot on this host, it is reproducable I would say. I would be happy to cooperate to debug this issue.

Currently I don't want to upgrade version of xray to a new one, as I think i will lose reproducable case.

Reproduction Method

Run xray server: /usr/local/bin/xray run -config /usr/local/etc/xray/config.json

Wait some time

Have goroutines snap: curl -o goroutine.out http://127.0.0.1:11111/debug/pprof/goroutine

Run view server, inspect: go tool pprof -http=0.0.0.0:6061 goroutine.out

Client config

url:

vless://{{ VLESS_UUID }}@{{ IP }}:443?security=reality&sni={{ SNI }}&fp=chrome&type=tcp&flow=xtls-rprx-vision&pbk={{ PUBLIC_KEY }}&sid={{ VLESS_SHORT_ID }}#{{ CONFIG_NAME }}

Server config

{
  "log": {
    "loglevel": "warning",
    "access": "/var/log/xray/access.log",
    "error": "/var/log/xray/error.log"
  },
  "inbounds": [
    {
      "port": 443,
      "protocol": "vless",
      "settings": {
        "clients": [
          {% for client in VLESS_CLIENTS %}
          {
            "id": "{{ client.vless_uuid }}",
            "email": "{{ client.vless_uuid }}",
            "flow": "xtls-rprx-vision",
            "level": 0
          }{% if not loop.last %},{% endif %}
          {% endfor %}
        ],
        "decryption": "none"
      },
      "streamSettings": {
        "network": "tcp",
        "security": "reality",
        "realitySettings": {
          "show": true,
          "dest": "{{ SNI }}:443",
          "xver": 0,
          "serverNames": [
            "{{ SNI }}"
          ],
          "privateKey": "{{ PRIVATE_KEY }}",
          "shortIds": [
            {% for client in VLESS_CLIENTS %}
            "{{ client.vless_sid }}"{% if not loop.last %},{% endif %}
            {% endfor %}
          ]
        }
      },
      "sniffing": {
        "enabled": true,
        "destOverride": [
          "http",
          "tls",
          "quic"
        ]
      },
      "tag": "main-vless"
    },
    {
      "port": 11111,
      "protocol": "dokodemo-door",
      "tag": "metrics_in",
      "listen": "127.0.0.1",
      "settings": {
        "address": "127.0.0.1"
      }
    }
  ],
  "outbounds": [
    {
      "protocol": "freedom",
      "settings": {},
      "streamSettings": {
        "sockopt": {
          "domainStrategy": "UseIP"
        }
      }
    },
    {
      "streamSettings": {
        "realitySettings": {
          "serverName": "yahoo.com",
          "shortId": "...",
          "publicKey": "...",
          "fingerprint": "chrome",
          "spiderX": ""
        },
        "tcpSettings": {
          "header": {
            "type": "none"
          }
        },
        "security": "reality",
        "network": "tcp"
      },
      "tag": "proxy",
      "protocol": "vless",
      "settings": {
        "vnext": [
          {
            "address": "...",
            "users": [
              {
                "id": "...",
                "flow": "xtls-rprx-vision",
                "encryption": "none"
              }
            ],
            "port": 443
          }
        ]
      }
    },
    {
      "tag": "block",
      "protocol": "blackhole",
      "settings": {}
    }
  ],
  "routing": {
    "domainStrategy": "IPIfNonMatch",
    "rules": [
      {
        "type": "field",
        "domain": [
          "geosite:youtube"
        ],
        "outboundTag": "proxy"
      },
      {
        "type": "field",
        "inboundTag": [
          "metrics_in"
        ],
        "outboundTag": "metrics_out"
      },
      {
        "type": "field",
        "protocol": "bittorrent",
        "outboundTag": "block"
      }
    ]
  },
  "policy": {
    "levels": {
      "0": {
        "statsUserUplink": true,
        "statsUserDownlink": true,
        "statsUserOnline": true
      }
    },
    "system": {
      "statsInboundUplink": true,
      "statsInboundDownlink": true,
      "statsOutboundUplink": true,
      "statsOutboundDownlink": true
    }
  },
  "metrics": {
    "tag": "metrics_out"
  },
  "stats": {},
  "dns": {
    "servers": [
      "94.140.14.14",
      "94.140.15.15",
      "1.1.1.1",
      "8.8.8.8"
    ]
  }
}

Client log

Not related to the task

Server log

Access / error: more or less same on both servers.

error log on the broken server (before OOM on the screen). The reason there are errors - i suppose because I use adblock dns (I am investigating it rn).

2025/05/30 16:54:28.997317 [Warning] [2577172105] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:54:33.598939 [Warning] [94012504] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:54:33.599332 [Warning] [94012504] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:54:33.707102 [Warning] [94012504] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:54:33.905545 [Warning] [94012504] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:54:34.207593 [Warning] [94012504] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:55:09.142436 [Warning] [2224415491] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:55:09.142764 [Warning] [2224415491] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:55:09.243845 [Warning] [2224415491] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:55:09.445370 [Warning] [2224415491] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:55:09.746221 [Warning] [2224415491] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:55:14.705352 [Warning] [2555639914] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:55:14.705596 [Warning] [2555639914] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:55:14.806839 [Warning] [2555639914] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:55:15.025218 [Warning] [2555639914] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:55:15.323742 [Warning] [2555639914] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:06.817241 [Warning] [2252804929] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:06.817538 [Warning] [2252804929] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:06.918452 [Warning] [2252804929] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:07.119329 [Warning] [2252804929] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:07.420386 [Warning] [2252804929] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:09.326297 [Warning] [2483592506] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:09.326865 [Warning] [2483592506] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:09.429161 [Warning] [2483592506] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:09.629720 [Warning] [2483592506] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:09.931701 [Warning] [2483592506] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:17.327856 [Warning] [93723005] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:17.328152 [Warning] [93723005] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:17.428859 [Warning] [93723005] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:17.629584 [Warning] [93723005] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:56:17.930524 [Warning] [93723005] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:58:50.653196 [Warning] [3348310388] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:58:50.656692 [Warning] [3348310388] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:58:50.757852 [Warning] [3348310388] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:58:50.958876 [Warning] [3348310388] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 16:58:51.260014 [Warning] [3348310388] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:00:58.362155 [Warning] [377717395] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:00:58.364946 [Warning] [377717395] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:00:58.465459 [Warning] [377717395] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:00:58.666876 [Warning] [377717395] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:00:58.967878 [Warning] [377717395] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:01:02.430239 [Warning] [171466443] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:01:02.430678 [Warning] [171466443] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:01:02.532401 [Warning] [171466443] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:01:02.733760 [Warning] [171466443] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:01:03.037266 [Warning] [171466443] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:01:07.297171 [Warning] [1860331353] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:01:07.297620 [Warning] [1860331353] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:01:07.398518 [Warning] [1860331353] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:01:07.600514 [Warning] [1860331353] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:01:07.902398 [Warning] [1860331353] transport/internet: failed to resolve ip > rcode: 3
2025/05/30 17:08:11.079372 [Warning] core: Xray 25.3.6 started

OOM:
May 30 17:08:07 ${HOST} kernel: Out of memory: Killed process 1196643 (xray) total-vm:2085504kB, anon-rss:870432kB, file-rss:0kB, shmem-rss:0kB, UID:65534 pgtables:1840kB oom_score_adj:

[Fangliding]

log level should be in debug
and needs heap pprof

[Fangliding]

而且也需要在最新版本尝试(release里的reality还存在性能问题 应该使用CI中的版本) 丢失可重现case是什么理由

[Nikita2305]

and needs heap pprof

Soon I will grab debug logs and then update xray version.

[Fangliding]

it's still goroutine

[Nikita2305]

it's still goroutine

There are two images, i decided to make two snapshots at the same time

[Nikita2305]

而且也需要在最新版本尝试(release里的reality还存在性能问题 应该使用CI中的版本) 丢失可重现case是什么理由

I updated version to the latest:

Xray 25.5.16 (Xray, Penetrates Everything.) 800b8b5 (go1.24.3 linux/amd64)
A unified platform for anti-censorship.

Issue wasn't fixed:

UPD: Restart of the whole host also didn't workout