New Node.js v18 fetch() HTTP API is not supported

[pimterry]

I'm opening this here to start discussion and see if there's a workaround. I imagine this might be an issue for Node.js itself, but it's worth considering if we can do anything in userland or what what exactly Node would need to expose.

Repro:

➜ export GLOBAL_AGENT_HTTP_PROXY=http://127.0.0.1:9000
➜ node
Welcome to Node.js v18.0.0.
Type ".help" for more information.
> require('global-agent').bootstrap();
true
> require('http').get('http://example.org') // Goes via proxy
...
> fetch('http://example.org') // Ignores proxy and always gets sent directly
...

Notably it is possible to configure Undici to always send requests via a proxy (using its setGlobalDispatcher method) but that doesn't seem to be accessible in Node anywhere.

Any ideas?

[pimterry]

I don't have a good general fix for global-agent, but just for reference the official-ish way to set a proxy for global fetch in node is to install a separate Undici dependency, and then use that API to set the global proxy agent, which will then be used by Node's internal Undici dependency too.

Something like this:

const Undici = requuire('undici');
const ProxyAgent = Undici.ProxyAgent;
const setGlobalDispatcher = Undici.setGlobalDispatcher;

setGlobalDispatcher(
    new ProxyAgent(process.env.HTTP_PROXY)
);

After that, global.fetch should automatically use your proxy settings. This is annoying as a fix since you need an external dep to be able to configure an internal API, but it's a usable workaround.

[aksjer]

Any ideas ?

[pimterry]

If you'd like to push this forward, the fundamental issue is with Node, here: nodejs/node#43187. Unless Node offers an API, exposing the functionality to do this, global-agent can't configure the proxy globally without adding a dependency on the whole of Undici (a large & heavy package to include for just this, especially since it's already included within Node.js itself already).

That said, Undici have said they're open to adding built-in support for HTTP_PROXY variables in the library itself: nodejs/undici#1650. That would solve this (you wouldn't need global-agent for fetch requests) although there's no obvious movement on that yet, and it's not totally clear when/if that would be enabled by default in Node itself once it's available (it would definitely be a significant breaking change).

[aksjer]

https://nextjs.org/docs/pages/building-your-application/configuring/custom-server
Next.js 14 custom server:

const { setGlobalDispatcher, ProxyAgent } = require('undici');
const { createServer } = require('http');
const { parse } = require('url');
const next = require('next');

setGlobalDispatcher(new ProxyAgent('http://127.0.0.1:9000'));

const hostname = 'localhost';
const port = 3000;
const app = next({ dev: process.env.NODE_ENV === 'development', hostname, port });
const handle = app.getRequestHandler();

app.prepare().then(() => {
  createServer(async (req, res) => {
    try {
      const parsedUrl = parse(req.url, true);
      await handle(req, res, parsedUrl);
    } catch (err) {
      console.error('Error occurred handling', req.url, err);
      res.statusCode = 500;
      res.end('internal server error');
    }
  })
    .once('error', (err) => {
      console.error(err);
      process.exit(1);
    })
    .listen(port, () => {
      console.log(`> Ready on http://${hostname}:${port}`);
    });
});

It works for prefetch (getServerSideProps) but not for rewrites of next.config.js which do not go through the proxy