LZ4 native module warning pollutes logs on Apple Silicon Macs

[robertcoopercode]

Description

When using @databricks/sql on Apple Silicon (ARM64) Macs, the library logs a verbose error/warning message about the LZ4 native module failing to load at startup. While the library gracefully falls back to a JavaScript implementation and continues to work, this warning is logged every time the application starts, polluting development logs.

Environment

• OS: macOS (Apple Silicon / ARM64)
• Node.js: v22.x
• @databricks/sql version: 1.12.0
• lz4 version: 0.6.5 (transitive dependency)

Error Message

LZ4 native module failed to load: Architecture or version mismatch Error: dlopen(/path/to/project/node_modules/.pnpm/lz4@0.6.5/node_modules/lz4/build/Release/xxhash.node, 0x0001): tried: '/path/to/project/node_modules/.pnpm/lz4@0.6.5/node_modules/lz4/build/Release/xxhash.node' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64e.v1' or 'arm64' or 'arm64'))
    at Object.Module._extensions..node (node:internal/modules/cjs/loader:1473:18)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1023:12)
    at Module.require (node:internal/modules/cjs/loader:1235:19)
    at require (node:internal/modules/helpers:176:18)
    at Object.<anonymous> (node_modules/.pnpm/lz4@0.6.5/node_modules/lz4/lib/utils.js:4:11)
    ...
    at tryLoadLZ4Module (node_modules/.pnpm/@databricks+sql@1.12.0/node_modules/@databricks/sql/lib/utils/lz4.ts:7:12)
    at Object.<anonymous> (node_modules/.pnpm/@databricks+sql@1.12.0/node_modules/@databricks/sql/lib/utils/lz4.ts:29:16)
    ...
{
  code: 'ERR_DLOPEN_FAILED'
}

Root Cause

The lz4 package (v0.6.5) does not provide prebuilt binaries for Apple Silicon, and the fallback compilation appears to produce x86_64 binaries instead of ARM64. The @databricks/sql library catches this error and falls back gracefully, but logs the full error with stack trace.

Expected Behavior

When the native LZ4 module fails to load and the library successfully falls back to a JavaScript implementation, this should either:

1. Not log anything (silent fallback)
2. Log a single-line debug/info message without the full stack trace (e.g., "LZ4 native module unavailable, using JavaScript fallback")
3. Only log at a debug level that can be filtered out

Actual Behavior

A full error with stack trace is logged to the console on every application startup, even though the library continues to function correctly.

Suggested Fix

In lib/utils/lz4.ts, consider:

• Reducing the log level of this message
• Removing the stack trace from the logged output
• Making the warning conditional on a debug/verbose flag

Workaround

Currently none - the warning is always logged when the native module fails to load.

---

Also note that another user mentioned a similar issue in a comment mentioning that this project should maybe not rely on an archived project.

[samikshya-db]

We have a contribution that solves the logging issue temporarily on this : #316. After this change, If we disable compression - you will never get the log (Note : compression is enabled by default on this driver).
So, once the above PR is merged and released - we should be good.

I also agree to the fact that the lz4 library needs to be replaced. We will work on that.