`ai-database` embed built nextjs/payload app into npm package

[nathanclevenger]

Embedding a Payload CMS Next.js App into an NPM Package

Embedding a Payload CMS app (built on Next.js) into an npm package means packaging the entire application (backend + frontend) so that it can be installed and run standalone with minimal setup. In this approach, the package will include a pre-built Next.js application (including Payload’s admin panel) and a startup script. The goal is for users to install the package (e.g. via npm install or npx) and run the app immediately, with a self-contained default configuration (such as using SQLite for the database if no DATABASE_URI is provided). This is similar to how some self-hosted Node.js CMS tools distribute themselves – for example, Ghost defaults to SQLite as the database backend out-of-the-box for a quick start . Below we outline how to structure the project, bundle it, configure defaults, and avoid common pitfalls when creating such an embedded package.

Building and Bundling the Payload + Next.js Application

Before packaging, you need to build the Payload CMS app for production. Payload 3.x runs fully within Next.js, so the standard Next.js build process produces everything needed . Key steps include:

• Production Build: Run next build (or the equivalent npm script) to generate a production build of the app. This will create a .next directory containing the compiled server code and static assets (the Payload admin panel is just a React app served by Next) . If your project uses TypeScript or custom server code, ensure it’s compiled (often Next.js handles this, but if you have separate scripts you may need a dist/ folder for compiled code as well). After building, you should have all the files needed to run the app in production, without requiring users to rebuild.
• Include All Necessary Assets: Make sure the package will include the build output. At minimum, a Next.js app needs the .next folder, your package.json, and the production dependencies to run . The Next.js build outputs the admin panel as static files (under .next/static) and server handler code. If you created the Payload app with older versions (where the admin was built to a separate build folder and server code to dist), include those as well. In short, do not .gitignore or exclude the build output when publishing the package – these compiled assets must be shipped with the package so the user doesn’t need to compile anything.
• Optional: Standalone Build: Next.js offers a “standalone” build mode which packages the app into a self-contained directory (including the necessary Node.js files). You can enable this by adding { output: 'standalone' } in next.config.js and then running next build. This produces a .next/standalone directory with a server.js file and a minimal set of files needed to run the app . In this mode, Next.js will bundle your app’s Node modules into the standalone folder for deployment. If you use this, be sure to also include the .next/static folder (which contains the admin UI assets) alongside the standalone files . Using the standalone output can simplify deployment since it creates a self-contained server bundle (the Next build will generate a server.js that you can run directly) . However, even if you don’t use standalone mode, you can still package the app by including the .next folder and listing your dependencies in package.json (the user’s installation will provide the needed Node modules).
• Verify the Build Artifacts: After building, confirm that the admin UI and server are included in the output. For example, you should have the Payload admin panel’s frontend (likely in .next/static or similar) and the Next.js server code. Payload’s docs note that to run the CMS, you need both the server code and the built admin panel present . In our case, those are packaged together. Double-check that no critical file is missed (e.g. public assets, Next config, etc.). A common best practice is to create a files list in package.json or use an .npmignore to explicitly include the build output and exclude source or unnecessary files, resulting in a lean package that still has everything needed to run.

Structuring the NPM Package and Start Script

Once the app is built, structure your npm package to include the build and provide a convenient startup command:

• Package Layout: You can include the build output at the root of the package or in a subdirectory. For simplicity, many projects keep the compiled output in the root (e.g. the .next folder at root alongside package.json). Ensure that your package.json (for the package) has proper dependencies listed for anything the app needs at runtime (e.g. next, react, payload, database adapter, etc.). When a user installs your package, these dependencies will be installed, so the compiled code can require them. (If you used the Next.js standalone output which already bundles Node modules, you might not need to list all of them, but it’s safest to have them as dependencies to avoid any missing module issues.)
• Bin Script: Define a command in your package.json using the "bin" field. For example, you might add:

"bin": {
  "mypayloadapp": "./start.js"
}

• where start.js is a script included in your package that starts the server. This allows users to run your app by typing npx mypayloadapp or using the command after global install. In start.js, use a Node.js shebang (#!/usr/bin/env node) and then launch the Next.js server. There are a couple of ways to do this:
  • If you did not use standalone output: you can programmatically start Next.js. One simple method is to spawn the Next CLI: e.g. require('child_process').execSync("npm run start", {stdio: 'inherit'}); assuming your package.json has a start script that calls next start. Alternatively, use Next’s Node API to start the server on a given port. Ensure process.env.NODE_ENV is set to "production" so that it runs the production build.
  • If you did use the standalone build: simply require or execute the generated server.js. For example:

#!/usr/bin/env node
require('./.next/standalone/server.js');

• • The standalone server.js will start the Next.js server (listeners, etc.) as soon as it’s run . (In a Docker example, the app is started with node server.js in the standalone directory – your bin script essentially does the same in the npm context.)
• Working Directory and Assets: One pitfall to watch out for is that the Next.js server might expect to be run from the project root (because it looks for the .next folder relative to the current working directory). If your bin script is invoked from an arbitrary directory (e.g. the user’s working directory), it may not find the build. A solution is to use process.chdir(__dirname) in your start script to switch into the package’s directory (where the build files are). Then calling node .next/standalone/server.js or next start will operate in the correct folder. Alternatively, construct absolute paths to the build files using __dirname. Ensuring the script runs the server in the right location will make the startup reliable.
• Port and Host: By default, Next will start on port 3000 and localhost. You can allow configuration of the port by respecting the PORT environment variable or adding an option. For example, Next’s start script looks at process.env.PORT or a -p flag. In a packaged app, you can document that users should set PORT if they want a custom port. In the Docker deployment example for Payload, they simply expose port 3000 and run with PORT=3000 by default , so 3000 is a safe default.
• Testing the Package: Before publishing, test the flow by doing npm pack (which creates a local tarball of your package) and then installing that tarball in a fresh environment or Docker container. Verify that running the bin command starts the app and that you can access the Payload admin UI at http://localhost:3000/admin. This helps catch any missing files or dependency issues. According to one Next.js discussion, the only files strictly needed to run a built app are the production .next folder, package.json, and node_modules with deps – try to simulate that scenario to ensure your package includes everything required.

Database Configuration and SQLite Fallback

To make the app self-contained, configure the Payload CMS to use SQLite by default when no external database is specified. Payload supports multiple database adapters (MongoDB, Postgres, and SQLite) . For SQLite support, you’ll need to install the official SQLite adapter (@payloadcms/db-sqlite) and use it in your Payload config .

Implementing the SQLite default:

• Payload Config: In your payload.config (or wherever you initialize Payload), set up the database adapter to switch based on an environment variable. For example, you can do something like:

import { sqliteAdapter } from '@payloadcms/db-sqlite';
import { mongooseAdapter } from 'payload/mongoose'; // if supporting Mongo

export default buildConfig({
  // ... other config ...
  db: process.env.DATABASE_URI 
       ? mongooseAdapter({ url: process.env.DATABASE_URI }) 
       : sqliteAdapter({ 
           client: { url: 'file:./payload.db' }  // SQLite file path
         }),
  // ...
});

• In this pseudocode, if DATABASE_URI is provided, it assumes a Mongo connection string and uses the Mongoose adapter; otherwise it falls back to SQLite with a file database. You could also support Postgres via a different adapter if needed. The key is that the sqliteAdapter takes a file path connection URL (prefixed with file: scheme) pointing to a SQLite database file. In an npm package scenario, using a relative path like 'file:./payload.db' will create the DB in the current working directory (which, if your start script sets the working dir to the package, would create the DB file in the package folder). You might prefer to store data in a known subdirectory or allow configuring the path via another env var. For simplicity, using the project root or working directory is fine for default behavior. The Payload SQLite adapter usage in code is straightforward – you pass it the client.url option from an env var , so adding a fallback with || 'file:./mydb.sqlite' in that env var is the easiest way to default to a file.
• Default Environment: You can ship a default .env.example with DATABASE_URI= commented out or pointed to a SQLite file to illustrate usage. For instance, the Payload docs suggest a connection string like DATABASE_URI=file:./your-database.db for SQLite in the env file . If no DATABASE_URI is provided, your config’s fallback will use SQLite automatically. This means a user who just installs and runs your package doesn’t need any env configuration to get started – the app will spin up using an on-disk SQLite database by default.
• Ensure Adapter is Installed: Remember to list @payloadcms/db-sqlite as a dependency and include it in your build. The create-payload-app CLI might have already set this up if you chose SQLite during setup . If not, install it and adjust the config as above. In your packaged app’s startup, also ensure that any required database initialization runs. Payload (with Drizzle ORM for SQLite) can auto-generate the database schema in development mode. In fact, when you first run a Payload project with SQLite, it will automatically create the .db file and setup the schema if it’s not present . This was demonstrated in Payload’s guide: the first time the server starts, Payload generates the SQLite DB file automatically and the app is ready to use . For a packaged app running in production mode, you should verify if Payload will apply the schema automatically. In Payload’s migration docs, the SQLite adapter supports automatic schema sync in dev, but in production you may need to run migrations . For a self-contained app, you can likely treat the first run as a “dev mode migration” to get the schema in place. One approach is to detect if the SQLite file is absent and invoke Payload’s migration programmatically or run the app once in dev mode. However, this might not be necessary – test it out: if your packaged app, when run fresh, successfully creates the SQLite tables and lets you use the admin, then it’s working. Otherwise, consider bundling a pre-migrated empty database or running payload migrate on install (which could be done in a postinstall script, though that’s uncommon). In most cases, simply starting the Payload server will handle it.
• Security and Secrets: Don’t forget other environment defaults. For example, Payload requires a PAYLOAD_SECRET in production. You might set a fallback like secret: process.env.PAYLOAD_SECRET || 'dev-secret' in your config . For a truly turnkey experience, you could generate a random secret at first run or instruct the user to set one. It’s best not to hard-code a fixed secret in a published package (for security, in case someone exposes their instance). Perhaps use a random value or require the user to supply it via env if running in production mode. In a dev/test context, an empty string might be allowed , but it’s better to handle it explicitly.

In summary, the app should default to SQLite when no DATABASE_URI is provided. This means the user doesn’t have to set up Mongo or Postgres to try the app. If the user does set DATABASE_URI (e.g. to a MongoDB connection string), ensure your app respects that – likely by switching to the appropriate adapter. Document in your README how to use an external database (they would install the needed adapter like mongoose for Mongo or the Postgres adapter, and set the env var). This gives flexibility: out-of-the-box it “just works” with SQLite, and it can connect to a real DB if configured.

Ensuring a Reliable Start After Installation

To guarantee that the Next.js app can be started reliably after the package is installed, follow these best practices:

• Match Build and Runtime: Make sure the versions of next, react, payload etc. in your package’s dependencies exactly match the versions you used to build. Discrepancies could cause the runtime to break (for instance, if the user installs a newer minor version of Next than what the .next build was produced with). Pin your dependency versions or test with version ranges to be safe. The Next.js deployment guide emphasizes copying the package.json and running npm install on the same versions used to build , which in our case means the package’s own install should bring in the correct versions.
• Include Node Engine Requirements: If Payload requires Node 18+, specify "engines": { "node": ">=18.x" } in your package.json. This warns users (or tools) if they try to install on an unsupported Node version.
• No Build on Install: The idea is to avoid requiring users to run a build. The package should contain pre-built JS. Ensure that any source TypeScript or JSX is compiled. If a user installs your package globally, npm will not run devDependencies build steps. So, perform all necessary build steps before publishing. You might set up a prepublish script that runs your build and then cleans unnecessary files. Common pitfall: forgetting to remove or transpile .ts files – if your package’s main entry is still TS, it won’t run. The user environment should only need to execute the provided JS.
• File Paths and Relative Resources: As mentioned, pay attention to file path issues. For example, if Payload uses a uploads directory or similar for file storage, ensure there’s a sensible default (perhaps a folder within the package or current working directory). Document these and allow configuration via env if needed (e.g. an env var for upload path or using the default ./uploads).
• Start in Production Mode: Running a Next.js app in production mode is important for performance and to use the built assets. When using next start or the standalone server.js, the app will automatically use the production build. Avoid using next dev in the packaged context. If you want to allow a development mode (for someone who wants to extend your package), that becomes more of a source inclusion scenario – typically not needed for an embedded app distributed via npm. Keep the package focused on running the production build reliably.
• Logging and Admin Access: When the user starts the app, they should see console logs or instructions (you can print a message in your start script) indicating that the server is running and perhaps what default URL to visit. For instance, log “Payload CMS is running. Open http://localhost:3000/admin to get started.” The first time they visit the admin, Payload will prompt to create an admin user. That process should work with SQLite out of the box. As a courtesy, mention default credentials or setup if your distribution customizes that (though Payload usually lets the user create their own admin on first run).
• Example from Directus: As a point of comparison, Directus (another headless CMS) provides an npm-based CLI tool to spin up an app. If using SQLite, Directus will create the database file automatically on bootstrap . Our packaged Payload app should behave similarly – creating the SQLite DB file and necessary tables on first run without errors. Indeed, the Payload SQLite guide confirms that simply starting the server will generate the SQLite file if it doesn’t exist . This means your users can immediately use the CMS after installation, achieving the “runs as-is” goal.

Common Pitfalls and How to Avoid Them

When embedding a Next.js/Payload app into a package, be mindful of these common pitfalls:

• Missing Static Assets or Build Files: One of the most frequent issues is a package that omits the Next.js build’s static files. If the admin panel’s JS/CSS assets aren’t included, the UI will break. Always include the .next/static directory (or build directory in older Payload setups) in your published package . Similarly, include any public files (images, etc.) your app might serve.
• Forgetting to Publish Critical Files: By default, npm will publish all files except those listed in .npmignore or excluded by package.json settings. Double-check that .next, your start script, and any config files are not accidentally being ignored. For example, if you used files in package.json, ensure it lists the build output. If you relied on .npmignore, ensure it doesn’t ignore your build folder (sometimes .gitignore is mistakenly used by npm if .npmignore is absent). A quick test is to run npm pack and inspect the tarball contents.
• Large Package Size: Including unnecessary files (like source maps, tests, or an entire node_modules directory) can bloat your package. You generally should not include node_modules in the package – let npm handle dependencies. An exception might be if using standalone output which has a trimmed node_modules; but even then, it’s better to let users install dependencies via package.json. The Next.js “standalone” build is mainly intended for Docker or manual deployment, and including that node_modules in an npm package could lead to duplication or conflicts. Keep the package lean by only including what’s needed at runtime.
• Platform-Specific Issues: Using SQLite means a native dependency (e.g. better-sqlite3 or libsql) under the hood. Ensure that during installation, the native dependency can install or has prebuilt binaries for common platforms. If a user on Windows or Linux installs your package, the @payloadcms/db-sqlite adapter should fetch or build the appropriate binary. Typically, the SQLite adapter uses libSQL (which might download a Wasm or lib package). Just be aware that the first install might trigger a compile step for any native modules (like image processing libraries or SQLite itself). This isn’t usually a problem, but it’s worth mentioning in docs if users need to have a C++ build toolchain for certain platforms. Most of the time, there are prebuilt binaries so it “just works.”
• Environment Variables in Different Environments: If users run the app in different ways (development vs production), the behavior might change. For instance, Payload will not auto-create database tables in production unless migrations are run. This could confuse users if not documented. A best practice is to either handle the migration in the package (as discussed) or clearly instruct how to initialize the database schema if needed. Since our aim is a zero-config start, ideally handle it internally. Consider running payload seed or providing an initial migration if Payload supports it via code.
• Updating the Package: If a user updates your package to a new version, and they were using the default SQLite file in the old package directory, they might “lose” data if the file was stored inside the previous version’s folder (especially for global installs, the package might install to a new directory or overwrite the old one). To avoid surprises, you might encourage storing the SQLite database outside of the package directory (e.g. in a user-provided path or a subfolder in working dir). For example, Ghost addresses this by having a /content directory for data which is outside the core package. In our simple setup, using a local file is fine for demos, but for real use, advise that the env var can point to an external database or a file path outside the node_modules. This is an advanced consideration, but important for data persistence across upgrades.
• Permissions and OS: If your package needs to write files (like the SQLite DB or upload files), make sure it has permission to do so in the locations used. Writing to a folder in node_modules (for global installs) might be problematic on some systems due to permissions. It’s another reason to consider using the current working directory or a user-specified path for such data. Document where the data is stored by default.

Examples and References

For inspiration, consider how other projects achieve embedded, self-contained apps:

• Ghost: Ghost is a Node.js CMS distributed via npm. When you install Ghost, it includes a built admin client and uses SQLite by default so that you can run ghost install local and get a working blog immediately . Users can later configure a different database in Ghost’s config file for production. Ghost’s structure separates the core (installed via npm) and a content directory for data. This influenced our approach of defaulting to SQLite and allowing overrides.
• Directus: Directus can be quickly started using npx directus init and npx directus start. If configured to use SQLite, Directus will create the DB file on first run . This reinforces the idea that an embedded app should handle its own DB initialization seamlessly. Our Payload package does the same by auto-initializing the SQLite DB on first run .
• Payload Official Templates: While not distributed as an npm package, the Payload CMS templates and examples show how to integrate with Next.js. For instance, the official public demo (on Payload’s GitHub) or the create-payload-app templates might provide clues on production setup. The Payload docs on Production Deployment note that you should run next build and then next start with the right files present – essentially what we’ve baked into the package. They also show using Docker with the Next standalone output , which we adapted for our packaging scenario.

By following these guidelines – packaging the production build, providing a startup script, defaulting to SQLite, and documenting how to configure overrides – you can create an npm package that users can install and run immediately. This gives the convenience of a one-command setup while still allowing more advanced configuration if needed. Just as Payload CMS prides itself on being a Next.js-first CMS, your packaged app can showcase that by running as a self-contained Next.js server that is only one npm install away for your users.

Sources:

• Next.js deployment discussion (James Mosier’s answer on required files)
• Payload CMS Docker deployment example (using Next.js standalone output)
• Payload SQLite adapter usage example
• Ghost documentation (defaulting to SQLite on first install)
• Payload guide on SQLite (auto-generating the DB file on first run)

[linear]

AI-56 `ai-database` embed built nextjs/payload app into npm package