Live dev environment

[geelen]

Was just considering the DX of working on FAB-level plugins after chatting to a few folks this week about the potential for this "FAB server layer" to form a really interesting layer of functionality & caching, between your user & your upstream API server. Starting with this thought bubble, but I realised that the workflow for building this kinda sucks:

Let's say you update one of your FAB plugins, and want to test it:

• If you're changing your client-side code as well, you need to yarn build:fab to build both your app and the FAB. Even for simple apps, that takes a little while.
• If you've only changed the FAB server component, yarn fab:build will do it and is fairly quick, but then you do need to rekick your fab serve fab.zip process and refresh the browser.

Was thinking a different workflow, but I wanted to put it out there to see what people thought of it before I lock down an api:

fab dev-server --proxy=3000 --port=3001

This would read your fab.config.json5 file, find all the plugins you're using, ignore (somehow, see below) input-{static,nextjs,sapper}, and things like plugin-rewire-assets, and actually plugin-render-html, since they're really tied to the compiled FAB. But it would live-reload the remaining plugins, and forward requests onto http://localhost:3000.

This should mean you load http://localhost:3001 as if it's your normal dev environment, but requests are moving through the FAB server code, meaning things like /api/graphql could be caught and all their logic iterated on. Hot module loading should still work with Webpack's dev server for client-side changes.

I'd need to stub out the cache module that I'm planning on introducing as part of RC2, which actually gives me a good way to dev on that, so I think this is going to happen.

Plus, I want to run fab.dev/blog as a server-rendered React Notion proxy so I can start to wean myself off gatsby. This would be a good way to test that.

[geelen]

Thinking about it, I could simply not invoke the build phase of any plugin, but then invoke the runtime phase of each builder with a ProtoFab that has no files or metadata. That'll mean plugin-rewire-assets and plugin-render-html are there they just have nothing to do.

[evanderkoogh]

I think the most important part is not to come up with a API in advance. There is so many edge cases here it most important to stay nimble and allow experimentation.

I have been thinking about a few different scenarios. One of which would be to check in ${plugin-name}-dev is a plugin that exists and then load that one instead of the real one.

But it is a pretty tricky one to do really well. Webpack server has a websocket (I think?) connection open to reload pages/modules.

But to then also ideally reload a page that isn't server by webpack is going to be interesting.

[taybenlor]

I like the idea of using the fab dev server as a proxy to your existing server, since that's how it behaves in production.

I can imagine issues when it comes to serving assets since your uncompiled assets may look very different to the fab compiled assets. So you may end up with very different paths. Probably this won't be an issue in most cases.

[yacinehmito]

I've given this some thought and would like to share where I am at right now.

Setting up the stage

I'll focus solely on the DX of running a FAB with assets that are dynamically compiled by some user-space tooling, typically a development server like the one run by react-scripts start from create-react-app. I am not looking to specifically address the case where the FAB server proxies the API server, although that may work too.

I'll also make some distinctions between:

• The FAB content, which are the server.js file and the _assets folder. This is part of the FAB standard.
• The FAB zip, which bundles the FAB content.
• The FAB server, which is the server run with server.js that leverages the content of the _assets folder.
• The fab tooling, which are the CLI and plugins that help produce and deploy a FAB zip, and run a FAB server locally.

FAB in uppercase letters always refer to the parts of the standard. fab in lowercase monospace always refer to the tooling.

Some definitions:

• "Development mode" refers to a setting where a FAB server running for the purpose of developing on the local machine.
• "Production mode" refers to a setting where a FAB server is running "normally" from a FAB zip.

Doing it "right" is a dead end

I started with the following design constraints:

1. The observable runtime behaviour of a FAB server in development mode should diverge as little as possible from the one in production mode. A corollary is that there should be as little divergence in the fab config between development and production modes.
2. A FAB server in development mode must optimise its start time, and can sacrifice bundle size or runtime performance in that pursuit.
3. Running a FAB server in development mode doesn't assume any pre-existing FAB zip or FAB content; building is handled at runtime, alongside the normal responsibilities of a FAB server.

I don't think it is practically possible to achieve this. Indeed, we are dealing with the following interfaces:

• A development server's downstream interface is the HTTP server itself. Its purpose is to abstract away the filesystem.
• FAB's upstream interface is the FAB zip or the FAB content, so it's quite file-centric.

Because FAB's upstream interface is file-centric and FABs have so far been built "cold" for production use, the fab tooling is also file-centric. In particular, one must specify all assets files (both their name and content) at build time by setting proto_fab appropriately.
Note that the runtime parts of fab on the other hand would not be affected when running in development mode, unless they are part of a plugin that forwards data from the build step, and that build step itself relies on the filesystem.

The real design challenge lies in this issue. The rest of the problems are comparatively trivial: setting up a watcher and an incremental compile step for the server part of a FAB is just engineering effort (and is partially done already with fab build --watch).

I had written the different design avenues I explored to fix or avoid the mismatch and why they are all a dead-end to me, but I removed it because it was a wall of text that led nowhere. I can share that on Discord.

Loosening the design constraints

Knowing that a solution following the stated design constraints cannot be practically achieved, let's relieve some.

The most critical one is the first:

The observable runtime behaviour of a FAB server in development mode should diverge as little as possible from the one in production mode.

Taking inspiration from recent developer tooling

Currently, there is a trend in developer tooling that purposefully does not follow that principle. Tools like Snowpack and Vite maximise the runtime behaviour between development mode (server through development server) and production mode (bundling the application into a folder of assets) in order to speed up development builds as much as possible.

The differentiation is maximised because these tools separate compilation from bundling:

• Compilation turns a set of source files into a set of assets that can be used by the platform. It is by design lazy and granular. It means that compilation happens only on demand, its result is entirely predictable, and it operates on the smallest resource possible. Compilation happens at runtime in development mode, and as part of bundling in production mode.
• Bundling turns the whole set of sources into the full folder of assets. It encompasses compilation, but does so much more as well. Bundling only happens in production mode, never in development.

Let's take this as a source of inspiration: could we separate plugins between what is akin to compilation and was is akin to bundling? Starting with some examples:

• We are given a plugin whose purpose is to set some HTTP redirects. It has every reason to run in development mode; it is neither compilation nor bundling, but is part of the expected behaviour of the server.
• We are given a plugin that performs variable injection in the HTML files at build time. This is something that belongs in "compilation": it operates file by file and the result is predictable (the filename stays the same).
• We are given a plugin that fingerprints assets. This is something that belongs in "bundling": it is only a performance improvement and the result is unpredictable (as it depends on the file content, which is not known in advance in development mode).

Note that the purpose of a plugin is not always indicative of at which step the code runs. A plugin that inline assets in server.js belongs in "bundling", but will require a runtime component.

Making the purpose of a plugin explicit

The learning here is that different plugins have different purposes, and it is their purposes that determine whether they should run in development. One could already get a sense of that, with some plugins prefixed with input- and @geelen suggesting that they would be specifically ignored. I was not comfortable with the proposition because it felt ad-hoc and hacky, but it pointed towards the right direction.

I'd like to propose a significant change, but much less ambitious that what I had initially proposed when keeping all design constraints. to clearly separate plugins by their purpose in the config.
To me, there are 3 different categories of purposes for plugins (but that can be challenged). They are:

• To serve something: Basically, to answer to whatever request. A plugin made to serve something only has a runtime component. It can run in development mode unchanged.
• To transform: It takes an existing resource and predictably transform it into another. A plugin made to transform something can have an arbitrary runtime component, as well as a limited build component. The build component should not have filesystem access, and declare upfront the kind of changes it wants to perform on the assets so that it can be called lazily in development mode.
• To bundle: It's not the same meaning as with the examples of Vite and Snowpack; it's bundling in the literal sense, as in Frontend Application Bundle. A plugin whose purpose is to bundle makes general changes to the assets and how they are served. It can have both a runtime component and an unlimited build component. Plugins that bundle do not run in development mode.

So, instead of having one plugins key in the fab config, there would be 3 different keys for the 3 different purposes.
How this would exactly affect the interface between plugins and fab is something that is left to be determined.

Once the purpose of plugins is clear, fab would expose the following interfaces:

• fab build create a FAB zip as it does right now, executing all plugins.
• fab serve starts a local server from a FAB zip.
• fab dev starts a local server that runs the build component of the transform plugins lazily. Because there are no assets to serve as-is, fab dev can also be provided either a folder for the "starting" assets, or the base URL of a development server that is serving these assets.

The description of the change is not very detailed. There are a lot of stuff to hammer out; it's just to start a discussion.

[geelen]

Thanks for this @yacinehmito. Lots to think about. Sorry it's taken me a couple of days to reply, I wanted to jot down a few points here and I think the best thing then is to set up a call to discuss it.

The observable runtime behaviour of a FAB server in development mode should diverge as little as possible from the one in production mode.

This is potentially an important distinction. FABs started life as a purely production-only artefact. In fact, the original design had no rewire-assets component: we were thinking we'd be building tooling to change a user's CRA/NextJS to output their assets into an _assets dir. I built that once for Linc and... decided not to pursue it!

As a result, and attempting to make fab (the tooling, I love your suggestion for caps/lowercase to mean standard vs tooling btw) as onboarding-friendly as possible, we shifted the focus to making things compatible with users' existing apps without needing internal restructuring. And so we got the plugins, etc. So fab has kind of grown into a project for building FABs without really considering what that would look like beyond just a code-level compatibility. I think it's probably time to really tackle what the development-time experience can be, so thanks for jumping on this.

Initial thoughts

Let's take this as a source of inspiration: could we separate plugins between what is akin to compilation and was is akin to bundling?

Love this delineation conceptually, but at first blush I can't think how exactly to capture that in code/tools.

The learning here is that different plugins have different purposes, and it is their purposes that determine whether they should run in development.

This I'm less sure about. I think it might be useful to drill into a concrete example here, which I'll come back to.

I'd like to propose a significant change, but much less ambitious that what was proposed earlier and that allows flexibility in how to make DX evolve: to clearly separate plugins by their purpose in the config.
...
To serve something: ...
To transform: ...
To bundle: ...

Love this. I think the challenge would be mapping the internal plugins to this model in a way that allowed them the best dev experience. But as for user-specified plugins, particularly the serve and transform ones (which would be the most common in user-land, I think), this should work well.

So, instead of having one plugins key in the fab config, there would be 3 different keys for the 3 different purposes.

I think you might run into ordering issues here? Unless you can guarantee that all serve plugins run before all transform ones, say. Otherwise I'd probably prefer a model where there's a single plugins field but the plugins know which stages they are responsible for, and activate/deactivate in each stage accordingly.

fab dev starts a local server that performs executes the build component of the transform plugins lazily. Because there are no assets to serve as-is, fab dev can also be provided either a folder for the "starting" assets, or the base URL of a development server that is serving these assets.

Love it. Would never have thought of backing onto a folder, but it makes sense. Effectively, input-static then becomes the "default" mode for a FAB (which is something I've toyed with, the idea that input could be a top-level param on the fab.config.json5, with values like './dist' for static, and 'nextjs' to imply @fab/input-nextjs).

So I think we're definitely thinking in the same lines wrt tooling, but I want to drill into a particular example of a plugin that might drive out the compile/bundle idea:

@fab/plugin-render-html

So, as it stands, this plugin does a couple of things:

• At build time, it finds all the .html files in the proto_fab.files and extracts them.
• For each .html file, it injects a a mustache template injection {{ FAB_ENV_INJECTION }} before the </head>
• Then it tokenises the template with the mustache compiler to turn the file into a .html.json
• It injects a runtime component that looks for the HTML requests (as well as handling fallback logic), fetches the .html.json and assembles the final result, replacing FAB_ENV_INJECTION with <script>window.FAB_SETTINGS = { ... }</script>.

This to me seems to match your concept of a transform plugin pretty much exactly, yeah? But the code as it is right now is pretty heavily divided into build and runtime steps. Would you envisage the plugin needing to change at all to support a fab dev mode? Or would it just have its build and runtime calls invoked on each... request? When the source filesystem changes?

Would love to discuss this use case with you. So far I'd assumed that this plugin couldn't really work except for a packaging step, but writing it out it's actually closer than I thought. And if this one works, then a lot could be made to work.

An alternative idea

Rather than push a plugin into a single "category" of serve|transform|bundle, I'm wondering if we could instead do something slightly different. Maybe the plugin could have an optional, separate dev or live entry point, which has one purpose: to respond to or announce changes to the proto_fab. I'll try to describe what I mean:

Let's take the following plugin chain:

input-static -> render-html -> rewire-assets

And a dev function signature like..

export async function dev(
  args: PluginArgs,
  proto_fab: ProtoFab<PluginMetadata>,
  onUpdate: (cb: (proto_fab: ProtoFab) => Promise<DevDirective>) => void,
  emitUpdate: (proto_fab: ProtoFab) => Promise<void>,
  onClose: (cb: () => Promise<void>) => void
): Promise<DevDirective> {
  // Default implementation if none present... maybe?

  // Invoke this plugin's build as normal
  await build(args, proto_fab)

  // Any changes to the proto_fab, rerun our build and emit
  onUpdate(async new_fab => {
    await build(args, new_fab)
    // Tell the dev server to propagate the proto_fab changes,
    // not to rebuild the `runtime.ts` file, but to reinitialise it
    return {
      propagateUpdate: true,
      compileRuntime: false,
      reloadRuntime: true
    }
  })

  onClose(() => {
    // noop
  })

  // Tell the dev server to include our runtime as normal
  return { compileRuntime: true }
}

Names could be changed, I'm just thinking of this as I'm writing it. But the idea being that each plugin, when booting, sets how it should behave and respond to dev-time changes, but preserves the chain-of-plugins system as-is.

It becomes pretty easy to implement a "bundle" type plugin, its dev callback is as simple as:

export asyn function dev() {
  // No build here
  
  // And tell the dev server not to even include
  // our `runtime.ts` file.
  return { compileRuntime: false }
}

On the other hand, something like input-static could set up a chokidar watcher on the input filesystem:

export async function dev(
  // as above
) {
  await build(args, proto_fab)

  // no onUpdate since we're the first plugin, instead
  // watch the filesystem
  
  const watcher = chokidar.watch(args.dir)
  
  watcher.on('all', debounce(async () => {
    const new_fab = proto_fab.clone()
    await build(args, new_fab)
    // Trigger a cascade of updates down the pipeline
    await emitUpdate(new_fab)
  })

  onClose(() => {
    await watcher.close()
  })

  return { compileRuntime: true }
}

Anything runtime-only, like your serve and a fair chunk of transform, would be covered by the default implementation above (without the build invocations).

The thought, then, would be that plugins might be able to detect which changes to a proto_fab they're interested in, and whether their runtime component should be reevaluated. If a change happens, all the onUpdate calls happen and nobody says [reload|compile]Runtime: true then we can avoid rebooting/recompiling the runtime code. The next request just hits the FAB like normal.

I also think this might be able to be extended so that user plugins by default watch themselves. So a ./proxy-something.ts file emits a emitUpdate event that causes a compileRuntime: true directive. We do what we can, then, with rollup to make that recompilation as fast as possible, but at the end we're always constructing a "real" FAB server.

Thoughts

I realise what I've done here is actually try to make a "live-reloaded FAB" that behaves as similarly as possible to a production FAB as possible. Which ends up being the thing you've already considered and dismissed, so maybe you're way ahead of me on this already! My gut feel is to preserve the dev/production alignment as much as practical, but there might be a good reason why that isn't a good goal that I'm not seeing.

Anyway, probably best to move to a video call or something to poke holes in.

[geelen]

From our call:

@yacinehmito: "If the development time story around FABs is not good, then you tend not to put things you need in development in your FAB"

This hit the nail on the head. There's so much stuff you could be doing with a little FAB server, so many things that currently need a full on Express/Hapi/Koa server (which aren't directly compatible with FABs) that would be awesome as little, shareable snippets of FAB server code, but if it's not easy to spin up & iterate on (independently of packaging & deploying) then it's not really a true "alternative" to existing JS server-side code.

I have a bunch more notes but I think I'd rather jot down where my head is at about how to proceed:

Probable

• fab init will add a npm script(s) for fab:dev (or potentially just dev?) that lets us preconfigure how, say, Create React App and FABs should work together in development mode. Something like:

"dev": "run-p -r dev:*",
"dev:start": "npm run start --port=19876",
"dev:fab": "fab dev --proxy=localhost:19876"

Alternatively, that could all be hidden behind a "smart" fab dev command that knows it's a CRA project and does magic stuff.

To start with, fab dev will just run fab build --watch and fab serve --watch behind the scenes. Improvements like the one described above can be implemented if/when that approach hits its limit.

Potential

• The concept of a "runtime only plugin" will start to become a "spec" of its own. For the moment I'll just be calling that a "FAB Server" spec, distinct from a "FAB Plugin". But it could have its own name, like "FSM"/"FSF" (FAB Server Middleware/Framework).

A "FAB Server" could run totally independently of a FAB, all it is is a dialect of a request/response interceptor that uses the Fetch API natively. But FABs would have zero-configuration support for a FAB Server. So you could think of your FAB-enriched app as being three parts:

• Your existing, client-side code, written in CRA or Next or Vue/Nuxt/Sapper, etc
• Your FAB Server app, which is any server-side endpoints you want to add to enhance your server, with the understanding that these will be running on the edge at deploy time
• Your FAB build config, with all the plugins required to make it compile/deploy as needed.

Then, in fab dev, only the first two would need to be active. So we bypass the whole "how to handle live-reloading build-time plugins" because most of the time it's not even needed.

Probable

• Irrespective of whether we pursue the "FAB Server is a separate thing" approach previously, one change I'm almost certain to implement is the idea that projects using fab have an implicit server.

That is to say, if a project has a fab.server.js (or fab-server.ts, src/server.ts, fab/server.ts, we should pick one), then it gets automatically included in a fab build. Potentially, a fab.config.json5 could even have an entry point server: './src/fab-server.ts' by default, but we don't treat the absence of that file as an error (unlike referencing a plugin).

This would mean that the server file has no access to any args that plugins do, and probably no metadata. Their runtime API might not just be the export default ({ Router }) but something slightly different, too.

Also, potentially, we could reframe @fab/plugin-precompile as an option on a user's "server" field. So something like:

{
  server: {
    entry: "src/server.ts",
    precompile: true, // or "webpack"
  }
}

That might be a nice way to reach out to folks with a fair bit of server-side code already, rather than the concept of a plugin-inside-a-plugin.

More thoughts

I'll jump back in when I think of them but it's dinner time now yo!

[taybenlor]

I thought I might throw in my own perspective on this, not on how the problem should be solved but rather on how I'm currently using the plugin system. This may help you make decisions.

I'm doing something different to the use cases you're describing and I'm kind of using the FAB Server as an alternative to express. This is all quite experimental and so I've made some weird decisions and using things in non-standard ways.

My app is kind of like a CMS that runs on the edge with Cloudflare workers (it's not that at all but it's the closest thing). My insight is that a CMS is mostly reads, not writes. So it doesn't make sense to waste server time on rendering templates and hitting the database every read, instead it only renders templates on writes. Reads come directly from the Cloudflare KV store.

I have a few different plugins:

• publish_document (build and runtime) - responsible for taking a JSON document and rendering it to public facing HTML
• store_document (runtime only) - stores JSON documents for private editing
• store_file (runtime only) - stores binary files / uploads (public read and private write)

The rest of my setup is basically the same as a static app. It's a series of templates rendered by eleventy, a SPA built in svelte, all built together and optimised by parcel.

The store_document and store_file plugins are relatively trivial so I'll quickly outline them. They take POST requests and store them in the Cloudflare KV store. They take GET requests and retrieve them from the KV store. They don't do much else. In the future all plugins will need some kind of authentication + authorisation setup, but I haven't explored that just yet.

The publish_document plugin is more complex.

Build publish_document
In the build phase it reads a html file from the proto_fab which is used as a template in the runtime phase. This html file includes all the CSS and JS imports from the original eleventy and parcel build steps. It takes this "built" template and combines it with the "unbuilt" templates used for document publishing on the server, so that they are correctly linked to their built CSS and JS. It takes all of this stuff out of the proto_fab.files and throws it into the proto_fab.metadata so that it's all in memory at runtime.

Runtime publish_document
In the runtime when the user chooses to POST a document the server takes the appropriate template from the metadata, combines it with the incoming JSON document and renders that into HTML and puts it into KV storage. When a GET request comes in for the published document it retrieves it straight from the KV store.

Right now, when I'm editing templates for the publish_document plugin I have to do a full build of my FAB to test it locally. To do that my build process goes:

1. Eleventy build static templates
2. Eleventy copy over source files (JS, CSS)
3. Parcel build static website (HTML, JS and CSS)
4. Fab build plugins, then build fab
5. Serve built fab

Ideally each of these would work in a chain, so if I edit a template, it is built, then parcel detects the built template and builds the static site, and then fab detects the change in the built static site and builds the plugin, then builds the fab and restarts/hot reloads the server.

I think re-thinking how FAB works to support more efficient development is a really good idea. And if doing that breaks my plugins thats not a big deal (since this is all experimental anyway). I just thought it might be helpful to get some insight into another use case.

Have a good weekend!

[evanderkoogh]

Hey @taybenlor, this is so so cool! @geelen and I have talked about exactly this use-case so often. We just haven't found the time to actually build it. Do you have this somewhere publicly where we can take a look at it? Would love to see it.

I am pretty sure that the case that you describe there is what we want to support as a minimum. I think it is easy to get stuck on how to support advanced use-cases like NextJS, but I think it would be wise to start supporting what Glen and I have started to call Static+. A mostly static deployment, but with some dynamic logic sprinkled in, exactly what you describe above.

That is a use-case which has been very much underserved in the current market and we can make the biggest difference. If we can think of a way that may also support NextJS and friends in the future, that would be great.

I'll have a think about this today and see if I can come up with a proposal.

[taybenlor]

Hey @taybenlor, this is so so cool! @geelen and I have talked about exactly this use-case so often. We just haven't found the time to actually build it. Do you have this somewhere publicly where we can take a look at it? Would love to see it.

Thanks! I genuinely think it's an interesting direction for web app development to go, especially as an architecture for "web apps on the edge". Unfortunately nothing I'm able to show publicly yet - but my MVP should be stood up in the next month or two and then I'd be happy to walk you through it.

[evanderkoogh]

Ok.. so here is my $0.02 on the subject.

I absolutely loved @yacinehmito's Setting the Stage, so here is my version:

The Purpose
The purpose of a local development mode in fab is to improve the development of FAB plugins or server-side code.

The Goal
The goal is to decrease the time it takes between editing a file and being able to see it in action running in a way that is as close to production as possible.

Now as Yacine has pointed out already the speed and closeness to production are impossible to achieve together. The closest to production is to run a full build, which is obviously not quick. So we are indeed building something that is imperfect, but can still be extremely valuable if we get it mostly right.

If we look at what needs to happen there are a few broad steps.

1. (optional) After editing, some external process needs to do some work
2. Interested configured plugins need to be notified that something has changed
3. Plugins need to be figure out what, if any, work they need to do and make those changes internally.
4. fab-dev-server needs to know when all plugins are done, so it can tell the browser to refresh.

What if all plugins had an optional dev mode? This would be mainly for what Yacine's called plugins with build functionality.
In dev mode they can do their thing, but in a quick, in-memory kind of way. And have code to figure out the minimum set of work to be done after a particular change.

The one thing that I am not too sure about is how the change detection/notification flow would go. First I thought it was from fab-dev-server to the plugins, but I think it would have to be the other way around. A plugin in dev mode would be able to watch a particular directory, or to spin up a dev server for the assets they are responsible for building.

So to take @taybenlor's example with the build-time publish_document plugin, in dev mode it would watch the output directory of parcel. And when a file changed it would figure out which templates it would need to update internally. And when it was done, it would notify the dev server that it should reload the current page.

And when we get to things like NextJS dev servers, we could subscribe to notifications for update if they already have that functionality, or add the functionality if it doesn't.

[geelen]

@taybenlor this is super awesome! I love that you're using the proto_fab.files and .metadata stuff, are you using Typescript? Did you have any trouble with the whole T extends PluginArgs = PluginArgs concepthere? I kinda implemented being like "I think this makes sense... right?" but haven't had any feedback on it yet.

Do you have any idea how long the turnaround time is for a fab build --watch [input_files] when you change either the HTML or JSON? The first cut of fab dev is probably going to just wrap that, so it'd be good to know what kind of ballpark you're seeing. I'm leaning towards implementing a granular cache on each of the build steps of the current plugins before a dedicated dev mode, since that has a secondary benefit of speeding up subsequent "real" production builds.

@evanderkoogh maybe you missed my comment above? I describe something very similar to that #188 (comment) under "An Alternative Idea". Basically drills down into some onUpdate, emitUpdate and onClose calls, a significantly smart dev server should be able to coordinate all the dev plugins and get a pretty good result.

That said, my chat with @yacinehmito kinda took a different direction. We were trying to think about how to give "runtime only" plugins a big iteration boost first, and potentially leave build --watch as "good enough" for anything involving build-time plugins. That was explicitly assuming that most of the user-land "plugins" that people might want to write would be runtime-only, which @taybenlor just kinda contradicted, ha!

So while I'm pretty sure on the very next step (create a fab dev entry point that just wraps build --watch && serve --watch, so we can then improve things behind the scenes gradually), the direction after that I'm less sure of. I guess the options as I see them are to do some/all of:

• Add a "server" entry to fab.config.json5 that's there by default, so you don't need to "add a plugin" to start adding behaviour.
• Talk about a "FAB Server" runtime spec as an alternative to Express/Koa (wanna run some JS on CF workers? here's a native, fetch-based composable middleware thing)
• Add dev entries to each of the plugins and get that loop super-tight so any plugin (in-built or user-authored) can be iterated on at dev time.
• Split up and identify plugins as one of several "categories" (serve/transform/package) which have different (but defined) dev-time behaviour.

Will keep noodling on it, but if you've got more suggestions/discussion, fire away!

[taybenlor]

are you using Typescript? Did you have any trouble with the whole T extends PluginArgs = PluginArgs concepthere?

I'm using Typescript for everything else, but decided to write the build step in JS because I didn't want to bother with compiling it. I was thinking of looking at that issue on getting build scripts in Typescript working.

I think the T extends PluginArgs thing totally makes sense, you want the plugin to be able to construct its own version of the pipeline.

Add a "server" entry to fab.config.json5 that's there by default, so you don't need to "add a plugin" to start adding behaviour.

Talk about a "FAB Server" runtime spec as an alternative to Express/Koa (wanna run some JS on CF workers? here's a native, fetch-based composable middleware thing)

This makes a lot of sense to me. I think you're right that small server-side changes are probably the main use of the plugin system, so making that much easier is going to be great.

I also think having a "FAB Server" spec would be super useful. I find the idea of building directly on top of CF primitives a bit worrying since it locks me into particular infrastructure. Having a spec also encourages other implementations on top of AWS, Azure, and Google Cloud.

I see a FAB server as being the perfect glue between web microservices. For example my plan is to implement authentication using Auth0 and payment using Stripe. Eventually if I need atomic persistence I plan to use a web service for that as well. That means I have no need of a "real" backend - I can just have the FAB server glue together the various "APIs as a Service" that I use.

PS: The types are so good in FAB, makes it really easy to quickly figure out what is going on.

[Edit] Sorry missed this:

Do you have any idea how long the turnaround time is for a fab build --watch [input_files] when you change either the HTML or JSON?

I haven't checked yet but I can get back to you when I next have some free time!

[geelen]

PS: The types are so good in FAB, makes it really easy to quickly figure out what is going on.

☺️

I also think having a "FAB Server" spec would be super useful. I find the idea of building directly on top of CF primitives a bit worrying since it locks me into particular infrastructure. Having a spec also encourages other implementations on top of AWS, Azure, and Google Cloud.

Yeah, effectively it'd be a "server-side service worker spec with cache", which just so happens to map nicely to CF Workers at the edge. If there's a true competitor to Workers down the track or just if you're not interested in edge compute you can run it anywhere. Same arguments for FABs proper, but for a server mindset.

I see a FAB server as being the perfect glue between web microservices. For example my plan is to implement authentication using Auth0 and payment using Stripe.

So one of the advantages of Express and friends is the abundance of published middleware. APIs like:

const middleware = require('some-api-service/middleware')

app.use(middleware)

This is something I hit a while ago, I think I was maybe trying to support divjoy with server-side code provided by FABs. I think the Mailchimp SDK kinda dropped-in in Express but had a direct dependency on request so was incompatible with FABs.

Whatever "FAB Server" spec we come up with, I think I'd like to design it with that kind of code-sharing in mind. In that respect, a hard dependency on building & deploying FABs is counter-productive, since you'd then need to convince people to use FABs (when there aren't good drop-in libraries) to get the critical mass to make drop-in libraries. On the other hand, a separate, modern (fetch-based) pure JS middleware definition is a bit lighter, and backwards compatible with an Express server.

So I think I have enough to proceed but I have a few open PRs I need to look at today. Biggest question I'd like feedback on right now is: what to call this "new" thing? "FAB server" works I suppose, but anything better?

Also, the API for accessing it. Atm we expose a default export that gets passed a FABRuntime, but that's very file-centric. I suppose we could keep it and provide an Express-compatible fallback using something like:

import express from 'express'
import { MiddlewareUtils } from '@fab/server-spec'
import server from './fab-server.ts' // (still use the default export idea)

const app = express()
app.use(MiddlewareUtils.toExpressMiddleware(server))
app.listen()

Is that... good enough? Or should we be encapsulating things further?