Build an AsyncRun-like interface for pre-stopified programs

[mwilliamson]

Hello! I'm trying to use stopify to compile node programs, but the examples given in the docs all seem to be for executing code in the browser. As a first example, I had the program:

console.log("hello")

I tried compiling it using node_modules/.bin/stopify in.js out.js --require-runtime, but running node out.js doesn't do anything. I tried, in a separate module, loading stopify and using it to execute the compiled script, but it seems like the API expects an uncompiled program to be passed in.

Is what I'm trying to do possible? If so, any help you could give in getting the example above to work, and also any advice on how to compile node applications that have multiple modules, would be much appreciated.

[arjunguha]

Here is an example of using it in Node:

> var stopify = require('./stopify');
undefined
> var runner = stopify.stopifyLocally(`console.log('hello, world');`, { /* compiler options */ }, { /* runtime options */ });
undefined
> runner.g = { console: console }; // makes console visible to stopified code
...
> runner.run(result => { console.log(result); });
hello, world
{ type: 'normal', value: undefined }
undefined

I am curious what you're trying. We've only ever used Node for unit tests.

[mwilliamson]

I'm writing a programming language, and one of the targets is JavaScript. One of the issues is my language has a synchronous, blocking style, which is awkward to translate into JavaScript: pure functions should still be synchronous, but functions that do IO and the like need to be asynchronous, and many higher-order functions need to be duplicated to support both (Bob Nystrom explains it better). I was hoping to use stopify to allow the output to be written in a synchronous style.

Thanks for the example. Is there a way to perform the compilation ahead of time? It seems like the CLI command I mentioned does so, but then it's not clear how to actually run the resulting code.

[arjunguha]

Got it. Stopify should support what you want. But, it looks like ahead-of-time compilation has bit-rotted. (Our primary use-cases are in the browser.) It should be a straightforward fix, and I'll address it this week. Sorry -- I can't quite get to it immediately.

[mwilliamson]

No worries, and thanks for the replies! If you don't mind one more question: what's the normal approach with modules? Would you normally transform the program into a single JavaScript file/string and pass that into stopify, or would you transform each individual module and have the import/require mechanism working as normal?

[jpolitz]

I've dealt with this a bit for Pyret, where we have a branch that's applying stopify in various ways to the output of the compiler.

The easiest thing to do is to generate one string and stopify it last. Stopify doesn't have support (yet) for handling top-level expressions/statements in modules that are required. require would just default to node require, which does its own (synchronous, as far as I understand it, though I don't know the state of the art with top-level yield and await) instantiation and caching.

We wanted to be able to do this more incrementally to support partial compilation and to support a mix of stopified and non-stopified code (e.g. we have some libraries that we know don't need to have stopify applied). We ended up implementing our own asynchronous require to handle blocking toplevel operations. I'm not proud of the readability of this code, but it does do transitive requires of modules; it also lazily invokes stopify on them programmatically:

https://github.com/brownplt/pyret-lang/blob/anchor/src/webworker/runner.ts#L104

Of course, if you don't have blocking/async operations at the toplevel in your language, you may be able to get away with the built-in, synchronous node require.

[mwilliamson]

Thanks, that's very helpful!

[arjunguha]

I may have a fix in #469. I'll post instructions as soon as I get Travis CI to be happy.

[arjunguha]

@mwilliamson I've fixed this on master, and can put out a release shortly. Would you mind building from source?

The documentation describes anAsyncRun interface that isn't setup for the Node runtime. Getting that to work will take a little more work. But, the code below uses a little more boilerplate that is needed to build any blocking function. I've shown a blocking sleep function, but you should be able to plug in anything else.

Here is a library implementation of a blocking sleep function.

// lib.js

/** Begin boilerplate that will be removable in the future. **/
var $__T = require("@stopify/continuations-runtime/dist/src/runtime/runtime");
var $__R = $__T.newRTS("lazy");

var saved = false;

function pauseImmediate(callback) {
    return $__R.captureCC((k) => {
        return $__R.endTurn(onDone => {
          saved = { k, onDone };
          callback();
        });
      });
}  

function continueImmediate(x) {
    if (saved === false) {
      throw new Error(`called continueImmediate before pauseImmediate`);
    }
    const { k, onDone } = saved;
    saved = false;
    return $__R.runtime(() => k(x), onDone);
}

function wrap(f) {
    $__R.runtime(() => f(), (result) => console.log(result));
}

/** End of boilerplate **/

function sleep(duration) {
    pauseImmediate(() => {
      setTimeout(() => continueImmediate({ type: 'normal', value: undefined }), duration);
    });
}

module.exports.sleep = sleep;
module.exports.wrap = wrap;

Here is an example program:

// example.js

// NOTE: A relative path will not work, because this is not
// a direct call to require. You can use an absolute path or put
// lib.js in a module (i.e., in node_modules).
var lib = require("/home/arjun/repos/Stopify/lib.js");

function main() {
    console.log('Hello, world');
    lib.sleep(1000);
    console.log('I slept');
}

// Generate this. Sorry. :(
lib.wrap(main);

Finally, build and run:

./stopify/bin/compile --require-runtime example.js example.compiled.js 
node example.compiled.js
beautified example.compiled.js
/home/arjun/repos/Stopify/lib.js
Hello, world
I slept
{ type: 'normal', value: undefined }

[mwilliamson]

Thanks, I've only a little time to play around this so far, but I'll hopefully have more time in the future. I did make one small change so that relative requires work, but I'm not sure what other (possibly incorrect!) effects this would have: master...mwilliamson:node-relative-require

[jpolitz]

I want to add something that we ran into recently that's related, which is that stopify assumes a single namespace for global variables. However, module, exports, and require can't be global since they have different behavior per-module that needs to be closed over if their evaluation is delayed by being inside a function.

We solved this with a trick I've seen elsewhere (for example in Google Caja), which is wrapping the code in a function with the appropriate arguments and then immediately applying:

https://github.com/brownplt/pyret-lang/blob/2807e54b2c8483ca390e6ce3754a54afe5735f02/src/webworker/runner.ts#L35

If we don't do this, it's not possible to get dynamic require to do the right thing with relative paths, or for module and exports to work right with some of the ways TypeScript generates code (which will generate code that closes over the global exports variable for uses of e.g. const export foo = 10 with foo used in a function). We still use .g to set the globals, but we immediately apply these three values – it may turn out that we need to immediately apply others as well to avoid further conflicts.

It seems like an opportunity for a configuration option to this AsyncRun to specify some names that shouldn't be treated as global but as “module”-local, even just by doing this trick on behalf of the caller.

I can look into doing this at some point, but wanted to get the idea down in a reasonable context.

[arjunguha]

It seems like an opportunity for a configuration option to this AsyncRun to specify some names that shouldn't be treated
as global but as “module”-local, even just by doing this trick on behalf of the caller.

I suppose you call evalAsync to load the code of each module. Is that correct?

If so, we could pass a set of module-level names to evalAsync. Would that work?

[jpolitz]

Basically, yes. I think it should be a set of names *and their values*, though. This makes it so a module that doesn't use return can still communicate via side effects on module.exports or similar (as the caller of evalSync shares a reference to that object).

…

On Mon, Sep 07, 2020 at 10:51 AM, Arjun Guha < ***@***.*** > wrote: > > > It seems like an opportunity for a configuration option to this AsyncRun > to specify some names that shouldn't be treated > as global but as “module”-local, even just by doing this trick on behalf > of the caller. > > I suppose you call evalAsync to load the code of each module. Is that correct? If so, we could pass a set of module-level names to evalAsync. Would that work? — You are receiving this because you commented. Reply to this email directly, view it on GitHub ( #468 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAA5IU5Y4LC2DMRQDA4LKWTSEUMQXANCNFSM4PBG5PQQ ).