[GR-54926] Import resolution hook

[sgammon]

Describe the issue
I would like a way to override the result of a module import in Truffle languages that support it, like JavaScript or Python. It is already possible to provide a custom filesystem and "override" Node modules, for example, but there is no generalized way to implement synthesized modules across languages.

Code snippet or code repository that reproduces the issue

1. I have a ProxyObject
2. I want to make it available "as a module" in a Truffle language, so the user can

import { someKey } from "some-import-intercepted-for-the-proxy-object";

Related context: ESM imports in JavaScript already return an object.

[woess]

How would you pass that ProxyObject to JS so that it can be resolved by the module lookup?

But my main question would be what can be achieved by such a hook that you couldn't be achieved by generating wrapper module code, e.g.:

export const { someKey } = globalThis.proxyObject;

Of course, that code would have to be language-specific.

[sgammon]

Hey @woess,

Proposing code might be a better way to communicate this idea. I've filed a PR with a sample API that works for us. Allow me to also answer your questions:

How would you pass that ProxyObject to JS so that it can be resolved by the module lookup?

I have added a JS module resolver "factory," which can be contributed by a Java embedding user. The steps to use it:

(1) The developer sets the context option js.module-loader-factory to 'handler' (similar to other settings)
(2) The developer implements their own JSModuleLoaderFactory and/or CommonJSResolverHook
(3) The developer installs their JSModuleLoaderFactory using JSEngine.setModuleLoaderFactory
(4) The developer installs their CommonJSResolverHook using JSEngine.setCjsResolverHook

After these steps, JSRealm consults the JSEngine for a module loader on-demand. The developer can extend NpmCompatibleEsModuleLoader and their implementation is consulted to load the ESM module.

Similarly, JSRealm consults JSEngine for CJS loads on-demand. The developer can implement CommonJSResolverHook and their implementation is consulted to load the CJS module.

CommonJSResolverHook and JSModuleLoaderFactory can both fall-back to built in behavior. For ESM this happens with inheritance or encapsulation; with CJS, calls are static anyway, so the user's implementation can simply call into the standard CJS load process.

But my main question would be what can be achieved by such a hook that you couldn't be achieved by generating wrapper module code [...]

Yes, this is what we do today, but it means we need to inject a VFS. Modules must be resolved against the VFS, and parsed, and so on, since we do not have access to a snapshot mechanism as in GraalNode.

The other problem here is that such symbols still need to be added to the global bindings, so they exist without the import. We've had trouble removing internal symbols like these before running actual non-internal guest code.

If we add global bindings and remove them before running guest code, we can't lazily parse JS modules, defeating the purpose of placing them in modules instead of the global bindings; if we don't remove the bindings used by the modules, there is no point to the imports.

Of course, that code would have to be language-specific

This is also a concern since we want to have the same functionality in Python. The proposed PR doesn't contemplate that yet but this would be another strong reason we would like to implement modules directly, and provide them to satisfy imports.

Yes, I know it is possible to mount symbols in the global context in both languages -- and we do this where needed/appropriate. But this does not scale well, because we end up polluting the user's own guest context for anything we add.

---

After applying this patch, and implementing downstream (elide-dev/elide#1227) we see a major improvement in module loading performance, and the Graal compiler seems to be able to "see" our modules at link and class init time much better than before.

Of course this is a rough API and I would very much like your feedback on it. I really do want to see this feature land in GraalJs, if possible, or some similar feature that allows the same overrides.

Other platforms which compete with GraalJs in the language-engine-as-a-platform space offer this functionality and we find ourselves needing it often; thus, not having this functionality has resulted in some pressure to switch away from Graal.

Thank you for your consideration :)

[sgammon]

I should also say that this API allows us to remove our hacks which patch the JSModuleLoader for TypeScript support. We could contribute that language implementation upstream.