[Meta] Plugin system for 2.0

[kdy1]

I will modify this as required.

Javascript plugins

If it's possible to make it fast, I'll support js plugins, too.

Rust-based plugins

Tasks

Task: Proxy swc_common (#2646)

• swc_common: Create a trait for thread-local APIs. (cargo feature plugin-base)
• swc_common: Implement the trait. (cargo feature plugin-rt)
• swc_common: Replace APIs in plugin mode. (cargo feature plugin-mode)
• Pass the trait to plugins and use it to replace swc_common APIs.

Task: rplugin: System to pass AST directly (#2671)

This is important for performance.

Task: Share more input, like SourceMap or SourceFile.

Task: Actaully use rplugin.

Task: Improve plugin runner

Task: Add plugin feature to swc

Task: Expose node apis

Design

I experimented lots of ways for plugin system, and decided to go with abi_stable. But there's a problem.

Problem: Identical API

I want to provide identical API as core transforms. And it requires sharing data stored in thread-local variables.

Triage: Dynamic library

I initially thought I need to make swc_common a dynamic library so plugins can invoke it. But dynamic linking does not fit for swc plugins. As swc_common depends on std, the version of rustc should be all same among plugins, and chaning version of rustc becomes a huge breaking change. I don't want to break all plugins by upgrading rustc. So wee need another way than a dynamic library.

---

I posted some questions on the rust community while trying this option, in case you are curious about what I tried.

• https://www.reddit.com/r/rust/comments/ql1c5m/questions_about_sharing_thread_locals_between/
• https://users.rust-lang.org/t/sharing-a-crate-with-crate-type-dylib/66768

I also experimented with extern "C". See https://github.com/kdy1/rust-dylib-test

Solution: Pass function to plugin with stable abi

Instead of making swc_common dynamic library, we can pass a big trait that contains functions related to thread-local variables in swc_common.

The plugin runner will pass this trait while invoking plugins. abi_stable can be used to do this.

For plugins, operations of swc_common will be replaced by the trait. In other words, functions in swc_common will work by calling a method in the passed trait object.

---

[kdy1]

Related PR: #2671

---

I made a crate to pass AST nodes as a special form to the plugin and this is the result I got.

Due to limitation of testing system,ast_clone_to_stable performsclone + to_stable andast_clone_to_stable_then_to_unstable performsclone + to_stable + to_unstable.
Time used for the clone is benchmarked by ast_clone, so you can subtract it.

I also profiled the performance of passing data to plugins using JSON

.

Note that in the real-world, clone is not required.

Deleted

I forgot to configure the memory allocator. So I updated the image above.

[devongovett]

Excited to see progress here! I think there's still one potential issue with this approach. Adding/updating/removing fields from AST nodes is potentially a breaking change when types are nested without boxes. For example:

struct Foo {
  foo: u32,
  bar: Bar,
  baz: u32
}

struct Bar {
  test: u32
}

Now say you add another field to the Bar struct. On its own this is ok if you add it at the end, but this changes the memory layout of Foo as well. Specifically, the fields after bar would be shifted. So any plugins compiled against an older version of SWC won't work anymore.

To avoid this you'd have to box every single nested struct (which might not be great for perf), and also make sure that you only ever add fields at the end.

Another way could be to make accessor functions and have plugins only use those instead of accessing fields directly. Eg get_bar(foo) instead of foo.bar. That would leave the implementation up to SWC core and allow older plugins to still work with a newer SWC (assuming fields aren't removed). Not sure the perf impact of doing that though, and you'd want to do some codegen to make this not super tedious for plugin devs to use.

Or maybe you just decide that AST changes can only be done in major versions. Not sure how often they occur.

[kdy1]

@devongovett Thank you for the feedback!

Actually abi_stable supports adding fields without breaking layout.

And It also supports making enums non-exhaustive so we can add variants without breaking plugins.

I'll make all fields of plugin ast private so adding fields is not a breaking change for rust, too.

---

Problem is that the current design makes 'unknown' variants and fields dropped by the plugin.

Thoughts:

• We can force people to use plugin ast for their transforms so it never breaks, but I'm not sure about this at the moment.

• Most of ast changes are typescript extensions. As plugins are invoked after stripping types out, so excluding AST nodes related to typescript from plugin ast is also possible.

• (Not sure) AST changes comes with new transforms to remove the added AST nodes.
  We might detect the version of plugin, and invoke the plugin after removing them. But I think this will make the code much more compilcated.

How do you think about these ideas?

[devongovett]

Ah interesting, I missed that. Looking at the docs it seems all field accesses end up becoming methods (returning Option if they were added after the last major), which is close to what I was suggesting.

We can force people to use plugin ast for their transforms so it never breaks, but I'm not sure about this at the moment.

What are the downsides of this? It seems like this would avoid some extra AST conversions as well, so maybe better perf too? I guess the difference would be that the API is slightly different for plugins vs in SWC core, but I think this is ok (and maybe expected?). I would think you would want to expose a more limited API to plugins anyway to avoid maintenance problems. Babel had this problem - they exposed everything in core to plugins, and were stuck not being able to make changes because plugins started depending on too many internals. I would recommend being quite intentional about what is exposed to plugins. Limit it at first and expand over time as things get figured out and requested by plugin authors.

Most of ast changes are typescript extensions. As plugins are invoked after stripping types out, so excluding AST nodes related to typescript from plugin ast is also possible.

I could imagine plugins wanting to run before stripping TS as well, e.g. a documentation generator, or code mod system.

We might detect the version of plugin, and invoke the plugin after removing them. But I think this will make the code much more compilcated.

Yeah that sounds hard.

[kdy1]

Ah interesting, I missed that. Looking at the docs it seems all field accesses end up becoming methods (returning Option if they were added after the last major), which is close to what I was suggesting.

Oh, I see. Yeah, something similar to method will be required to handle such cases. (Logically, didn't think about how does it work)

What are the downsides of this? It seems like this would avoid some extra AST conversions as well, so maybe better perf too? I guess the difference would be that the API is slightly different for plugins vs in SWC core, but I think this is ok (and maybe expected?). I would think you would want to expose a more limited API to plugins anyway to avoid maintenance problems. Babel had this problem - they exposed everything in core to plugins, and were stuck not being able to make changes because plugins started depending on too many internals. I would recommend being quite intentional about what is exposed to plugins. Limit it at first and expand over time as things get figured out and requested by plugin authors.

Exactly. It will avoid lots of conversions. I was simply trying to expose identical API as core transforms, but I think you are right. The problem of babel is quite interesting. Thank you so much for sharing it.

---

I concluded that exposing plugin ast to plugins is better than exposing normal ast to the plugin, thanks to your comment. I'll implement a visitor generator for plugin ast.

[kdy1]

I found that prefix type requires using _Ref types everywhere, because of the layout issue you mentioned.
I think this can make implementing plugin harder, so I'll investigate.

[kdy1]

One downside of using plugin ast is split of the libraries - normal ast vs plugin ast

[dwoznicki]

Which libraries would use the normal ast? Anything outside of the core? Also, are you planning for the plugin ast to be a perfect subset of the normal ast, or will it have to be a little different?

[kdy1]

normal ast is ast used by core.
I'm not sure about the plugin ast at the moment.

[dwoznicki]

If the two ASTs are similar enough, perhaps you could make a single swc-ast crate and use #[cfg] to create normal and plugin compilation targets.