shapeshifter: replace shift with acorn

[felixroos]

It could be more practical to move to babel. Reasons:

• non hacky support for top level await and ES2020
• rich plugin system

as briefly discussed in #171 :

Maybe we could move to another parser / transformer lib in the future, like babel standalone (which is based on acorn, which supports ES2020) or if that's too heavy just the packages we need (@babel/parser @babel/traverse @bable/type @babel/generate + any plugin packages we need, like top level await.

I think the permanent solution is to switch to @babel/parser, it's strongly optimized for this kind of quick and simple AST manipulation, extremely battle-tested & well documented + you can even support sexy experimental JS syntax if you like (e.g. https://github.com/tc39/proposal-pipeline-operator)

would probably also fix #79

edit: another (faster) alternative would be SWC: https://swc.rs/docs/usage/wasm

edit: see shootout below

[felixroos]

just made a little shootout between

• shift
• babel
• acorn
• swc

shift

shift is what is currently used in strudel.

Size: 426.52 KiB / gzip: 72.50 KiB

show code

import { codeGen } from 'shift-codegen';
import { parseScript } from 'shift-parser';
import { replace } from 'shift-traverser';
import { LiteralStringExpression } from 'shift-ast';

const before = 'sequence(C3,g3)';

const isNote = (name) => /^[a-gC-G][bs]?[0-9]$/.test(name);

const after = codeGen(
  replace(parseScript(before), {
    enter(node, parent) {
      if (node.type === 'IdentifierExpression' && isNote(node.name)) {
        return new LiteralStringExpression({ value: node.name });
      }
    },
  }),
);

console.log(after);

babel standalone

A babel build for the browser.

Size: 2665.97 KiB / gzip: 586.26 KiB

show code

import * as Babel from '@babel/standalone';

const input = 'const getMessage = () => "Hello World";';
const output = Babel.transform(input, { presets: ['env'] }).code;

babel

not really designed for the browser, but potentially more lightweight than standalone when it works..

Size: probably 312kB gzipped, according to "Import Cost" plugin. The packages are not really meant for the browser so they fail with vite / rollup. Code used:

show code

import { parse } from '@babel/parser';
import traverse from '@babel/traverse';
import generate from '@babel/generator';

function transform(code, options, visitor) {
  const ast = parse(code, options);
  console.log(ast);
  traverse(ast, visitor);
  return generate(ast, { sourceMaps: true });
}

const input = `
const x = await y();
let n = 1
`;

const output = transform(
  input,
  { sourceType: 'module' },
  {
    enter(path) {
      if (path.isIdentifier({ name: 'n' })) {
        path.node.name = 'x';
      }
    },
  },
);
console.log(output);

This code actually runs in the dev server, but the build fails..

swc

The docs are pretty thin, also, the wasm file is 20,1MB ....

acorn

A popular estree parser written in JS.

Size: 206.41 KiB / gzip: 57.56 KiB

show code

import escodegen from 'escodegen';
import { parse } from 'acorn';
import { walk } from 'estree-walker';

const input = `let x = 2;const y = 5`;

const ast = parse(input, {
  ecmaVersion: 2022,
  allowAwaitOutsideFunction: true,
});

let n = 0;

console.log(ast);

walk(ast, {
  enter(node, parent, prop, index) {
    console.log(node);
    if (node.type === 'Identifier') {
      node.name = 'v' + n;
      n++;
    }
  },
  leave(node, parent, prop, index) {},
});

console.log(ast);

const output = escodegen.generate(ast);

console.log(output);

Ranking

1. acorn

• smallest (57.56 KiB)
• supports new es features like TLA
• is maintained
• implements widely used estree AST format

2. shift

• small (72kB)
• does not support newer es features like TLA
• not fully maintained
• implements a conceptually better (they say), but less used AST

3. babel standalone

• pretty big (586.26 KiB)
• does support newer es feature like TLA
• fully maintained
• implements a custom estree fork (they say)
• monolithic package with tones of features (of which we don't need most)

4. swc

• insanely huge (21MB)
• did not even test after seeing the sparse docs and huge wasm file