From Framework to notebook-kit: Cell‑Level Rendering and SSG Integration

[pkel]

Hi everybody and thanks for your fantastic work on the Observable ecosystem.

We work with Observable Framework to publish data-driven content (both dashboards and stories) as a static site. We will now try to scale our page to more dashboards (about 15-pages backed by 80 time series). The data is from many different domains, so the project will involve many people. Their skills range from "domain expert w/o programming knowledge" to "can do data analysis and visualization in Jupyter Notebooks" to "builds and publishes static web pages and does small adjustments to the default themes". We also consider to hire HTML/CSS experts for deeper adjustment to the theme.

For us, Framework has many strengths:

• secure and simple deployment as static site
• data loaders in arbitrary languages (Python!) help to convince and onboard domain experts; good decoupling between data-provider and content-editor
• integration of all the plotting and data transformation libraries enables quick iteration during design (e.g. duckdb transformation in JS for exploration, then move it server-side to optimize asset size and page load times)
• the tools for reactive programming (generators, inputs, identifying unbound variables in cells, and keeping them up to date) hits a sweet spot. People accustomed to Jupyter notebooks grok this quickly; much of the complexity is hidden. (Implementation details leak through sometimes, but IMO it's a good tradeoff overall)

We've also encountered a few rough edges that Observable Notebooks 2.0 and notebook-kit might be able to smooth out:

Rough Edge 1: We encounter a barrier between drafting content (editing notebooks) and publishing (editing markdown files, git versioning, npm run dev). The most actionable way so far was to let the domain experts draft their content in Jupyter/Python; then reimplement everything for publication in Framework. Ideally, both would use the same stack. To me it looks like Observable Notebooks 2.0 can be that technology: draft content in observable notebooks (online or with the new Desktop app) then hand it over to a developer for publication.

Rough Edge 2: As the guy who is responsible for the publication on the static site, I sometimes fight the markdown-first syntax of framework pages and/or the default theme. I think most of this is related to the interpretation of whitespace: How and why does it wrap stuff in <div>, <emph>, or <p>? How can I insert a code block inside a HTML structure, without it adding <p>? Why are code blocks inside html tags sometimes not recognized as such? Simple things like that. For me it would be easier if the entire page would be defined as html. Observable 2.0 seems to be doing exactly that.

Rough Edge 3: We need custom, reusable HTML components. E.g. for "plot on a flip-able dashboard card with additional info on the backside; screen reader shows backside info text for accessibility". The framework docs mention jsx and htl support. Unfortunately both methods render on the client and thus break some of the things we really like about Framework.

After briefly looking at Observable Notebook 2.0, Observable Desktop, and notebook-kit it looks like you are on the best way to resolve these issues. I see the following opportunities.

---

Opportunity 1: Extend notebook-kit such that it compiles Observable Notebook cells in arbitrary HTML documents

What I mean is, the <notebook>...</notebook> HTML is currently pretty strict: mostly a list of interactive cells. It would be convenient if a future notebook-kit can take a .html file as input, look for observable notebook cells, do its free-variable analysis + observable runtime magic, and spit out a reactive version of the page. This would allow us to use notebook-kit as a post-processor for any conventional static site generator. Rough Edge 3 could be offloaded to existing tools entirely. E.g. using Hugo Shortcode syntax, I could do:

---
theme: dashboard
title: Example Dashboard
grid-cols: 4
---

```js observable hidden
const data = ...
```

{{< card colspan=2 info="Text on the backside of the card" title="Card's Title" >}}
```js observable
Plot.rectY(data, {x: "letter", y: "frequency"}).plot()
```
{{< /card >}

...

which Hugo compiles to

<html>
  <head>...</head>
  <body>
    <h1>Example Dashboard</h2>
    <div class="grid grid-cols-4">
    <script observable hidden>
      const data = ...
    </script>
    <div class="card grid-colspan-2 with-info">
      <h2>Card's title</h2>
      <div class="info">
        Text on the backside of the card
      </div>
      <script observable>
        Plot.rectY(data, {x: "letter", y: "frequency"}).plot()
      </script>
    </div>
    ...
  </body>
</html>

before notebook-kit expands the simplified js into actual js, probably also adding some resources to <head>.

The pros here would be the unrestricted choice of static site generator, clear separation of client code and build-time templating, inheritance of static site generator features like related content, tagging, tag-pages, multi-author, multi-language, and so on.

---

Opportunity 2: Vite(press) plugin to render individual notebook cells

I see there is already an integration with vite, to render entire notebooks into an existing static site project. So I wonder, is it maybe more feasible to implement what I propose in Opportunity 1 directly in a js-based static site generator like vitepress instead of generic html post-processor? I don't mind about the syntax; instead of Hugo shortcodes I can as well write (SSR-compatible) vue components.

---

Looking forward to hear your opinions on this. Do you have similar problems and how do you solve them? What am I missing?

[mbostock]

Thanks for the great feedback! It’s lovely to learn in more detail about your experience, and we appreciate the suggestions.

Allowing notebook HTML to contain arbitrary structure is an interesting idea. Our current use of HTML as a serialization format isn’t intended to be user-extensible; it’s really just HTML so that it can leverage the default syntax highlighting and editing affordances of current code editors (and e.g. previewing HTML source files on GitHub). I don’t think this suggestion would work well with the Desktop editor — it wouldn’t be able to render or preserve the arbitrary content surrounding cells, or know where to insert new cells. So that would be a challenge.

Instead the approach we have been taking is to have a separate HTML template into which the notebook is rendered (but only outside of Desktop). This provides a stronger distinction between the content of the notebook and the rendered appearance as an application, which allows us to offer a better editing experience while still having customization when publishing.

So I think I’d like to continue that approach, but we could definitely make the HTML template more expressive and offer more control over where individual cells are rendered. As well as separate HTML templates for each notebook.

Also, I’ll point out that since Notebook Kit is open-source, you can do all of what you describe yourself by writing the code. 😅 Our primary focus is supporting our commercial business, so we’d love to know what we could offer in terms of an integrated editing, publishing, and collaborating service that you would pay for.