collaborative editing

[eilseq]

I think would be interesting considering an additional package to the structure that provides real-time audio and data communication. A typical use case would be a session of multiple players collaborating on the same session.

With this application in mind, I think the library should allow a user to initiate a public session and invite other players using the generated link. This means that the library would be in charge of defining a session and return a URL invitation.

The negotiation between peers can happen without a server, despite of the topology. There are few libraries that makes this process very easy. They take over most of the required coordination to instantiate a p2p stream, reducing a lot the amount of code needed to define a real-time session.

Suggested: https://github.com/peers/peerjs

[felixroos]

We definitely need collaborative editing! This issue should possibly wait for #28 , as it is tightly coupled with the editor implementation. Omicron666 also mentioned we could use yjs, which is also used by glicol and gibber. There are already codemirror 6 bindings in the making.

[eilseq]

Perhaps we can start with audio streaming. The main goal would be to capture and stream external sources, eventually driven but the underline collaborative code. This is a typical use case in ensembles based on external midi gear or additional SynthDefs in supercollider. I would say that in terms of architecture, might also be a good starting point.

[felixroos]

Audio streaming alone would be nice too, but we should keep in mind that if we add collaborative editing later, it would ideally run through the same p2p connection as the audio (?). I imagine collaborative editing can be tricky to get right when writing from scratch (opposed to using readymade solution like yjs), but I might be wrong. At least something to be aware of..

[eilseq]

It makes sense, but I still wouldn't prioritise the collaborative editor over the underlying real-time communication strategy. For example: if instead of PeerJS we use y-webrtc I think we would be able to retrieve some sort of session context that y-* packages rely on.

Main example for y-codemirror, clearly expose a provider based on webrtc:

import * as Y from 'yjs'
import { CodemirrorBinding } from 'y-codemirror'
import { WebrtcProvider } from 'y-webrtc'
import CodeMirror from 'codemirror'

const ydoc = new Y.Doc()
const provider = new WebrtcProvider('codemirror-demo-room', ydoc)
const yText = ydoc.getText('codemirror')
const yUndoManager = new Y.UndoManager(yText)

const editor = CodeMirror(editorDiv, {
  mode: 'javascript',
  lineNumbers: true
})

const binding = new CodemirrorBinding(yText, editor, provider.awareness, { yUndoManager })

In the class WebrtcProvider, the peer field is an instance of Peer from the simple-peer library. According to this example, Peer have an easy entry point for attaching a media stream to an existing instance. In the context of WebrtcProvider, that instance would be located under provider.peer:

var Peer = require('simple-peer')

// get video/voice stream
navigator.mediaDevices.getUserMedia({
  video: true,
  audio: true
}).then(gotMedia).catch(() => {})

function gotMedia (stream) {
  var peer1 = new Peer({ initiator: true, stream: stream })
  var peer2 = new Peer()

  peer1.on('signal', data => {
    peer2.signal(data)
  })

  peer2.on('signal', data => {
    peer1.signal(data)
  })

  peer2.on('stream', stream => {
    // got remote video stream, now let's show it in a video tag
    var video = document.querySelector('video')

    if ('srcObject' in video) {
      video.srcObject = stream
    } else {
      video.src = window.URL.createObjectURL(stream) // for older browsers
    }

    video.play()
  })
}

[felixroos]

great! looks like compatibility is given with minor adjustments. so nothing to worry with peerjs i guess.

[eilseq]

Something like this would probably do. Can be a function provided by the package:

...
const ydoc = new Y.Doc()
const provider = new WebrtcProvider('codemirror-demo-room', ydoc)
const yText = ydoc.getText('codemirror')
const yUndoManager = new Y.UndoManager(yText)

...
export const bindCodemirrorEditor = (editor) => 
    new CodemirrorBinding(yText, editor, provider.awareness, { yUndoManager })

[eilseq]

I'm un-assigning myself. Due to conflict of interest with my current position I cannot publish open-source code related to this particular subject.

[felixroos]

tried out yjs the other day: https://codesandbox.io/s/collab-editor-demo-cge8hx?file=/src/index.js

[boourns]

I use yjs on sequencer.party for syncronizing any text documents, as well as the state of each WAM plugin. It is stable, in that when calling the functions, they return as expected and do what you would expect.

However, because every document edit is saved for reversability, documents grow in size quickly and subsequent edits begin to take longer and longer. This quickly collides with audio scheduling on the main thread and you will have sequencer skips every time the document is updated.

The solution for me was to move all yjs calls to their own worker thread. I used and enjoyed https://threads.js.org.

[felixroos]

I use yjs on sequencer.party for syncronizing any text documents, as well as the state of each WAM plugin. It is stable, in that when calling the functions, they return as expected and do what you would expect.

However, because every document edit is saved for reversability, documents grow in size quickly and subsequent edits begin to take longer and longer. This quickly collides with audio scheduling on the main thread and you will have sequencer skips every time the document is updated.

The solution for me was to move all yjs calls to their own worker thread. I used and enjoyed https://threads.js.org.

good to know, thanks for the hint. there is also comlink

[felixroos]

just dumping this from chat here, so it won't get forgotten

yaxu — 03/01/2023 11:57 AM
Just thinking about collaborative editing again. I think this is the state of things:

• There's this issue ostensibly about sharing audio but mostly about collaborative editing: collaborative editing #33
• Flok is also working nicely (although having some issues on chrome for mac), and could be enhanced to add trigger highlighting
• Looking at @Froos 's demo, it all looks pretty straightforward to implement with codemirror 6 https://codesandbox.io/s/collab-editor-demo-cge8hx?file=/src/index.js

froos — 03/01/2023 12:02 PM
sharing a single source of truth with multiple people might only be usable with block based evaluation
so either use the flok / estuary approach of one editor per user oder make sure you can evaluate individual blocks
I kind of like the idea of having a single editor instead of multiple
it would still be cool to work a bit on the flok integration

yaxu — 03/01/2023 12:45 PM
yes agreed on single editor + block-based eval would be nice.. although there could be a kind of hybrid like feedforward where blocks are automatically treated as separate patterns by the editor.. so can act a bit like separate editors (can be muted, visualised separately etc)

froos — 03/01/2023 12:53 PM
jep.. I don't have a clear picture of how it should work. it should generally be avoided to accidentally evaluate the stuff another person is writing at the moment

yaxu — 03/01/2023 12:56 PM
yes unless there is some kind of traffic light system, where instead of evaluating, you indicate you are ready for the next evaluation.. but that doesn't happen until everyone is ready
not sure how practical that would be and yep block-based probably better