[Galene] Background blur in Galene

[Juliusz Chroboczek <jch@irif.fr>]

Hi,

I've just merged an implementation of background blur based on Google's
MediaPipe library into master.  Unless I got something wrong, the
implementation has all of the features that I had planned:

  - we don't ever contact Google's servers, all code is hosted locally;

  - the library is loaded lazily, Galene still loads speedily and there's
    no extra delay unless you select background blur;

  - everything happens on the client, no unblurred video ever reaches the
    server;

  - the heavy lifting happens in a separate thread (a "web worker"), so
    the interface remains responsive when blurring;

  - when the client gets overwhelmed, we start dropping frames rather than
    building a backlog;

  - the Google's library is an optional install, if it's not present, then
    the background blur menu entry is disabled.

On my six-year-old laptop, I'm getting roughly 8fps after blurring, which
is noticeably jerky but usable.  A single core is pinned at 100%, and the
UI remains responsive.  Performance is likely to improve when the Google
boys and girls implement WebGPU support in TFLite.

Acknowledgements to Francis Bolduc, who showed me that this is possible.


Try it out
==========

Go to https://galene.org:8443/group/public/, open the side menu, choose
Filters -> Background blur, hit Present.


Install it on your server
=========================

First, update your installation of Galene to the latest master.  Then,
perform the procedure described here:

  https://galene.org/INSTALL.html#optional-install-background-blur

It is okay to do that while the server is running.


Possible extensions
===================

Now that we have MediaPipe working within Galene, it should be easy to add
additional filters, such as replacing the background with Dracula's
castle, adding a top hat, or dying the speaker's hair.

I don't find such features useful, but please let me know if you think
otherwise.


Technical details
=================

It was hard work, so please indulge me while I describe how the sausage
was made.  Please feel free to skip the rest of this message.


Local installation of MediaPipe
-------------------------------

Google's documentation suggests that you should load MediaPipe off their
CDN, which is of course inacceptable for Galene.  Since running a local
copy is not documented, it took me a fair amount of time to work out the
following:

  - MediaPipe sources are at https://github.com/google-ai-edge/mediapipe;
    however, I never managed to compile their code, apparently I have the
    wrong version of the TypeScript compiler;

  - compiled MediaPipe code can be downloaded using npm, as described in
    the INTALL document; I have no evidence that it corresponds to the
    github code, but I also have no evidence that Google are evil;

  - even though MediaPipe's JavaScript is an ECMAScript module, it doesn't
    work when invoked from a web worker implemented itself as an
    ECMAScript module; after fighting with that for a couple of days,
    I switched to a traditional (pre-ECMAScript 6) web worker;

  - MediaPipe's JavaScript uses the "eval" function (!), which is not
    allowed in Galene; there's a special case for that in the Go code:

    https://github.com/jech/galene/blob/master/webserver/webserver.go#L214
    https://github.com/jech/galene/blob/master/webserver/webserver.go#L97


Lazy loading
------------

I rather like the fast load times of Galene, and so didn't want to load
the MediaPipe code until the user explicitly requests background blur.
Additionally, since some people might not trust Google, I wanted to keep
the installation optional.

I was initially under the impression that lazy loading required switching
all of Galene's code to ECMAScript modules, and started doing just that.
Fortunately, it turns out that there are at least two ways of loading
JavaScript lazily from traditional JavaScript:

  - dynamic import, which loads an ECMASCript module into non-module code;

  - web workers, which are only instantiated at the time the WebWorker
    constructor is called.

We use both: when the user selects background blur, we instantiate a web
worker, and the web worker performs a dynamic import of the MediaPipe
library.  The effect is that the first time a user chooses background
blur, their video will freeze for up to a few seconds while the libraries
are being loaded; however, since this happens in the web worker, the user
interface remains responsive and the other videos keep playing.  (After
that, there are no further delays as long as the libraries remain in the
browser's cache.)


Running segmentation and building the blurred frame
---------------------------------------------------

Whenever we have a new frame, we check whether the worker is busy.  If
that's the case, we drop the frame, and continue on our merry way.  If the
worker is not busy, we copy the current frame to a bitmap, and pass it to
the worker.

MediaPipe's segmentation algorithm produces a mask as a Uint8Array, which
the worker converts to an alpha mask and returns it to Galene's main loop.
We then perform blurring and compositing using an HTML Canvas, which (at
last on my machine) happens entirely on the GPU.


-- Juliusz Chroboczek