[Galene] Background blur in Galene

[Galene] Background blur in Galene

[Juliusz Chroboczek]

Hi,

I've just merged an implementation of background blur based on Google's
MediaPipe library into master.  Unless I got something wrong, the
implementation has all of the features that I had planned:

  - we don't ever contact Google's servers, all code is hosted locally;

  - the library is loaded lazily, Galene still loads speedily and there's
    no extra delay unless you select background blur;

  - everything happens on the client, no unblurred video ever reaches the
    server;

  - the heavy lifting happens in a separate thread (a "web worker"), so
    the interface remains responsive when blurring;

  - when the client gets overwhelmed, we start dropping frames rather than
    building a backlog;

  - the Google's library is an optional install, if it's not present, then
    the background blur menu entry is disabled.

On my six-year-old laptop, I'm getting roughly 8fps after blurring, which
is noticeably jerky but usable.  A single core is pinned at 100%, and the
UI remains responsive.  Performance is likely to improve when the Google
boys and girls implement WebGPU support in TFLite.

Acknowledgements to Francis Bolduc, who showed me that this is possible.


Try it out
==========

Go to https://galene.org:8443/group/public/, open the side menu, choose
Filters -> Background blur, hit Present.


Install it on your server
=========================

First, update your installation of Galene to the latest master.  Then,
perform the procedure described here:

  https://galene.org/INSTALL.html#optional-install-background-blur

It is okay to do that while the server is running.


Possible extensions
===================

Now that we have MediaPipe working within Galene, it should be easy to add
additional filters, such as replacing the background with Dracula's
castle, adding a top hat, or dying the speaker's hair.

I don't find such features useful, but please let me know if you think
otherwise.


Technical details
=================

It was hard work, so please indulge me while I describe how the sausage
was made.  Please feel free to skip the rest of this message.


Local installation of MediaPipe
-------------------------------

Google's documentation suggests that you should load MediaPipe off their
CDN, which is of course inacceptable for Galene.  Since running a local
copy is not documented, it took me a fair amount of time to work out the
following:

  - MediaPipe sources are at https://github.com/google-ai-edge/mediapipe;
    however, I never managed to compile their code, apparently I have the
    wrong version of the TypeScript compiler;

  - compiled MediaPipe code can be downloaded using npm, as described in
    the INTALL document; I have no evidence that it corresponds to the
    github code, but I also have no evidence that Google are evil;

  - even though MediaPipe's JavaScript is an ECMAScript module, it doesn't
    work when invoked from a web worker implemented itself as an
    ECMAScript module; after fighting with that for a couple of days,
    I switched to a traditional (pre-ECMAScript 6) web worker;

  - MediaPipe's JavaScript uses the "eval" function (!), which is not
    allowed in Galene; there's a special case for that in the Go code:

    https://github.com/jech/galene/blob/master/webserver/webserver.go#L214
    https://github.com/jech/galene/blob/master/webserver/webserver.go#L97


Lazy loading
------------

I rather like the fast load times of Galene, and so didn't want to load
the MediaPipe code until the user explicitly requests background blur.
Additionally, since some people might not trust Google, I wanted to keep
the installation optional.

I was initially under the impression that lazy loading required switching
all of Galene's code to ECMAScript modules, and started doing just that.
Fortunately, it turns out that there are at least two ways of loading
JavaScript lazily from traditional JavaScript:

  - dynamic import, which loads an ECMASCript module into non-module code;

  - web workers, which are only instantiated at the time the WebWorker
    constructor is called.

We use both: when the user selects background blur, we instantiate a web
worker, and the web worker performs a dynamic import of the MediaPipe
library.  The effect is that the first time a user chooses background
blur, their video will freeze for up to a few seconds while the libraries
are being loaded; however, since this happens in the web worker, the user
interface remains responsive and the other videos keep playing.  (After
that, there are no further delays as long as the libraries remain in the
browser's cache.)


Running segmentation and building the blurred frame
---------------------------------------------------

Whenever we have a new frame, we check whether the worker is busy.  If
that's the case, we drop the frame, and continue on our merry way.  If the
worker is not busy, we copy the current frame to a bitmap, and pass it to
the worker.

MediaPipe's segmentation algorithm produces a mask as a Uint8Array, which
the worker converts to an alpha mask and returns it to Galene's main loop.
We then perform blurring and compositing using an HTML Canvas, which (at
last on my machine) happens entirely on the GPU.


-- Juliusz Chroboczek

[Galene] Re: Background blur in Galene

[Kenichiro MATOHARA]

Hello.

This is great.
I ran the Galene server on my Raspberry Pi tested it with Firefox on
my laptop (CPU: Intel Core i7-10510U/OS: Debian sid), and it worked
fine.
I'll try using it at the meeting tomorrow.

By the way, I found a small mistake in the INSTALL file.

$ git diff
diff --git a/INSTALL b/INSTALL
index 390037b..f60597b 100644
--- a/INSTALL
+++ b/INSTALL
@@ -38,6 +38,7 @@ disable the menu entries for background blur.
 Optionally install Google's MediaPipe library:
 ```
 mkdir mediapipe
+cd mediapipe
 npm pack @mediapipe/tasks-vision
 tar xzf mediapipe-tasks-vision-*.tgz
 rm -f ../static/third-party/tasks-vision
@@ -45,7 +46,7 @@ mv package ../static/third-party/tasks-vision
 cd ../static/third-party/tasks-vision
 mkdir models
 cd models
-wget wget https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter/float16/latest/selfie_segmenter.tflite
+wget https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter/float16/latest/selfie_segmenter.tflite
 cd ../../../../
 ```

2024年12月14日(土) 0:04 Juliusz Chroboczek <jch@irif.fr>:
>
> Hi,
>
> I've just merged an implementation of background blur based on Google's
> MediaPipe library into master.  Unless I got something wrong, the
> implementation has all of the features that I had planned:
>
>   - we don't ever contact Google's servers, all code is hosted locally;
>
>   - the library is loaded lazily, Galene still loads speedily and there's
>     no extra delay unless you select background blur;
>
>   - everything happens on the client, no unblurred video ever reaches the
>     server;
>
>   - the heavy lifting happens in a separate thread (a "web worker"), so
>     the interface remains responsive when blurring;
>
>   - when the client gets overwhelmed, we start dropping frames rather than
>     building a backlog;
>
>   - the Google's library is an optional install, if it's not present, then
>     the background blur menu entry is disabled.
>
> On my six-year-old laptop, I'm getting roughly 8fps after blurring, which
> is noticeably jerky but usable.  A single core is pinned at 100%, and the
> UI remains responsive.  Performance is likely to improve when the Google
> boys and girls implement WebGPU support in TFLite.
>
> Acknowledgements to Francis Bolduc, who showed me that this is possible.
>
>
> Try it out
> ==========
>
> Go to https://galene.org:8443/group/public/, open the side menu, choose
> Filters -> Background blur, hit Present.
>
>
> Install it on your server
> =========================
>
> First, update your installation of Galene to the latest master.  Then,
> perform the procedure described here:
>
>   https://galene.org/INSTALL.html#optional-install-background-blur
>
> It is okay to do that while the server is running.
>
>
> Possible extensions
> ===================
>
> Now that we have MediaPipe working within Galene, it should be easy to add
> additional filters, such as replacing the background with Dracula's
> castle, adding a top hat, or dying the speaker's hair.
>
> I don't find such features useful, but please let me know if you think
> otherwise.
>
>
> Technical details
> =================
>
> It was hard work, so please indulge me while I describe how the sausage
> was made.  Please feel free to skip the rest of this message.
>
>
> Local installation of MediaPipe
> -------------------------------
>
> Google's documentation suggests that you should load MediaPipe off their
> CDN, which is of course inacceptable for Galene.  Since running a local
> copy is not documented, it took me a fair amount of time to work out the
> following:
>
>   - MediaPipe sources are at https://github.com/google-ai-edge/mediapipe;
>     however, I never managed to compile their code, apparently I have the
>     wrong version of the TypeScript compiler;
>
>   - compiled MediaPipe code can be downloaded using npm, as described in
>     the INTALL document; I have no evidence that it corresponds to the
>     github code, but I also have no evidence that Google are evil;
>
>   - even though MediaPipe's JavaScript is an ECMAScript module, it doesn't
>     work when invoked from a web worker implemented itself as an
>     ECMAScript module; after fighting with that for a couple of days,
>     I switched to a traditional (pre-ECMAScript 6) web worker;
>
>   - MediaPipe's JavaScript uses the "eval" function (!), which is not
>     allowed in Galene; there's a special case for that in the Go code:
>
>     https://github.com/jech/galene/blob/master/webserver/webserver.go#L214
>     https://github.com/jech/galene/blob/master/webserver/webserver.go#L97
>
>
> Lazy loading
> ------------
>
> I rather like the fast load times of Galene, and so didn't want to load
> the MediaPipe code until the user explicitly requests background blur.
> Additionally, since some people might not trust Google, I wanted to keep
> the installation optional.
>
> I was initially under the impression that lazy loading required switching
> all of Galene's code to ECMAScript modules, and started doing just that.
> Fortunately, it turns out that there are at least two ways of loading
> JavaScript lazily from traditional JavaScript:
>
>   - dynamic import, which loads an ECMASCript module into non-module code;
>
>   - web workers, which are only instantiated at the time the WebWorker
>     constructor is called.
>
> We use both: when the user selects background blur, we instantiate a web
> worker, and the web worker performs a dynamic import of the MediaPipe
> library.  The effect is that the first time a user chooses background
> blur, their video will freeze for up to a few seconds while the libraries
> are being loaded; however, since this happens in the web worker, the user
> interface remains responsive and the other videos keep playing.  (After
> that, there are no further delays as long as the libraries remain in the
> browser's cache.)
>
>
> Running segmentation and building the blurred frame
> ---------------------------------------------------
>
> Whenever we have a new frame, we check whether the worker is busy.  If
> that's the case, we drop the frame, and continue on our merry way.  If the
> worker is not busy, we copy the current frame to a bitmap, and pass it to
> the worker.
>
> MediaPipe's segmentation algorithm produces a mask as a Uint8Array, which
> the worker converts to an alpha mask and returns it to Galene's main loop.
> We then perform blurring and compositing using an HTML Canvas, which (at
> last on my machine) happens entirely on the GPU.
>
>
> -- Juliusz Chroboczek
> _______________________________________________
> Galene mailing list -- galene@lists.galene.org
> To unsubscribe send an email to galene-leave@lists.galene.org



-- 
KenichiroMATOHARA
mailto:matoken@gmail.com
https://matoken.org/
.O.
..O
OOO

[Galene] Re: Background blur in Galene

[Juliusz Chroboczek]

> it worked fine.

[...]

> I'll try using it at the meeting tomorrow.

Please do, and let us know how it goes.

> By the way, I found a small mistake in the INSTALL file.

Thanks, applied.

-- Juliusz

[Galene] Re: Background blur in Galene

[Dirk-Willem van Gulik]

On 13 Dec 2024, at 22:45, Kenichiro MATOHARA <matoken@gmail.com> wrote:

> I ran the Galene server on my Raspberry Pi tested it with Firefox on
> my laptop (CPU: Intel Core i7-10510U/OS: Debian sid), and it worked

Brave & Chrome on various recent flavours of OSX works fine.

Small observation - the colours seem very a bit faded-out/dull when in the main window; and brighten up to normal in the ‘picture in picture’ display.

Dw

[Galene] Re: Background blur in Galene

[Juliusz Chroboczek]

>> I ran the Galene server on my Raspberry Pi tested it with Firefox on
>> my laptop (CPU: Intel Core i7-10510U/OS: Debian sid), and it worked

> Brave & Chrome on various recent flavours of OSX works fine.

Good.

> Small observation - the colours seem very a bit faded-out/dull when in
> the main window; and brighten up to normal in the ‘picture in picture’
> display.

Strange.  Are you sure it's not due to the screen's brightness changing
when you switch to PiP, and the light reflecting on your face?

-- Juliusz