[Galene] Re: Background blur in Galene

[Kenichiro MATOHARA <matoken@gmail.com>]

Hello.

This is great.
I ran the Galene server on my Raspberry Pi tested it with Firefox on
my laptop (CPU: Intel Core i7-10510U/OS: Debian sid), and it worked
fine.
I'll try using it at the meeting tomorrow.

By the way, I found a small mistake in the INSTALL file.

$ git diff
diff --git a/INSTALL b/INSTALL
index 390037b..f60597b 100644
--- a/INSTALL
+++ b/INSTALL
@@ -38,6 +38,7 @@ disable the menu entries for background blur.
 Optionally install Google's MediaPipe library:
 ```
 mkdir mediapipe
+cd mediapipe
 npm pack @mediapipe/tasks-vision
 tar xzf mediapipe-tasks-vision-*.tgz
 rm -f ../static/third-party/tasks-vision
@@ -45,7 +46,7 @@ mv package ../static/third-party/tasks-vision
 cd ../static/third-party/tasks-vision
 mkdir models
 cd models
-wget wget https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter/float16/latest/selfie_segmenter.tflite
+wget https://storage.googleapis.com/mediapipe-models/image_segmenter/selfie_segmenter/float16/latest/selfie_segmenter.tflite
 cd ../../../../
 ```

2024年12月14日(土) 0:04 Juliusz Chroboczek <jch@irif.fr>:
>
> Hi,
>
> I've just merged an implementation of background blur based on Google's
> MediaPipe library into master.  Unless I got something wrong, the
> implementation has all of the features that I had planned:
>
>   - we don't ever contact Google's servers, all code is hosted locally;
>
>   - the library is loaded lazily, Galene still loads speedily and there's
>     no extra delay unless you select background blur;
>
>   - everything happens on the client, no unblurred video ever reaches the
>     server;
>
>   - the heavy lifting happens in a separate thread (a "web worker"), so
>     the interface remains responsive when blurring;
>
>   - when the client gets overwhelmed, we start dropping frames rather than
>     building a backlog;
>
>   - the Google's library is an optional install, if it's not present, then
>     the background blur menu entry is disabled.
>
> On my six-year-old laptop, I'm getting roughly 8fps after blurring, which
> is noticeably jerky but usable.  A single core is pinned at 100%, and the
> UI remains responsive.  Performance is likely to improve when the Google
> boys and girls implement WebGPU support in TFLite.
>
> Acknowledgements to Francis Bolduc, who showed me that this is possible.
>
>
> Try it out
> ==========
>
> Go to https://galene.org:8443/group/public/, open the side menu, choose
> Filters -> Background blur, hit Present.
>
>
> Install it on your server
> =========================
>
> First, update your installation of Galene to the latest master.  Then,
> perform the procedure described here:
>
>   https://galene.org/INSTALL.html#optional-install-background-blur
>
> It is okay to do that while the server is running.
>
>
> Possible extensions
> ===================
>
> Now that we have MediaPipe working within Galene, it should be easy to add
> additional filters, such as replacing the background with Dracula's
> castle, adding a top hat, or dying the speaker's hair.
>
> I don't find such features useful, but please let me know if you think
> otherwise.
>
>
> Technical details
> =================
>
> It was hard work, so please indulge me while I describe how the sausage
> was made.  Please feel free to skip the rest of this message.
>
>
> Local installation of MediaPipe
> -------------------------------
>
> Google's documentation suggests that you should load MediaPipe off their
> CDN, which is of course inacceptable for Galene.  Since running a local
> copy is not documented, it took me a fair amount of time to work out the
> following:
>
>   - MediaPipe sources are at https://github.com/google-ai-edge/mediapipe;
>     however, I never managed to compile their code, apparently I have the
>     wrong version of the TypeScript compiler;
>
>   - compiled MediaPipe code can be downloaded using npm, as described in
>     the INTALL document; I have no evidence that it corresponds to the
>     github code, but I also have no evidence that Google are evil;
>
>   - even though MediaPipe's JavaScript is an ECMAScript module, it doesn't
>     work when invoked from a web worker implemented itself as an
>     ECMAScript module; after fighting with that for a couple of days,
>     I switched to a traditional (pre-ECMAScript 6) web worker;
>
>   - MediaPipe's JavaScript uses the "eval" function (!), which is not
>     allowed in Galene; there's a special case for that in the Go code:
>
>     https://github.com/jech/galene/blob/master/webserver/webserver.go#L214
>     https://github.com/jech/galene/blob/master/webserver/webserver.go#L97
>
>
> Lazy loading
> ------------
>
> I rather like the fast load times of Galene, and so didn't want to load
> the MediaPipe code until the user explicitly requests background blur.
> Additionally, since some people might not trust Google, I wanted to keep
> the installation optional.
>
> I was initially under the impression that lazy loading required switching
> all of Galene's code to ECMAScript modules, and started doing just that.
> Fortunately, it turns out that there are at least two ways of loading
> JavaScript lazily from traditional JavaScript:
>
>   - dynamic import, which loads an ECMASCript module into non-module code;
>
>   - web workers, which are only instantiated at the time the WebWorker
>     constructor is called.
>
> We use both: when the user selects background blur, we instantiate a web
> worker, and the web worker performs a dynamic import of the MediaPipe
> library.  The effect is that the first time a user chooses background
> blur, their video will freeze for up to a few seconds while the libraries
> are being loaded; however, since this happens in the web worker, the user
> interface remains responsive and the other videos keep playing.  (After
> that, there are no further delays as long as the libraries remain in the
> browser's cache.)
>
>
> Running segmentation and building the blurred frame
> ---------------------------------------------------
>
> Whenever we have a new frame, we check whether the worker is busy.  If
> that's the case, we drop the frame, and continue on our merry way.  If the
> worker is not busy, we copy the current frame to a bitmap, and pass it to
> the worker.
>
> MediaPipe's segmentation algorithm produces a mask as a Uint8Array, which
> the worker converts to an alpha mask and returns it to Galene's main loop.
> We then perform blurring and compositing using an HTML Canvas, which (at
> last on my machine) happens entirely on the GPU.
>
>
> -- Juliusz Chroboczek
> _______________________________________________
> Galene mailing list -- galene@lists.galene.org
> To unsubscribe send an email to galene-leave@lists.galene.org



-- 
KenichiroMATOHARA
mailto:matoken@gmail.com
https://matoken.org/
.O.
..O
OOO