From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mail.toke.dk; spf=pass (mailfrom) smtp.mailfrom=irif.fr (client-ip=2001:660:3301:8000::1:2; helo=korolev.univ-paris7.fr; envelope-from=jch@irif.fr; receiver=) Authentication-Results: mail.toke.dk; dkim=pass (2048-bit key; unprotected) header.d=irif.fr header.i=@irif.fr header.a=rsa-sha256 header.s=dkim-irif header.b=F9bbir/J Received: from korolev.univ-paris7.fr (korolev.univ-paris7.fr [IPv6:2001:660:3301:8000::1:2]) by mail.toke.dk (Postfix) with ESMTPS id 2BCE9AB4EEE for ; Fri, 13 Dec 2024 16:04:19 +0100 (CET) Received: from mailhub.math.univ-paris-diderot.fr (mailhub.math.univ-paris-diderot.fr [81.194.30.253]) by korolev.univ-paris7.fr (8.14.4/8.14.4/relay1/82085) with ESMTP id 4BDDbnI4030377 for ; Fri, 13 Dec 2024 14:37:49 +0100 Received: from mailhub.math.univ-paris-diderot.fr (localhost [127.0.0.1]) by mailhub.math.univ-paris-diderot.fr (Postfix) with ESMTP id B66CFAABEC for ; Fri, 13 Dec 2024 14:37:49 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=irif.fr; h= content-type:content-type:mime-version:user-agent:subject :subject:from:from:message-id:date:date:resent-date:resent-from :received:received; s=dkim-irif; t=1734097068; x=1734961069; bh= W27QK37CVx0os0KgNFyqXQzqnFjQ5Lg+jhCL1bm/X3Q=; b=F9bbir/J7EoAZj6c Xc8tNoL557Sms7W1PSfkLN1USNTVLDX3rKq3ByQkg2xqzgoghQFCzTBEeNZMfLR0 228OmFEOTD+V0KjLkz01UtmNVgHN33jpspspGGFQ+32G/ecg74W+RvNX0xvVCajH T6toI/BEAG8qtAlAumuVqW7aevTnFluvfRDhQB4K3A3fXTbOerrGTJgNxPMQazhF zaITE0gVWhPYSiGm19wuL2SsQoy9TRPuaViQCRxS6bJdN6KspS30noSgPjDzIEn3 /TBO1hUPCH3WI5vpteIyk66Ehg7lFF1F6O+OWBVI8E90fwtYkunjJLxk6cg9pHDf X2mpgw== X-Virus-Scanned: amavisd-new at math.univ-paris-diderot.fr Received: from mailhub.math.univ-paris-diderot.fr ([127.0.0.1]) by mailhub.math.univ-paris-diderot.fr (mailhub.math.univ-paris-diderot.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id pSiuTbLfTPkj for ; Fri, 13 Dec 2024 14:37:48 +0100 (CET) Received: from pirx.irif.fr (unknown [37.175.127.120]) (Authenticated sender: jch) by mailhub.math.univ-paris-diderot.fr (Postfix) with ESMTPSA id 428B6AAF0D for ; Fri, 13 Dec 2024 14:37:48 +0100 (CET) Resent-From: Juliusz Chroboczek Resent-Date: Fri, 13 Dec 2024 14:37:47 +0100 Resent-To: galene@lists.galene.org Date: Fri, 13 Dec 2024 13:37:00 +0100 Message-ID: <87pllv69oz.wl-jch@irif.fr> From: Juliusz Chroboczek To: galene@lists.galene.org User-Agent: Wanderlust/2.15.9 (Almost Unreal) Emacs/29.4 Mule/6.0 MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII Resent-Message-Id: <20241213133749.B66CFAABEC@mailhub.math.univ-paris-diderot.fr> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (korolev.univ-paris7.fr [194.254.61.138]); Fri, 13 Dec 2024 14:37:50 +0100 (CET) X-Miltered: at korolev with ID 675C38AD.000 by Joe's j-chkmail (http : // j-chkmail dot ensmp dot fr)! X-j-chkmail-Enveloppe: 675C38AD.000 from mailhub.math.univ-paris-diderot.fr/mailhub.math.univ-paris-diderot.fr/null/mailhub.math.univ-paris-diderot.fr/ X-j-chkmail-Score: MSGID : 675C38AD.000 on korolev.univ-paris7.fr : j-chkmail score : . : R=. U=. O=. B=0.000 -> S=0.000 X-j-chkmail-Status: Ham Message-ID-Hash: GAOABKWVGQTGODGPX4F6XYLUZWJE7THV X-Message-ID-Hash: GAOABKWVGQTGODGPX4F6XYLUZWJE7THV X-MailFrom: jch@irif.fr X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list Subject: [Galene] Background blur in Galene List-Id: =?utf-8?q?Gal=C3=A8ne_videoconferencing_server_discussion_list?= Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi, I've just merged an implementation of background blur based on Google's MediaPipe library into master. Unless I got something wrong, the implementation has all of the features that I had planned: - we don't ever contact Google's servers, all code is hosted locally; - the library is loaded lazily, Galene still loads speedily and there's no extra delay unless you select background blur; - everything happens on the client, no unblurred video ever reaches the server; - the heavy lifting happens in a separate thread (a "web worker"), so the interface remains responsive when blurring; - when the client gets overwhelmed, we start dropping frames rather than building a backlog; - the Google's library is an optional install, if it's not present, then the background blur menu entry is disabled. On my six-year-old laptop, I'm getting roughly 8fps after blurring, which is noticeably jerky but usable. A single core is pinned at 100%, and the UI remains responsive. Performance is likely to improve when the Google boys and girls implement WebGPU support in TFLite. Acknowledgements to Francis Bolduc, who showed me that this is possible. Try it out ========== Go to https://galene.org:8443/group/public/, open the side menu, choose Filters -> Background blur, hit Present. Install it on your server ========================= First, update your installation of Galene to the latest master. Then, perform the procedure described here: https://galene.org/INSTALL.html#optional-install-background-blur It is okay to do that while the server is running. Possible extensions =================== Now that we have MediaPipe working within Galene, it should be easy to add additional filters, such as replacing the background with Dracula's castle, adding a top hat, or dying the speaker's hair. I don't find such features useful, but please let me know if you think otherwise. Technical details ================= It was hard work, so please indulge me while I describe how the sausage was made. Please feel free to skip the rest of this message. Local installation of MediaPipe ------------------------------- Google's documentation suggests that you should load MediaPipe off their CDN, which is of course inacceptable for Galene. Since running a local copy is not documented, it took me a fair amount of time to work out the following: - MediaPipe sources are at https://github.com/google-ai-edge/mediapipe; however, I never managed to compile their code, apparently I have the wrong version of the TypeScript compiler; - compiled MediaPipe code can be downloaded using npm, as described in the INTALL document; I have no evidence that it corresponds to the github code, but I also have no evidence that Google are evil; - even though MediaPipe's JavaScript is an ECMAScript module, it doesn't work when invoked from a web worker implemented itself as an ECMAScript module; after fighting with that for a couple of days, I switched to a traditional (pre-ECMAScript 6) web worker; - MediaPipe's JavaScript uses the "eval" function (!), which is not allowed in Galene; there's a special case for that in the Go code: https://github.com/jech/galene/blob/master/webserver/webserver.go#L214 https://github.com/jech/galene/blob/master/webserver/webserver.go#L97 Lazy loading ------------ I rather like the fast load times of Galene, and so didn't want to load the MediaPipe code until the user explicitly requests background blur. Additionally, since some people might not trust Google, I wanted to keep the installation optional. I was initially under the impression that lazy loading required switching all of Galene's code to ECMAScript modules, and started doing just that. Fortunately, it turns out that there are at least two ways of loading JavaScript lazily from traditional JavaScript: - dynamic import, which loads an ECMASCript module into non-module code; - web workers, which are only instantiated at the time the WebWorker constructor is called. We use both: when the user selects background blur, we instantiate a web worker, and the web worker performs a dynamic import of the MediaPipe library. The effect is that the first time a user chooses background blur, their video will freeze for up to a few seconds while the libraries are being loaded; however, since this happens in the web worker, the user interface remains responsive and the other videos keep playing. (After that, there are no further delays as long as the libraries remain in the browser's cache.) Running segmentation and building the blurred frame --------------------------------------------------- Whenever we have a new frame, we check whether the worker is busy. If that's the case, we drop the frame, and continue on our merry way. If the worker is not busy, we copy the current frame to a bitmap, and pass it to the worker. MediaPipe's segmentation algorithm produces a mask as a Uint8Array, which the worker converts to an alpha mask and returns it to Galene's main loop. We then perform blurring and compositing using an HTML Canvas, which (at last on my machine) happens entirely on the GPU. -- Juliusz Chroboczek