From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qt1-x836.google.com (mail-qt1-x836.google.com [IPv6:2607:f8b0:4864:20::836]) by mail.toke.dk (Postfix) with ESMTPS id 3E123AB4FC9 for ; Fri, 13 Dec 2024 22:45:55 +0100 (CET) Authentication-Results: mail.toke.dk; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=GPYglDSh Received: by mail-qt1-x836.google.com with SMTP id d75a77b69052e-46772a0f8fbso19560461cf.3 for ; Fri, 13 Dec 2024 13:45:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734126353; x=1734731153; darn=lists.galene.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=vG3UBUn2bvAr9cgCV5IbDATrpE5yIcDwQ6dxBcu5W54=; b=GPYglDShBnOgYCxCqdICzAYgS9xpmHi2kx25zpn1GY5JWyy+zZCkebraV2HQQhEjRJ zfwhGsUiZBLNMhbrGylcjy//TN7AVWmeHCHwnDoACP2Lm4LdrnawFSb8C5RBodMxB06x fXqeePpYjz10t6UJv/vSnTuAnNtZN8hRrLfgPYBlAyplM8ZSAuViYiHGo9togXNO6Hqi lZp1kaiVQWWWjyphi6LAdvprUHrTmiU2SNiWH0JBOVGOyh0AnhNEk21kvqza2sxqW+Hq VrK8tcTboAXy5HvKhC2cslHQ3tuQU+BcyGavCD4ok2wFDqInCE8E1ME2y/SQ4UhO+Ewc XNLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734126353; x=1734731153; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vG3UBUn2bvAr9cgCV5IbDATrpE5yIcDwQ6dxBcu5W54=; b=Dvls2sAMmswnguPn/z6lWEpkdgm4nZJT3ww+nUDho4fkVYxaydoDlzSebEGYrlkfUn PcdnIcZod+J8sCd78RMI5x56ciQB9112GBdHWJK1CwQm14AnfuQfvlWVq3t2LCuOm7PN A+kU/NRqS9Mf9woxBs36513WS9R5bvWQEbAplNZCB4QDW4knYU6f+l+mN8X39BhnrDHJ DGUgfI3pJkSQupd0vKUobEtjQxqpsAqV3LJSmYqwFF8Lm8VC65ArN4LSPOpa0qt8IWaR 0z3Bx75fQPpftdKWQJUn2cEBAJ2Wh29enHKg9xuQcEVt/xawk3C9HG/lb78i+MXa3jgh m2fA== X-Gm-Message-State: AOJu0Yz7SwPbUxqCi4diPYaLQoQuj2hTiw5n9tvahaxmfIxxHHCkPXpg suIz/FPfgmrnZsHQqfKTOeqZO/ZGppS2QMX+xn/4WkjOs+7Yy8QpLtVIOfL40IQ4tlHxkDSR3aH XDca9btB+DYtBj1JgV+UspQ8nLRQ= X-Gm-Gg: ASbGnct2EEK+uRWeBbaOMrpVCr2bFdM+KqxK+Me9QHNc8ERM603oLY1GNn987SA7y05 JH4XREJfCj1Vk97kNx8LAoYbaEpN4tk5P1QRMDw== X-Google-Smtp-Source: AGHT+IHK4Sf17ZNwVUaxy5kKnsQxJHk6Ups/jFHHRGR3QJZ6u0U+BS5QbjHQNNoQ6y2yh4Vr6I5nYdx8lR1caWFlT4I= X-Received: by 2002:a05:622a:40e:b0:467:613d:9a1 with SMTP id d75a77b69052e-467a5856f83mr59429111cf.48.1734126353339; Fri, 13 Dec 2024 13:45:53 -0800 (PST) MIME-Version: 1.0 References: <87pllv69oz.wl-jch@irif.fr> In-Reply-To: <87pllv69oz.wl-jch@irif.fr> From: Kenichiro MATOHARA Date: Sat, 14 Dec 2024 06:45:42 +0900 Message-ID: To: Juliusz Chroboczek Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Message-ID-Hash: OROTTOQ6JB3OKRPT3EDOUUWCY53GU4GS X-Message-ID-Hash: OROTTOQ6JB3OKRPT3EDOUUWCY53GU4GS X-MailFrom: matoken@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: galene@lists.galene.org X-Mailman-Version: 3.3.10 Precedence: list Subject: [Galene] Re: Background blur in Galene List-Id: =?utf-8?q?Gal=C3=A8ne_videoconferencing_server_discussion_list?= Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hello. This is great. I ran the Galene server on my Raspberry Pi tested it with Firefox on my laptop (CPU: Intel Core i7-10510U/OS: Debian sid), and it worked fine. I'll try using it at the meeting tomorrow. By the way, I found a small mistake in the INSTALL file. $ git diff diff --git a/INSTALL b/INSTALL index 390037b..f60597b 100644 --- a/INSTALL +++ b/INSTALL @@ -38,6 +38,7 @@ disable the menu entries for background blur. Optionally install Google's MediaPipe library: ``` mkdir mediapipe +cd mediapipe npm pack @mediapipe/tasks-vision tar xzf mediapipe-tasks-vision-*.tgz rm -f ../static/third-party/tasks-vision @@ -45,7 +46,7 @@ mv package ../static/third-party/tasks-vision cd ../static/third-party/tasks-vision mkdir models cd models -wget wget https://storage.googleapis.com/mediapipe-models/image_segmenter/= selfie_segmenter/float16/latest/selfie_segmenter.tflite +wget https://storage.googleapis.com/mediapipe-models/image_segmenter/selfi= e_segmenter/float16/latest/selfie_segmenter.tflite cd ../../../../ ``` 2024=E5=B9=B412=E6=9C=8814=E6=97=A5(=E5=9C=9F) 0:04 Juliusz Chroboczek : > > Hi, > > I've just merged an implementation of background blur based on Google's > MediaPipe library into master. Unless I got something wrong, the > implementation has all of the features that I had planned: > > - we don't ever contact Google's servers, all code is hosted locally; > > - the library is loaded lazily, Galene still loads speedily and there's > no extra delay unless you select background blur; > > - everything happens on the client, no unblurred video ever reaches the > server; > > - the heavy lifting happens in a separate thread (a "web worker"), so > the interface remains responsive when blurring; > > - when the client gets overwhelmed, we start dropping frames rather tha= n > building a backlog; > > - the Google's library is an optional install, if it's not present, the= n > the background blur menu entry is disabled. > > On my six-year-old laptop, I'm getting roughly 8fps after blurring, which > is noticeably jerky but usable. A single core is pinned at 100%, and the > UI remains responsive. Performance is likely to improve when the Google > boys and girls implement WebGPU support in TFLite. > > Acknowledgements to Francis Bolduc, who showed me that this is possible. > > > Try it out > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > Go to https://galene.org:8443/group/public/, open the side menu, choose > Filters -> Background blur, hit Present. > > > Install it on your server > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D > > First, update your installation of Galene to the latest master. Then, > perform the procedure described here: > > https://galene.org/INSTALL.html#optional-install-background-blur > > It is okay to do that while the server is running. > > > Possible extensions > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > Now that we have MediaPipe working within Galene, it should be easy to ad= d > additional filters, such as replacing the background with Dracula's > castle, adding a top hat, or dying the speaker's hair. > > I don't find such features useful, but please let me know if you think > otherwise. > > > Technical details > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > It was hard work, so please indulge me while I describe how the sausage > was made. Please feel free to skip the rest of this message. > > > Local installation of MediaPipe > ------------------------------- > > Google's documentation suggests that you should load MediaPipe off their > CDN, which is of course inacceptable for Galene. Since running a local > copy is not documented, it took me a fair amount of time to work out the > following: > > - MediaPipe sources are at https://github.com/google-ai-edge/mediapipe; > however, I never managed to compile their code, apparently I have the > wrong version of the TypeScript compiler; > > - compiled MediaPipe code can be downloaded using npm, as described in > the INTALL document; I have no evidence that it corresponds to the > github code, but I also have no evidence that Google are evil; > > - even though MediaPipe's JavaScript is an ECMAScript module, it doesn'= t > work when invoked from a web worker implemented itself as an > ECMAScript module; after fighting with that for a couple of days, > I switched to a traditional (pre-ECMAScript 6) web worker; > > - MediaPipe's JavaScript uses the "eval" function (!), which is not > allowed in Galene; there's a special case for that in the Go code: > > https://github.com/jech/galene/blob/master/webserver/webserver.go#L21= 4 > https://github.com/jech/galene/blob/master/webserver/webserver.go#L97 > > > Lazy loading > ------------ > > I rather like the fast load times of Galene, and so didn't want to load > the MediaPipe code until the user explicitly requests background blur. > Additionally, since some people might not trust Google, I wanted to keep > the installation optional. > > I was initially under the impression that lazy loading required switching > all of Galene's code to ECMAScript modules, and started doing just that. > Fortunately, it turns out that there are at least two ways of loading > JavaScript lazily from traditional JavaScript: > > - dynamic import, which loads an ECMASCript module into non-module code= ; > > - web workers, which are only instantiated at the time the WebWorker > constructor is called. > > We use both: when the user selects background blur, we instantiate a web > worker, and the web worker performs a dynamic import of the MediaPipe > library. The effect is that the first time a user chooses background > blur, their video will freeze for up to a few seconds while the libraries > are being loaded; however, since this happens in the web worker, the user > interface remains responsive and the other videos keep playing. (After > that, there are no further delays as long as the libraries remain in the > browser's cache.) > > > Running segmentation and building the blurred frame > --------------------------------------------------- > > Whenever we have a new frame, we check whether the worker is busy. If > that's the case, we drop the frame, and continue on our merry way. If th= e > worker is not busy, we copy the current frame to a bitmap, and pass it to > the worker. > > MediaPipe's segmentation algorithm produces a mask as a Uint8Array, which > the worker converts to an alpha mask and returns it to Galene's main loop= . > We then perform blurring and compositing using an HTML Canvas, which (at > last on my machine) happens entirely on the GPU. > > > -- Juliusz Chroboczek > _______________________________________________ > Galene mailing list -- galene@lists.galene.org > To unsubscribe send an email to galene-leave@lists.galene.org --=20 KenichiroMATOHARA mailto:matoken@gmail.com https://matoken.org/ .O. ..O OOO