[Galene] What I've learnt about background blur

[Juliusz Chroboczek <jch@irif.fr>]

What I've learned while implementing background blur.


There are multiple segmentation techniques
==========================================

The first step is to segment the image into background and foreground.
There are multiple techniques to do that, and they yield different
results.

In addition to the traditional techniques, such as chroma keying, there
are at least two ML-based approaches: semantic segmentation and depth
estimation.  Semantic segmentation is trained to recognise human figures,
while depth estimation guesses the distance of every pixel from the
camera.

Both approaches work surprisingly well (computer vision is really close to
magic).  They have different privacy issues: semantic segmentation will
unblur the background when a human walks by, and depth estimation will
unblur objects lying on your desk.  (The obvious solution would be to do
both and combine the results, but that seems a little overkill.)

I've used semantic segmentation in Galene, since that's what MediaPipe
implements.  I haven't experimented with depth estimation.


Blending is not that obvious
============================

The obvious postprocessing techniques is to blur the original image in
order to construct the blurred backgound.  That's what both Jitsi Meet [1]
and LiveKit [2] do, and that's what I did initially.  It turns out that it
doesn't yield very good results: it causes the foreground to bleed into
the background, causing a halo around the foreground figure.

Thankfully, the problem is well known in the Photoshop community, and the
simple solution is to mask before blurring: this way, you're blurring the
background only, and avoid bleeding the foreground into the background.
That's what Galene does now, and I find the results satisfactory, at least
on 640x400 videos.  (I think this can be improved still, but I'm currently
waiting for my mana bar to replenish.)

Interestingly, Google Meet apparently does something more complicated:
they use a WebGL shader to vary the amount of blurring depending on
something or something else, and use a bilateral filter instead of
a Gaussian filter.  Unfortunately, their posting on the subject [3] does
not make a lot of sense to me (they're applying a bilateral filter to
a monochrome mask?  where does the CoC come from?), I suspect that the
technical writer did not fully undertand what the engineers told them.


[1]: https://github.com/jitsi/jitsi-meet/blob/master/react/features/stream-effects/virtual-background/JitsiStreamBackgroundEffect.ts#L102

[2]: https://github.com/livekit/track-processors-js/blob/main/src/transformers/BackgroundTransformer.ts#L143

[3]: https://research.google/blog/background-features-in-google-meet-powered-by-web-ml/


-- Juliusz