[Galene] Filtering outgoing video

[Galene] Filtering outgoing video

[Juliusz Chroboczek]

Dear all,

One of my colleagues is using a pocket mirror in front of her laptop's
webcam in order to stream her (physical) desktop.  Teaching during
lockdown when you've got no blackboard at home is *hard*.

Obviously, if you use a mirror, then your writing comes out mirrored, so
you need to mirror it again before sending it out.  She's been using some
proprietary webcam software to do that, so I've just added the ability to
filter the outgoing video through an arbitrary Javascript function.

There's no user interface yet.  At the chat prompt, type

  /set filter "mirror-h"

or

  /set filter "mirror-v"

*before* you hit Ready.  The video should come out mirrored.

Known limitation: the filtering is unconditionally done at 30fps rather
than being synchronised with the webcam's framerate.  If anyone knows how
to fix that, please let me know.

I'd be grateful if somebody competent at Javascript could check my code.

-- Juliusz

[Galene] Re: Filtering outgoing video

[Toke Høiland-Jørgensen]

Juliusz Chroboczek <jch@irif.fr> writes:

> Dear all,
>
> One of my colleagues is using a pocket mirror in front of her laptop's
> webcam in order to stream her (physical) desktop.  Teaching during
> lockdown when you've got no blackboard at home is *hard*.
>
> Obviously, if you use a mirror, then your writing comes out mirrored, so
> you need to mirror it again before sending it out.  She's been using some
> proprietary webcam software to do that, so I've just added the ability to
> filter the outgoing video through an arbitrary Javascript function.
>
> There's no user interface yet.  At the chat prompt, type
>
>   /set filter "mirror-h"
>
> or
>
>   /set filter "mirror-v"
>
> *before* you hit Ready.  The video should come out mirrored.

It's quite common for other video conferencing solutions to mirror your
own view of yourself, without affecting what is sent to others (so it
looks like you're looking in a mirror when you see yourself). Any chance
this could be extended to allow that as well?

-Toke

[Galene] Re: Filtering outgoing video

[Juliusz Chroboczek]

> It's quite common for other video conferencing solutions to mirror your
> own view of yourself, without affecting what is sent to others (so it
> looks like you're looking in a mirror when you see yourself).

We already do that.  (Except in full-screen and in PiP mode, there's
a pull request for that.)

[Galene] Re: Filtering outgoing video

[Toke Høiland-Jørgensen]

Juliusz Chroboczek <jch@irif.fr> writes:

>> It's quite common for other video conferencing solutions to mirror your
>> own view of yourself, without affecting what is sent to others (so it
>> looks like you're looking in a mirror when you see yourself).
>
> We already do that.  (Except in full-screen and in PiP mode, there's
> a pull request for that.)

Hah, yes indeed, so it seems! Seems I didn't even notice that, even
though I spent two hours in a video meeting today :D

-Toke

[Galene] Re: Filtering outgoing video

[Antonin Décimo]

Cool !

> I'd be grateful if somebody competent at Javascript could check my code.

Firefox raises an exception:

    Uncaught (in promise)
    Exception { name: "NS_ERROR_NOT_INITIALIZED", message: "", result: 3253927937, filename: "https://localhost:8443/galene.js", lineNumber: 807, columnNumber: 0, data: null, stack: "Filter@https://localhost:8443/galene.js:807:38\naddLocalMedia@https://localhost:8443/galene.js:937:17\nasync*gotJoined@https://localhost:8443/galene.js:1638:23\nServerConnection.prototype.connect/</this.socket.onmessage@https://localhost:8443/protocol.js:293:33\nEventHandlerNonNull*ServerConnection.prototype.connect/<@https://localhost:8443/protocol.js:257:9\nServerConnection.prototype.connect@https://localhost:8443/protocol.js:224:18\nserverConnect@https://localhost:8443/galene.js:2521:32\n@https://localhost:8443/galene.js:2422:9\nEventHandlerNonNull*@https://localhost:8443/galene.js:2413:10\n" }


    this.canvas = document.createElement('canvas');

    /** .... */
    this.captureStream = this.canvas.captureStream(0);

The problem is that the context isn't created before captureStream.


Have you considered using WebGL and shaders for filters instead? We
may benefit from acceleration and we could add trippy shaders too!

-- Antonin

[Galene] Re: Filtering outgoing video

[Juliusz Chroboczek]

> The problem is that the context isn't created before captureStream.

Grr.  I wonder what happens if I destroy and recreate the context.

> Have you considered using WebGL and shaders for filters instead?

Anything you can do in Javascript in less than 30ms, you can put in the
filter.  Before you go wild, wait until I implement pacing (dropping
frames if you go over 30ms -- right now, you might hang the UI.)

-- Juliusz

[Galene] Re: Filtering outgoing video

[Juliusz Chroboczek]

>> I'd be grateful if somebody competent at Javascript could check my code.

> Firefox raises an exception:

Grr.  Firefox bug, https://bugzilla.mozilla.org/show_bug.cgi?id=1257440

(I'll work around it, just wanted to share my bad mood with the list.)

[Galene] Re: Filtering outgoing video

[Juliusz Chroboczek]

> Grr.  Firefox bug, https://bugzilla.mozilla.org/show_bug.cgi?id=1257440

I've worked around another firefox bug, this should be fixed now.  I've
also added a user interface.

If you want to play with WebGL, set contextType to 'webgl2'.

-- Juliusz

[Galene] Re: Filtering outgoing video

[Michael Ströder]

On 1/13/21 12:14 PM, Juliusz Chroboczek wrote:
>>> I'd be grateful if somebody competent at Javascript could check my code.
> 
>> Firefox raises an exception:
> 
> Grr.  Firefox bug, https://bugzilla.mozilla.org/show_bug.cgi?id=1257440

Ah, already five years with status "NEW"...

Did you find a solution? With recent git HEAD firefox 84.0.2 just shows
colored stripes when enabling one of the filters.

Ciao, Michael.