[Galene] Keeping the developer honest: why native clients are important

[Galene] Keeping the developer honest: why native clients are important

[Juliusz Chroboczek]

Hi,

I've just had a lengthy phone discussion on the subject, so I thought I'd
summarise for the list.

Simplifying somewhat, there used to be two kinds of protocols:
undocumented proprietary protocols, such as SMB, and documented, open
protocols, such as NFS.  The proprietary protocols used to have more
features, but they tended to be messy, since they were not designed for
multiple implementations.  The open protocols tended to be cleaner and
relatively easy to implement, but had fewer features.

At some point, we were convinced that open protocols would win in the end,
but Web applications arrived.  In a web app, the client and the server are
bundled together, and even when the software is open-source, the protocol
is proprietary: it is not designed for third-party applications, it is
undocumented, and it tends to change with every version of the software.
(There are notable exceptions, such as Github, who have published their
protocol, and keep it reasonably stable.)

Galene's protocol is carefully designed to be open:

  1. it is designed to be implementable by non-web clients: it only relies
     on WebSocket and WebRTC, and at no point do we carry HTML or other
     complex formats in a message payload (that's why you cannot have
     italics in Galene's chat);

  2. it is documented; not very well, granted, but it's documented;

  3. it's versioned, and does not change in an incompatible manner between
     versions;

  4. there is a negotiation mechanism that allows the client and the server
     to choose a common protocol version.

Point 1 is the most important, and it's the reason why I've spent
countless hours implementing a native Android client: if the client side
of the protocol is implementable in Java, then it's probably implementable
in Python, in Rust, in C++, or whatever other overly complex language will
be fashionable tomorrow.

There are other advantages to having native clients:

  - the native client can integrate with the platform better than a web
    app can: the current Android client can do screen sharing, which is
    impossible in a mobile browser;

  - the native client is much more reactive than any web browser, which is
    nice if you're not willing to spend 1700€ on the latest Apple shiny;

  - a native client gives potentially better security when accessing an
    untrusted instance of Galene, since the client code is downloaded from
    a trusted source, not from the untrusted server; to be fair, that's
    not really relevant until I implement end-to-end encryption, but
    that's not trivial to do securely, and since we're not trying to sell
    snake oil, it's not worth implementing unless it actually guarantees
    security (I'm looking at you, Zoom).

Of course, we should go further, we should actually strive towards
a standardised protocol for video conferencing.  WHIP (which Galene
implements) is a step in that direction, but it only provides ingress:
it's useful for hooking an IP camera or OBS Studio to an arbitrary server,
but it's not a complete solution for conferencing.  I'm following the IETF
MOQ group, which is working on a media stack based on QUICK, but it seems
dominated by Google and the CDN vendors, so I expect it to be geared
towards Youtube and Netflix rather than videoconferencing.

So for now, the best we can do is ensure that Galene's protocol remains
easy to implement, ensure that there exist native applications, and try to
connect with the rest of the open-source videoconerencing ecosystem.  And
if you've got free time on your hands, you could do worse than hack
together a desktop client for Galene.

-- Juliusz

[Galene] Re: Keeping the developer honest: why native clients are important

[Sean DuBois]

On Sat, Sep 14, 2024 at 03:55:41PM +0200, Juliusz Chroboczek wrote:
> Hi,
>
> I've just had a lengthy phone discussion on the subject, so I thought I'd
> summarise for the list.
>
> Simplifying somewhat, there used to be two kinds of protocols:
> undocumented proprietary protocols, such as SMB, and documented, open
> protocols, such as NFS.  The proprietary protocols used to have more
> features, but they tended to be messy, since they were not designed for
> multiple implementations.  The open protocols tended to be cleaner and
> relatively easy to implement, but had fewer features.
>
> At some point, we were convinced that open protocols would win in the end,
> but Web applications arrived.  In a web app, the client and the server are
> bundled together, and even when the software is open-source, the protocol
> is proprietary: it is not designed for third-party applications, it is
> undocumented, and it tends to change with every version of the software.
> (There are notable exceptions, such as Github, who have published their
> protocol, and keep it reasonably stable.)
>
> Galene's protocol is carefully designed to be open:
>
>   1. it is designed to be implementable by non-web clients: it only relies
>      on WebSocket and WebRTC, and at no point do we carry HTML or other
>      complex formats in a message payload (that's why you cannot have
>      italics in Galene's chat);
>
>   2. it is documented; not very well, granted, but it's documented;
>
>   3. it's versioned, and does not change in an incompatible manner between
>      versions;
>
>   4. there is a negotiation mechanism that allows the client and the server
>      to choose a common protocol version.
>
> Point 1 is the most important, and it's the reason why I've spent
> countless hours implementing a native Android client: if the client side
> of the protocol is implementable in Java, then it's probably implementable
> in Python, in Rust, in C++, or whatever other overly complex language will
> be fashionable tomorrow.
>
> There are other advantages to having native clients:
>
>   - the native client can integrate with the platform better than a web
>     app can: the current Android client can do screen sharing, which is
>     impossible in a mobile browser;
>
>   - the native client is much more reactive than any web browser, which is
>     nice if you're not willing to spend 1700€ on the latest Apple shiny;
>
>   - a native client gives potentially better security when accessing an
>     untrusted instance of Galene, since the client code is downloaded from
>     a trusted source, not from the untrusted server; to be fair, that's
>     not really relevant until I implement end-to-end encryption, but
>     that's not trivial to do securely, and since we're not trying to sell
>     snake oil, it's not worth implementing unless it actually guarantees
>     security (I'm looking at you, Zoom).
>
> Of course, we should go further, we should actually strive towards
> a standardised protocol for video conferencing.  WHIP (which Galene
> implements) is a step in that direction, but it only provides ingress:
> it's useful for hooking an IP camera or OBS Studio to an arbitrary server,
> but it's not a complete solution for conferencing.  I'm following the IETF
> MOQ group, which is working on a media stack based on QUICK, but it seems
> dominated by Google and the CDN vendors, so I expect it to be geared
> towards Youtube and Netflix rather than videoconferencing.
>
> So for now, the best we can do is ensure that Galene's protocol remains
> easy to implement, ensure that there exist native applications, and try to
> connect with the rest of the open-source videoconerencing ecosystem.  And
> if you've got free time on your hands, you could do worse than hack
> together a desktop client for Galene.
>
> -- Juliusz

Native clients are very important. I hope with WHIP/WHEP they start to
get more attention.

  - Better Accessibility. A native client can be designed to accommodate particular needs.
    The user could then point them at any service.

  - Auditable/Learnable. Modern Javascript is equivalent to shipping a
    stripped binary. It is going to be even harder with
    WebCodecs/WebTransport/WASM. I would love to see Open Source clients
    where the community can iterate and improve on Congestion
    Control/Error Correct etc... would be great if that even forces
    companies to put resources on it.

  - User Freedom. No vendor lock-in! If your conferencing service starts
    doing bad things, just point your client at a different URL.

----

I would like to extend WHIP/WHEP to cover a few more use cases. I am not
sure where to start though.

* Conference Calling. I have one WHIP session to upload, then one
  (or many) WHEP sessions to download. Maybe some sort of 'pseudo WHEP'
  resource that I can listen for SSE events on? That would announce the
  streams as they join/leave?

* Autoconf. I want to push video into OBS. Or send video OBS<->OBS into
  the same network. Remove dependence on proprietary software to do
  video mixing in the LAN.

[Galene] Re: Keeping the developer honest: why native clients are important

[Juliusz Chroboczek]

> I would like to extend WHIP/WHEP to cover a few more use cases. I am not
> sure where to start though.
> 
> * Conference Calling. I have one WHIP session to upload, then one
>   (or many) WHEP sessions to download. Maybe some sort of 'pseudo WHEP'
>   resource that I can listen for SSE events on? That would announce the
>   streams as they join/leave?

Another problem is that people really want to have a chat when
videoconferencing, and there's no standard, simple chat protocol.  (Don't
get me started on XMPP.)

> * Autoconf. I want to push video into OBS. Or send video OBS<->OBS into
>   the same network. Remove dependence on proprietary software to do
>   video mixing in the LAN.

How do you push video into OBS?  Can OBS act as a WHIP endpoint?

-- Juliusz