[Galene] Re: fq-codel trashing

[Dave Taht <dave.taht@gmail.com>]

On Tue, Jan 12, 2021 at 10:10 AM Juliusz Chroboczek <jch@irif.fr> wrote:
>
> > IF you have spare cpu, you can also run cake native no shaper and
> > strip gro (maxpacket 64k indicates gro is on).
>
> GRO will only trigger for HTTP, WebSocket and TURN traffic, I believe;
> most of Galène's traffic is RTP over UDP, with packets under 1200 bytes.
>
> We just had a meeting with 70 people and at around 40 cameras switched on,
> Galène became unusable — there were too many voice drops, which indicates
> two issues:

i have generally found vms perform badly for lots of small packets and r/t.

please try cake. And collect a capture on the underlying hw if possible.

>   * I need to think of a better way of prioritising voice over video when
>     under load;

I take it (I really didn't understand) that unbundling these two types
is not currently feasible
in the javascript or webbrowser. ?Still.

I am pretty intersted in this layer of stuff but since adding crypto
to rtp it's got really hard to look at it.

>   * there are fairness issues — some clients were receiving okay-ish
>     audio, others were not.

collisions in fq_codel start to occur as the sqrt(1024) so at 70 users
the odds are 2-3 of your sessions were colliding.

Cake uses 8 way set associative. 0 collisions at this load.

I imagine you were not tracking the actual backlog in fq_codel during
this conference?
(while :; do tc -s qdisc show dev bla >> whatever; done

all that said, I tend to point fingers at loss at the virtio layer and
underlying hw first just because I'm defensive about
people blaming fq_codel for anything.  :) I also hate vms for r/t traffic.

you had a lot of GRO traffic it looked like. you can disable gso/gro
via ethtool and stick with fq_codel or use cake with
the
>
> Galène recovered after some people switched their cameras off, I didn't
> need to restart anything.  At the highest point, Galène was at 270% CPU,
> and the TURN server was using another 50%.  That's on a four-core VM.

but it does sound like more virtual cores will help at this load?

you in a position to profile? you are context switching like crazy
most likely also.

> > On the other hand, gro is saves on context switches in vms.
>
>     $ PRODUCT INSTALL GALENE /CONFIGURATION=DEFAULT /LOG
>
> -- Juliusz



-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729