Galène videoconferencing server discussion list archives
 help / color / mirror / Atom feed
From: Dave Taht <dave.taht@gmail.com>
To: Juliusz Chroboczek <jch@irif.fr>
Cc: galene@lists.galene.org, Dave Taht <dave@taht.net>
Subject: [Galene] Re: fq-codel trashing
Date: Tue, 12 Jan 2021 11:05:48 -0800	[thread overview]
Message-ID: <CAA93jw7Tg+7_210BD4-gAAjuEw5kVt=axybA5M14EJk8qGOe1w@mail.gmail.com> (raw)
In-Reply-To: <871reqqb52.wl-jch@irif.fr>

On Tue, Jan 12, 2021 at 10:10 AM Juliusz Chroboczek <jch@irif.fr> wrote:
>
> > IF you have spare cpu, you can also run cake native no shaper and
> > strip gro (maxpacket 64k indicates gro is on).
>
> GRO will only trigger for HTTP, WebSocket and TURN traffic, I believe;
> most of Galène's traffic is RTP over UDP, with packets under 1200 bytes.
>
> We just had a meeting with 70 people and at around 40 cameras switched on,
> Galène became unusable — there were too many voice drops, which indicates
> two issues:

i have generally found vms perform badly for lots of small packets and r/t.

please try cake. And collect a capture on the underlying hw if possible.

>   * I need to think of a better way of prioritising voice over video when
>     under load;

I take it (I really didn't understand) that unbundling these two types
is not currently feasible
in the javascript or webbrowser. ?Still.

I am pretty intersted in this layer of stuff but since adding crypto
to rtp it's got really hard to look at it.

>   * there are fairness issues — some clients were receiving okay-ish
>     audio, others were not.

collisions in fq_codel start to occur as the sqrt(1024) so at 70 users
the odds are 2-3 of your sessions were colliding.

Cake uses 8 way set associative. 0 collisions at this load.

I imagine you were not tracking the actual backlog in fq_codel during
this conference?
(while :; do tc -s qdisc show dev bla >> whatever; done

all that said, I tend to point fingers at loss at the virtio layer and
underlying hw first just because I'm defensive about
people blaming fq_codel for anything.  :) I also hate vms for r/t traffic.

you had a lot of GRO traffic it looked like. you can disable gso/gro
via ethtool and stick with fq_codel or use cake with
the
>
> Galène recovered after some people switched their cameras off, I didn't
> need to restart anything.  At the highest point, Galène was at 270% CPU,
> and the TURN server was using another 50%.  That's on a four-core VM.

but it does sound like more virtual cores will help at this load?

you in a position to profile? you are context switching like crazy
most likely also.

> > On the other hand, gro is saves on context switches in vms.
>
>     $ PRODUCT INSTALL GALENE /CONFIGURATION=DEFAULT /LOG
>
> -- Juliusz



-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

  reply	other threads:[~2021-01-12 19:06 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-12 13:46 [Galene] " Juliusz Chroboczek
2021-01-12 15:55 ` [Galene] " Toke Høiland-Jørgensen
2021-01-12 16:01   ` Dave Taht
2021-01-12 17:38   ` Juliusz Chroboczek
2021-01-12 17:42     ` Dave Taht
2021-01-12 18:10       ` Juliusz Chroboczek
2021-01-12 19:05         ` Dave Taht [this message]
2021-01-12 19:52           ` Michael Ströder
2021-01-12 21:02           ` Juliusz Chroboczek
2021-01-12 19:29         ` Michael Ströder
2021-01-12 21:22           ` Juliusz Chroboczek
2021-01-13 19:09             ` Michael Ströder
2021-01-14 12:59               ` Juliusz Chroboczek
2021-01-14 13:03                 ` Michael Ströder
2021-01-14 13:10                   ` Juliusz Chroboczek
2021-01-14 13:23                     ` Michael Ströder

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://lists.galene.org/postorius/lists/galene.lists.galene.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAA93jw7Tg+7_210BD4-gAAjuEw5kVt=axybA5M14EJk8qGOe1w@mail.gmail.com' \
    --to=dave.taht@gmail.com \
    --cc=dave@taht.net \
    --cc=galene@lists.galene.org \
    --cc=jch@irif.fr \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox