[Galene] Re: fq-codel trashing

[Juliusz Chroboczek <jch@irif.fr>]

> please try cake.

Both my servers are used in production right now, so I'm not very keen on
experimenting.  I should have some funding for Galène soon.

> And collect a capture on the underlying hw if possible.

I don't have access to the underlying hardware, it's a cloud provider.

>> * I need to think of a better way of prioritising voice over video when
>> under load;

> I take it (I really didn't understand) that unbundling these two types
> is not currently feasible in the javascript or webbrowser.

[...]

> collisions in fq_codel start to occur as the sqrt(1024) so at 70 users
> the odds are 2-3 of your sessions were colliding.

I don't think the network is the problem.  The problem is that there's
a lot of buffering in Pion, and as I push packets down the layers, I have
no way of finding out the amount already queued -- so when we're short on
CPU, the video tends to crowd out the audio.

I'm not sure what the solution is.

>> Galène recovered after some people switched their cameras off, I didn't
>> need to restart anything.  At the highest point, Galène was at 270% CPU,
>> and the TURN server was using another 50%.  That's on a four-core VM.

> but it does sound like more virtual cores will help at this load?

According to profiling, 60% CPU time is spent in the write system call.

> you are context switching like crazy most likely also.

I'll measure next time, but I doubt it: Go's scheduler operates almost
entirely in userspace.

-- Juliusz