Galène videoconferencing server discussion list archives
 help / color / mirror / Atom feed
* [Galene] we broke galene head pretty badly today
@ 2021-06-02  4:39 Dave Taht
  2021-06-02 10:48 ` [Galene] " Juliusz Chroboczek
  2021-07-05 20:46 ` Juliusz Chroboczek
  0 siblings, 2 replies; 10+ messages in thread
From: Dave Taht @ 2021-06-02  4:39 UTC (permalink / raw)
  To: galene

16 guests, my video would freeze, some had audio to each other, others didn't.

-- 
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/

Dave Täht CTO, TekLibre, LLC

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [Galene] Re: we broke galene head pretty badly today
  2021-06-02  4:39 [Galene] we broke galene head pretty badly today Dave Taht
@ 2021-06-02 10:48 ` Juliusz Chroboczek
  2021-06-02 11:06   ` Dave Taht
  2021-07-05 20:46 ` Juliusz Chroboczek
  1 sibling, 1 reply; 10+ messages in thread
From: Juliusz Chroboczek @ 2021-06-02 10:48 UTC (permalink / raw)
  To: Dave Taht; +Cc: galene

> 16 guests, my video would freeze, some had audio to each other, others didn't.

What was the CPU load on the server?

Did you increase the file descriptor limit before stating the server?

-- Juliusz

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [Galene] Re: we broke galene head pretty badly today
  2021-06-02 10:48 ` [Galene] " Juliusz Chroboczek
@ 2021-06-02 11:06   ` Dave Taht
  0 siblings, 0 replies; 10+ messages in thread
From: Dave Taht @ 2021-06-02 11:06 UTC (permalink / raw)
  To: Juliusz Chroboczek; +Cc: galene

On Wed, Jun 2, 2021 at 3:48 AM Juliusz Chroboczek <jch@irif.fr> wrote:
>
> > 16 guests, my video would freeze, some had audio to each other, others didn't.

If we can get another big meeting together just to test...

> What was the CPU load on the server?

didn't look.

> Did you increase the file descriptor limit before stating the server?

yes.

>
> -- Juliusz



-- 
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/

Dave Täht CTO, TekLibre, LLC

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [Galene] Re: we broke galene head pretty badly today
  2021-06-02  4:39 [Galene] we broke galene head pretty badly today Dave Taht
  2021-06-02 10:48 ` [Galene] " Juliusz Chroboczek
@ 2021-07-05 20:46 ` Juliusz Chroboczek
  2021-07-05 21:01   ` Juliusz Chroboczek
  1 sibling, 1 reply; 10+ messages in thread
From: Juliusz Chroboczek @ 2021-07-05 20:46 UTC (permalink / raw)
  To: Dave Taht; +Cc: galene

> 16 guests, my video would freeze, some had audio to each other, others didn't.

Yeah.

Galène currently reacts badly to being out of CPU.  I've tried to design
it so it would drop frames and keep the audio going, but I've failed.

I need to get back at the problem, especially since the simulcast support
in 0.4 makes it possible to drop frames without corruption, but I first
need to find a way to reproduce it.  I'll try setting up Galène on the
tiny ARM board you gave me and downclock it as much as possible, perhaps
that'll help.

-- Juliusz




^ permalink raw reply	[flat|nested] 10+ messages in thread

* [Galene] Re: we broke galene head pretty badly today
  2021-07-05 20:46 ` Juliusz Chroboczek
@ 2021-07-05 21:01   ` Juliusz Chroboczek
  2021-07-06  0:41     ` Dave Taht
  0 siblings, 1 reply; 10+ messages in thread
From: Juliusz Chroboczek @ 2021-07-05 21:01 UTC (permalink / raw)
  To: Dave Taht; +Cc: galene

> Galène currently reacts badly to being out of CPU.  I've tried to design
> it so it would drop frames and keep the audio going, but I've failed.

Just to be clear, I'm not looking at the load average or CPU utilisation,
since these measures are typically unreliable on modern CPUs (which vary
their speed dynamically) let alone on VMs.  I need a quick way to answer
the following question: I've got this packet that I can drop without
corrupting the video, should I drop it or not.

Timestamp the packets at ingress time and drop anything droppable that's
too old?  Any other ideas?

-- Julius

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [Galene] Re: we broke galene head pretty badly today
  2021-07-05 21:01   ` Juliusz Chroboczek
@ 2021-07-06  0:41     ` Dave Taht
  2021-07-06 10:52       ` Juliusz Chroboczek
  0 siblings, 1 reply; 10+ messages in thread
From: Dave Taht @ 2021-07-06  0:41 UTC (permalink / raw)
  To: Juliusz Chroboczek; +Cc: galene

yes a codel-ly approach would be helpful, but what stage of the
pipeline is costing the most cpu?

What I saw in another set of tests was context switching cost a lot
and that perhaps recvmmsg and sendmmsg would help there.

Still, when you run out of cpu, dropping whole frames would be good.
you could interleave dropping a frame per sender perhaps?

do we know what other sfus do?

On Mon, Jul 5, 2021 at 2:01 PM Juliusz Chroboczek <jch@irif.fr> wrote:
>
> > Galčne currently reacts badly to being out of CPU.  I've tried to design
> > it so it would drop frames and keep the audio going, but I've failed.
>
> Just to be clear, I'm not looking at the load average or CPU utilisation,
> since these measures are typically unreliable on modern CPUs (which vary
> their speed dynamically) let alone on VMs.  I need a quick way to answer
> the following question: I've got this packet that I can drop without
> corrupting the video, should I drop it or not.
>
> Timestamp the packets at ingress time and drop anything droppable that's
> too old?  Any other ideas?
>
> -- Julius



-- 
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/

Dave Täht CTO, TekLibre, LLC

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [Galene] Re: we broke galene head pretty badly today
  2021-07-06  0:41     ` Dave Taht
@ 2021-07-06 10:52       ` Juliusz Chroboczek
  2021-07-06 13:44         ` Dave Taht
  0 siblings, 1 reply; 10+ messages in thread
From: Juliusz Chroboczek @ 2021-07-06 10:52 UTC (permalink / raw)
  To: Dave Taht; +Cc: galene

> yes a codel-ly approach would be helpful, but what stage of the
> pipeline is costing the most cpu?

Galène is fairly well optimised, on a machine with hardware crypto, most
of the time is spent in the write system call.  I'll post a profile when
I'm next at the lab.

> Still, when you run out of cpu, dropping whole frames would be good.
> you could interleave dropping a frame per sender perhaps?

I can drop every second frame without causing video corruption, and I can
put backpressure on the sender, the code is already there.  The only
difficulty is how to determine that I'm under load.

-- Juliusz

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [Galene] Re: we broke galene head pretty badly today
  2021-07-06 10:52       ` Juliusz Chroboczek
@ 2021-07-06 13:44         ` Dave Taht
  2021-07-06 13:47           ` Dave Taht
  0 siblings, 1 reply; 10+ messages in thread
From: Dave Taht @ 2021-07-06 13:44 UTC (permalink / raw)
  To: Juliusz Chroboczek; +Cc: galene

On Tue, Jul 6, 2021 at 3:52 AM Juliusz Chroboczek <jch@irif.fr> wrote:
>
> > yes a codel-ly approach would be helpful, but what stage of the
> > pipeline is costing the most cpu?
>
> Galčne is fairly well optimised, on a machine with hardware crypto, most
> of the time is spent in the write system call.  I'll post a profile when
> I'm next at the lab.

What hardware is the weakest and when does it start to fail?

I can also start setting up simple videoconferences. It would be
really good to have a tool
that could create virtual sessions to load things up. I got tired of
bigbuckbunny a long time ago
but I've often thought it would be amusing to try and sync up a 4+ piece band.

>
> > Still, when you run out of cpu, dropping whole frames would be good.
> > you could interleave dropping a frame per sender perhaps?
>
> I can drop every second frame without causing video corruption, and I can
> put backpressure on the sender, the code is already there.  The only
> difficulty is how to determine that I'm under load.

timestamp a packet or frame on entry (as early as possible), check
time again on exit (after the write). If the delta is greater than the
framerate, start dropping frames.

st

>
> -- Juliusz



-- 
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/

Dave Täht CTO, TekLibre, LLC

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [Galene] Re: we broke galene head pretty badly today
  2021-07-06 13:44         ` Dave Taht
@ 2021-07-06 13:47           ` Dave Taht
  2021-07-06 15:56             ` Juliusz Chroboczek
  0 siblings, 1 reply; 10+ messages in thread
From: Dave Taht @ 2021-07-06 13:47 UTC (permalink / raw)
  To: Juliusz Chroboczek; +Cc: galene

I would still rather like it if sendmmsg could be used in conjunction
with user-space sfq across the participants. This gives you natural
packet pacing, cuts write context switch time by up to a factor of 42.

On Tue, Jul 6, 2021 at 6:44 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> On Tue, Jul 6, 2021 at 3:52 AM Juliusz Chroboczek <jch@irif.fr> wrote:
> >
> > > yes a codel-ly approach would be helpful, but what stage of the
> > > pipeline is costing the most cpu?
> >
> > Galčne is fairly well optimised, on a machine with hardware crypto, most
> > of the time is spent in the write system call.  I'll post a profile when
> > I'm next at the lab.
>
> What hardware is the weakest and when does it start to fail?
>
> I can also start setting up simple videoconferences. It would be
> really good to have a tool
> that could create virtual sessions to load things up. I got tired of
> bigbuckbunny a long time ago
> but I've often thought it would be amusing to try and sync up a 4+ piece band.
>
> >
> > > Still, when you run out of cpu, dropping whole frames would be good.
> > > you could interleave dropping a frame per sender perhaps?
> >
> > I can drop every second frame without causing video corruption, and I can
> > put backpressure on the sender, the code is already there.  The only
> > difficulty is how to determine that I'm under load.
>
> timestamp a packet or frame on entry (as early as possible), check
> time again on exit (after the write). If the delta is greater than the
> framerate, start dropping frames.
>
> st
>
> >
> > -- Juliusz
>
>
>
> --
> Latest Podcast:
> https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/
>
> Dave Täht CTO, TekLibre, LLC



-- 
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/

Dave Täht CTO, TekLibre, LLC

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [Galene] Re: we broke galene head pretty badly today
  2021-07-06 13:47           ` Dave Taht
@ 2021-07-06 15:56             ` Juliusz Chroboczek
  0 siblings, 0 replies; 10+ messages in thread
From: Juliusz Chroboczek @ 2021-07-06 15:56 UTC (permalink / raw)
  To: Dave Taht; +Cc: galene

> I would still rather like it if sendmmsg could be used in conjunction
> with user-space sfq across the participants. This gives you natural
> packet pacing, cuts write context switch time by up to a factor of 42.

The actual sending of packets happens roughly 17 layers below the packet
dropping and queueing logic -- recall that WebRTC falls back transparently
to TURN encapsulation and even TCP when needed, and that's handled in the
lower layers.

But please do try your hand at it:

  https://github.com/pion/ice/issues/128

-- Juliusz

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2021-07-06 15:56 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-02  4:39 [Galene] we broke galene head pretty badly today Dave Taht
2021-06-02 10:48 ` [Galene] " Juliusz Chroboczek
2021-06-02 11:06   ` Dave Taht
2021-07-05 20:46 ` Juliusz Chroboczek
2021-07-05 21:01   ` Juliusz Chroboczek
2021-07-06  0:41     ` Dave Taht
2021-07-06 10:52       ` Juliusz Chroboczek
2021-07-06 13:44         ` Dave Taht
2021-07-06 13:47           ` Dave Taht
2021-07-06 15:56             ` Juliusz Chroboczek

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox