Galène videoconferencing server discussion list archives
 help / color / mirror / Atom feed
* [Galene] testing latency under load on webrtc with galene?
@ 2021-03-02  1:14 Dave Taht
  2021-03-02 14:59 ` [Galene] " Jeroen van Veen
  2021-03-02 16:01 ` [Galene] Re: [Bloat] " Luca Muscariello
  0 siblings, 2 replies; 5+ messages in thread
From: Dave Taht @ 2021-03-02  1:14 UTC (permalink / raw)
  To: bloat, galene

Given that we have a worldwide network of flent servers...

Given how easy galene is to hack on... and a 10 minute install...

given some webrtc scripting... a few more stats... some javascript...
skull sweat... funding...

It seems plausible to be able to construct a suite of tests that could
indeed track jitter
delay and loss across the internet over webrtc. Perpetually uploading
bigbuckbunny or some
other suitable movie might be an option, but I have a fondness for
extracting a sub 30 second segment from  "Max Headroom", which if
those here have not seen it, predicted much of the fine mess we're all
in now.

I guess my question is mostly, is a "headless" test feasible? In the
context of samknow's lack of webrtc test... lowpowered hw....

-- 
"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman

dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [Galene] Re: testing latency under load on webrtc with galene?
  2021-03-02  1:14 [Galene] testing latency under load on webrtc with galene? Dave Taht
@ 2021-03-02 14:59 ` Jeroen van Veen
  2021-03-02 16:01 ` [Galene] Re: [Bloat] " Luca Muscariello
  1 sibling, 0 replies; 5+ messages in thread
From: Jeroen van Veen @ 2021-03-02 14:59 UTC (permalink / raw)
  To: Dave Taht; +Cc: bloat, galene

Not sure if this helps, but in Chromium there is an option
to fake media devices; e.g. chromium --use-fake-device-for-media-stream
Together with a headless testing library like Puppeteer or Playwright,
it should be quite easy to write a script that load-tests a Galene server using
several Chrome instances. Not really sure if the WebRTC api uncovers
things like jitter and packet loss though.

Anyway, all it requires is some basic setup to click through the
Galene UI. In that case, https://playwright.dev/docs/cli/ might help.

- Jeroen


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
Op dinsdag, maart 2, 2021 2:14 AM, Dave Taht <dave.taht@gmail.com> schreef:

> Given that we have a worldwide network of flent servers...
>
> Given how easy galene is to hack on... and a 10 minute install...
>
> given some webrtc scripting... a few more stats... some javascript...
> skull sweat... funding...
>
> It seems plausible to be able to construct a suite of tests that could
> indeed track jitter
> delay and loss across the internet over webrtc. Perpetually uploading
> bigbuckbunny or some
> other suitable movie might be an option, but I have a fondness for
> extracting a sub 30 second segment from "Max Headroom", which if
> those here have not seen it, predicted much of the fine mess we're all
> in now.
>
> I guess my question is mostly, is a "headless" test feasible? In the
> context of samknow's lack of webrtc test... lowpowered hw....
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> "For a successful technology, reality must take precedence over public
> relations, for Mother Nature cannot be fooled" - Richard Feynman
>
> dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729
>
> Galene mailing list -- galene@lists.galene.org
> To unsubscribe send an email to galene-leave@lists.galene.org



^ permalink raw reply	[flat|nested] 5+ messages in thread

* [Galene] Re: [Bloat] testing latency under load on webrtc with galene?
  2021-03-02  1:14 [Galene] testing latency under load on webrtc with galene? Dave Taht
  2021-03-02 14:59 ` [Galene] " Jeroen van Veen
@ 2021-03-02 16:01 ` Luca Muscariello
  2021-03-04 19:09   ` Sean DuBois
  1 sibling, 1 reply; 5+ messages in thread
From: Luca Muscariello @ 2021-03-02 16:01 UTC (permalink / raw)
  To: Dave Taht; +Cc: bloat, galene

[-- Attachment #1: Type: text/plain, Size: 1958 bytes --]

Dave,

we have done extensive WebRTC (and several other online meeting
apps) testing lately and this paper
https://ieeexplore.ieee.org/document/9153228 reports a
methodology for WebRTC based on Chromium and Selenium Grid and
as test orchestrator Jitsi Torture.

I would avoid feeding clients with BBB as video as it is not
representative of a meeting as encoders are optimized for
different kinds of video. There are several video samples
out there.

We have scaled up clients to hundreds with this methodology.
The paper is short so many details have been omitted but there
are not many other options to do this kind of test at scale.

Luca







describes the methodology



On Tue, Mar 2, 2021 at 2:15 AM Dave Taht <dave.taht@gmail.com> wrote:

> Given that we have a worldwide network of flent servers...
>
> Given how easy galene is to hack on... and a 10 minute install...
>
> given some webrtc scripting... a few more stats... some javascript...
> skull sweat... funding...
>
> It seems plausible to be able to construct a suite of tests that could
> indeed track jitter
> delay and loss across the internet over webrtc. Perpetually uploading
> bigbuckbunny or some
> other suitable movie might be an option, but I have a fondness for
> extracting a sub 30 second segment from  "Max Headroom", which if
> those here have not seen it, predicted much of the fine mess we're all
> in now.
>
> I guess my question is mostly, is a "headless" test feasible? In the
> context of samknow's lack of webrtc test... lowpowered hw....
>
> --
> "For a successful technology, reality must take precedence over public
> relations, for Mother Nature cannot be fooled" - Richard Feynman
>
> dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 4442 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [Galene] Re: [Bloat] testing latency under load on webrtc with galene?
  2021-03-02 16:01 ` [Galene] Re: [Bloat] " Luca Muscariello
@ 2021-03-04 19:09   ` Sean DuBois
  2021-03-04 21:22     ` Luca Muscariello
  0 siblings, 1 reply; 5+ messages in thread
From: Sean DuBois @ 2021-03-04 19:09 UTC (permalink / raw)
  To: Luca Muscariello; +Cc: Dave Taht, bloat, galene

I would really love to build a general WebRTC tester. Spinning up lots of
Chromium instances is super expensive. Even if you do the mock webcam
you are doing way more then needed.

I created[0] which spins up many PeerConnections to load test a server.
I think this would be a great alternative to the testing that exists
today.

* You can send pre-encoded video or RTP packets directly.
* You can receive RTP and not pay the costs of processing
* You get direct access to Receiver Reports, Transport Wide Congestion
  Control and Receiver Estimated Max Bandwidth.
* Way easier to deploy. We can build a small static binary.

If anyone is interested would love to help them kick off a project. We
can make it general enough to test Galene, Janus, Jitsi, Ion etc... If
it gets acceptance from the greater WebRTC community it could really
grow!

[0] https://github.com/pion/rtsp-bench/blob/master/client/main.go

On Tue, Mar 02, 2021 at 05:01:42PM +0100, Luca Muscariello wrote:
> Dave,
>
> we have done extensive WebRTC (and several other online meeting
> apps) testing lately and this paper
> https://ieeexplore.ieee.org/document/9153228 reports a
> methodology for WebRTC based on Chromium and Selenium Grid and
> as test orchestrator Jitsi Torture.
>
> I would avoid feeding clients with BBB as video as it is not
> representative of a meeting as encoders are optimized for
> different kinds of video. There are several video samples
> out there.
>
> We have scaled up clients to hundreds with this methodology.
> The paper is short so many details have been omitted but there
> are not many other options to do this kind of test at scale.
>
> Luca
>
>
>
>
>
>
>
> describes the methodology
>
>
>
> On Tue, Mar 2, 2021 at 2:15 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> > Given that we have a worldwide network of flent servers...
> >
> > Given how easy galene is to hack on... and a 10 minute install...
> >
> > given some webrtc scripting... a few more stats... some javascript...
> > skull sweat... funding...
> >
> > It seems plausible to be able to construct a suite of tests that could
> > indeed track jitter
> > delay and loss across the internet over webrtc. Perpetually uploading
> > bigbuckbunny or some
> > other suitable movie might be an option, but I have a fondness for
> > extracting a sub 30 second segment from  "Max Headroom", which if
> > those here have not seen it, predicted much of the fine mess we're all
> > in now.
> >
> > I guess my question is mostly, is a "headless" test feasible? In the
> > context of samknow's lack of webrtc test... lowpowered hw....
> >
> > --
> > "For a successful technology, reality must take precedence over public
> > relations, for Mother Nature cannot be fooled" - Richard Feynman
> >
> > dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >

> _______________________________________________
> Galene mailing list -- galene@lists.galene.org
> To unsubscribe send an email to galene-leave@lists.galene.org


^ permalink raw reply	[flat|nested] 5+ messages in thread

* [Galene] Re: [Bloat] testing latency under load on webrtc with galene?
  2021-03-04 19:09   ` Sean DuBois
@ 2021-03-04 21:22     ` Luca Muscariello
  0 siblings, 0 replies; 5+ messages in thread
From: Luca Muscariello @ 2021-03-04 21:22 UTC (permalink / raw)
  To: Sean DuBois; +Cc: Dave Taht, bloat, galene

[-- Attachment #1: Type: text/plain, Size: 3938 bytes --]

Yes, Chromium is expensive as it took 360 cores for 100 clients.
We had to use a test application like the one you've shared
to scale to thousands.

At the time we started running these experiments I'm not sure
your code was already available.
We've also used jitsi hammer until it was discontinued.

As long as you can send a video sample and as long as the
test app includes rate adaptation, it's almost everything
one could ask for.

Luca





On Thu, Mar 4, 2021 at 8:09 PM Sean DuBois <sean@siobud.com> wrote:

> I would really love to build a general WebRTC tester. Spinning up lots of
> Chromium instances is super expensive. Even if you do the mock webcam
> you are doing way more then needed.
>
> I created[0] which spins up many PeerConnections to load test a server.
> I think this would be a great alternative to the testing that exists
> today.
>
> * You can send pre-encoded video or RTP packets directly.
> * You can receive RTP and not pay the costs of processing
> * You get direct access to Receiver Reports, Transport Wide Congestion
>   Control and Receiver Estimated Max Bandwidth.
> * Way easier to deploy. We can build a small static binary.
>
> If anyone is interested would love to help them kick off a project. We
> can make it general enough to test Galene, Janus, Jitsi, Ion etc... If
> it gets acceptance from the greater WebRTC community it could really
> grow!
>
> [0] https://github.com/pion/rtsp-bench/blob/master/client/main.go
>
> On Tue, Mar 02, 2021 at 05:01:42PM +0100, Luca Muscariello wrote:
> > Dave,
> >
> > we have done extensive WebRTC (and several other online meeting
> > apps) testing lately and this paper
> > https://ieeexplore.ieee.org/document/9153228 reports a
> > methodology for WebRTC based on Chromium and Selenium Grid and
> > as test orchestrator Jitsi Torture.
> >
> > I would avoid feeding clients with BBB as video as it is not
> > representative of a meeting as encoders are optimized for
> > different kinds of video. There are several video samples
> > out there.
> >
> > We have scaled up clients to hundreds with this methodology.
> > The paper is short so many details have been omitted but there
> > are not many other options to do this kind of test at scale.
> >
> > Luca
> >
> >
> >
> >
> >
> >
> >
> > describes the methodology
> >
> >
> >
> > On Tue, Mar 2, 2021 at 2:15 AM Dave Taht <dave.taht@gmail.com> wrote:
> >
> > > Given that we have a worldwide network of flent servers...
> > >
> > > Given how easy galene is to hack on... and a 10 minute install...
> > >
> > > given some webrtc scripting... a few more stats... some javascript...
> > > skull sweat... funding...
> > >
> > > It seems plausible to be able to construct a suite of tests that could
> > > indeed track jitter
> > > delay and loss across the internet over webrtc. Perpetually uploading
> > > bigbuckbunny or some
> > > other suitable movie might be an option, but I have a fondness for
> > > extracting a sub 30 second segment from  "Max Headroom", which if
> > > those here have not seen it, predicted much of the fine mess we're all
> > > in now.
> > >
> > > I guess my question is mostly, is a "headless" test feasible? In the
> > > context of samknow's lack of webrtc test... lowpowered hw....
> > >
> > > --
> > > "For a successful technology, reality must take precedence over public
> > > relations, for Mother Nature cannot be fooled" - Richard Feynman
> > >
> > > dave@taht.net <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729
> > > _______________________________________________
> > > Bloat mailing list
> > > Bloat@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/bloat
> > >
>
> > _______________________________________________
> > Galene mailing list -- galene@lists.galene.org
> > To unsubscribe send an email to galene-leave@lists.galene.org
>
>

[-- Attachment #2: Type: text/html, Size: 6443 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-03-04 21:22 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-02  1:14 [Galene] testing latency under load on webrtc with galene? Dave Taht
2021-03-02 14:59 ` [Galene] " Jeroen van Veen
2021-03-02 16:01 ` [Galene] Re: [Bloat] " Luca Muscariello
2021-03-04 19:09   ` Sean DuBois
2021-03-04 21:22     ` Luca Muscariello

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox