From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by mail.toke.dk (Postfix) with ESMTPS id 4842A80736B for ; Thu, 4 Mar 2021 22:22:17 +0100 (CET) Authentication-Results: mail.toke.dk; dkim=pass (1024-bit key) header.d=ieee.org header.i=@ieee.org header.b=gNV+IzJ0 Received: by mail-wm1-x329.google.com with SMTP id w7so9270490wmb.5 for ; Thu, 04 Mar 2021 13:22:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ieee.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=xf83WzOhDxWDAqgIHcFYxMrj15AhgPDFF4J1WTvmTrg=; b=gNV+IzJ0lZlo+c4uafpiBGLy6LzTwE/yxXXyXcF1sY+WUuGKGB/qSvpGEcU4BERZig xUEydG6nJnvkGedsmTyeK3QJZMgDwf9qcNhGVVmdFlDknkreEqiDTHTwVRe6bNtomsuH Lj9a8jZ+KW72m7k0AdPwpdGcj06pmwkjnckro= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=xf83WzOhDxWDAqgIHcFYxMrj15AhgPDFF4J1WTvmTrg=; b=Bx2obDe4VSoJ7s13dobGUo2hteGf/ApquhTuQgxBup9CD5x3RwCalqsGsb79WBPO+1 TeMOmhDIMJgvOVsU+2Pe7WAs7t24ydvVsGiufYQFIE/uL7nEQuLdLyqU2hc1XAsDJSgs wFVBdVniToD6vTpbWBy4Dup5M1AB2gmk3DHk0XfM5eqa4XBPZWypWOab6JiMsvEQcHcL K7CY1OO8j7X/CGAvP7fvdyQoMAUhSDpaYBXR0u7pY+sxhc/v3+B6MmcWPag3FCQDUoVs XBS36FjEMOGhuNGJc3OLm1usm1WdmOwgMLSpnAuOsi1alw9czfwbW/NUmwUIoL4HdXV3 +mAg== X-Gm-Message-State: AOAM533kynByoORlKZ1U1njdVfTKcYbzCCb7//thU5QjtN8iNMdwFcQZ 2OXtdKzmej5g5+s5gVvT8+Mud7UP3k3palQvfiTmng== X-Google-Smtp-Source: ABdhPJxnlPzNJOMuPSE7lGmB4Vl5GtOIMduw0C5GEILgR2WOj0tynbPlYfr6XTik/ZObl23KYve2Vuw19QfxIP9dNT0= X-Received: by 2002:a7b:c050:: with SMTP id u16mr5738512wmc.90.1614892935752; Thu, 04 Mar 2021 13:22:15 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Luca Muscariello Date: Thu, 4 Mar 2021 22:22:04 +0100 Message-ID: To: Sean DuBois Content-Type: multipart/alternative; boundary="000000000000c9534405bcbc8f77" X-MailFrom: muscariello@ieee.org X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation Message-ID-Hash: G3EEXJSIKBOAVNAONU2HWPOWBZCU7MND X-Message-ID-Hash: G3EEXJSIKBOAVNAONU2HWPOWBZCU7MND X-Mailman-Approved-At: Thu, 04 Mar 2021 23:55:17 +0100 CC: Dave Taht , bloat , galene@lists.galene.org X-Mailman-Version: 3.3.2 Precedence: list Subject: [Galene] Re: [Bloat] testing latency under load on webrtc with galene? List-Id: =?utf-8?q?Gal=C3=A8ne_videoconferencing_server_discussion_list?= Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: --000000000000c9534405bcbc8f77 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Yes, Chromium is expensive as it took 360 cores for 100 clients. We had to use a test application like the one you've shared to scale to thousands. At the time we started running these experiments I'm not sure your code was already available. We've also used jitsi hammer until it was discontinued. As long as you can send a video sample and as long as the test app includes rate adaptation, it's almost everything one could ask for. Luca On Thu, Mar 4, 2021 at 8:09 PM Sean DuBois wrote: > I would really love to build a general WebRTC tester. Spinning up lots of > Chromium instances is super expensive. Even if you do the mock webcam > you are doing way more then needed. > > I created[0] which spins up many PeerConnections to load test a server. > I think this would be a great alternative to the testing that exists > today. > > * You can send pre-encoded video or RTP packets directly. > * You can receive RTP and not pay the costs of processing > * You get direct access to Receiver Reports, Transport Wide Congestion > Control and Receiver Estimated Max Bandwidth. > * Way easier to deploy. We can build a small static binary. > > If anyone is interested would love to help them kick off a project. We > can make it general enough to test Galene, Janus, Jitsi, Ion etc... If > it gets acceptance from the greater WebRTC community it could really > grow! > > [0] https://github.com/pion/rtsp-bench/blob/master/client/main.go > > On Tue, Mar 02, 2021 at 05:01:42PM +0100, Luca Muscariello wrote: > > Dave, > > > > we have done extensive WebRTC (and several other online meeting > > apps) testing lately and this paper > > https://ieeexplore.ieee.org/document/9153228 reports a > > methodology for WebRTC based on Chromium and Selenium Grid and > > as test orchestrator Jitsi Torture. > > > > I would avoid feeding clients with BBB as video as it is not > > representative of a meeting as encoders are optimized for > > different kinds of video. There are several video samples > > out there. > > > > We have scaled up clients to hundreds with this methodology. > > The paper is short so many details have been omitted but there > > are not many other options to do this kind of test at scale. > > > > Luca > > > > > > > > > > > > > > > > describes the methodology > > > > > > > > On Tue, Mar 2, 2021 at 2:15 AM Dave Taht wrote: > > > > > Given that we have a worldwide network of flent servers... > > > > > > Given how easy galene is to hack on... and a 10 minute install... > > > > > > given some webrtc scripting... a few more stats... some javascript... > > > skull sweat... funding... > > > > > > It seems plausible to be able to construct a suite of tests that coul= d > > > indeed track jitter > > > delay and loss across the internet over webrtc. Perpetually uploading > > > bigbuckbunny or some > > > other suitable movie might be an option, but I have a fondness for > > > extracting a sub 30 second segment from "Max Headroom", which if > > > those here have not seen it, predicted much of the fine mess we're al= l > > > in now. > > > > > > I guess my question is mostly, is a "headless" test feasible? In the > > > context of samknow's lack of webrtc test... lowpowered hw.... > > > > > > -- > > > "For a successful technology, reality must take precedence over publi= c > > > relations, for Mother Nature cannot be fooled" - Richard Feynman > > > > > > dave@taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729 > > > _______________________________________________ > > > Bloat mailing list > > > Bloat@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/bloat > > > > > > _______________________________________________ > > Galene mailing list -- galene@lists.galene.org > > To unsubscribe send an email to galene-leave@lists.galene.org > > --000000000000c9534405bcbc8f77 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Yes, Chromium is expensive as it took 360 cores for 100= clients.
We had to use a test application like the one you've shared=C2=A0
to scale to t= housands.

At = the time we started running these experiments I'm not sure
your code was alr= eady available.
We've also used jitsi hammer until it was discontinued.

As long as you ca= n send a video sample and as long as the=C2=A0
test app includes rate adaptation, it= 's almost everything
one could ask for.

Luca





On Thu, Mar 4, 2021 at 8:09 PM Sean DuBois <sean@siobud.com> wrote:
I would really lov= e to build a general WebRTC tester. Spinning up lots of
Chromium instances is super expensive. Even if you do the mock webcam
you are doing way more then needed.

I created[0] which spins up many PeerConnections to load test a server.
I think this would be a great alternative to the testing that exists
today.

* You can send pre-encoded video or RTP packets directly.
* You can receive RTP and not pay the costs of processing
* You get direct access to Receiver Reports, Transport Wide Congestion
=C2=A0 Control and Receiver Estimated Max Bandwidth.
* Way easier to deploy. We can build a small static binary.

If anyone is interested would love to help them kick off a project. We
can make it general enough to test Galene, Janus, Jitsi, Ion etc... If
it gets acceptance from the greater WebRTC community it could really
grow!

[0] https://github.com/pion/rtsp-bench/= blob/master/client/main.go

On Tue, Mar 02, 2021 at 05:01:42PM +0100, Luca Muscariello wrote:
> Dave,
>
> we have done extensive WebRTC (and several other online meeting
> apps) testing lately and this paper
> https://ieeexplore.ieee.org/document/9153228 re= ports a
> methodology for WebRTC based on Chromium and Selenium Grid and
> as test orchestrator Jitsi Torture.
>
> I would avoid feeding clients with BBB as video as it is not
> representative of a meeting as encoders are optimized for
> different kinds of video. There are several video samples
> out there.
>
> We have scaled up clients to hundreds with this methodology.
> The paper is short so many details have been omitted but there
> are not many other options to do this kind of test at scale.
>
> Luca
>
>
>
>
>
>
>
> describes the methodology
>
>
>
> On Tue, Mar 2, 2021 at 2:15 AM Dave Taht <dave.taht@gmail.com> wrote:
>
> > Given that we have a worldwide network of flent servers...
> >
> > Given how easy galene is to hack on... and a 10 minute install...=
> >
> > given some webrtc scripting... a few more stats... some javascrip= t...
> > skull sweat... funding...
> >
> > It seems plausible to be able to construct a suite of tests that = could
> > indeed track jitter
> > delay and loss across the internet over webrtc. Perpetually uploa= ding
> > bigbuckbunny or some
> > other suitable movie might be an option, but I have a fondness fo= r
> > extracting a sub 30 second segment from=C2=A0 "Max Headroom&= quot;, which if
> > those here have not seen it, predicted much of the fine mess we&#= 39;re all
> > in now.
> >
> > I guess my question is mostly, is a "headless" test fea= sible? In the
> > context of samknow's lack of webrtc test... lowpowered hw....=
> >
> > --
> > "For a successful technology, reality must take precedence o= ver public
> > relations, for Mother Nature cannot be fooled" - Richard Fey= nman
> >
> > dave@taht.net<= /a> <Dave T=C3=A4ht> CTO, TekLibre, LLC Tel: 1-831-435-0729
> > _______________________________________________
> > Bloat mailing list
> > = Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >

> _______________________________________________
> Galene mailing list -- galene@lists.galene.org
> To unsubscribe send an email to galene-leave@lists.galene.org

--000000000000c9534405bcbc8f77--