From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-il1-x12f.google.com (mail-il1-x12f.google.com [IPv6:2607:f8b0:4864:20::12f]) by mail.toke.dk (Postfix) with ESMTPS id E68787C9210 for ; Mon, 11 Jan 2021 07:24:11 +0100 (CET) Authentication-Results: mail.toke.dk; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=g+5AWCIe Received: by mail-il1-x12f.google.com with SMTP id r17so17210794ilo.11 for ; Sun, 10 Jan 2021 22:24:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=X6jGYTW42/1lxMfdpabD0HORi+Jniyt0Jy3yMuNKCPo=; b=g+5AWCIeW8TH7L8BM39jbvNkwLnTacLtPXFexf+MBVodLMAELLKKaYabRCkAwAechW RBjXH/mV4dr2pt5FcSDvqaD8zfwSUYkdLNqRjXHpJzgIENtcCom3wNThNwE1YyUbmevL D4lx2wCQaEgnt68QmIQTqBOJWNMr8HwqY6CD7cmMAqeFvw1fCfC8AJuLrAcx9+Gpd4h+ g5iNnbSRWIN8ibmptLxVLRryAMxR3F80jxGBQypBX1w6Lrtz5HAZQCt3gUEoiLgxlYOF vf/c8j1Zh0+7G8Y9BdSUxsIAhkcmL4BWvveUbGNjeGvEhoTV2rQdwffIjjB76A6VeSno TyKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=X6jGYTW42/1lxMfdpabD0HORi+Jniyt0Jy3yMuNKCPo=; b=iVeuDsphBGCoVzzGrQQb48mIDDfbO9LWdAHk9Fb8u4CxAhms+Vu++V1QqzNoaePxzV k/y+YRogWRhZBLJeojz+hd4uLZ1Cw5JiK1z0NgE52Z2/OcRA8MFOr8TttFWrPUcLoiHx tsWZkfxCJzqySBkHu14AbdFBIRHKej2mz3nNs+hhMfVMcjdRjKurOYEOWK9y4+7BsQ/q rXrtJoIpUX0SQxfY13ixmuIanRDNXFjPxxITExmdjz1nfqK/11ue92Vf61DJm41RrmEI Ay2Je7mBzJd7ilI5DxtVa968AdFw5dYFCVcunf8a1eLAjIe470djksLlPfhpPuFCz3T9 mawA== X-Gm-Message-State: AOAM531xH5f3h2Wz4WTUQL6sBTdK5ZAIxrK8tomMJXVwWQgfHZ//mXgk AQ0R1WipVU8+0U7s2FU8+U0pPaaz+/dTQN3Hn8s= X-Google-Smtp-Source: ABdhPJwoSRm+xTm7O3WII2TAruhvUUb6dENMDaNkvlBt87h1b4XAi7vBs2kg+JrVu7vEVJbFgkZ8thdbM5wVopEhoKQ= X-Received: by 2002:a92:da0f:: with SMTP id z15mr6089787ilm.287.1610346248881; Sun, 10 Jan 2021 22:24:08 -0800 (PST) MIME-Version: 1.0 References: <875z485swt.wl-jch@irif.fr> <87wnwnha9w.fsf@toke.dk> <87v9c77e20.wl-jch@irif.fr> <87turrh6y2.fsf@toke.dk> <87mtxj789e.wl-jch@irif.fr> <87o8hzgr4j.fsf@toke.dk> <871rev6w4a.wl-jch@irif.fr> <87im87gdy3.fsf@toke.dk> <87wnwmnsj9.wl-jch@irif.fr> <877dokhpja.fsf@toke.dk> <87lfd04ye5.wl-jch@irif.fr> <87eeismnwy.fsf@toke.dk> In-Reply-To: From: Dave Taht Date: Sun, 10 Jan 2021 22:23:56 -0800 Message-ID: To: =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Message-ID-Hash: 6YDK3I5DWSOUTX4VR2ETLXGBKJRLTMBE X-Message-ID-Hash: 6YDK3I5DWSOUTX4VR2ETLXGBKJRLTMBE X-MailFrom: dave.taht@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header CC: Juliusz Chroboczek , galene@lists.galene.org X-Mailman-Version: 3.3.2 Precedence: list Subject: [Galene] Re: Congestion control and WebRTC [was: Logging] List-Id: =?utf-8?q?Gal=C3=A8ne_videoconferencing_server_discussion_list?= Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: On Sun, Jan 10, 2021 at 4:30 PM Dave Taht wrote: > > On Sun, Jan 10, 2021 at 2:23 PM Toke H=C3=B8iland-J=C3=B8rgensen wrote: > > > >> In the presence of an FQ-CoDel'ed bottleneck it may be better to put > > >> audio and video on two separate 5-tuples: That would cause the audio > > >> stream to be treated as a 'sparse flow' with queueing priority and f= ewer > > >> drops when congested. > > > > > > Uh-huh. I'll send you a patch to do that, in case you find the time = to > > > test it. > > > > Sounds good, thanks! > > > > >> (As an aside, is there a reference for the codec constraints in > > >> browsers? And is it possible to tweak codec parameters, say to burn = some > > >> bandwidth to enable really high-fidelity audio for special use cases= ? Or > > >> is Opus so good that it doesn't matter?) > > > > > > A typical laptop microphone has rather poor frequency response, so Op= us at > > > 48kbit/s is as good as the original. It's just not worth reducing th= e > > > audio rate upon congestion, it's the video rate that gets reduced. > > There have been some good work (across town) on jacktrip, for > multi-party collaborative audio. > > https://www.npr.org/2020/11/21/937043051/musicians-turn-to-new-software-t= o-play-together-online > > but zoom and google etc simply don't cut it. > > I am personally extremely interested in high quality, positional > collaborative 3D audio, with the video component highly secondary. > > Here's the low end device I've been using of late, mounted to the roof > of my boat. > > https://zoomcorp.com/en/us/handheld-recorders/handheld-recorders/h3-vr-36= 0-audio-recorder/ > > An example of what can be achieved with it, mixed down to binaural, > instead of 5:1 > http://www.taht.net/~d/circle/wish_youwerehere_Binaural.mp3 > > You can clearly hear the airplane go *overhead*, totally > coincidentally. in the outro. I note that I'm not in best voice on > that recording, and one of the big songs I've mostly been recording > this year is over here: https://www.youtube.com/watch?v=3DtUun-jFFoU4 > > > > > > > Right, I see. Looking at the commit that introduced codec support, it > > looks pretty straight-forward to crank up the bitrate; maybe I'll > > experiment with that a bit (but not using my laptop's microphone). > > Cool. What comes out of that device is 4 channels of various > ambisonics encodings. Whether opus can feed that into a decoder on the > other side (imagine trying to sit next to the drummer, or piano > player) dunno. Can galene, or a web browser, carry multichannel opus audio? > > > > As to the video rate, you've got plenty of exciting knobs. > > > > > > 1. Congestion control. As implemented in modern browsers, WebRTC use= s two > > > congestion controllers: a fairly traditional loss-based controller, a= nd an > > > interesting delay-based one. This is described here: > > > > > > https://tools.ietf.org/html/draft-ietf-rmcat-gcc-02 > > I liked this one. Except that I wanted to add either classic or SCE > based ecn to the whole thing. > > > > Unlike some other video servers, that mere forward congestion indicti= ons > > > from the receivers to the sender, Gal=C3=A8ne terminates congestion c= ontrol on > > > both legs. currently obeys congestion indication from the receivers,= and > > > implements the loss-based congestion controller for data received fro= m the > > > sender. > > > > > > https://github.com/jech/galene/blob/master/rtpconn/rtpconn.go#L956 > > > > > > We do not currently implement the delay-based controller, which cause= s > > > collapse if the sender is on a low-rate bufferbloated network. That = is > > > why Gal=C3=A8ne's client limits the rate to 700kbit/s by default (in = the > > > =C2=AB Send =C2=BB entry in the side menu). > > > > Right, but the browsers do? > > > > > Implementing the delay-based controller is number 1 on my wishlist. = Your > > > help would be greatly appreciated. > > > > Can't promise any hacking time, unfortunately, at least not short-term. > > Happy to test out stuff, though :) > > > > > 2. Sender-side tweaks. The sender has a number of knobs they can twe= ak, > > > notably maximum bitrate (separately for each track), and hints about > > > whether to prefer framerate or image quality upon congestion. The se= nder > > > can also pick the webcam resolution, and they can request downscaling > > > before encoding. > > > > Ah, hence the "blackboard mode" - gotcha! > > > > > 3. SVC. The technology that excites me right now is scalable video c= oding > > > (SVC), which I believe will make simulcast obsolete. With VP8, the > > > sender can request that some frames should not be used as reference f= or > > > intra prediction; these =C2=AB discardable =C2=BB frames can be dropp= ed by the > > > server without causing corruption. VP9 implements full scalability: > > > temporal scalability, as in VP8, spatial scalability, where the codec > > > generates a low resolution flow and a high resolution flow that uses = the > > > low resolution flow for intra prediction, and quality scalability, wh= ere > > > the codec generates frames with varying quality. > > > > > > https://en.wikipedia.org/wiki/Scalable_Video_Coding > > > > > > I'm currently planning to skip simulcasting, which I feel is an > > > obsolescent technology, and experiment with SVC instead. Implementin= g the > > > delay-based controller is a higher prioerity, though. > > > > Uh, hadn't heard about that before; neat! > > > > >> Another packet-based optimisation that could be interesting to try o= ut > > >> is to mark packets containing video key frames as ECN-capable. > > > > > > Keyframes can be huge (120 packets is not unusual), it wouldn't be > > > resonable to mark such a burst as ECN-capable without actually reacti= ng to > > > CE. And if we drop part of the keyframe, we'll NACK the missing pack= ets > > > and recover 20ms + 1RTT later. > > > > Hmm, right, okay I see what you mean... > > > > >>> Ideally you'd also actually respond to CE markings, > > >> RFC 6679. I don't know if it's implemented in browsers. > > > > > > It is not. > > > > Ah, too bad :( > > > > -Toke > > > > -- > "For a successful technology, reality must take precedence over public > relations, for Mother Nature cannot be fooled" - Richard Feynman > > dave@taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729 --=20 "For a successful technology, reality must take precedence over public relations, for Mother Nature cannot be fooled" - Richard Feynman dave@taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729