From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-il1-x12e.google.com (mail-il1-x12e.google.com [IPv6:2607:f8b0:4864:20::12e]) by mail.toke.dk (Postfix) with ESMTPS id 126227C8FB2 for ; Mon, 11 Jan 2021 01:30:17 +0100 (CET) Authentication-Results: mail.toke.dk; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PZ2+qNal Received: by mail-il1-x12e.google.com with SMTP id q1so16654117ilt.6 for ; Sun, 10 Jan 2021 16:30:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=Go/s8NENfbCm/bn5ZzhWQovt/E8jYiv/SbOECgfCqpg=; b=PZ2+qNaliJAofnfI/05Z7GlB53nsDGRzS3TzFPpwMZW5VNY2zA1E1Fn23Gz13+zXd1 xLW4DueiuUBCohub6Yrwj5wMwqbsMcMlryQ5kiy92P7H0APzEQiL5OzsRTlgoLvgNAcl 8kU8Spkkl5OWOt9RjofbtpD9DZe9hGjofPv/dfyya8vaoyr2qskA+wvCywdOj0HsD7PL oD8tV/3ja1JGlBh0Omqzh/H0L3W7/AS3ZXzVDkTBZIkcZNWvGkeK8Y2miCv40Cn1SN8l FZvKcLC1KrwTwRP3acb0wVKLFowcXeEGa83e8YvX/Z+JWrIg5qlDgkvwMbRXn1EFMnJ3 dlgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Go/s8NENfbCm/bn5ZzhWQovt/E8jYiv/SbOECgfCqpg=; b=brYh/pPDxzfGU70WJBUAg3flUHCYL8ryENDMaTYkObyUHvV+lju+GFtjmcMUF3KBot oYQj073nJMEgXbkxJJGJURFZhXIzH2yaMX2V0jbM0gb/rW7899Elxk87PJdljZL6cAD9 JmCLpg4bkgJGv3ThzyRav1SCrUj7unXSJWdA9eB2UHGWDlgvo3ep10x8E+gr3FEFCl5Y /N4kXJRgz3CpbfN/+PWBJUYEW2NPwxghSnQdya4shGppuaNMj9PBYcadaDKeVYLW1uNO 2miG1l8noaEyEO9FKswMRerx6yijj1VIQGti2cQMgAz4L2EUT68G/0GK4Ovgw+hjyoGw nGSQ== X-Gm-Message-State: AOAM531uexlOA3JYHegP1/X4BiLWu7R28DDdFaOMcfkx8ar5nBQNlAKY yRc+9l2lnU6oztIqoCTpDoeWZdxHVRxPILE+NLY= X-Google-Smtp-Source: ABdhPJwthisAKnQ1iO0+hfVs2TzXbCt18s60L6BjneTnxauoTBVCrLqcmdNC+KeVqHjFeev1Dzclv0KDDXmAeK0+rMs= X-Received: by 2002:a92:ba08:: with SMTP id o8mr13363455ili.249.1610325015354; Sun, 10 Jan 2021 16:30:15 -0800 (PST) MIME-Version: 1.0 References: <875z485swt.wl-jch@irif.fr> <87wnwnha9w.fsf@toke.dk> <87v9c77e20.wl-jch@irif.fr> <87turrh6y2.fsf@toke.dk> <87mtxj789e.wl-jch@irif.fr> <87o8hzgr4j.fsf@toke.dk> <871rev6w4a.wl-jch@irif.fr> <87im87gdy3.fsf@toke.dk> <87wnwmnsj9.wl-jch@irif.fr> <877dokhpja.fsf@toke.dk> <87lfd04ye5.wl-jch@irif.fr> <87eeismnwy.fsf@toke.dk> In-Reply-To: <87eeismnwy.fsf@toke.dk> From: Dave Taht Date: Sun, 10 Jan 2021 16:30:03 -0800 Message-ID: To: =?UTF-8?B?VG9rZSBIw7hpbGFuZC1Kw7hyZ2Vuc2Vu?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-MailFrom: dave.taht@gmail.com X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation Message-ID-Hash: PW3EXKMLRUY6TVQ4FUKKTICEDFUEHYG6 X-Message-ID-Hash: PW3EXKMLRUY6TVQ4FUKKTICEDFUEHYG6 X-Mailman-Approved-At: Mon, 11 Jan 2021 01:38:49 +0100 CC: Juliusz Chroboczek , galene@lists.galene.org X-Mailman-Version: 3.3.2 Precedence: list Subject: [Galene] Re: Congestion control and WebRTC [was: Logging] List-Id: =?utf-8?q?Gal=C3=A8ne_videoconferencing_server_discussion_list?= Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: On Sun, Jan 10, 2021 at 2:23 PM Toke H=C3=B8iland-J=C3=B8rgensen wrote: > >> In the presence of an FQ-CoDel'ed bottleneck it may be better to put > >> audio and video on two separate 5-tuples: That would cause the audio > >> stream to be treated as a 'sparse flow' with queueing priority and few= er > >> drops when congested. > > > > Uh-huh. I'll send you a patch to do that, in case you find the time to > > test it. > > Sounds good, thanks! > > >> (As an aside, is there a reference for the codec constraints in > >> browsers? And is it possible to tweak codec parameters, say to burn so= me > >> bandwidth to enable really high-fidelity audio for special use cases? = Or > >> is Opus so good that it doesn't matter?) > > > > A typical laptop microphone has rather poor frequency response, so Opus= at > > 48kbit/s is as good as the original. It's just not worth reducing the > > audio rate upon congestion, it's the video rate that gets reduced. There have been some good work (across town) on jacktrip, for multi-party collaborative audio. https://www.npr.org/2020/11/21/937043051/musicians-turn-to-new-software-to-= play-together-online but zoom and google etc simply don't cut it. I am personally extremely interested in high quality, positional collaborative 3D audio, with the video component highly secondary. Here's the low end device I've been using of late, mounted to the roof of my boat. https://zoomcorp.com/en/us/handheld-recorders/handheld-recorders/h3-vr-360-= audio-recorder/ An example of what can be achieved with it, mixed down to binaural, instead of 5:1 http://www.taht.net/~d/circle/wish_youwerehere_Binaural.mp3 You can clearly hear the airplane go *overhead*, totally coincidentally. in the outro. I note that I'm not in best voice on that recording, and one of the big songs I've mostly been recording this year is over here: https://www.youtube.com/watch?v=3DtUun-jFFoU4 > > Right, I see. Looking at the commit that introduced codec support, it > looks pretty straight-forward to crank up the bitrate; maybe I'll > experiment with that a bit (but not using my laptop's microphone). Cool. What comes out of that device is 4 channels of various ambisonics encodings. Whether opus can feed that into a decoder on the other side (imagine trying to sit next to the drummer, or piano player) dunno. > > As to the video rate, you've got plenty of exciting knobs. > > > > 1. Congestion control. As implemented in modern browsers, WebRTC uses = two > > congestion controllers: a fairly traditional loss-based controller, and= an > > interesting delay-based one. This is described here: > > > > https://tools.ietf.org/html/draft-ietf-rmcat-gcc-02 I liked this one. Except that I wanted to add either classic or SCE based ecn to the whole thing. > > Unlike some other video servers, that mere forward congestion indiction= s > > from the receivers to the sender, Gal=C3=A8ne terminates congestion con= trol on > > both legs. currently obeys congestion indication from the receivers, a= nd > > implements the loss-based congestion controller for data received from = the > > sender. > > > > https://github.com/jech/galene/blob/master/rtpconn/rtpconn.go#L956 > > > > We do not currently implement the delay-based controller, which causes > > collapse if the sender is on a low-rate bufferbloated network. That is > > why Gal=C3=A8ne's client limits the rate to 700kbit/s by default (in th= e > > =C2=AB Send =C2=BB entry in the side menu). > > Right, but the browsers do? > > > Implementing the delay-based controller is number 1 on my wishlist. Yo= ur > > help would be greatly appreciated. > > Can't promise any hacking time, unfortunately, at least not short-term. > Happy to test out stuff, though :) > > > 2. Sender-side tweaks. The sender has a number of knobs they can tweak= , > > notably maximum bitrate (separately for each track), and hints about > > whether to prefer framerate or image quality upon congestion. The send= er > > can also pick the webcam resolution, and they can request downscaling > > before encoding. > > Ah, hence the "blackboard mode" - gotcha! > > > 3. SVC. The technology that excites me right now is scalable video cod= ing > > (SVC), which I believe will make simulcast obsolete. With VP8, the > > sender can request that some frames should not be used as reference for > > intra prediction; these =C2=AB discardable =C2=BB frames can be dropped= by the > > server without causing corruption. VP9 implements full scalability: > > temporal scalability, as in VP8, spatial scalability, where the codec > > generates a low resolution flow and a high resolution flow that uses th= e > > low resolution flow for intra prediction, and quality scalability, wher= e > > the codec generates frames with varying quality. > > > > https://en.wikipedia.org/wiki/Scalable_Video_Coding > > > > I'm currently planning to skip simulcasting, which I feel is an > > obsolescent technology, and experiment with SVC instead. Implementing = the > > delay-based controller is a higher prioerity, though. > > Uh, hadn't heard about that before; neat! > > >> Another packet-based optimisation that could be interesting to try out > >> is to mark packets containing video key frames as ECN-capable. > > > > Keyframes can be huge (120 packets is not unusual), it wouldn't be > > resonable to mark such a burst as ECN-capable without actually reacting= to > > CE. And if we drop part of the keyframe, we'll NACK the missing packet= s > > and recover 20ms + 1RTT later. > > Hmm, right, okay I see what you mean... > > >>> Ideally you'd also actually respond to CE markings, > >> RFC 6679. I don't know if it's implemented in browsers. > > > > It is not. > > Ah, too bad :( > > -Toke --=20 "For a successful technology, reality must take precedence over public relations, for Mother Nature cannot be fooled" - Richard Feynman dave@taht.net CTO, TekLibre, LLC Tel: 1-831-435-0729