From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mail.toke.dk; spf=pass (mailfrom) smtp.mailfrom=irif.fr (client-ip=2001:660:3301:8000::1:2; helo=korolev.univ-paris7.fr; envelope-from=jch@irif.fr; receiver=) Received: from korolev.univ-paris7.fr (korolev.univ-paris7.fr [IPv6:2001:660:3301:8000::1:2]) by mail.toke.dk (Postfix) with ESMTPS id 3EECA84CBDD for ; Sat, 15 May 2021 13:16:10 +0200 (CEST) Received: from potemkin.univ-paris7.fr (potemkin.univ-paris7.fr [IPv6:2001:660:3301:8000::1:1]) by korolev.univ-paris7.fr (8.14.4/8.14.4/relay1/82085) with ESMTP id 14FBGAAJ012741 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 15 May 2021 13:16:10 +0200 Received: from mailhub.math.univ-paris-diderot.fr (mailhub.math.univ-paris-diderot.fr [81.194.30.253]) by potemkin.univ-paris7.fr (8.14.4/8.14.4/relay2/82085) with ESMTP id 14FBGAT4025729 for ; Sat, 15 May 2021 13:16:10 +0200 Received: from mailhub.math.univ-paris-diderot.fr (localhost [127.0.0.1]) by mailhub.math.univ-paris-diderot.fr (Postfix) with ESMTP id A76B41075F9 for ; Sat, 15 May 2021 13:16:10 +0200 (CEST) X-Virus-Scanned: amavisd-new at math.univ-paris-diderot.fr Received: from mailhub.math.univ-paris-diderot.fr ([127.0.0.1]) by mailhub.math.univ-paris-diderot.fr (mailhub.math.univ-paris-diderot.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id G-rA5euEgCIe for ; Sat, 15 May 2021 13:16:07 +0200 (CEST) Received: from pirx.irif.fr (unknown [78.194.40.74]) (Authenticated sender: jch) by mailhub.math.univ-paris-diderot.fr (Postfix) with ESMTPSA id C25591075F6 for ; Sat, 15 May 2021 13:16:07 +0200 (CEST) Date: Sat, 15 May 2021 13:16:07 +0200 Message-ID: <87k0o0nt6w.wl-jch@irif.fr> From: Juliusz Chroboczek To: galene@lists.galene.org User-Agent: Wanderlust/2.15.9 (Almost Unreal) Emacs/28.0 Mule/6.0 MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (korolev.univ-paris7.fr [IPv6:2001:660:3301:8000::1:2]); Sat, 15 May 2021 13:16:10 +0200 (CEST) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (potemkin.univ-paris7.fr [194.254.61.141]); Sat, 15 May 2021 13:16:10 +0200 (CEST) X-Miltered: at korolev with ID 609FAD7A.000 by Joe's j-chkmail (http : // j-chkmail dot ensmp dot fr)! X-Miltered: at potemkin with ID 609FAD7A.000 by Joe's j-chkmail (http : // j-chkmail dot ensmp dot fr)! X-j-chkmail-Enveloppe: 609FAD7A.000 from potemkin.univ-paris7.fr/potemkin.univ-paris7.fr/null/potemkin.univ-paris7.fr/ X-j-chkmail-Enveloppe: 609FAD7A.000 from mailhub.math.univ-paris-diderot.fr/mailhub.math.univ-paris-diderot.fr/null/mailhub.math.univ-paris-diderot.fr/ X-j-chkmail-Score: MSGID : 609FAD7A.000 on korolev.univ-paris7.fr : j-chkmail score : . : R=. U=. O=. B=0.000 -> S=0.000 X-j-chkmail-Score: MSGID : 609FAD7A.000 on potemkin.univ-paris7.fr : j-chkmail score : . : R=. U=. O=. B=0.000 -> S=0.000 X-j-chkmail-Status: Ham X-j-chkmail-Status: Ham Message-ID-Hash: ZQQNX6VENHXF3FAXIZOERHBVVE5SCXIY X-Message-ID-Hash: ZQQNX6VENHXF3FAXIZOERHBVVE5SCXIY X-MailFrom: jch@irif.fr X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.4 Precedence: list Subject: [Galene] Merged simulcast and SVC into master List-Id: =?utf-8?q?Gal=C3=A8ne_videoconferencing_server_discussion_list?= Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Dear all, I've just merged all of my simulcast and SVC code into master. This means that the server is now feature-complete for 0.4; all that's needed is some work on the client. The code is running on https://galene.org:8444, and I need you to test, especially with larger groups. (https://galene.org:8443 is still running the 0.3 branch.) It's been a fair amount of work, please be patient while I ramble a bit and describe the different pieces. 1. The sender checks if there are at least three users in the group. If that's the case, it sends two video tracks for each video -- one scaled down stream at 100kbs, and the normal stream. Additionally, both streams have a special "scalable" or "layered" structure, where frames are indexed with an integer (the "temporal index" or "layer number") and only depend on frames with an equal or lower layer index. If you're visually inclined, see figure 3 of https://www.eetimes.com/on2s-truemotion-vp7-video-codec-and-golden-frames/ Caveats: we disable the feature on Firefox, which uses a different protocol for simulcast that Pion doesn't support yet. It is unknown whether it works on Safari. I've seen strange behaviour with VP9, I need to experiment some more. 2. When distributing flows to clients, the server picks among the available tracks and sends just one track. The choice of track is controlled by the receiving client (using the "request" and "streamRequest" protocol messages, which are exported by protocol.js as the ServerConnection.request and Stream.request methods). 3. Instead of using videos at the slowest rate among all clients, we pick the rate as min(slowest * 2^(layers - 1), fastest) Note that this degenerates to the slowest rate if the video is not scalable, just like before. The ratio might need to be tweaked, I haven't done much testing yet. Caveat: since the video rate is now higher than it used to be, this might put more load on your server. I'll implement a per-group throughput limit if this turns out to be a problem. 3. The server detects whether a given flow is scalable. If the client is congested, the server uses the layered structure to decimate the flow -- a congested client only receives frames from the lower layers. The nice feature is that the resulting flow is a perfectly compliant VP8 flow, and no special support is needed on the receiver. We switch layers at most once every 500ms, which is the rate at which we get congestion indications from the receiver. There are no visible artifacts when switching -- SVC is as close to magic as you can get. The rewriting has some overhead, both in space and time. Both are negligible: the space cost is on the order of 1kB per flow, while the cost of rewriting the packets is just a couple percent in my tests. We detect the case where no rewriting is necessary, and take a fast path. Caveat: the result is that congested clients receive jerky video instead of the video quality decreasing for everyone equally. This is good for lectures, but not necessarily for dancing lessons. VP9 and AV1 allow spacial scalability (reducing the resolution instead of the framerate), and this appears to be implemented in Chromium 92 for AV1 (but not for VP9). Caveat: the highest layer consists of frames that can be individually discarded, which implies that we could do finer congestion control by discarding just a fraction of the higher-layer frames. We don't do that right now, we simply fall back to the intermediate layer 4. The client dynamically selects the flow it requests. Currently, the logic is as follows - if the video is in full screen, we request the high-resolution flow; - if the video's (scrollWidht*scrollHeight) is less than 76800, we request the low-resolution flow; - otherwise we request the high-resolution flow. This can be overridden in the side menu (choose "Receive: low"). Caveat: a client switching flows requests a new keyframe, which causes a degradation of quality for all other clients. Perhaps it's worth n adding some hysteresis, we'll see. Caveat: if the browser window is small, the server sends the high-resolution flow and the client which immediately requests the low-resolution flow. A solution would be to use two-step negotiation, which would add one RTT to every flow establishment. I'll consider that, but not for the next version. Caveat: in the current implementation, switching flows causes visible flicker. A solution would be to create a new