* [Galene] Merged simulcast and SVC into master
@ 2021-05-15 11:16 Juliusz Chroboczek
2021-05-15 18:12 ` [Galene] " Juliusz Chroboczek
0 siblings, 1 reply; 2+ messages in thread
From: Juliusz Chroboczek @ 2021-05-15 11:16 UTC (permalink / raw)
To: galene
Dear all,
I've just merged all of my simulcast and SVC code into master. This means
that the server is now feature-complete for 0.4; all that's needed is some
work on the client.
The code is running on https://galene.org:8444, and I need you to test,
especially with larger groups. (https://galene.org:8443 is still running
the 0.3 branch.)
It's been a fair amount of work, please be patient while I ramble a bit
and describe the different pieces.
1. The sender checks if there are at least three users in the group. If
that's the case, it sends two video tracks for each video -- one scaled
down stream at 100kbs, and the normal stream.
Additionally, both streams have a special "scalable" or "layered"
structure, where frames are indexed with an integer (the "temporal
index" or "layer number") and only depend on frames with an equal or
lower layer index. If you're visually inclined, see figure 3 of
https://www.eetimes.com/on2s-truemotion-vp7-video-codec-and-golden-frames/
Caveats: we disable the feature on Firefox, which uses a different
protocol for simulcast that Pion doesn't support yet. It is unknown
whether it works on Safari. I've seen strange behaviour with VP9,
I need to experiment some more.
2. When distributing flows to clients, the server picks among the
available tracks and sends just one track. The choice of track is
controlled by the receiving client (using the "request" and
"streamRequest" protocol messages, which are exported by protocol.js
as the ServerConnection.request and Stream.request methods).
3. Instead of using videos at the slowest rate among all clients, we pick
the rate as
min(slowest * 2^(layers - 1), fastest)
Note that this degenerates to the slowest rate if the video is not
scalable, just like before. The ratio might need to be tweaked,
I haven't done much testing yet.
Caveat: since the video rate is now higher than it used to be, this
might put more load on your server. I'll implement a per-group
throughput limit if this turns out to be a problem.
3. The server detects whether a given flow is scalable. If the client is
congested, the server uses the layered structure to decimate the
flow -- a congested client only receives frames from the lower layers.
The nice feature is that the resulting flow is a perfectly compliant
VP8 flow, and no special support is needed on the receiver.
We switch layers at most once every 500ms, which is the rate at which
we get congestion indications from the receiver. There are no visible
artifacts when switching -- SVC is as close to magic as you can get.
The rewriting has some overhead, both in space and time. Both are
negligible: the space cost is on the order of 1kB per flow, while the
cost of rewriting the packets is just a couple percent in my tests. We
detect the case where no rewriting is necessary, and take a fast path.
Caveat: the result is that congested clients receive jerky video
instead of the video quality decreasing for everyone equally. This is
good for lectures, but not necessarily for dancing lessons. VP9 and
AV1 allow spacial scalability (reducing the resolution instead of the
framerate), and this appears to be implemented in Chromium 92 for AV1
(but not for VP9).
Caveat: the highest layer consists of frames that can be individually
discarded, which implies that we could do finer congestion control by
discarding just a fraction of the higher-layer frames. We don't do
that right now, we simply fall back to the intermediate layer
4. The client dynamically selects the flow it requests. Currently, the
logic is as follows
- if the video is in full screen, we request the high-resolution flow;
- if the video's (scrollWidht*scrollHeight) is less than 76800, we
request the low-resolution flow;
- otherwise we request the high-resolution flow.
This can be overridden in the side menu (choose "Receive: low").
Caveat: a client switching flows requests a new keyframe, which causes
a degradation of quality for all other clients. Perhaps it's worth
n adding some hysteresis, we'll see.
Caveat: if the browser window is small, the server sends the
high-resolution flow and the client which immediately requests the
low-resolution flow. A solution would be to use two-step negotiation,
which would add one RTT to every flow establishment. I'll consider
that, but not for the next version.
Caveat: in the current implementation, switching flows causes visible
flicker. A solution would be to create a new <video> element instead
of changing the stream mid-video. Another solution would be to stitch
streams in the server rather than switch streams in the client, but
that would mean that both streams need to use the same codec; I'm
hoping that at some point I can use VP8 for the low-resolution stream
and AV1 for the high-resolution once, which would enable sending
high-resolution AV1 to clients that support it and low-resolution VP8
to the others.
Enjoy,
-- Juliusz
^ permalink raw reply [flat|nested] 2+ messages in thread
* [Galene] Re: Merged simulcast and SVC into master
2021-05-15 11:16 [Galene] Merged simulcast and SVC into master Juliusz Chroboczek
@ 2021-05-15 18:12 ` Juliusz Chroboczek
0 siblings, 0 replies; 2+ messages in thread
From: Juliusz Chroboczek @ 2021-05-15 18:12 UTC (permalink / raw)
To: galene
> VP9 and AV1 allow spacial scalability (reducing the resolution
> instead of the framerate), and this appears to be implemented in
> Chromium 92 for AV1 (but not for VP9).
This turns out to be incorrect. Chromium supports spacial scalability for
VP9, although it doesn't advertise it.
On the other hand, I'm getting errors from the AV1 encoder if I try:
[25554:28:0515/195635.968545:ERROR:video_stream_encoder.cc(1750)] Failed to encode frame. Error code: -7
-- Juliusz
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2021-05-15 18:12 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-15 11:16 [Galene] Merged simulcast and SVC into master Juliusz Chroboczek
2021-05-15 18:12 ` [Galene] " Juliusz Chroboczek
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox