[Galene] about packet pacers

[Galene] about packet pacers

[Guillaume Denis]

[-- Attachment #1: Type: text/plain, Size: 1491 bytes --]

Hello Galène,
hello Toke (I might have a question for you below!)

For a bit of context I am working on a project [1] that uses pion, in a
different fashion than Galène since we transform audio/video streams and
add effects to them, implying that we decode and reencode them server side
(with GStreamer).

My initial question was about how to adapt encoder settings (bitrate) to
network conditions, for instance not to send a 200KB/s video to a 100KB/s
capable network.
Juliusz pointed Google Congestion Control Algorithm [2] and Galène's
implementation out to me: thanks to RTCP receiver and REMB reports, and
thanks to the GCC algorithm, I can compute a desirable bitrate.

Now this bitrate can be used to monitor encoder settings. This article
https://www.aitrans.online/static/paper/Gcc-analysis.pdf also mentions that
pacers and padders should be adapted.

Toke, would you have some input regarding this:
- do you think that in this use case monitoring a pacer with the target
bitrate would have real added value?
- I guess the pacer must act before the RTP packetizer so that timing
information present in RTP headers are correct/not modified by the pacer.
Is this correct?
- are you aware of any reusable pacer implementation? I am also wondering
if it would make more sense to be part of GStreamer processing or in pion

Thanks,
Guillaume

[1] https://github.com/creamlab/ducksoup
[2] https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-gcc-02

[-- Attachment #2: Type: text/html, Size: 1903 bytes --]

[Galene] Re: about packet pacers

[Juliusz Chroboczek]

I'll let Toke (or Dave?) answer the other points, but here's one I think
I understand.

> - I guess the pacer must act before the RTP packetizer so that timing
> information present in RTP headers are correct/not modified by the pacer. Is
> this correct?

As far as I understand, the pacer is applied after the RTP packetiser.
For each frame (or "picture" in VP9) produduced by the codec, the RTP
packetiser will produce a burst of packets with the same timestamp.  If
these packets are sent in quick succession, they will temporarily congest
the sending interface or the bottleneck router.

The role of the pacer, as I undestand it, is to schedule the set of
packets in a single frame so that they go out at regular intervals.  The
effect is a slight increase in delay (since the frame needs to be
reassembled at the receiver before it can be processed), in jitter (since
some packets might be larger than others), but a reduction in short-term
network congestion and therefore in packet loss.

The pacer does not rewrite RTP timestamps: the RTP timestamp describes
where the packets fit in the reconstructed video, it has little to do with
the time at which the packet is actually sent.  (If the actual sending
time is needed, there's the abs-send-time extension, which should be
inserted after the pacer.)

Galene does not use a pacer, since it merely forwards packets that have
already been properly disciplined by the sender.  (There is one exception,
it's when we send a cached keyframe to a new receiver -- we currently send
the keyframe as a burst, we should probably be conditioning it.)

-- Juliusz

[Galene] Re: about packet pacers

[Toke Høiland-Jørgensen]

Guillaume Denis <gdenispro@gmail.com> writes:

> Toke, would you have some input regarding this:
> - do you think that in this use case monitoring a pacer with the target
> bitrate would have real added value?

Hmm, so the idea here would be that we know the bitrate of the stream
from the codec setting, and from that we can compute the packet-level
bitrate (by adding the packet overhead) and pace out the packets at that
rate? As Juliusz says, that could alleviate some short-term congestion
if the codec produces very bursty packet streams, which may be a win.
One question is whether the codec can be relied upon to stay within a
certain bitrate and over what timescales? If the pacer rate is set
wrong, it could end up bottlenecking the codec output at the local
sender, causing a queue to build there, which would probably be bad...

> - I guess the pacer must act before the RTP packetizer so that timing
> information present in RTP headers are correct/not modified by the pacer.
> Is this correct?

This one I think Juliusz answered in the negative (which is good,
because I had no idea ;))

> - are you aware of any reusable pacer implementation? I am also
> wondering if it would make more sense to be part of GStreamer
> processing or in pion

Well it would need to be at whatever level produces the packet data;
you'd probably want to queue up fully-formed packets and buffer them to
the send rate. Not really aware of any reusable pacer implementations.
On Linux you can use the SO_TXTIME socket option to get the kernel to
pace the packets for you; but that only works if there's a qdisc
installed that can handle the pacing (such as sch_fq), and it's hardly
cross-platform, so dunno if that will work?

-Toke

[Galene] Re: about packet pacers

[Juliusz Chroboczek]

>> Toke, would you have some input regarding this:
>> - do you think that in this use case monitoring a pacer with the target
>> bitrate would have real added value?

> Hmm, so the idea here would be that we know the bitrate of the stream
> from the codec setting, and from that we can compute the packet-level
> bitrate (by adding the packet overhead) and pace out the packets at that
> rate? [...]
> One question is whether the codec can be relied upon to stay within a
> certain bitrate and over what timescales?

No, but we know where the current frame ends, and we know when the next
frame is due.  So we can spread the current frame smoothly over the
interval between the current frame and the next one.  That will break when
we get variable frame-rate codecs, but we'll cross that bridge when we get
there.

This is not what the GCC draft recommends[1]: what they suggest is to
consider the target bitrate (the one that the congestion controller yields)
and send a burst of up to 5ms*rate every 5ms.

[1] https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-gcc-02#section-4

> On Linux you can use the SO_TXTIME socket option to get the kernel to
> pace the packets for you; but that only works if there's a qdisc
> installed that can handle the pacing (such as sch_fq), and it's hardly
> cross-platform, so dunno if that will work?

I'd rather we did the pacing in the application, since that will allow us
to drop packets more selectively: if we drop a packet due to congestion
control, then we want to continue dropping until the end of the frame.
What's more, both the VP8 and VP9 codecs can be configured in a mode where
they regularly schedule a "droppable" frame that is not used for
intra-prediction (it's part of the simulcast implementation).

-- Juliusz

[Galene] Re: about packet pacers

[Guillaume Denis]

[-- Attachment #1: Type: text/plain, Size: 2377 bytes --]

Toke, Juliusz, thanks for your input.

As a followup and even if it's off topic for Galene users I'll share this
implementation of the GCC algorithm based on RTCP packets:
https://github.com/creamlab/ducksoup/blob/main/sfu/mixer.go#L203
(with a resulting periodical update of GStreamer encoder bitrates)

Using pacers is a bit too networky for me right now, I'll see how much I
need it.
Thanks, Guillaume

Le sam. 7 août 2021 à 01:01, Juliusz Chroboczek <jch@irif.fr> a écrit :

> >> Toke, would you have some input regarding this:
> >> - do you think that in this use case monitoring a pacer with the target
> >> bitrate would have real added value?
>
> > Hmm, so the idea here would be that we know the bitrate of the stream
> > from the codec setting, and from that we can compute the packet-level
> > bitrate (by adding the packet overhead) and pace out the packets at that
> > rate? [...]
> > One question is whether the codec can be relied upon to stay within a
> > certain bitrate and over what timescales?
>
> No, but we know where the current frame ends, and we know when the next
> frame is due.  So we can spread the current frame smoothly over the
> interval between the current frame and the next one.  That will break when
> we get variable frame-rate codecs, but we'll cross that bridge when we get
> there.
>
> This is not what the GCC draft recommends[1]: what they suggest is to
> consider the target bitrate (the one that the congestion controller yields)
> and send a burst of up to 5ms*rate every 5ms.
>
> [1]
> https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-gcc-02#section-4
>
> > On Linux you can use the SO_TXTIME socket option to get the kernel to
> > pace the packets for you; but that only works if there's a qdisc
> > installed that can handle the pacing (such as sch_fq), and it's hardly
> > cross-platform, so dunno if that will work?
>
> I'd rather we did the pacing in the application, since that will allow us
> to drop packets more selectively: if we drop a packet due to congestion
> control, then we want to continue dropping until the end of the frame.
> What's more, both the VP8 and VP9 codecs can be configured in a mode where
> they regularly schedule a "droppable" frame that is not used for
> intra-prediction (it's part of the simulcast implementation).
>
> -- Juliusz
>

[-- Attachment #2: Type: text/html, Size: 3030 bytes --]

[Galene] Re: about packet pacers

[Juliusz Chroboczek]

> As a followup and even if it's off topic for Galene users I'll share
> this implementation of the GCC algorithm based on RTCP packets:
> https://github.com/creamlab/ducksoup/blob/main/sfu/mixer.go#L203 (with
> a resulting periodical update of GStreamer encoder bitrates)

It's only the sender side implementation, which is the easy bit (and which
Galene already implements).

-- Juliusz

[Galene] Re: about packet pacers

[Juliusz Chroboczek]

> It's only the sender side implementation, which is the easy bit (and which
> Galene already implements).

I've now looked at the code in more detail, and their loss-based
controller is copy-pasted from Galene, even the comments are the same.

-- Juliusz

[Galene] Re: about packet pacers

[Guillaume Denis]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]

Hello Juliusz,

Yes I copy/pasted it from Galène (and even the comments since I didn't mean
to hide it). Sorry if it wasn't clear (and for the noise here), I added a
reference to Galène to outline it, in a genuine comment ;)

The particularity in DuckSoup is that this bitrate estimation is used to
control encoder settings in GStreamer (to be improved, but there is a first
working version).

Guillaume

Le mer. 18 août 2021 à 23:32, Juliusz Chroboczek <jch@irif.fr> a écrit :

> > It's only the sender side implementation, which is the easy bit (and
> which
> > Galene already implements).
>
> I've now looked at the code in more detail, and their loss-based
> controller is copy-pasted from Galene, even the comments are the same.
>
> -- Juliusz
>

[-- Attachment #2: Type: text/html, Size: 1093 bytes --]

[Galene] Re: about packet pacers

[Juliusz Chroboczek]

> Yes I copy/pasted it from Galène

Sure, it's fine to copy code from Galène.  (I would have appreciated it if
you had cited your source, though, just as you would do in a scientific
publication.)

I understood your comment as implying that Ducksoup would be a good source
of inspiration for Galene.  After examining it, it looks to me like it
implements roughly the same parts of GCC as Galène, but uses the
information in a different manner.

> The particularity in DuckSoup is that this bitrate estimation is used to
> control encoder settings in GStreamer

Ducksoup uses a rather original architecture.  At a cursory glance, it
looks like it reencodes all streams at the server, MCU-style, but without
combining the streams in a single composite video, SFU-style.  I think it
would be useful to put this information in the README.

-- Juliusz

[Galene] Re: about packet pacers

[Dave Taht]

I was terribly busy when this thread went by.

On Fri, Aug 6, 2021 at 4:01 PM Juliusz Chroboczek <jch@irif.fr> wrote:
>
> >> Toke, would you have some input regarding this:
> >> - do you think that in this use case monitoring a pacer with the target
> >> bitrate would have real added value?
>
> > Hmm, so the idea here would be that we know the bitrate of the stream
> > from the codec setting, and from that we can compute the packet-level
> > bitrate (by adding the packet overhead) and pace out the packets at that
> > rate? [...]
> > One question is whether the codec can be relied upon to stay within a
> > certain bitrate and over what timescales?
>
> No, but we know where the current frame ends, and we know when the next
> frame is due.  So we can spread the current frame smoothly over the
> interval between the current frame and the next one.  That will break when
> we get variable frame-rate codecs, but we'll cross that bridge when we get
> there.
>
> This is not what the GCC draft recommends[1]: what they suggest is to
> consider the target bitrate (the one that the congestion controller yields)
> and send a burst of up to 5ms*rate every 5ms.

That draft is old, and I don't know the right answer, but I'm pretty sure that's
not the right answer.

> [1] https://datatracker.ietf.org/doc/html/draft-ietf-rmcat-gcc-02#section-4
>
> > On Linux you can use the SO_TXTIME socket option to get the kernel to
> > pace the packets for you; but that only works if there's a qdisc
> > installed that can handle the pacing (such as sch_fq), and it's hardly
> > cross-platform, so dunno if that will work?

I would like to try this - as the simplest possible implementation to
experiment with.

>
> I'd rather we did the pacing in the application, since that will allow us
> to drop packets more selectively: if we drop a packet due to congestion
> control, then we want to continue dropping until the end of the frame.

To be clear this is when you are dealing with galene being internally congested,
blocking so much on write that it needs to drop a frame?

> What's more, both the VP8 and VP9 codecs can be configured in a mode where
> they regularly schedule a "droppable" frame that is not used for
> intra-prediction (it's part of the simulcast implementation).

oh, excellent. Are you using this at all at this point, to probe for
more bandwidth.

>
> -- Juliusz
> _______________________________________________
> Galene mailing list -- galene@lists.galene.org
> To unsubscribe send an email to galene-leave@lists.galene.org



-- 
Fixing Starlink's Latencies: https://www.youtube.com/watch?v=c9gLo6Xrwgw

Dave Täht CEO, TekLibre, LLC