* [Galene] Building a streaming gateway for Galène
@ 2021-04-06 11:05 Alexandre IOOSS
2021-04-06 12:57 ` [Galene] " Juliusz Chroboczek
0 siblings, 1 reply; 12+ messages in thread
From: Alexandre IOOSS @ 2021-04-06 11:05 UTC (permalink / raw)
To: galene
Hello all,
For the past two weeks I have been working on streaming from a
broadcasting software such as OBS-Studio (that uses FFMpeg) to Galène.
This would enable users to use Galène for low-latency video streaming
and as a robust self-hosted Twitch alternative for conferences.
I believe we don't want to have to do any video/audio conversion on the
server side, so I see multiple possibilities to stream to Galène:
0. Using OBS Virtual v4l2 camera, but it is not portable and has audio
sync issues.
1. Streaming to a gateway that will initiate the WebRTC session with Galène.
2. Implement WebRTC support in OBS-Studio as an output plugin.
The point 2 was already tried by
<https://github.com/CoSMoSoftware/OBS-studio-webrtc> but it seems to be
problematic as they are using the WebRTC library from Chromium. I don't
expect this OBS fork to be merged upstream soon.
Last week I tried to implement a gateway using GStreamer webrtcbin
plugin, <https://github.com/erdnaxe/galene-stream>. Although I am able
to stream to Galène, I observe video drops. I am circumventing the issue
by forcing the VP8 encoder to insert a keyframe every 5 frames, but that
seems a terrible hack.
Right now I am a bit lost, I don't know if I should either try to
replace GStreamer webrtcbin with Pion
(https://github.com/pion/example-webrtc-applications/tree/master/gstreamer-send),
or if something is wrong with GStreamer/Galène. Does anyone have advice?
Best regards,
--
Alexandre Iooss
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Galene] Re: Building a streaming gateway for Galène
2021-04-06 11:05 [Galene] Building a streaming gateway for Galène Alexandre IOOSS
@ 2021-04-06 12:57 ` Juliusz Chroboczek
2021-04-06 16:10 ` [Galene] webrtc security cameras? Dave Taht
2021-04-06 17:44 ` [Galene] Re: Building a streaming gateway for Galène Alexandre IOOSS
0 siblings, 2 replies; 12+ messages in thread
From: Juliusz Chroboczek @ 2021-04-06 12:57 UTC (permalink / raw)
To: Alexandre IOOSS; +Cc: galene
> For the past two weeks I have been working on streaming from a
> broadcasting software such as OBS-Studio (that uses FFMpeg) to Galène.
This would be a great addition! We have quite a few users that use OBS,
and, as you mention, going through the v4l virtual camera has some
disadvantages.
> 0. Using OBS Virtual v4l2 camera, but it is not portable and has audio
> sync issues.
> 1. Streaming to a gateway that will initiate the WebRTC session with Galène.
> 2. Implement WebRTC support in OBS-Studio as an output plugin.
I would tend to agree, (1) is probably the best solution, since it avoids
bloating either Galène or OBS.
> Last week I tried to implement a gateway using GStreamer webrtcbin
> plugin, <https://github.com/erdnaxe/galene-stream>. Although I am able
> to stream to Galène, I observe video drops. I am circumventing the issue
> by forcing the VP8 encoder to insert a keyframe every 5 frames, but that
> seems a terrible hack.
I'll have a look at your code as soon as I have some time. In the
meantime, there are three things that you need to do:
1. send periodic RTCP Sender Reports with accurate NTP times;
2. react to RTCP PLI packets by inserting a keyframe;
3. react to RTCP NACK packets by resending the packets listed in the NACK.
(1) is essential for proper lipsynch. (2) is what allows a stream to
recover after a catastrophic loss event. (3) is what allows a stream to
recover rapidly after moderate amounts of loss.
> Right now [...] I don't know if I should...
I'll have a look, think it over, then get back to you.
-- Juliusz
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Galene] webrtc security cameras?
2021-04-06 12:57 ` [Galene] " Juliusz Chroboczek
@ 2021-04-06 16:10 ` Dave Taht
2021-04-06 17:07 ` [Galene] " Juliusz Chroboczek
2021-04-06 17:44 ` [Galene] Re: Building a streaming gateway for Galène Alexandre IOOSS
1 sibling, 1 reply; 12+ messages in thread
From: Dave Taht @ 2021-04-06 16:10 UTC (permalink / raw)
To: Juliusz Chroboczek; +Cc: Alexandre IOOSS, galene
Whilst we are doing speculative stuff, one thing that irks me is that
most of the security cameras in the world, either phone home to china,
or to google, with no sources available in either case. This bothers
me a lot, not just for baby cameras, but as I'd rather send video from
up to downstairs through my home router only in general, and log
locally...
I am curious if any cameras exist that can do webrtc more "right" and come with
open sources. I am well aware of the raspi, but am too lazy to want to build a
housing for it or run power to it.... poe would be good....
What I am thinking of doing longer term is trying to get to where I could blast
raw-er video packets through a video distribution box in the studio, so at last,
finally, my long cherished "jamophone" could come to life.
See also: https://lola.conts.it/
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Galene] Re: webrtc security cameras?
2021-04-06 16:10 ` [Galene] webrtc security cameras? Dave Taht
@ 2021-04-06 17:07 ` Juliusz Chroboczek
0 siblings, 0 replies; 12+ messages in thread
From: Juliusz Chroboczek @ 2021-04-06 17:07 UTC (permalink / raw)
To: Dave Taht; +Cc: Alexandre IOOSS, galene
> I am curious if any cameras exist that can do webrtc more "right" and come with
> open sources. I am well aware of the raspi, but am too lazy to want to build a
> housing for it or run power to it.... poe would be good....
There do exist quality cameras that connect locally to a server that you
control, but they use RTMP, which is an obsolescent protocol, running over
TCP, and that cannot easily be extended to handle modern codecs.
If I understand Alexandre correctly, he's building an RTMP-to-Galène
gateway. Such a gateway should work with off-the-shelf cameras, or at
least should be easy to tweak so that it works with them.
In the longer term, we should work on replacing RTMP with a modern
protocol. That's what the WISH working group is chartered to do at the
IETF, and, given the competent people I've seen participating, I have at
least some hope that they'll be able to deliver.
In the short term, let's work with Alexandre on getting OBS to
interoperate with Galène. Once that's done, we'll see what we can do
about off-the-shelf hardware.
-- Juliusz
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Galene] Re: Building a streaming gateway for Galène
2021-04-06 12:57 ` [Galene] " Juliusz Chroboczek
2021-04-06 16:10 ` [Galene] webrtc security cameras? Dave Taht
@ 2021-04-06 17:44 ` Alexandre IOOSS
2021-04-06 23:19 ` Juliusz Chroboczek
1 sibling, 1 reply; 12+ messages in thread
From: Alexandre IOOSS @ 2021-04-06 17:44 UTC (permalink / raw)
To: Juliusz Chroboczek; +Cc: galene
On 4/6/21 2:57 PM, Juliusz Chroboczek wrote:
> I'll have a look at your code as soon as I have some time. In the
> meantime, there are three things that you need to do:
>
> 1. send periodic RTCP Sender Reports with accurate NTP times;
> 2. react to RTCP PLI packets by inserting a keyframe;
> 3. react to RTCP NACK packets by resending the packets listed in the NACK.
>
> (1) is essential for proper lipsynch. (2) is what allows a stream to
> recover after a catastrophic loss event. (3) is what allows a stream to
> recover rapidly after moderate amounts of loss.
When dumping traffic in Wireshark, I see only RTCP "Sender report" and
"Receiver report" packets. Sender reports are periodic, and sent by
groups of 2 (video then audio):
```
Real-time Transport Control Protocol (Sender Report)
10.. .... = Version: RFC 1889 Version (2)
..0. .... = Padding: False
...0 0000 = Reception report count: 0
Packet type: Sender Report (200)
Length: 6 (28 bytes)
Sender SSRC: 0x9ce1015b (2631991643)
Timestamp, MSW: 2797058655 (0xa6b7ba5f)
Timestamp, LSW: 1448315617 (0x56538ae1)
[MSW and LSW as NTP timestamp: Aug 20, 1988 08:44:15.337212257 UTC]
RTP timestamp: 866227088
Sender's packet count: 2194185357
Sender's octet count: 2612991028
[RTCP frame length check: OK - 28 bytes]
Real-time Transport Control Protocol (Sender Report)
10.. .... = Version: RFC 1889 Version (2)
..0. .... = Padding: False
...0 0000 = Reception report count: 0
Packet type: Sender Report (200)
Length: 6 (28 bytes)
Sender SSRC: 0x3a724a90 (980568720)
Timestamp, MSW: 4079490799 (0xf32816ef)
Timestamp, LSW: 2870575121 (0xab198011)
[MSW and LSW as NTP timestamp: Apr 10, 2029 07:53:19.668357853 UTC]
RTP timestamp: 2166723788
Sender's packet count: 468654353
Sender's octet count: 2838781192
[RTCP frame length check: OK - 28 bytes]
```
I don't really understand what you mean by "accurate NTP times". Streams
from Firefox or GStreamer seem to use random NTP timestamps.
When dumping Firefox WebRTC traffic, I also see "Generic RTP Feeback:
NACK" packets, which confirms the fact that GStreamer is not doing NACK
in my script. I should see if it's possible to enable it.
I am still quite new to WebRTC, so be aware that I might have done
something stupid with the GStreamer pipeline.
Thank you for pointing out this could be a potential problem,
--
Alexandre Iooss
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Galene] Re: Building a streaming gateway for Galène
2021-04-06 17:44 ` [Galene] Re: Building a streaming gateway for Galène Alexandre IOOSS
@ 2021-04-06 23:19 ` Juliusz Chroboczek
2021-04-07 20:40 ` Alexandre IOOSS
0 siblings, 1 reply; 12+ messages in thread
From: Juliusz Chroboczek @ 2021-04-06 23:19 UTC (permalink / raw)
To: Alexandre IOOSS; +Cc: galene
> I don't really understand what you mean by "accurate NTP times". Streams
> from Firefox or GStreamer seem to use random NTP timestamps.
The NTP timestamp is used by the receiver to synchronise audio with video.
It does not need to be accurate, but the NTP timestamps of correlated
tracks (audio and video for a single video) should use the same clock,
otherwise the receiver won't be able to resynchronise them. This is
particularly important for talking heads (lectures and conferences).
> When dumping Firefox WebRTC traffic, I also see "Generic RTP Feeback:
> NACK" packets, which confirms the fact that GStreamer is not doing NACK
> in my script. I should see if it's possible to enable it.
This will improve things a lot. You'll need to keep a history of recently
sent packets, I don't know if gstreamer can do that automatically.
As to keyframes — you should send periodic keyframes, but you shouldn't do
it too often (Chrome sends one every 120s). On the other hand, you should
send a keyframe fairly quickly (within 500ms or so) whenever you receive
a PLI (Picture Loss Indication) RTCP packet.
> I am still quite new to WebRTC, so be aware that I might have done
> something stupid with the GStreamer pipeline.
It's a complicated protocol stack, don't worry too much about getting all
the bits right straight away.
-- Juliusz
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Galene] Re: Building a streaming gateway for Galène
2021-04-06 23:19 ` Juliusz Chroboczek
@ 2021-04-07 20:40 ` Alexandre IOOSS
2021-04-07 22:01 ` Juliusz Chroboczek
0 siblings, 1 reply; 12+ messages in thread
From: Alexandre IOOSS @ 2021-04-07 20:40 UTC (permalink / raw)
To: Juliusz Chroboczek; +Cc: galene
On 4/7/21 1:19 AM, Juliusz Chroboczek wrote:>> When dumping Firefox
WebRTC traffic, I also see "Generic RTP Feeback:
>> NACK" packets, which confirms the fact that GStreamer is not doing NACK
>> in my script. I should see if it's possible to enable it.
>
> This will improve things a lot. You'll need to keep a history of recently
> sent packets, I don't know if gstreamer can do that automatically.
>
> As to keyframes — you should send periodic keyframes, but you shouldn't do
> it too often (Chrome sends one every 120s). On the other hand, you should
> send a keyframe fairly quickly (within 500ms or so) whenever you receive
> a PLI (Picture Loss Indication) RTCP packet.
I have good and bad news. The good news is that GStreamer webrtcbin
supports NACK and PLI. I added those lines:
# Enable WebRTC negative acknowledgement and FEC
transceiver_count = self.webrtc.emit("get-transceivers").len
for i in range(transceiver_count):
transceiver = self.webrtc.emit("get-transceiver", i)
transceiver.set_property("do-nack", True)
transceiver.set_property("fec-type", GstWebRTC.WebRTCFECType.ULP_RED)
Now I see "Generic RTP Feeback: NACK" packets in Wireshark and the SDP
offer being sent to Galène also looks better:
[...]
a=sendrecv
a=rtpmap:97 VP8/90000
a=rtcp-fb:97 nack
a=rtcp-fb:97 nack pli
a=framerate:30
a=rtpmap:98 red/90000
a=rtpmap:99 ulpfec/90000
a=rtpmap:100 rtx/90000
a=fmtp:100 apt=98
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=97
a=ssrc-group:FID 1611068776 2492991179
[...]
a=sendrecv
a=rtpmap:96 OPUS/48000/2
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=fmtp:96 sprop-maxcapturerate=48000;sprop-stereo=1
a=rtpmap:102 red/48000
a=rtpmap:103 ulpfec/48000
a=rtpmap:104 rtx/48000
a=fmtp:104 apt=102
a=rtpmap:105 rtx/48000
a=fmtp:105 apt=96
a=ssrc-group:FID 2413482691 909406007
[...]
The bad news is that the stream is still dropping. If I print some
statistics during streaming, GStreamer reports `pli-count=0` and
`nack-count=0`, so something seems broken.
Looking the issue online I found
<https://mediasoup.discourse.group/t/broadcasting-a-vp8-rtp-stream-from-gstreamer/93/18>:
> GStreamer first looks at the sender ssrc. If it gets NULL then it
> tries with media ssrc. [...] somehow, GStreamer is getting “something”
> when looking for sender ssrc = 0, which is crazy.
Looking at the Wireshark dump, Galène seems also to use "0" as sender
SSRC, so that might be the issue. I might try to do a dirty hack
somewhere to confirm this hypothesis.
Does Galène has ulpfec support (https://github.com/pion/webrtc/issues/1418)?
Should I do PLI and ulpfec on audio and video in this specific
application or is it a terrible idea?
Thank you again for all your advice!
--
Alexandre Iooss
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Galene] Re: Building a streaming gateway for Galène
2021-04-07 20:40 ` Alexandre IOOSS
@ 2021-04-07 22:01 ` Juliusz Chroboczek
2021-04-07 22:05 ` Juliusz Chroboczek
0 siblings, 1 reply; 12+ messages in thread
From: Juliusz Chroboczek @ 2021-04-07 22:01 UTC (permalink / raw)
To: Alexandre IOOSS; +Cc: galene
> Looking at the Wireshark dump, Galène seems also to use "0" as sender
> SSRC, so that might be the issue. I might try to do a dirty hack
> somewhere to confirm this hypothesis.
Galène doesn't choose the SSRC -- the SSRC is chosen by the offerer.
Perhaps you forgot to initialise something?
> Does Galène has ulpfec support
No, since it's supported by neither Chrome nor Firefox. In the presence
of NACK, it's not very useful for video (the small freezes caused by NACK
recovery are acceptable), but it could dramatically improve audio quality.
I intend to implement ulpfec as soon as it's supported in Chrome.
-- Juliusz
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Galene] Re: Building a streaming gateway for Galène
2021-04-07 22:01 ` Juliusz Chroboczek
@ 2021-04-07 22:05 ` Juliusz Chroboczek
2021-04-09 13:42 ` Alexandre IOOSS
0 siblings, 1 reply; 12+ messages in thread
From: Juliusz Chroboczek @ 2021-04-07 22:05 UTC (permalink / raw)
To: Alexandre IOOSS; +Cc: galene
>> Looking at the Wireshark dump, Galène seems also to use "0" as sender
>> SSRC, so that might be the issue. I might try to do a dirty hack
>> somewhere to confirm this hypothesis.
> Galène doesn't choose the SSRC -- the SSRC is chosen by the offerer.
You can find out the SSRC that Galène thinks the offerer has chosen by
adding the following line:
diff --git a/rtpconn/rtpconn.go b/rtpconn/rtpconn.go
index eec5d97..6b7e8cb 100644
--- a/rtpconn/rtpconn.go
+++ b/rtpconn/rtpconn.go
@@ -509,6 +509,7 @@ func newUpConn(c group.Client, id string, labels map[string]string, offer string
up := &rtpUpConnection{id: id, pc: pc, labels: labels}
pc.OnTrack(func(remote *webrtc.TrackRemote, receiver *webrtc.RTPReceiver) {
+ log.Println("SSID", remote.SSRC())
up.mu.Lock()
mid := getTrackMid(pc, remote)
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Galene] Re: Building a streaming gateway for Galène
2021-04-07 22:05 ` Juliusz Chroboczek
@ 2021-04-09 13:42 ` Alexandre IOOSS
2021-04-09 13:55 ` Juliusz Chroboczek
2021-04-09 14:17 ` Juliusz Chroboczek
0 siblings, 2 replies; 12+ messages in thread
From: Alexandre IOOSS @ 2021-04-09 13:42 UTC (permalink / raw)
To: Juliusz Chroboczek; +Cc: galene
On 4/8/21 12:05 AM, Juliusz Chroboczek wrote:
>>> Looking at the Wireshark dump, Galène seems also to use "0" as sender
>>> SSRC, so that might be the issue. I might try to do a dirty hack
>>> somewhere to confirm this hypothesis.
>
>> Galène doesn't choose the SSRC -- the SSRC is chosen by the offerer.
>
> You can find out the SSRC that Galène thinks the offerer has chosen by
> adding the following line:
>
> diff --git a/rtpconn/rtpconn.go b/rtpconn/rtpconn.go
> index eec5d97..6b7e8cb 100644
> --- a/rtpconn/rtpconn.go
> +++ b/rtpconn/rtpconn.go
> @@ -509,6 +509,7 @@ func newUpConn(c group.Client, id string, labels map[string]string, offer string
> up := &rtpUpConnection{id: id, pc: pc, labels: labels}
>
> pc.OnTrack(func(remote *webrtc.TrackRemote, receiver *webrtc.RTPReceiver) {
> + log.Println("SSID", remote.SSRC())
> up.mu.Lock()
>
> mid := getTrackMid(pc, remote)
>
Ok so everything is fine with SSRC. I was worried because Wireshark does
not seem to decode properly the RTP packets (encryption?).
I also confirm that keyframe requests are received by GStreamer and I
can observe the signal going back to the VP8 encoder.
I did some digging in GStreamer code and from what I understand:
1. PLI are working fine, and I can confirm that Galène manages to ask
for keyframes.
2. NACK are received but ignored because my GStreamer webrtcbin element
doesn't have "RTX" enabled.
*
<https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/blob/1.18/ext/webrtc/gstwebrtcbin.c#L5310>
is reached,
*
<https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/blob/1.18/ext/webrtc/gstwebrtcbin.c#L5324>
is never reached.
This implies that `transport_stream_get_pt(stream, "RTX")=0`, which
should not happen. I don't really understand why it's happening but
maybe it has something to do with the SDP response of Galène not
including the "a=rtpmap:100 rtx/90000"?
> The purpose of the sender RTX object is to keep a history of RTP
> packets up to a configurable limit [...]. It will listen for upstream
> custom retransmission events (GstRTPRetransmissionRequest) that comes
> from downstream (#GstRtpSession). When receiving a request it will
> look up the requested seqnum in its list of stored packets. If the
> packet is available, it will create a RTX packet according to RFC 4588
> and send this as an auxiliary stream. RTX is SSRC-multiplexed.
--
https://gstreamer.freedesktop.org/documentation/rtpmanager/rtprtxsend.html
Do you confirm that "RTX" (https://tools.ietf.org/html/rfc4588) is the
feature missing?
--
Alexandre
erdnaxe@crans.org
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Galene] Re: Building a streaming gateway for Galène
2021-04-09 13:42 ` Alexandre IOOSS
@ 2021-04-09 13:55 ` Juliusz Chroboczek
2021-04-09 14:17 ` Juliusz Chroboczek
1 sibling, 0 replies; 12+ messages in thread
From: Juliusz Chroboczek @ 2021-04-09 13:55 UTC (permalink / raw)
To: Alexandre IOOSS; +Cc: galene
> 1. PLI are working fine, and I can confirm that Galène manages to ask
> for keyframes.
Excellent. This means you can reduce the rate of periodic keyframes to
something pretty minimal (like one per minute), and expect the stream to
recover within a second or so after a packet loss, even if NACK recovery
is broken.
> 2. NACK are received but ignored because my GStreamer webrtcbin element
> doesn't have "RTX" enabled.
Ah... right. Galène doesn't use a separate RTX track for retransmissions,
it expects the client to honour retransmissions on the main audio track.
Let me explain. There are two ways you can do retransmissions in WebRTC:
by simply resending the original packet, or by using an auxiliary "RTX"
track that is only used for retransmissions. Since Pion v2 didn't support
RTX tracks, Galène takes the former approach -- this makes it slightly
more difficult to properly account for lost packets, but it's much
simpler. Browsers support both, and are happy to retransmit packets over
the main track when there is no dedicated RTX track.
Now that Galène uses Pion v3, it should be possible to add support for
dedicated RTX tracks to it. Don't hold your breath, though, I've got
a few deadlines looming and am very busy with other stuff right now.
Sorry.
-- Juliusz
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Galene] Re: Building a streaming gateway for Galène
2021-04-09 13:42 ` Alexandre IOOSS
2021-04-09 13:55 ` Juliusz Chroboczek
@ 2021-04-09 14:17 ` Juliusz Chroboczek
1 sibling, 0 replies; 12+ messages in thread
From: Juliusz Chroboczek @ 2021-04-09 14:17 UTC (permalink / raw)
To: Alexandre IOOSS; +Cc: galene
> Ok so everything is fine with SSRC. I was worried because Wireshark does
> not seem to decode properly the RTP packets (encryption?).
Most likely encryption. WebRTC uses SRTP, only the packet type, the SSRC
and the RTP timestamp (but not the RTCP timestamp) are sent in the clear.
(Which has always made me nervous. If you have access to accurate
timestamps and packet lengths, how much audio can you recover? Should
Galène be sending cover traffic?)
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-04-09 14:17 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-06 11:05 [Galene] Building a streaming gateway for Galène Alexandre IOOSS
2021-04-06 12:57 ` [Galene] " Juliusz Chroboczek
2021-04-06 16:10 ` [Galene] webrtc security cameras? Dave Taht
2021-04-06 17:07 ` [Galene] " Juliusz Chroboczek
2021-04-06 17:44 ` [Galene] Re: Building a streaming gateway for Galène Alexandre IOOSS
2021-04-06 23:19 ` Juliusz Chroboczek
2021-04-07 20:40 ` Alexandre IOOSS
2021-04-07 22:01 ` Juliusz Chroboczek
2021-04-07 22:05 ` Juliusz Chroboczek
2021-04-09 13:42 ` Alexandre IOOSS
2021-04-09 13:55 ` Juliusz Chroboczek
2021-04-09 14:17 ` Juliusz Chroboczek
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox