Hello all, For the past two weeks I have been working on streaming from a broadcasting software such as OBS-Studio (that uses FFMpeg) to Galène. This would enable users to use Galène for low-latency video streaming and as a robust self-hosted Twitch alternative for conferences. I believe we don't want to have to do any video/audio conversion on the server side, so I see multiple possibilities to stream to Galène: 0. Using OBS Virtual v4l2 camera, but it is not portable and has audio sync issues. 1. Streaming to a gateway that will initiate the WebRTC session with Galène. 2. Implement WebRTC support in OBS-Studio as an output plugin. The point 2 was already tried by <https://github.com/CoSMoSoftware/OBS-studio-webrtc> but it seems to be problematic as they are using the WebRTC library from Chromium. I don't expect this OBS fork to be merged upstream soon. Last week I tried to implement a gateway using GStreamer webrtcbin plugin, <https://github.com/erdnaxe/galene-stream>. Although I am able to stream to Galène, I observe video drops. I am circumventing the issue by forcing the VP8 encoder to insert a keyframe every 5 frames, but that seems a terrible hack. Right now I am a bit lost, I don't know if I should either try to replace GStreamer webrtcbin with Pion (https://github.com/pion/example-webrtc-applications/tree/master/gstreamer-send), or if something is wrong with GStreamer/Galène. Does anyone have advice? Best regards, -- Alexandre Iooss
> For the past two weeks I have been working on streaming from a > broadcasting software such as OBS-Studio (that uses FFMpeg) to Galène. This would be a great addition! We have quite a few users that use OBS, and, as you mention, going through the v4l virtual camera has some disadvantages. > 0. Using OBS Virtual v4l2 camera, but it is not portable and has audio > sync issues. > 1. Streaming to a gateway that will initiate the WebRTC session with Galène. > 2. Implement WebRTC support in OBS-Studio as an output plugin. I would tend to agree, (1) is probably the best solution, since it avoids bloating either Galène or OBS. > Last week I tried to implement a gateway using GStreamer webrtcbin > plugin, <https://github.com/erdnaxe/galene-stream>. Although I am able > to stream to Galène, I observe video drops. I am circumventing the issue > by forcing the VP8 encoder to insert a keyframe every 5 frames, but that > seems a terrible hack. I'll have a look at your code as soon as I have some time. In the meantime, there are three things that you need to do: 1. send periodic RTCP Sender Reports with accurate NTP times; 2. react to RTCP PLI packets by inserting a keyframe; 3. react to RTCP NACK packets by resending the packets listed in the NACK. (1) is essential for proper lipsynch. (2) is what allows a stream to recover after a catastrophic loss event. (3) is what allows a stream to recover rapidly after moderate amounts of loss. > Right now [...] I don't know if I should... I'll have a look, think it over, then get back to you. -- Juliusz
Whilst we are doing speculative stuff, one thing that irks me is that most of the security cameras in the world, either phone home to china, or to google, with no sources available in either case. This bothers me a lot, not just for baby cameras, but as I'd rather send video from up to downstairs through my home router only in general, and log locally... I am curious if any cameras exist that can do webrtc more "right" and come with open sources. I am well aware of the raspi, but am too lazy to want to build a housing for it or run power to it.... poe would be good.... What I am thinking of doing longer term is trying to get to where I could blast raw-er video packets through a video distribution box in the studio, so at last, finally, my long cherished "jamophone" could come to life. See also: https://lola.conts.it/
> I am curious if any cameras exist that can do webrtc more "right" and come with
> open sources. I am well aware of the raspi, but am too lazy to want to build a
> housing for it or run power to it.... poe would be good....
There do exist quality cameras that connect locally to a server that you
control, but they use RTMP, which is an obsolescent protocol, running over
TCP, and that cannot easily be extended to handle modern codecs.
If I understand Alexandre correctly, he's building an RTMP-to-Galène
gateway. Such a gateway should work with off-the-shelf cameras, or at
least should be easy to tweak so that it works with them.
In the longer term, we should work on replacing RTMP with a modern
protocol. That's what the WISH working group is chartered to do at the
IETF, and, given the competent people I've seen participating, I have at
least some hope that they'll be able to deliver.
In the short term, let's work with Alexandre on getting OBS to
interoperate with Galène. Once that's done, we'll see what we can do
about off-the-shelf hardware.
-- Juliusz
On 4/6/21 2:57 PM, Juliusz Chroboczek wrote:
> I'll have a look at your code as soon as I have some time. In the
> meantime, there are three things that you need to do:
>
> 1. send periodic RTCP Sender Reports with accurate NTP times;
> 2. react to RTCP PLI packets by inserting a keyframe;
> 3. react to RTCP NACK packets by resending the packets listed in the NACK.
>
> (1) is essential for proper lipsynch. (2) is what allows a stream to
> recover after a catastrophic loss event. (3) is what allows a stream to
> recover rapidly after moderate amounts of loss.
When dumping traffic in Wireshark, I see only RTCP "Sender report" and
"Receiver report" packets. Sender reports are periodic, and sent by
groups of 2 (video then audio):
```
Real-time Transport Control Protocol (Sender Report)
10.. .... = Version: RFC 1889 Version (2)
..0. .... = Padding: False
...0 0000 = Reception report count: 0
Packet type: Sender Report (200)
Length: 6 (28 bytes)
Sender SSRC: 0x9ce1015b (2631991643)
Timestamp, MSW: 2797058655 (0xa6b7ba5f)
Timestamp, LSW: 1448315617 (0x56538ae1)
[MSW and LSW as NTP timestamp: Aug 20, 1988 08:44:15.337212257 UTC]
RTP timestamp: 866227088
Sender's packet count: 2194185357
Sender's octet count: 2612991028
[RTCP frame length check: OK - 28 bytes]
Real-time Transport Control Protocol (Sender Report)
10.. .... = Version: RFC 1889 Version (2)
..0. .... = Padding: False
...0 0000 = Reception report count: 0
Packet type: Sender Report (200)
Length: 6 (28 bytes)
Sender SSRC: 0x3a724a90 (980568720)
Timestamp, MSW: 4079490799 (0xf32816ef)
Timestamp, LSW: 2870575121 (0xab198011)
[MSW and LSW as NTP timestamp: Apr 10, 2029 07:53:19.668357853 UTC]
RTP timestamp: 2166723788
Sender's packet count: 468654353
Sender's octet count: 2838781192
[RTCP frame length check: OK - 28 bytes]
```
I don't really understand what you mean by "accurate NTP times". Streams
from Firefox or GStreamer seem to use random NTP timestamps.
When dumping Firefox WebRTC traffic, I also see "Generic RTP Feeback:
NACK" packets, which confirms the fact that GStreamer is not doing NACK
in my script. I should see if it's possible to enable it.
I am still quite new to WebRTC, so be aware that I might have done
something stupid with the GStreamer pipeline.
Thank you for pointing out this could be a potential problem,
--
Alexandre Iooss
> I don't really understand what you mean by "accurate NTP times". Streams > from Firefox or GStreamer seem to use random NTP timestamps. The NTP timestamp is used by the receiver to synchronise audio with video. It does not need to be accurate, but the NTP timestamps of correlated tracks (audio and video for a single video) should use the same clock, otherwise the receiver won't be able to resynchronise them. This is particularly important for talking heads (lectures and conferences). > When dumping Firefox WebRTC traffic, I also see "Generic RTP Feeback: > NACK" packets, which confirms the fact that GStreamer is not doing NACK > in my script. I should see if it's possible to enable it. This will improve things a lot. You'll need to keep a history of recently sent packets, I don't know if gstreamer can do that automatically. As to keyframes — you should send periodic keyframes, but you shouldn't do it too often (Chrome sends one every 120s). On the other hand, you should send a keyframe fairly quickly (within 500ms or so) whenever you receive a PLI (Picture Loss Indication) RTCP packet. > I am still quite new to WebRTC, so be aware that I might have done > something stupid with the GStreamer pipeline. It's a complicated protocol stack, don't worry too much about getting all the bits right straight away. -- Juliusz
On 4/7/21 1:19 AM, Juliusz Chroboczek wrote:>> When dumping Firefox WebRTC traffic, I also see "Generic RTP Feeback: >> NACK" packets, which confirms the fact that GStreamer is not doing NACK >> in my script. I should see if it's possible to enable it. > > This will improve things a lot. You'll need to keep a history of recently > sent packets, I don't know if gstreamer can do that automatically. > > As to keyframes — you should send periodic keyframes, but you shouldn't do > it too often (Chrome sends one every 120s). On the other hand, you should > send a keyframe fairly quickly (within 500ms or so) whenever you receive > a PLI (Picture Loss Indication) RTCP packet. I have good and bad news. The good news is that GStreamer webrtcbin supports NACK and PLI. I added those lines: # Enable WebRTC negative acknowledgement and FEC transceiver_count = self.webrtc.emit("get-transceivers").len for i in range(transceiver_count): transceiver = self.webrtc.emit("get-transceiver", i) transceiver.set_property("do-nack", True) transceiver.set_property("fec-type", GstWebRTC.WebRTCFECType.ULP_RED) Now I see "Generic RTP Feeback: NACK" packets in Wireshark and the SDP offer being sent to Galène also looks better: [...] a=sendrecv a=rtpmap:97 VP8/90000 a=rtcp-fb:97 nack a=rtcp-fb:97 nack pli a=framerate:30 a=rtpmap:98 red/90000 a=rtpmap:99 ulpfec/90000 a=rtpmap:100 rtx/90000 a=fmtp:100 apt=98 a=rtpmap:101 rtx/90000 a=fmtp:101 apt=97 a=ssrc-group:FID 1611068776 2492991179 [...] a=sendrecv a=rtpmap:96 OPUS/48000/2 a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=fmtp:96 sprop-maxcapturerate=48000;sprop-stereo=1 a=rtpmap:102 red/48000 a=rtpmap:103 ulpfec/48000 a=rtpmap:104 rtx/48000 a=fmtp:104 apt=102 a=rtpmap:105 rtx/48000 a=fmtp:105 apt=96 a=ssrc-group:FID 2413482691 909406007 [...] The bad news is that the stream is still dropping. If I print some statistics during streaming, GStreamer reports `pli-count=0` and `nack-count=0`, so something seems broken. Looking the issue online I found <https://mediasoup.discourse.group/t/broadcasting-a-vp8-rtp-stream-from-gstreamer/93/18>: > GStreamer first looks at the sender ssrc. If it gets NULL then it > tries with media ssrc. [...] somehow, GStreamer is getting “something” > when looking for sender ssrc = 0, which is crazy. Looking at the Wireshark dump, Galène seems also to use "0" as sender SSRC, so that might be the issue. I might try to do a dirty hack somewhere to confirm this hypothesis. Does Galène has ulpfec support (https://github.com/pion/webrtc/issues/1418)? Should I do PLI and ulpfec on audio and video in this specific application or is it a terrible idea? Thank you again for all your advice! -- Alexandre Iooss
> Looking at the Wireshark dump, Galène seems also to use "0" as sender > SSRC, so that might be the issue. I might try to do a dirty hack > somewhere to confirm this hypothesis. Galène doesn't choose the SSRC -- the SSRC is chosen by the offerer. Perhaps you forgot to initialise something? > Does Galène has ulpfec support No, since it's supported by neither Chrome nor Firefox. In the presence of NACK, it's not very useful for video (the small freezes caused by NACK recovery are acceptable), but it could dramatically improve audio quality. I intend to implement ulpfec as soon as it's supported in Chrome. -- Juliusz
>> Looking at the Wireshark dump, Galène seems also to use "0" as sender >> SSRC, so that might be the issue. I might try to do a dirty hack >> somewhere to confirm this hypothesis. > Galène doesn't choose the SSRC -- the SSRC is chosen by the offerer. You can find out the SSRC that Galène thinks the offerer has chosen by adding the following line: diff --git a/rtpconn/rtpconn.go b/rtpconn/rtpconn.go index eec5d97..6b7e8cb 100644 --- a/rtpconn/rtpconn.go +++ b/rtpconn/rtpconn.go @@ -509,6 +509,7 @@ func newUpConn(c group.Client, id string, labels map[string]string, offer string up := &rtpUpConnection{id: id, pc: pc, labels: labels} pc.OnTrack(func(remote *webrtc.TrackRemote, receiver *webrtc.RTPReceiver) { + log.Println("SSID", remote.SSRC()) up.mu.Lock() mid := getTrackMid(pc, remote)
On 4/8/21 12:05 AM, Juliusz Chroboczek wrote: >>> Looking at the Wireshark dump, Galène seems also to use "0" as sender >>> SSRC, so that might be the issue. I might try to do a dirty hack >>> somewhere to confirm this hypothesis. > >> Galène doesn't choose the SSRC -- the SSRC is chosen by the offerer. > > You can find out the SSRC that Galène thinks the offerer has chosen by > adding the following line: > > diff --git a/rtpconn/rtpconn.go b/rtpconn/rtpconn.go > index eec5d97..6b7e8cb 100644 > --- a/rtpconn/rtpconn.go > +++ b/rtpconn/rtpconn.go > @@ -509,6 +509,7 @@ func newUpConn(c group.Client, id string, labels map[string]string, offer string > up := &rtpUpConnection{id: id, pc: pc, labels: labels} > > pc.OnTrack(func(remote *webrtc.TrackRemote, receiver *webrtc.RTPReceiver) { > + log.Println("SSID", remote.SSRC()) > up.mu.Lock() > > mid := getTrackMid(pc, remote) > Ok so everything is fine with SSRC. I was worried because Wireshark does not seem to decode properly the RTP packets (encryption?). I also confirm that keyframe requests are received by GStreamer and I can observe the signal going back to the VP8 encoder. I did some digging in GStreamer code and from what I understand: 1. PLI are working fine, and I can confirm that Galène manages to ask for keyframes. 2. NACK are received but ignored because my GStreamer webrtcbin element doesn't have "RTX" enabled. * <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/blob/1.18/ext/webrtc/gstwebrtcbin.c#L5310> is reached, * <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/blob/1.18/ext/webrtc/gstwebrtcbin.c#L5324> is never reached. This implies that `transport_stream_get_pt(stream, "RTX")=0`, which should not happen. I don't really understand why it's happening but maybe it has something to do with the SDP response of Galène not including the "a=rtpmap:100 rtx/90000"? > The purpose of the sender RTX object is to keep a history of RTP > packets up to a configurable limit [...]. It will listen for upstream > custom retransmission events (GstRTPRetransmissionRequest) that comes > from downstream (#GstRtpSession). When receiving a request it will > look up the requested seqnum in its list of stored packets. If the > packet is available, it will create a RTX packet according to RFC 4588 > and send this as an auxiliary stream. RTX is SSRC-multiplexed. -- https://gstreamer.freedesktop.org/documentation/rtpmanager/rtprtxsend.html Do you confirm that "RTX" (https://tools.ietf.org/html/rfc4588) is the feature missing? -- Alexandre erdnaxe@crans.org
> 1. PLI are working fine, and I can confirm that Galène manages to ask > for keyframes. Excellent. This means you can reduce the rate of periodic keyframes to something pretty minimal (like one per minute), and expect the stream to recover within a second or so after a packet loss, even if NACK recovery is broken. > 2. NACK are received but ignored because my GStreamer webrtcbin element > doesn't have "RTX" enabled. Ah... right. Galène doesn't use a separate RTX track for retransmissions, it expects the client to honour retransmissions on the main audio track. Let me explain. There are two ways you can do retransmissions in WebRTC: by simply resending the original packet, or by using an auxiliary "RTX" track that is only used for retransmissions. Since Pion v2 didn't support RTX tracks, Galène takes the former approach -- this makes it slightly more difficult to properly account for lost packets, but it's much simpler. Browsers support both, and are happy to retransmit packets over the main track when there is no dedicated RTX track. Now that Galène uses Pion v3, it should be possible to add support for dedicated RTX tracks to it. Don't hold your breath, though, I've got a few deadlines looming and am very busy with other stuff right now. Sorry. -- Juliusz
> Ok so everything is fine with SSRC. I was worried because Wireshark does
> not seem to decode properly the RTP packets (encryption?).
Most likely encryption. WebRTC uses SRTP, only the packet type, the SSRC
and the RTP timestamp (but not the RTCP timestamp) are sent in the clear.
(Which has always made me nervous. If you have access to accurate
timestamps and packet lengths, how much audio can you recover? Should
Galène be sending cover traffic?)