Over the previous couple of weeks, GStreamer’s RTP stack received a few new and fairly helpful options. As it’s tough to configure, principally as a result of there being so many various attainable configurations, I made a decision to write down about this a bit with some instance code.
The options are RFC 6051-style speedy synchronization of RTP streams, which can be utilized for inter-stream (e.g. audio/video) synchronization in addition to inter-device (i.e. community) synchronization, and the flexibility to simply retrieve absolute sender clock instances per packet on the receiver facet.
Observe that every of this was already attainable earlier than with GStreamer through completely different mechanisms with completely different trade-offs. Clearly, not with the ability to have working audio/video synchronization can be merely not acceptable and I beforehand talked about the best way to do inter-device synchronization with GStreamer earlier than, for instance on the GStreamer Convention 2015 in Düsseldorf.
The instance code beneath will make use of the GStreamer RTSP Server library however may be utilized to any form of RTP workflow, together with WebRTC, and are written in Rust however the identical will also be achieved in another language. The complete code may be present in this repository.
And for reference, the merge requests to allow all this are [1], [2] and [3]. You in all probability don’t need to backport these to an older model of GStreamer although as there are dependencies on varied different adjustments elsewhere. All the following wants at the very least GStreamer from the git fundamental department as of immediately, or the upcoming 1.22 launch.
Baseline Sender / Receiver Code
The start line of the instance code may be discovered right here within the baseline department. All of the essential steps are commented so it ought to be comparatively self-explanatory.
Sender
The sender is beginning an RTSP server on the native machine on port 8554 and gives a media with H264 video and Opus audio on the mount level /check. It may be began with
$ cargo run -p rtp-rapid-sync-example-send
After beginning the server it may be accessed through GStreamer with e.g. gst-play-1.0 rtsp://127.0.0.1:8554/check or equally through VLC or another software program that helps RTSP.
This doesn’t do something particular but however lays the muse for the next steps. It creates an RTSP server occasion with a customized RTSP media manufacturing facility, which in flip creates customized RTSP media situations. All this isn’t wanted at this level but however will permit for the mandatory customization later.
One essential side right here is that the bottom time of the media’s pipeline is ready to zero
pipeline.set_base_time(gst::ClockTime::ZERO); pipeline.set_start_time(gst::ClockTime::NONE);
This permits the timeoverlay aspect that’s positioned within the video a part of the pipeline to render the clock time over the video frames. We’re going to make use of this later to verify on the receiver that the clock time on the sender and the one retrieved on the receiver are the identical.
let video_overlay = gst::ElementFactory::make("timeoverlay", None)
.context("Creating timeoverlay")?;
[...]
video_overlay.set_property_from_str("time-mode", "running-time");
It truly solely helps rendering the operating time of every buffer, however in a reside pipeline with the bottom time set to zero the operating time and pipeline clock time are the identical. See the documentation for some extra particulars concerning the time ideas in GStreamer.
Total this creates the next RTSP stream producer bin, which can be used additionally in all the next steps:
Receiver
The receiver is an easy playbin pipeline that performs an RTSP URI given through command-line parameters and runs till the stream is completed or an error has occurred.
It may be run with the next as soon as the sender is began
$ cargo run -p rtp-rapid-sync-example-send -- "rtsp://192.168.1.101:8554/check"
Please don’t overlook to exchange the IP with the IP of the machine that’s truly operating the server.
All of the code ought to be acquainted to anybody who ever wrote a GStreamer software in Rust, aside from one half that may want a bit extra clarification
pipeline.connect_closure(
"source-setup",
false,
glib::closure!(|_playbin: &gst::Pipeline, supply: &gst::Component| {
supply.set_property("latency", 40u32);
}),
);
playbin goes to create an rtspsrc, and at that time it is going to emit the source-setup sign in order that the applying can do any further configuration of the supply aspect. Right here we’re connecting a sign handler to that sign to do precisely that.
By default rtspsrc introduces a latency of two seconds of latency, which is much more than what’s normally wanted. For reside, non-VOD RTSP streams this worth ought to be across the community jitter and right here we’re configuring that to 40 milliseconds.
Retrieval of absolute sender clock instances
Now as step one we’re going to retrieve absolutely the sender clock instances for every video body on the receiver. They are going to be rendered by the receiver on the backside of every video body and also will be printed to stdout. The adjustments between the earlier model of the code and this model may be seen right here and the ultimate code right here within the sender-clock-time-retrieval department.
When operating the sender and receiver as earlier than, the video from the receiver ought to look much like the next
The higher time that’s rendered on the video frames is rendered by the sender, the underside time is rendered by the receiver and each ought to at all times be the identical except one thing is damaged right here. Each instances are the pipeline clock time when the sender created/captured the video body.
On this configuration absolutely the clock instances of the sender are offered to the receiver through the NTP / RTP timestamp mapping offered by the RTCP Sender Stories. That’s additionally the rationale why it takes about 5s for the receiver to know the sender’s clock time as RTCP packets are usually not scheduled fairly often and solely after about 5s by default. The RTCP interval may be configured on rtpbin along with many different issues.
Sender
On the sender-side the configuration adjustments are somewhat small and never even completely needed.
rtpbin.set_property_from_str("ntp-time-source", "clock-time");
By default the RTP NTP time used within the RTCP packets is predicated on the native machine’s walltime clock transformed to the NTP epoch. Whereas this works superb, this isn’t the clock that’s used for synchronizing the media and as such there can be drift between the RTP timestamps of the media and the NTP time from the RTCP packets, which can be reset each time the receiver receives a brand new RTCP Sender Report from the sender.
As an alternative, we configure rtpbin right here to make use of the pipeline clock because the supply for the NTP timestamps used within the RTCP Sender Stories. This doesn’t give us (by default at the very least, see later) an precise NTP timestamp nevertheless it doesn’t have the drift drawback talked about earlier than. With out additional configuration, on this pipeline the used clock is the monotonic system clock.
rtpbin.set_property("rtcp-sync-send-time", false);
rtpbin usually makes use of the time when a packet is shipped out for the NTP / RTP timestamp mapping within the RTCP Sender Stories. That is modified with this property to as a substitute use the time when the video body / audio pattern was captured, i.e. it doesn’t embrace all of the latency launched by encoding and different processing within the sender pipeline.
This doesn’t make any massive distinction on this situation however normally one would have an interest within the seize clock instances and never the ship clock instances.
Receiver
On the receiver-side there are a number of extra adjustments. Initially now we have to opt-in to rtpjitterbuffer placing a reference timestamp metadata on each obtained packet with the sender’s absolute clock time.
pipeline.connect_closure(
"source-setup",
false,
glib::closure!(|_playbin: &gst::Pipeline, supply: &gst::Component| {
supply.set_property("latency", 40u32);
supply.set_property("add-reference-timestamp-meta", true);
}),
);
rtpjitterbuffer will begin placing the metadata on packets as soon as it is aware of the NTP / RTP timestamp mapping, i.e. after the primary RTCP Sender Report is obtained on this case. Between the Sender Stories it’ll interpolate the clock instances. The traditional timestamps (PTS) on every packet are usually not affected by this and are nonetheless primarily based on no matter clock is used domestically by the receiver for synchronization.
To truly make use of the reference timestamp metadata we add a timeoverlay aspect as video-filter on the receiver:
let timeoverlay =
gst::ElementFactory::make("timeoverlay", None).context("Creating timeoverlay")?;
timeoverlay.set_property_from_str("time-mode", "reference-timestamp");
timeoverlay.set_property_from_str("valignment", "backside");
pipeline.set_property("video-filter", &timeoverlay);
This can then render the sender’s absolute clock instances on the backside of every video body, as seen within the screenshot above.
And final we additionally add a pad probe on the sink pad of the timeoverlay aspect to retrieve the reference timestamp metadata of every video body after which printing the sender’s clock time to stdout:
let sinkpad = timeoverlay
.static_pad("video_sink")
.anticipate("Did not get timeoverlay sinkpad");
sinkpad
.add_probe(gst::PadProbeType::BUFFER, |_pad, information| {
if let Some(gst::PadProbeData::Buffer(ref buffer)) = information.knowledge {
if let Some(meta) = buffer.meta::<gst::ReferenceTimestampMeta>() {
println!("Have sender clock time {}", meta.timestamp());
} else {
println!("Don't have any sender clock time");
}
}
gst::PadProbeReturn::Okay
})
.anticipate("Failed so as to add pad probe");
Fast synchronization through RTP header extensions
The primary drawback with the earlier code is that the sender’s clock instances are solely recognized as soon as the primary RTCP Sender Report is obtained by the receiver. There are numerous methods to configure rtpbin to make this occur sooner (e.g. by decreasing the RTCP interval or by switching to the AVPF RTP profile) however in any case the data can be transmitted exterior the precise media knowledge stream and it will probably’t be assured that it’s truly recognized on the receiver from the very first obtained packet onwards. That is after all not an issue in each use-case, however for the instances the place it’s there’s a resolution for this drawback.
RFC 6051 defines an RTP header extension that enables to transmit the NTP timestamp that corresponds an RTP packet immediately along with this very packet. And that’s what the subsequent adjustments to the code are making use of.
The adjustments between the earlier model of the code and this model may be seen right here and the ultimate code right here within the rapid-synchronization department.
Sender
So as to add the header extension on the sender-side it’s only needed so as to add an occasion of the corresponding header extension implementation to the payloaders.
let hdr_ext = gst_rtp::RTPHeaderExtension::create_from_uri(
"urn:ietf:params:rtp-hdrext:ntp-64",
)
.context("Creating NTP 64-bit RTP header extension")?;
hdr_ext.set_id(1);
video_pay.emit_by_name::<()>("add-extension", &[&hdr_ext]);
This primary instantiates the header extension primarily based on the uniquely outlined URI for it, then units its ID to 1 (see RFC 5285) after which provides it to the video payloader. The identical is then executed for the audio payloader.
By default this may add the header extension to each RTP packet that has a distinct RTP timestamp than the earlier one. In different phrases: on the primary packet that corresponds to an audio or video body. By way of properties on the header extension this may be configured however usually the default ought to be enough.
Receiver
On the receiver-side no adjustments would truly be needed. Using the header extension is signaled through the SDP (see RFC 5285) and it will likely be robotically made use of inside rtpbin as one other supply of NTP / RTP timestamp mappings along with the RTCP Sender Stories.
Nevertheless, we configure one further property on rtpbin
supply.connect_closure(
"new-manager",
false,
glib::closure!(|_rtspsrc: &gst::Component, rtpbin: &gst::Component| {
rtpbin.set_property("min-ts-offset", gst::ClockTime::from_mseconds(1));
}),
);
Inter-stream audio/video synchronization
The rationale for configuring the min-ts-offset property on the rtpbin is that the NTP / RTP timestamp mapping will not be solely used for offering the reference timestamp metadata however it is usually used for inter-stream synchronization by default. That’s, for getting right audio / video synchronization.
With RTP alone there isn’t any mechanism to synchronize a number of streams in opposition to one another because the packet’s RTP timestamps of various streams don’t have any correlation to one another. This isn’t an excessive amount of of an issue as normally the packets for audio and video are obtained roughly on the similar time however there’s nonetheless some inaccuracy in there.
One method to repair that is to make use of the NTP / RTP timestamp mapping for every stream, both from the RTCP Sender Stories or from the RTP header extension, and that’s what’s made use of right here. And since the mapping is offered fairly often through the RTP header extension however the RTP timestamps are solely correct as much as clock fee (1/90000s for video and 1/48000s) for audio on this case, we configure a threshold of 1ms for adjusting the inter-stream synchronization. With out this it could be adjusted virtually constantly by a really small quantity backwards and forwards.
Different approaches for inter-stream synchronization are offered by RTSP itself earlier than streaming begins (through the RTP-Information header), however as a result of a bug that is presently not made use of by GStreamer.
Yet one more method can be through the clock info offered by RFC 7273, about which I already wrote beforehand and which can be supported by GStreamer. This additionally permits inter-device, community synchronization and used for that goal as a part of e.g. AES67, Ravenna, SMPTE 2022 / 2110 and plenty of different protocols.
Inter-device community synchronization
Now for the final half, we’re going so as to add precise inter-device synchronization to this instance. The adjustments between the earlier model of the code and this model may be seen right here and the ultimate code right here within the network-sync department. This doesn’t use the clock info offered through RFC 7273 (which might be an alternative choice) however makes use of the identical NTP / RTP timestamp mapping that was mentioned above.
When beginning the receiver a number of instances on completely different (or the identical) machines, every of them ought to play again the media synchronized to one another and precisely 2 seconds after the corresponding audio / video frames are produced on the sender.
For this, each sender and all receivers are utilizing an NTP clock (pool.ntp.org on this case) as a substitute of the native monotonic system clock for media synchronization (i.e. because the pipeline clock). As an alternative of an NTP clock it could even be attainable to another mechanism for community clock synchronization, e.g. PTP or the GStreamer netclock.
println!("Syncing to NTP clock");
clock
.wait_for_sync(gst::ClockTime::from_seconds(5))
.context("Syncing NTP clock")?;
println!("Synced to NTP clock");
This code instantiates a GStreamer NTP clock after which synchronously waits as much as 5 seconds for it to synchronize. If that fails then the applying merely exits with an error.
Sender
On the sender facet all that’s wanted is to configure the RTSP media manufacturing facility, and as such the pipeline used inside it, to make use of the NTP clock
manufacturing facility.set_clock(Some(&clock));
This causes all media contained in the sender’s pipeline to be synchronized in keeping with this NTP clock and to additionally use it for the NTP timestamps within the RTCP Sender Stories and the RTP header extension.
Receiver
On the receiver facet the identical has to occur
pipeline.use_clock(Some(&clock));
As well as a pair extra settings should be configured on the receiver although. Initially we configure a static latency of two seconds on the receiver’s pipeline.
pipeline.set_latency(gst::ClockTime::from_seconds(2));
That is needed as GStreamer can’t know the latency of each receiver (e.g. completely different decoders is likely to be used), and in addition as a result of the sender latency can’t be robotically recognized. Every audio / video body can be timestamped on the receiver with the NTP timestamp when it was captured / created, however since then all of the latency of the sender, the community and the receiver pipeline has handed and for this some compensation should occur.
Which worth to make use of right here relies upon loads on the general setup, however 2 seconds is a (very) protected guess on this case. The worth solely must be bigger than the sum of sender, community and receiver latency and in the long run has the impact that the receiver is displaying the media precisely that a lot later than the sender has produced it.
And final we even have to inform rtpbin that
- sender and receiver clock are synchronized to one another, i.e. on this case each are utilizing precisely the identical NTP clock, and that no translation to the pipeline’s clock is critical, and
- that the outgoing timestamps on the receiver ought to be precisely the sender timestamps and that this conversion ought to occur primarily based on the NTP / RTP timestamp mapping
supply.set_property_from_str("buffer-mode", "synced");
supply.set_property("ntp-sync", true);
And that’s it.
A cautious reader can even have observed that all the above would additionally work with out the RTP header extension, however then the receivers would solely be synchronized as soon as the primary RTCP Sender Report is obtained. That’s what the test-netclock.c / test-netclock-client.c instance from the GStreamer RTSP server is doing.
As standard with RTP, the above is by far not the one manner of doing this and GStreamer additionally helps varied different synchronization mechanisms. Which one is the right one for a particular use-case is dependent upon a variety of components.
