[asterisk-dev] Adding the support for NACK in asterisk

Matthew Jordan mjordan at digium.com
Mon Dec 29 09:46:32 CST 2014

On Tue, Dec 23, 2014 at 11:01 AM, Nitesh Bansal <nitesh.bansal at gmail.com>
> Hi Matt,
> Thanks for this nice explanation.
> Well,  we still don't know if we are going to do it or not, it is still a
> pending decision.
> But i do have some comments ref your response, responding inline.
> General comment: By adding the support for NACK, i meant just adding the
> ability
> to forward the RTCP FB messages, i do agree that adding the capability in
> asterisk
> to generate NACKs on its own will make it much more complicated?

Let's assume you aren't in a packet-to-packet bridge provided by
bridge_native_rtp, as in that case, the RTCP messages should be
auto-magically forwarded anyway.

In a core bridge, forwarding may be more complicated than generating the
RTCP FB messages.

Remember that Asterisk is a B2BUA. As such, the general placement of the
RTP stack in the architecture will look something like this:

        -------------| Bridging Core |-------------
       |             -----------------             |
------------------                        ------------------
| Channel Core   |                        | Channel Core   |
------------------                        ------------------
| Channel Driver |                        | Channel Driver |
------------------                        ------------------
| RTP instance A |                        | RTP instance B |
------------------                        ------------------

Let's say RTP instance A wanted to forward RTCP FB messages to whatever it
is talking to on the other side of the bridge. There are a few issues with
doing this:
(1) The channel on the other side of the bridge may - or may not - be an
RTP capable channel driver.
(2) You may have more than one channel on the other side of the bridge
(such as is the case in a conference).
(3) Reaching across a bridge from the RTP instance requires:
     - Grabbing the channel through the driver/core (and dealing with the
locking situation between the channel and the driver pvt)
     - Grabbing the bridge_channel structure safely, which mediates between
the bridge and the channel
     - Grabbing the bridge structure safely
     - Grabbing the other channel on the other side of the bridge safely
     - (You can see where this is going)
   Suffice to say, there are a lot of locks you have to grab safely to
"own" the whole picture of the world and pass the information over.
Alternatively, you can pass a control frame (which is the right way to do
this), but does mean more custom control frames, and all the channel
drivers have to be updated to understand said control frame.
(4) Forwarding a FB message across a bridge - particularly in a multi-party
scenario - may result in the wrong information being requested (and more
confusion besides). Say, for example, we have three devices talking to
Asterisk, one of which - say device A - is suffering from significant
packet loss. As a result, device A tells Asterisk via a FB message that it
is missing some packets. However, who does Asterisk forward that to? The
answer has to be both parties: the FB message itself was not destined for a
single party on the other side of a bridge, but for Asterisk itself.
However, both parties are not providing the actual video stream that is
sent to device A - Asterisk chooses which stream of video is being sent to
device A. As such - particularly if the source of the stream updated during
a period of packet loss - we won't know what set of RTP packets should be
re-transmitted to device A. Even if we could make that determination, if
both devices do re-transmit their RTP, then the RTP implementation for each
of those devices will probably drop the packets, as the sequence
number/timestamp of received RTP from those devices will already be passed
the received packets.

Asterisk fares poorly as a proxy.

Asterisk is a B2BUA: the device talking to Asterisk through RTP instance A
is talking to Asterisk, not directly to the device talking to RTP instance
B. Ideally, Asterisk would be generating the feedback RTCP reports, as it
is the thing that is actually receiving, decoding, and generating the RTP
packets being sent to the device(s) on the other side of the bridge.

> (1) You'll need to modify rtp_engine and res_rtp_asterisk to
> understand compound RTCP packets properly. Currently, we always assume
> that the RTCP packets contain a single SR/RR: that means modifications
> to the parsing in ast_rtcp_read, as well as the conversion of received
> RTCP reports in rtp_engine into JSON for delivery to RTP Stasis topic
> consumers. Note that this would be an opportune time to make the
> parsing in ast_rtcp_read not be quite so terrible as it is today.
> (2) Going along with #1, you'll need to add additional RTCP report
> types - both when reading and when writing. Currently, Asterisk only
> supports SR/RR. The RFC mandates FR reports, along with SDES packets.
> You'll also need to change the algorithm for when reports are
> scheduled. Since RFC 4585 modifies the scheduling algorithm, you'll
> pretty much have to abandon how scheduling is set up currently in
> ast_rtp_read. There are different modes of operation, each with slight
> variations to the algorithm for how/when reports should be sent. A
> state machine would probably be handy for this.
> Nitesh -- since asterisk is just going to forward the RTCP FB reports,
> it
> be sufficient to parse the RTCP FB messages, adapt the packet header like
> and other stuff and just forward it, without modifying the current
> algorithm?

See previous comments on why I think forwarding the FB messages is probably
wrong for this scenario.

> (3) NACK mode. I would be very careful here: as a B2BUA, Asterisk
> *always* assumes there are only two participants in an RTP session -
> and making Asterisk realize that this isn't the case is going to be
> more challenging than you may think. Asterisk has a lot of
> implementation 'shortcuts' it takes in RTCP statistic management that
> it takes because it knows that there is only one other RTP participant
> other than itself: if you go down the road of wanting to maintain
> additional statistics beyond the RTP instance and whoever it is
> talking to, the scope of this would expand dramatically.
> Nitesh -- I am not sure if i understood your comment, why does it need to
> maintain
> addiitonal info beyond RTP instance, in my opinion, for forwarding RTCP FB
> reports,
> all we need is the SSRC and sequence numbers?
> (4) If I'm reading the RFC correctly, you _may_ not need to buffer the
> RTP packets themselves; rather, you need to inform the other side of
> the RTP packets you've lost. Generally, this means keeping a limited
> history of which RTP sequence numbers you've received, and which ones
> are missed. When constructing the NACK message (in 5a), you'd see if
> you had a missed RTP sequence number, then look to see if any of the
> 16 RTP packets after that one were also missed. After reporting this,
> the buffer could be flushed - how long the buffer is probably depends
> on the frequency of report sending, which is variable as per (2).
> Nitesh -- I have made this assumption considering that many traditional
> SIP phones don't support NACK and even if they do, they don't send the
> retransmission as specified in the RFC 4588.
> This draft http://tools.ietf.org/pdf/draft-ietf-straw-b2bua-rtcp-02.pdf
> talks
> about the issue of handling the NACK messages in B2BUA environment.
> Here is a piece of text from Section 4 of this draft:
> It is worth noting that some additional care may be needed, with
> respect to RTCP, when acting as an intermediary between two secure
> sessions. Specifically, issues may arise when relaying NACK feedback
> originated by a user who failed to receive some RTP packets, that
> were instead received by the B2BUA: such an issue might occur when
> the packet loss happens between the user and the B2BUA itself, and
> not between the RTP packet sender and the B2BUA. Such a situation
> might result in the RTP media sender retransmitting the encrypted
> packet, which would then be rejected by the B2BUA as a replayed one.
> However, this issue is well known and addressed in [
> RFC4588
> ], which
> both the B2BUA and the involved participants in the communication
> SHOULD make use of to prevent it from happening. This mechanism
> allows for a Redundancy RTP Stream to be used for the purpose, which
> would prevent the replay error. Of course, all recommendations given
> previously with respect to managing SSRCs across the B2BUA still
> apply here as well: in fact, such a redundant RTP stream would make
> use of a different SSRC, that would need to be taken care of both at
> the RTP and the SDP level. If [
> RFC4588
> ] is not supported, the B2BUA
> SHOULD handle NACK packets directly, and only forward feedback on
> lost packets it has not access to

I think this is more of an indictment against merely relaying NACK feedback
messages then anything else. If Asterisk forwarded FB messages then I can
see this scenario arising. If, however, Asterisk generates the feedback
messages itself, I'm not sure how this would apply. Asterisk already
completely decodes and re-encodes encrypted media streams between two
endpoint. If Asterisk generates the feedback messages itself as well, then
an RTP packet sender would not request a re-send of packets from Asterisk
that Asterisk was unaware of.

The scheduling issues would - I think - still be a problem. How much, in
practice, the scheduling issues really matter is another concern. It may be
that maintaining the existing scheduling is sufficient - I'd be curious to
know what the real impact of the existing scheduling on FB reports would be.

> (5) RTCP feedback messages. The transport layer feedback messages
> would obviously be provided by the RTP engine and/or res_rtp_asterisk.
> However, the other layers - payload/application - should have
> corresponding APIs in rtp_engine. In particular, the codec layer
> (payload) would be interesting, as it would require the ability for
> codec modules (or something else in the media 'section' of Asterisk)
> to take an RTP instance and register a handler for providing feedback
> messages at opportune times. The application layer would be similar,
> except that currently, we probably wouldn't have an implementation.
> You may want to consider that as 'future work', as it is probably
> sufficient to be a project in its own right.
> Nitesh -- Do we require to register any handlers if we just care about
> forwarding
> the RTCP FB messages?

I think if you don't concern yourself with application or codec level
feedback mechanisms, then this part of the RFC could be ignored and the
need for registered handlers (or a framework to support it) could go away.

>   (a) Generic NACK: these messages should be implemented in such a
> fashion that res_rtp_asterisk can have one be constructed based on the
> statistics in (4). A part of me wishes all of this were implemented in
> RTP engine, so that other RTP implementations could take advantage of
> the RTCP work. That's a consideration that should be given some
> thought, given how much of the RTCP implementation would have to be
> ripped apart by this work anyway.
>   (b) PLI/SLI/RPSI: I'm not sure these apply to Asterisk very well. We
> don't decode video, so it's kind of immaterial. You may want to have
> the ability for Asterisk to pass through these types of RTCP messages
> from a video decoder to a video encoder; that would entail
> understanding the RTCP messages, enqueuing a control frame onto the
> channel, and - in the far end - taking that control frame and turning
> it into the appropriate information on the other side of the bridge.
> There's bridging considerations to think through here on how the
> control frame is propagated, particularly for multi-party bridges.
> If you're still interested in doing this work, please do let me us
> know! I think the first major hurdle here is to hammer out how all of
> this can be effectively introduced into the RTP engine/implementation
> in Asterisk - as a large part of what makes this so difficult is the
> non-extensible nature of Asterisk's RTP implementation. It may be that
> we can help out some with that part, such that the actual act of
> implementing the RFC and its various feedback mechanisms is a bit
> easier.

While I am leaning towards "don't forward the FB messages", I think a
concrete example of feedback messages that you think should be forwarded
might help here. Right now, I'm having a hard time envisioning the specific
scenario where you have two or more endpoints providing video to Asterisk
and where Asterisk can simply relay the entire feedback message and not
generate the message itself. I could be mistaken, but it feels like while
this could work in a simple two-party core bridge, there are a lot of other
scenarios where this goes wrong quickly.


Matthew Jordan
Digium, Inc. | Engineering Manager
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20141229/295121cf/attachment-0001.html>

More information about the asterisk-dev mailing list