[asterisk-bugs] [JIRA] (ASTERISK-28139) RTP Stream Incorrect Payload Type Causes Asterisk To Drop Calls

[Paul Brooks (JIRA)]

    [ https://issues.asterisk.org/jira/browse/ASTERISK-28139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=245304#comment-245304 ] 

Paul Brooks commented on ASTERISK-28139:
----------------------------------------

Michael, thank you for sharing your similar experience. I might want to tweak that asymmetric_rtp_codec setting in a test Asterisk 16 setup to see if it does resolve the issue we were having. I just peaked at the running settings for both our Asterisk 14 and Asterisk 16 versions, and the asymmetric_rtp_codec setting is turned off on our endpoints in both versions. Coincidentally, in troubleshooting our issue I noticed the asymmetric_rtp_codec setting and thought about trying it, but after seeing it described as allowing simultaneous different codecs for send/receive, I remember thinking it was the opposite of what I needed and looked past it. Your testimonial gives me hope that there might be an immediate solution on the horizon. However, if the asymmetric_rtp_codec setting does resolve our issue, I think it should only be temporary because it would be masking the real problem. George says Asterisk is supposed to already drop unrecognized rtp packets but in my case Asterisk is dropping the call, not the bad packet!

George, I'll look into gathering a pcap related to a bad call for you. It might take a little time because I musk as Verizon  to provide their SIP pcap from their end to supplement the pcap I obtained with the rtp packets. Luckily in my experience they appear to pcap SIP activity for all their live traffic, so I shouldn't have difficulty getting it from them. The particular pcap I captured only shows our SIP replies as encrypted on the interface, which wouldn't be of much use to you. We can see Verizon's sip packets as both unencrypted and encrypted, but the outgoing packets only show the encrypted version. Once I obtain Verizon's pcap I'll forward it to you along with my pcap of the same call and the warning messaged logged by Asterisk for the call.

In the meantime, George can you weigh in on your thoughts about Michael's suggested use of asymmetric_rtp_codec to alleviate my probem? Seeing how he experienced the same issue with version 13.21.1, it makes me rethink my theory about this bug being related to Asterisk 15's new stream framework. My version 14 predates his 13.21, so maybe this issue is related to some other change made to Asterisk during the past 12 months or so?

One thing I noticed about Verizon's codec negotiation that seemed different compared to others I've seen is that fact that their offer isn't followed by individual rtpmap lines like I normally see.

For example, I typically see multiple rtpmap lines following a media offer, like this

m=audio 11998 RTP/AVP 0 18 8 101
a=rtpmap:0 PCMU/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:8 PCMA/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sendrecv
a=ptime:20

by contrast, Verizon's offers don't have those additional rtpmap lines. Theirs looks like this,

m=audio 38852 RTP/AVP 18 0 8 101
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=ptime:20
a=fmtp:18 annexb=no

Could this have anything to do with with what's tripping up Asterisk when the bad g729 packet arrives?




> RTP Stream Incorrect Payload Type Causes Asterisk To Drop Calls
> ---------------------------------------------------------------
>
>                 Key: ASTERISK-28139
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-28139
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Resources/res_pjsip
>    Affects Versions: 16.0.0
>         Environment: Debian 9.5
>            Reporter: Paul Brooks
>            Assignee: Paul Brooks
>              Labels: pjsip
>
> I represent a company offering hosted PBX service to various small businesses. We use Asterisk as our host. Last week we upgraded our Asterisk 14.6.2 server to the new Asterisk 16.0.0. We began to see many calls dropping upon answer. After a thorough investigation we had to roll back to the older version of Asterisk to alleviate the problem. Hopefully this issue can be recognized as a bug and corrected in a near-future version of Asterisk 16 so that we can attempt to upgrade to Asterisk 16 once again.
> The cause of the dropped calls turned out to be Asterisk's behavior in handling packet(s) received with the wrong payload type in the audio stream. One of our underlying carriers is Verizon Wholesale, and they deliver calls to us originated as either toll free or local dialed. Our dropped calls always involved Verizon's channels so we opened a ticket with them. Their support team helped us pinpoint the source of the dropped call problem.
> Our interconnection to Verizon is somewhat complex: they deliver calls to us over any one of 14 endpoints (in some seemingly random, quasi-round robin way). Every time a call is dropped by Asterisk 16, Asterisk logs errors similar to this,
> WARNING[11710][C-0000185e]: translate.c:485 ast_translator_build_path: No translator path: (ending codec is not valid)
> WARNING[11710][C-0000185e]: translate.c:485 ast_translator_build_path: No translator path: (starting codec is not valid)
> WARNING[11723][C-0000185e]: channel.c:5578 set_format: Unable to find a codec translation path: (g729) -> (ulaw)
> Our Asterisk is configured with all endpoints allowing ulaw only. Verizon's INVITE messages always offer media as 18 0 8 101 (g729, ulaw, alaw). Examination of SIP messages related to captured dropped calls between Verizon and our Asterisk shows normal negotiation. When the call is answered by Asterisk, it sends Verizon an answer message specifying type 0 for the media (ulaw/g711). But in a high percentage of the calls, Verizon will send a single g729 packet at the front of the resulting g711 stream. Verizon admitted to us that this is an old bug present in some of their DSP cards. They assert that they have an open case with their vendor involved but don't expect a fix anytime soon. Further, they point out that, "As a SIP wholesale connection, [Verizon will] interface [us] not only with Verizon PSTN gateways, but other wholesalers, retail customers, DSCIP carriers etc . . . [Verizon] cannot police the integrity of every RTP stream sent to [our] endpoint."
> Verizon pointed out to us further that RFC 3550 is pretty clear on the handling of incorrect payload types: "A receiver MUST ignore packets with payload types that it does not understand."
> So, from my perspective, assuming the RFC does say that Asterisk should be ignoring rtp packets containing the incorrect payload type, I must ask, why is Asterisk dropping the call when it encounters such a packet? It must be a bug. So I am reporting it as such. I must emphasize here that our previous versions of Asterisk that ran the same Verizon endpoints did not experience this dropped call issue.
> My hunch is that this undesirable behavior made its way into Asterisk beginning with version 15 when its framework for handling audio streams was significantly modified. It is well documented that prior to version 15, Asterisk had a "very loose" stream implementation that used a single pipe, and beginning with version 15, the framework was tightened up to allow for multiple streams for a single call. Here are a couple references on this topic: https://wiki.asterisk.org/wiki/display/AST/New+in+15, https://blogs.asterisk.org/2017/09/20/asterisk-15-multi-stream-media-sfu/
> I suspect now that Asterisk 15,16 allows multiple streams for a single call, it is being confused into thinking there is a simultaneous g729 stream alongside the g711 stream when it encounters the invalid g729 packet at the beginning of Verizon's g711 stream (I performed a packet capture on a live dropped call and indeed I saw the g729 packet at the beginning of the g711 stream).
> I believe Asterisk's implementation of this multi-stream concept needs to somehow be tweaked to recognize when a call has negotiated only a single media stream and therefore be prepared to discard any invalid packets that pop up into a live media stream. It appears that Asterisk 14 and prior had no trouble ignoring such invalid packets. Asterisk's 16 behavior of abruptly hanging up a call whenever it encounters an invalid media packets seems untenable.



--
This message was sent by Atlassian JIRA
(v6.2#6252)