[asterisk-dev] [Code Review] Fix race condition that may lead to a crash in ast_hangup() procedure.

[Matthew Nicholson]

> On 2009-09-15 13:27:42, Russell Bryant wrote:
> > /tags/1.6.1.1/include/asterisk/cdr.h, lines 185-187
> > <https://reviewboard.asterisk.org/r/362/diff/2/?file=6389#file6389line185>
> >
> >     I don't think this needs to be duplicated.

This has been removed.


> On 2009-09-15 13:27:42, Russell Bryant wrote:
> > /tags/1.6.1.1/main/channel.c, lines 1618-1622
> > <https://reviewboard.asterisk.org/r/362/diff/2/?file=6390#file6390line1618>
> >
> >     As I stated earlier, I don't really like the approach of brute force try to remove it twice.  Each attempt is a linked list iteration.  It's not free.  I would be okay with doing something like this in trunk, because at least there, it's an ao2_unlink(), which will execute much faster.
> >     
> >     In 1.4 and other versions where the channel list is a singly linked list like this, I guess we can just use a channel flag to know if it has already been removed from the channel list.  I kind of feel like this is ugly, too, but at least it doesn't waste cycles.

Added a flag.


- Matthew


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/362/#review1082
-----------------------------------------------------------


On 2009-09-15 13:40:04, Matthew Nicholson wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/362/
> -----------------------------------------------------------
> 
> (Updated 2009-09-15 13:40:04)
> 
> 
> Review request for Asterisk Developers.
> 
> 
> Summary
> -------
> 
> This patch fixes a race condition that can occur if a channel is retrieved from the channel list while it is being hung up.  This situation can be caused if the pbx thread calls ast_hangup on a channel while another thread attempts to retrieve the channel from the channel list.  One example where this occurs involves the 'Bridge' manager action.
> 
> An approximation of the hangup procedure in ast_hangup() follows:
>  1. lock the channel
>  2. free some fields in the channel
>  3. call chan->tech->hangup()
>  4. unlock the channel
>  5. generate the 'Hangup' manager event 
>  6. do CDR stuff
>  7. remove the channel from the channel list
>  8. lock and unlock the channel
>  9. free the remaining channel fields
> 
> During step 3, the channel tech_pvt field will be freed and set to NULL.  After step 4 and before step 7, the channel may be retrieved from the channel list and used.  Problems can occur if any of the freed fields are used.  In this specific instance the tech_pvt field is dereferenced causing asterisk to crash.  My purposal for fixing the problem involves moving step 7 and inserting it before step 3.
> 
> This problem potentially exists in all supported versions of asterisk.
> 
> 
> This addresses bug 15316.
>     https://issues.asterisk.org/view.php?id=15316
> 
> 
> Diffs
> -----
> 
>   /tags/1.6.1.1/include/asterisk/cdr.h 218291 
>   /tags/1.6.1.1/include/asterisk/channel.h 218291 
>   /tags/1.6.1.1/main/channel.c 218291 
> 
> Diff: https://reviewboard.asterisk.org/r/362/diff
> 
> 
> Testing
> -------
> 
> To test this, I replicated the customer's environment from bug 15316 (this involves sending a Bridge manager action at the proper time).  In order to achieve the proper timing for the race condition to manifest itself, I added a sleep() command after step 3 in the procedure above and a shorter sleep withing the Bridge action (action_bridge()) just before retrieving a channel from the channel list.  While the Bridge action thread is sleeping, I hang up the channel requested for the bridge action causing the hangup procedure to run through step 3 then sleep.  The Bridge action thread then wakes and retrieves the channel from the channel list while the hangup thread is sleeping.  The Bridge thread now has a partially torn down channel with a NULL tech_pvt.  Eventually the Bridge thread executes ast_rtp_new_source() which attempts to access the tech_pvt and crashes asterisk.
> 
> After moving step 7 and inserting it before step 3, the crash no longer occurs and the Bridge thread never finds the channel in the channel list.
> 
> 
> Thanks,
> 
> Matthew
> 
>