[asterisk-dev] [Code Review] 4188: segfault when playing back voicemail under high concurrency with an IMAP backend

[Ben Smithurst]

> On Nov. 18, 2014, 1:25 p.m., Matt Jordan wrote:
> > From http://www.washington.edu/imap/documentation/internal.txt.html:
> > 
> > {quote}
> > 			* * * IMPORTANT * * *
> > 
> >      Any multi-threaded application should test stream->lock prior to
> > calling any c-client stream functions.  Any attempt to call a
> > mail_xxx() function while one is already in progress on the same
> > stream will cause the application to fail in unpredictable ways.
> > 
> >      Note that this check is insufficient in a preemptive-scheduling
> > multi-tasking application due to the possibility of a timing race.
> > Such applications must be written so that only one process accesses
> > the stream, or to have a higher level lock.
> > 
> >      Since MAIL operations will not finish until they are completed, a
> > single-tasking application does not have to worry about this problem,
> > except in the callback invoked from MAIL (e.g. mm_exists(), etc.) in which
> > case the stream is *always* locked.
> > {quote}
> > 
> > So, a few things from this:
> > (1) Based on the description from the UW IMAP documentation, locking vms->lock is sufficient *if* other mail accesses also lock the same vms object for that stream. If that isn't happening, then your patch probably won't fix anything, since both mail_open calls were already protected by the vms->lock and your global lock doesn't protect any other mail_XXXX accesses. The only way your patch would affect anything is if there were two different vms objects being used to open the same stream at the same time.
> > (2) Your backtrace on the issue doesn't show concurrent accesses to vm_execmain - although a lot of symbols are missing - so right now it's impossible to tell who the offending accesses are. It's also impossible to know if it is due to two concurrent accesses of mail_open with different vms objects, or something else.
> > 
> > Generally, I'm not this patch fixes your issue.

The concern we found was that some code called by mail_open uses static variables, specifically the ip_nametoaddr function - at least the ip6_unix.c implementation of it, which is where we're seeing crashes.  We believe the problem may be 2 DIFFERENT streams are being opened at once, correctly locked individually, but ultimately causing a conflict with a static variable deep inside the uw-imap code.


- Ben


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/4188/#review13790
-----------------------------------------------------------


On Nov. 17, 2014, 3:03 p.m., David Duncan Ross Palmer wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/4188/
> -----------------------------------------------------------
> 
> (Updated Nov. 17, 2014, 3:03 p.m.)
> 
> 
> Review request for Asterisk Developers.
> 
> 
> Bugs: ASTERISK-24516
>     https://issues.asterisk.org/jira/browse/ASTERISK-24516
> 
> 
> Repository: Asterisk
> 
> 
> Description
> -------
> 
> Asterisk segfaults when playing back voicemail under high concurrency with an IMAP backend
> 
> 
> Diffs
> -----
> 
>   /trunk/apps/app_voicemail.c 427675 
> 
> Diff: https://reviewboard.asterisk.org/r/4188/diff/
> 
> 
> Testing
> -------
> 
> Rapid load testing performed with {{SIPp}}:
> {code}
> sipp -sn uac -d 7000 -s ‘*1’ 127.0.0.1 -l 400 -mp 5606
> {code}
> 
> 
> Thanks,
> 
> David Duncan Ross Palmer
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20141121/b03be0c5/attachment-0001.html>