[asterisk-dev] [Code Review] 3402: http: response body frequently missing when running in virtual machine

[Scott Griepentrog]

> On March 26, 2014, 3:38 p.m., Matt Jordan wrote:
> > /branches/12/main/http.c, line 472
> > <https://reviewboard.asterisk.org/r/3402/diff/1/?file=56722#file56722line472>
> >
> >     I'm not sure why the send-only shutdown of ser->fd is needed here.
> >     
> >     When the file stream is closed via fclose(ser->f), the file descriptor is implicitly closed as well.
> >     
> >     From http://linux.die.net/man/3/fdopen:
> >     
> >     "The file descriptor is not dup'ed, and will be closed when the stream created by fdopen() is closed."
> >     
> >     I'm not sure why the fflush is strictly necessary here as well, as closing should flush the file stream.

This is strictly a workaround for what I presume to be a bug in the socket/tcp buffer handling (i.e. linux kernel).  In my testing, a low level write() to ser->fd resulted in the same exact failure scenario: data was written to the socket just prior to close that did not end up being transmitted by TCP -- but only when Asterisk is running in a VM; and further only when the far end of the socket originates from a machine other than that same vm.

Thus: I don't believe fdopen or c stdio FILE buffering has anything to do with this.  Neither is it technically a fault of Asterisk, which is correctly writing data to the socket/stream before closing it.  It should be noted that this is also not a fault of the virtual environment, as two different implementations (Virtual Box & KVM/Qemu) both exhibited the same exact results.  The failure is too specifically the last write issued to the socket prior to close, and would appear to be a race condition triggered by minor differences resulting from running within a virtual machine.

The shutdown() call is the key to working around this problem.  It triggers the socket buffering and/or TCP implementation to close things out and transmit the last of the data reliably prior to closing the socket for real.  The fflush() issued just prior to that is only to insure that the FILE buffering is flushed out to the file descriptor prior to the shutdown call.  The shutdown() call does not actually close the socket, only informs TCP that there will be nothing more sent, which resolves this issue.


- Scott


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/3402/#review11384
-----------------------------------------------------------


On March 26, 2014, 3:13 p.m., Scott Griepentrog wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/3402/
> -----------------------------------------------------------
> 
> (Updated March 26, 2014, 3:13 p.m.)
> 
> 
> Review request for Asterisk Developers.
> 
> 
> Bugs: SWP-6767
>     https://issues.asterisk.org/jira/browse/SWP-6767
> 
> 
> Repository: Asterisk
> 
> 
> Description
> -------
> 
> When running asterisk in a virtual machine, responses to ARI requests would frequently be missing.  A race condition related to closing the socket immediately after writing data onto it is resolved in this patch by insuring the output stream is flushed, and then informing TCP of the shutdown prior to the close.
> 
> 
> Diffs
> -----
> 
>   /branches/12/main/http.c 411242 
> 
> Diff: https://reviewboard.asterisk.org/r/3402/diff/
> 
> 
> Testing
> -------
> 
> Tested on Ubuntu server 12.04 with Sam's json api test from issue.
> 
> 
> Thanks,
> 
> Scott Griepentrog
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20140326/6721e9cc/attachment.html>