[asterisk-dev] Dahdi resource limit

[Shaun Ruffell]

On Tue, Apr 03, 2012 at 09:26:49PM +0200, Olle E. Johansson wrote:
> 
> 3 apr 2012 kl. 21:11 skrev Shaun Ruffell:
> 
> > On Tue, Apr 03, 2012 at 06:22:36PM +0200, Olle E. Johansson wrote:
> >> 
> >> 3 apr 2012 kl. 17:01 skrev Shaun Ruffell:
> >> 
> >>> On Tue, Apr 03, 2012 at 10:18:22AM +0200, Olle E. Johansson wrote:
> >>>> 
> >>>> I am lost how to debug this and provide a good report. Dahdi
> >>>> developers - are you aware of some sort of limit apart from the
> >>>> file handles that can lead to this issue?
> >>> 
> >>> 2.5.0 was the first release to add a module parameter to specify the
> >>> maximum number of psuedo channels that can be opened (and lowered
> >>> limit from hard coded 1024 for all channels to 512 for only pseudo
> >>> channels). Perhaps your bumping into that? 
> >>> 
> >>> The change went in in r9610 "dahdi: Add module parameter to limit
> >>> number of pseudo channels." [1]
> >>> 
> >>> [1] http://svnview.digium.com/svn/dahdi?view=revision&revision=9610
> >> 
> >> Thanks for the feedback, Shaun!
> >> 
> >> As I wrote before - we have configured Dahdi to for 4000 file handles,
> >> but still hit some kind of limit around 1400 open ones.
> > 
> > Ahh, I wasn't exactly sure what you meant by "we have configured
> > Dahdi for 4000 file handles".  The 'max_pseudo_channels' only limits
> > the total number of psuedo channels that can be created, not the
> > total number of open descriptors to DAHDI objects.
> > 
> > So regarding your original question about what limits may be in the
> > driver: There is a hard coded limit of 1024 conferences which you
> > may be hitting if you're dropping each psuedo you create into it's
> > own conference. See DAHDI_MAX_CONF include/dahdi/user.h if you want
> > to bump it up, but I still think you should be CPU bound before you
> > hit that limit.
> > 
> > Since all the conferencing is done on a single CPU, I can't open
> > more than about 1500 psuedo channels and still have a usable system.
> > Too much time is eaten up in the context of the timer interrupt
> > which is handling all the mixing, transmit, and receives for those
> > pseudo channels.
> > 
> > Try the following to get a sense of how many psuedo channels you
> > might be able to open:
> > 
> > # /etc/init.d/dahdi stop
> > # modprobe dahdi max_pseudo_channels=4000
> > # ulimit -n 4000
> > 
> > Then run the following through your python interpreter:
> > 
> >  import sys
> >  import time
> > 
> >  chans = []
> >  x = 0
> > 
> >  while True:
> >      chans.append(file("/dev/dahdi/pseudo", "rw"))
> >      time.sleep(0.01)
> >      x = x + 1
> >      sys.stdout.write("%d " % (x))
> >      sys.stdout.flush()
> > 
> > When the CPU that is running the timer gets pegged out, this process
> > can take awhile to kill, but everything will go back to normal once
> > it is.
> > 
> > The load on the system is even more pronounced when those pseudos
> > are actually in conference with one another.
> > 
> > Does this help?
> Absolutely, thank you! 
> Together with Kevin's earlier explanation of the pseudo and timer usage I know have a more complete picture.
> 
> Now, the question is where we go from here. I guess not doing mixing in Dahdi is my solution,
> which for 1.4 means one of the out-of-tree solutions.
> 
> I still wonder where the leakage is, but that is another issue. It's not one per conference, seems more random like that.

Olle,

Another thing that you may do in the interim to increase the number
of meetme conferences you can process is change DAHDI_CHUNKSIZE size
and only build dahdi.ko.

If I change DAHDI_CHUNKSIZE to 80 (10 ms of audio in each chunk) I
can hold open 4000 pseudo channels and sit at about 75% CPU consumed
in software interrupt context. That's not too bad if you have other
cores to do the real work. 80 is probably the max you'll want to do
if audio is coming from userspace in 20ms frames.

You'll need to change DAHDI_CHUNKSIZE in include/dahdi/kernel.h,
increase DAHDI_MAX_CONFERENCE to an appropriate number, and edit
drivers/dahdi/Kbuild, commenting out all the board drivers (some of
them are not designed to work with chunksizes greater than
8bytes/1ms).

Just an idea that might get you closer to your target,
Shaun

-- 
Shaun Ruffell
Digium, Inc. | Linux Kernel Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: www.digium.com & www.asterisk.org