[asterisk-dev] Dahdi resource limit

[Olle E. Johansson]

4 apr 2012 kl. 19:09 skrev Shaun Ruffell:

> On Tue, Apr 03, 2012 at 09:26:49PM +0200, Olle E. Johansson wrote:
>> 
>> 3 apr 2012 kl. 21:11 skrev Shaun Ruffell:
>> 
>>> On Tue, Apr 03, 2012 at 06:22:36PM +0200, Olle E. Johansson wrote:
>>>> 
>>>> 3 apr 2012 kl. 17:01 skrev Shaun Ruffell:
>>>> 
>>>>> On Tue, Apr 03, 2012 at 10:18:22AM +0200, Olle E. Johansson wrote:
>>>>>> 
>>>>>> I am lost how to debug this and provide a good report. Dahdi
>>>>>> developers - are you aware of some sort of limit apart from the
>>>>>> file handles that can lead to this issue?
>>>>> 
>>>>> 2.5.0 was the first release to add a module parameter to specify the
>>>>> maximum number of psuedo channels that can be opened (and lowered
>>>>> limit from hard coded 1024 for all channels to 512 for only pseudo
>>>>> channels). Perhaps your bumping into that? 
>>>>> 
>>>>> The change went in in r9610 "dahdi: Add module parameter to limit
>>>>> number of pseudo channels." [1]
>>>>> 
>>>>> [1] http://svnview.digium.com/svn/dahdi?view=revision&revision=9610
>>>> 
>>>> Thanks for the feedback, Shaun!
>>>> 
>>>> As I wrote before - we have configured Dahdi to for 4000 file handles,
>>>> but still hit some kind of limit around 1400 open ones.
>>> 
>>> Ahh, I wasn't exactly sure what you meant by "we have configured
>>> Dahdi for 4000 file handles".  The 'max_pseudo_channels' only limits
>>> the total number of psuedo channels that can be created, not the
>>> total number of open descriptors to DAHDI objects.
>>> 
>>> So regarding your original question about what limits may be in the
>>> driver: There is a hard coded limit of 1024 conferences which you
>>> may be hitting if you're dropping each psuedo you create into it's
>>> own conference. See DAHDI_MAX_CONF include/dahdi/user.h if you want
>>> to bump it up, but I still think you should be CPU bound before you
>>> hit that limit.
>>> 
>>> Since all the conferencing is done on a single CPU, I can't open
>>> more than about 1500 psuedo channels and still have a usable system.
>>> Too much time is eaten up in the context of the timer interrupt
>>> which is handling all the mixing, transmit, and receives for those
>>> pseudo channels.
>>> 
>>> Try the following to get a sense of how many psuedo channels you
>>> might be able to open:
>>> 
>>> # /etc/init.d/dahdi stop
>>> # modprobe dahdi max_pseudo_channels=4000
>>> # ulimit -n 4000
>>> 
>>> Then run the following through your python interpreter:
>>> 
>>> import sys
>>> import time
>>> 
>>> chans = []
>>> x = 0
>>> 
>>> while True:
>>>     chans.append(file("/dev/dahdi/pseudo", "rw"))
>>>     time.sleep(0.01)
>>>     x = x + 1
>>>     sys.stdout.write("%d " % (x))
>>>     sys.stdout.flush()
>>> 
>>> When the CPU that is running the timer gets pegged out, this process
>>> can take awhile to kill, but everything will go back to normal once
>>> it is.
>>> 
>>> The load on the system is even more pronounced when those pseudos
>>> are actually in conference with one another.
>>> 
>>> Does this help?
>> Absolutely, thank you! 
>> Together with Kevin's earlier explanation of the pseudo and timer usage I know have a more complete picture.
>> 
>> Now, the question is where we go from here. I guess not doing mixing in Dahdi is my solution,
>> which for 1.4 means one of the out-of-tree solutions.
>> 
>> I still wonder where the leakage is, but that is another issue. It's not one per conference, seems more random like that.
> 
> Olle,
> 
> Another thing that you may do in the interim to increase the number
> of meetme conferences you can process is change DAHDI_CHUNKSIZE size
> and only build dahdi.ko.
> 
> If I change DAHDI_CHUNKSIZE to 80 (10 ms of audio in each chunk) I
> can hold open 4000 pseudo channels and sit at about 75% CPU consumed
> in software interrupt context. That's not too bad if you have other
> cores to do the real work. 80 is probably the max you'll want to do
> if audio is coming from userspace in 20ms frames.
> 
> You'll need to change DAHDI_CHUNKSIZE in include/dahdi/kernel.h,
> increase DAHDI_MAX_CONFERENCE to an appropriate number, and edit
> drivers/dahdi/Kbuild, commenting out all the board drivers (some of
> them are not designed to work with chunksizes greater than
> 8bytes/1ms).
> 
> Just an idea that might get you closer to your target,
> Shaun

Great. Yes, I'm only using DAHDI and no hardware cards, so this sounds like something we want to test.

Happy Easter!

/O