[Asterisk-Users] Asterisk & SMP: Is irqbalance Redundant on 2.6 Kernels? - Resolved

[Matt Roth]

I contacted the developer of irqbalance (Eric Dorland) and he responded
as follows:

"Certainly it can't cause any serious problems, just performance ones.
And yes, it is my understanding that later 2.6 can do irq balancing
themselves if the right option is set when compiling it."

Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer

Matt Roth wrote:

> Asterisk users,
>
> I posted the following email to the Fedora users list 
> <https://www.redhat.com/archives/fedora-list/2006-March/msg04154.html> 
> and it got no responses, so now I'm calling on your expertise.  Please 
> take a look at it and share your knowledge on the subject with me.
>
> Additionally, let me know if you believe what I am trying to do is 
> critically flawed. It's possible that it would be better to allocate 
> one processor to handle all of the IRQs and reserve the others for 
> running processes.  I have some concerns about retransmits if packet 
> data is split across the CPUs and re-assembled in a different order 
> than expected.  I'm open to any suggestions on how to best balance 
> Asterisk's load on an SMP system.
>
> ============================================================
>
> List users,
>
> I have a server running Fedora Core 3 with the 2.6.12-1.1376_FC3smp
> kernel.  It has four physical CPUs and I would like to balance the IRQs
> across them.  It is a large scale VOIP server running Asterisk, so my
> main interest is balancing the IRQs generated by the ethernet device.
> Obviously, there is heavy network activity, so if the IRQs are not
> balanced the server will be CPU bound by the lone processor assigned to
> handle them.
>
> Initially, I solved this problem by adding the following line to
> "/etc/rc.local" (82 is the IRQ of the ethernet device):
>
>  echo 0f > /proc/irq/82/smp_affinity
>
> This worked like a charm.  The IRQs from the ethernet device were
> balanced across the four processors and each of their idle times was
> comparable.  Unfortunately, when I duplicated this on our backup machine
> (adjusting the IRQ accordingly), I discovered that something was
> overwriting the value that I echoed to the "smp_affinity" file.  The
> problem was that the primary machine had a service that included a
> fifteen second sleep in its init script.  This service was started
> *after* the irqbalance daemon and *before* rc.local.  Apparently
> irqbalance runs on a ten second interval and (this part is speculation
> based on my experience, as it is not the most well documented daemon out
> there) it only updates the smp affinity of ethernet devices on the first
> poll.  All this led to irqbalance overwriting the value I echoed to the
> "smp_affinity" file on the backup machine, because it did not have the
> service with the sleep in its init script installed.
>
> My first question is, what is the point of the irqbalance daemon if the
> kernel can handle balancing IRQs by itself?
>
> Googling for irqbalance brought me to the documentation for the Debian
> package <http://packages.debian.org/unstable/utils/irqbalance>, which
> states "Useful mostly just for 2.4 kernels, or 2.6 kernels with
> CONFIG_IRQBALANCE turned off."  I grepped through the kernel source, but
> could not find this constant so as an experiment I turned off the
> irqbalance daemon and rebooted the system.  Every "smp_affinity" file
> under "/proc/irq/*/" now contained the default value of "ffffffff" and I
> confirmed that all IRQs were being balanced across the CPUs by checking
> "/proc/interrupts".
>
> Is it possible that irqbalance is just a holdover from the 2.4 kernel
> days?  Can it be turned off on my kernel without any adverse affects?
>
> I also ran the same experiment on a multi-processor SUSE box running the
> 2.6.5-52-smp kernel.  This machine did not have irqbalance installed and
> the values in the "smp_affinity" files were all "ff".  On this machine,
> all of the interrupts were being handled by CPU0. Apparently, SUSE's
> kernels are compiled without IRQ balancing enabled.
>
> I've also done a little digging through "kernel-parameters.txt" and I
> came across a few interesting lines:
>
>  acpi=     noirq -- do not use ACPI for IRQ routing
>
>  acpi_irq_balance   [HW,ACPI] ACPI will balance active IRQs -- default
> in APIC mode
>
>  pci=      noacpi   [IA-32] Do not use ACPI for IRQ routing or for PCI
> scanning.
>
>  noapic             [SMP,APIC] Tells the kernel to not make use of any
> IOAPICs that may be present in the system.
>
>  noirqbalance       [IA-32,SMP,KNL] Disable kernel irq balancing
>
> This is all more than a little confusing to me.  It appears that the
> kernel's IRQ balancing is contingent on ACPI and APIC, but I'm far from
> positive about this conclusion.  If someone could explain to me how
> these different components interact with IRQ balancing, I would
> appreciate it.  My fear is that in an attempt to slim down the server I
> will turn off a component that will disable or silently cripple IRQ
> balancing by the kernel.
>
> Thank you,
>
> Matthew Roth
> InterMedia Marketing Solutions
> Software Engineer and Systems Developer
> _______________________________________________
> --Bandwidth and Colocation provided by Easynews.com --
>
> Asterisk-Users mailing list
> To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-users
>