[asterisk-dev] Kernel modules => mainline kernel

[Kevin P. Fleming]

John T. Guthrie III wrote:
> I was wondering, what are the plans, if any towards getting the Zaptel linux
> kernel modules included into the mainline linux kernel?

(reviving an old thread I haven't had time to jump into yet)

I'm going to respond to the top message of this thread, even though this
response will cover questions raised in various other replies.

First, let me say that we _have_ discussed this possibility at length
within the Digium organization. There are three primary issues that have
kept us from proceeding down this road (they are not impossible to
solve, just difficult):

1) Support for both 2.4 and 2.6 kernels is important for current Zaptel
users. There are still large numbers of users using Asterisk and Zaptel
on 2.4 kernels, and until very recently (with the TC400B driver) we've
supported every single piece of Digium hardware on 2.4 kernels.
Maintaining two code bases (one in the mainline 2.6 kernel and one
outside {since the 2.4 kernel is frozen for new features}) would
certainly be more difficult than the current situation.

2) Release cycles are an issue as well; if the drivers are part of the
mainline 2.6 kernel, then adding drivers for new hardware is tied to
_both_ the kernel release cycle (which would impact release of new
hardware products and having their drivers available when the hardware
ships) _AND_ distribution release cycles, because users who use Zaptel
via the mainline/distro kernel will have to wait until the distribution
has validated/backported/etc. the new drivers into their kernel
releases. For the 'extremely stable' distributions like
RHEL/CentOS/Debian/Ubuntu LTS/etc. that resist backports of new
features, it would require convincing those distro maintainers of the
worth of backporting the new drivers and the users would have to wait
months for them to be available.

3) Licensing is also another concern. If the primary Zaptel package for
2.6 kernels is in the mainline kernel, then it is open to modification
by anyone who submits patches to the mainline kernel. Since Digium
desires to maintain dual-license control over Zaptel (just as we do for
Asterisk), this could complicate the patch acceptance and merge process.
Even if we were granted 'maintainer' status over all of Zaptel in the
mainline kernel, it would still be relatively likely that someone would
propose a patch and be unwilling to license it for Digium's use. We'd
rather not deal with that situation if we can avoid it (even though
there are people in the kernel community who already deal with it and
have offered to help us).

Note that this is _not_ a hard-and-fast decision, nor am I the one to
make this decision solely if the decision were to be made. I'm only
communicating what we've already discussed internally and with the
Fedora maintainers who brought this up with us most recently.

Now, on to the technical issues raised in this thread:

A) 'stable API' - I have no idea where this comment comes from; there
were very few API changes between Zaptel 1.2 and Zaptel 1.4. We've added
some functionality (transcoder and HPEC support) since Zaptel 1.4 was
branched, but this did not change the API to userspace in any
non-backward-compatible way.

B) 'single application' - I can't speak to whether this is true or not,
but Zaptel is in no way tied to Asterisk. I'm sure there are other
applications that use Zaptel, although there may not be any with as many
users as Asterisk has.

C) 'kernel version compatibility' - I think we've had a pretty good
track record over the past year with maintaining compatibility with new
kernel releases. I personally keep my systems up to date as soon as new
mainline kernels are released, and I have personally updated Zaptel to
match the last three or four kernel API changes that affected us. There
have been some issues with the XPP drivers as Oron and Tzafrir can
attest to, but the drivers for Digium hardware have been reasonably easy
to keep updated.

D) 'larger chunk size/fewer interrupts' - One point that was overlooked
is that you cannot change to a 10ms/20ms/etc. Zaptel chunk size without
severely impacting software echo cancellation. If all Zaptel users had
echo cancellation that occurred before the data was transferred off the
card into kernelspace, then we could easily allow each Zaptel channel to
 have a configurable chunk size and transfer data in natural size
frames. However, that is not the case, and won't be the case for the
foreseeable future. As far as I know, waiting more than one or two
milliseconds to echo-cancel the received data is likely to break any
good echo canceler.

E) 'single interrupt/HPET/etc' - I'd be happy to see such a change
appear. I agree that there is no reason we need to process the data from
each card separately; everything runs at 1KHz, so any timing source that
can provide 1KHz reliably should be adequate. I'd also agree with Oron
that the Zaptel core 'timing' the individual drivers makes a lot of
sense, instead of the other way around. We'd need to be able to
configure the core with a list of possible timing sources and their
relative priority, but that wouldn't be terribly difficult to do.

F) broken math - This example (for 4 quad T1/E1 cards in E1 mode) was
offered:

So 4 Quad-PRI cards with this driver will generate 4000 interrupts/sec.
   Let's compare the data volume:
     8bytes * 30channels * 4ports * 2[R/W] * 1000/sec <= 2Kb/sec of PCM.

   Hmm... 4000 interrupts/sec for a 2Kb/sec data flow, good deal indeed.

However, this math is quite wrong. For 16 E1s, the math is like this:

8 bytes per channel
30 channels per span
4 spans per card
4 cards in the box
2 directions of data flow
1000 interrupts per second

That totals 7.68MB of data per second, using 4000 interrupts. That's
still vastly more interrupts than a NIC would use, but NICs don't have
timing and latency sensitivity that TDM does. Dropping that to 1000
interrupts per second would put us at 7.68KB of data processed per
interrupt, which is a much more reasonable number.

All the discussions regarding performance improvement, interrupt
mitigation and other technical topics don't really relate to whether the
drivers are in the mainline kernel or not; it's certainly possible for
our developers to learn from what has been done in the kernel's drivers
and take advantage of those techniques. Up to this point we have not
concentrated on _improving_ Zaptel but instead on keeping it compatible
with kernel changes and our new hardware products, but that will be
changing in the near future.