[asterisk-dev] zaptel, menuselect and autoconf

Thu Jul 5 04:06:49 CDT 2007

Warning: long post follow. 

I have been looking at separating the kernelspace of zaptel from
userspace: 
http://bugs.digium.com/view.php?id=7117
http://svn.digium.com/view/zaptel/team/tzafrir/kernelmove_14/

As you can see from the commits in that branch (rev. 2651),
I ended up removing menuselect completely and using autoconf. Kevin and
Russel did not like this idea. I'd like to point out here what problems 
I have encountered and suggest some solutions to the problems of
menuselect.

As I worked on cleaning up the build system of Zaptel I noticed that
there are two major sources of mess:

1. Mixing userspace, kernel 2.4 and kernel 2.6 makefiles in the same
file. Sorting that out has been the main focus of the branch.

2. menuselect:
  2.1. Configuration still needs much parsing after inclusion (the
  common $(filter ) and even more common $(filter-out ) ).
  2.2. A number of include files.
  2.3. Configueration changes at make runtime

This is why a fairly recent version of gnu make is needed, and also why
'make clean install' will try to build and install everything,
regardless of the modules you have disabled in the menu.

As there was great objection to simply removing menuselect, I tried
figuring out what is it exactly that menuselect does.

So following is my understanding as to the role of menuselect.

Menuselect is mainly intended to allow the user to easily select the
build-time configuration. 

Is build-time configuration really needed?
"""""""""""""""""""""""""""""""""""""""""
libpri and asterisk-gui don't use menuselect even today. And really, 
there's nothing to configure there. 

While asterisk-addons uses it, I believe that for the few options 
there the configuration through a configure script should do the trick. 

Zaptel is a borderline case. I like the configure syntax interface for
in better: it makes the common cases (all, or just a specific module you
need) easier and simpler than menuselect. But if folks really insisted
on menuselect there, then whatever.

Asterisk is really the application menuselect was designed for. However,
is there really a point for the common user to disable building modules?
Again: "make the common task simple, make everything possible". One
problem with menuselect of today is that it makes it all too easy to 
accidentally building a module.

Hence my standard instructions for building asterisk are "./configure;
make", to avoid letting the builder shooting himself in the foot.

Why is menuselect run from the Makefile?
"""""""""""""""""""""""""""""""""""""""
Menuselect is run after the configure script. It is run from the
makefile. 

In the reverted commit above, all that logic has moved from the makefile
to the autoconf script. What is the problem then?

menuselect cannot apply its selection of modules. Menuselect can't even
fix the lists of modules to satisfy dependencies.

So I'll just run menuselect dependency checkingand be done with it,
right? But this requires a built menuselect and mxml. And hence can only
be run after ./configure .

So I tried to rewite the dependency checking in an independent script I
could run from the configure script. I played a bit with some XML 
parsers and realised that any of them will pull some non-standard 
libraries along the way.

So the answer is:
Because we store the configuration as XML and need to build the XML
parser in order to read it.

What does Menuselect do?
""""""""""""""""""""""""
In a default run of the makefile it is only used with the option
--check-deps. In that case it will report if the configuration is OK or
not. Alternatively it will run a menu and generate cinfiguration. It is
not possible to ask it to just fix partial configuration.

The main data file for menuselect is menuselect-tree which is a
generated XML file. It is generated by a few simple shell scripts for a
few "XML" files in the toplevel directories as well as data in modules.
Those .xml files are not real XML files but rather snippers. In the
process they are concatenated into the menuselect-tree XML file.

Menuselect allows selecting items not to build. It will only generate a
legal configuration: one in which all dependencies are satisfied. It
also has an option to just go ever the current configuration and check
of its configuration are satisfied.

But then again, the XML data we use is not so complex. It is a tree with
an exact depth of 3: a single toplevel menu, in it categories, an in
each category there are members.

So I would suggest selecting one of the following two alternatives:

1. Depend on some XML library to be available on the build system and
use an existing parser program / module in the configure script, or:

2. Use simpler format for the data.

(1) seems to me unrealistic. Hence I have opted for (2).

Simpler Configuration Data Format
"""""""""""""""""""""""""""""""""
I have an initial implementation of this in the branch
zaptel/team/tzafrir/kernelmove_conf .

The data format I chose was to place each category in its own file under
the configuration directory (build_conf/ ).  Here is the file 
build_conf/kernel. I should point out that this file is completely 
generated by a small perl script from data embedded in source files.

It is currently extracted on every run of autoconf. It seems that the 
overhead for that is much lower than e.g. the time it takes it to 
generate outputs, or to test for libusb. I suspect that even for 
Asterisk the overhead won't be that big.

  Depends: zaptel
  DisplayName: Dynamic Span Support
  Item: ztdynamic
  PullOnly: yes

  Depends: zaptel
  DisplayName: Transcoder Support
  Item: zttranscode
  PullOnly: yes

  Depends: zaptel
  DisplayName: Digium Wildcard TE4XXP / TE2XXP
  Item: wct4xxp/

  Depends: zaptel, zttranscode
  DisplayName: Digium Wildcard TC400B
  Item: wctc4xxp/

Currently all entries except Item are copied verbatim from the
specially-formatted comment in the code (I kept the same comment style),
e.g: in kernel/wcfxo.c:

  /*** MODULEINFO
  Depends: zaptel
  DisplayName: Digium Wildcard X100P
   ***/

I have not yet tried to isolate it from the other changes in the
kernelmove branch (moving modules into kernel/ ) but the changes are
basically the added:

  build_tools/config_info
  build_tools/update_config (still too hard_wired)

And the following changes:

  configure.ac - kernel modules and utilities configuration
  All kernel modules have embedded configuration.

Configure prints a nice summary in the end, so I know what It'll do.
By default it will just build "all modules" as in today.

That is:

  ./configure

will eventually print:

  configure: Will build the following:
  configure: Utilities: fxotune fxstest kernel/xpp/utils/ sethdlc-new ztcfg ztdiag ztmonitor zttest zttool .
  configure: Kernel Modules and subdirectories: pciradio tor2 torisa wcfxo wct1xxp wct4xxp/ wctc4xxp/ wctdm wctdm24xxp wcte11xp wcte12xp wcusb xpp/ zaptel ztd-eth ztd-loc ztdummy ztdynamic zttranscode .
  configure: Kernel 2.6.18-4-686 from /lib/modules/2.6.18-4-686/build .
  configure: *** Zaptel build successfully configured ***

If I use:

  ./configure --with-modules=wctdm

I will eventually get:

  configure: Will build the following:
  configure: Utilities: fxotune fxstest kernel/xpp/utils/ sethdlc-new ztcfg ztdiag ztmonitor zttest zttool .
  configure: Kernel Modules and subdirectories: wctdm zaptel .
  configure: Kernel 2.6.18-4-686 from /lib/modules/2.6.18-4-686/build .
  configure: *** Zaptel build successfully configured ***

Take a look at configure.ac . Compare the ugly manual code that handles
utilities to the simple code that handles kernel modules. All the work
is moved to a simple perl script (config_info).

The makefile becomes much simpler.

This model is still a bit too simplistic to handle dependencies between
categories, as present in Asterisk. That would require a slightly more
complex handling of the lists. Maybe separate the dependency checks from
the rest of the cleanup. Certainly pass items from several categories.
Still, not too complex.

Conf changes vs. kernelmove
"""""""""""""""""""""""""""
I starte this work in the kernelmove branch, but I consider it
independent. And specifically, I have reverted those changes from the
kernelmove branch.

I keep those changes in a branch that follows kernelmove and not
zaptel 1.4 or trunk proper simply because kernelmove has a saner
makefile to start with.

-- 
               Tzafrir Cohen       
icq#16849755                    jabber:tzafrir at jabber.org
+972-50-7952406           mailto:tzafrir.cohen at xorcom.com       
http://www.xorcom.com  iax:guest at local.xorcom.com/tzafrir