[asterisk-dev] slow slow menuselect (and proposal for a fix)

[Kevin P. Fleming]

Luigi Rizzo wrote:

(note that I have already committed a fix for the problems outlined
here, just finally got a chance to respond... see below)

> The reason seems to be that, if i am not mistaken, menuselect itself
> does a full pass on the entire set of sources to check dependencies.
> There is a second full pass, with awk being run on each source file
> while making up menuselect-tree.  This seems to happen every time
> I run 'make' in the top level, even without any change in the
> sources.

Actually, this is not quite true. menuselect itself is not aware of the
source files at all, it is fed all the information it needs via the
menuselect-tree file and some other input files.

What you are seeing is the prep_moduledeps script in build_tools, which
uses awk twice to get the MODINFO and MAKEOPTS blocks from the file, and
grep once to get the file's description from the AST_MODULE_INFO macro
call. Unfortunately this latter option requires scanning the entire file
since the macro call is always at the end of the file.

It should not be happening every time you run 'make'... if you are
seeing the 'Generating input for menuselect...' output every time you
run a normal 'make', then something is wrong on your system and make
cannot tell what files need to be rebuilt due to dependencies having
changed.


> The attempt to put in the same file (the .c source) code, documentation
> and build dependencies is commendable. But in my opinion it is completely
> unreasonable that we have to process 300 files and 310k lines of code
> every time we run 'make', only to collect the 1000 or so lines
> that we care about (and find that they are unchanged).

See my answer above... your analysis is flawed, in that it is not
supposed to be run every time you compile the source code. With that
said, however, there was significant room for improvement, because as it
stood it would regenerate the entire set of metadata for any change in
any source file that it cared about, which was certainly not optimal.
Now this information is cached and only regenerated for the files that
have actually changed.

> I see two steps to remove this inefficiency:
>  
> 1. (simple): remove the dependency on menuselect.makeopts from a number
>         of targets, introducing it only as a dependency for makeopts
> 
> 2. (slightly harder): remove the MODULEINFO and MAKEOPTS sections
>         from foo.c source files, and put them somewhere else, either
>         in some foo.opts files that go along with them, and/or in a central
>         place for each directory

No, we won't do this. Splitting up the information about the
dependencies for a source file from the file itself will only lead to
developers making mistakes and not keeping things updated, or users
applying patches to source files and not patching the metadata files, or
worse.

> There are at least two reasons for the latter change:
> - prevent rebuilding menuselect-tree and running menuselect every time
>   a simple change (e.g. bugfix or the like) is made to a C source file;
> - considerably speed up rebuilding menuselect-tree and running menuselect,
>   because now we have fewer and much shorter files to deal with.

Please review the changes I've already committed; on my laptop, changing
a single source file with the previous build system required 1.83
seconds to run 'make menuselect.makeopts', but with the improved build
system this only requires 0.37 seconds. Of course there is the time
required to initially build the metadata cache files when a new checkout
is made, but we can't make everything perfect :-)

(Oh, and I've also modified the script used when we package Asterisk
into a tarball so that all these metadata cache files will be part of
the tarball as well, so that users using numbered releases won't have to
wait for the metadata parsing step either.)

-- 
Kevin P. Fleming
Director of Software Technologies
Digium, Inc. - "The Genuine Asterisk Experience" (TM)