[asterisk-dev] Final Preview: docs.asterisk.org

[asterisk at phreaknet.org]

On 6/29/2023 3:14 PM, George Joseph wrote:
> I think we're at a point where the new documentation site can go 
> live.  The dynamic documentation is integrated and the README file has 
> been expanded greatly with information on how the site is built and 
> how you can build it yourself.

Hi, Geroge,
     Just had a chance to look at this this afternoon. The instructions 
for the dynamic doc generation definitely made my head hurt a little 
bit, but I have a few thoughts after putzing around a little bit.

My initial thought was that some of the make targets in the Makefile 
could be split up a little bit. The version-dynamic target both 
downloads documentation source and does the actual build of the 
documentation. They could be split into different targets:

  * In my case, I don't want to download the upstream Asterisk
    documentation, I want to use the local core-en-us.xml, which is a
    superset of the documentation available upstream.
  * Since I'm not downloading the docs, I don't think I need the "gh"
    tool, and so it doesn't need to be installed for purely generating
    documentation. (Also, could be documented as a pre-req, if doing the
    download step)

Then again, when you boil it down to that, it seems like it really comes 
down to just:

  * utils/astxml2markdown.py
  * mike deploy

So it seems I'm better off avoiding the makefile and just running the 
individual commands needed. I'll end up wrapping it in a script anyways, 
but just wondering if there's anything else about the process here I've 
missed that wouldn't be conducive to that?

In particular, it doesn't seem that finagling with Git repositories is 
necessary at all to build the dynamic docs. I was able to get it to work 
with just this:

git clone https://github.com/asterisk/documentation.git --depth 1
cd documentation
pip3 install -r requirements.txt
mv docs/favicon.ico docs/index.md .
rm -rf docs/*
mv favicon.ico index.md docs
utils/astxml2markdown.py 
--file=/usr/src/asterisk-20.3.0/doc/core-en_US.xml 
--xslt=utils/astxml2markdown.xslt --directory=docs/ --branch=20 
--version=20.3.0
mike deploy -F mkdocs.yml 20
rm -rf /var/www/html/asterisk && mv site /var/www/html/asterisk
cd .. && rm -rf documentation

It seems to work really well. There were just a couple surprises or 
annoying aspects:

  * Even with --depth 1, the documentation repo takes a hot minute to
    clone, due to all the large PDF artifacts in it, though a tarball of
    the repo wouldn't help either if it came with all the static
    artifacts anyways. Could probably work around that by cloning it
    using svn instead of git, but I was too lazy to do that today.
  * For just turning markdown into HTML, mike is pretty slow, it takes
    over half a minute just for the dynamic docs (compared to ~1 second
    for my previous method, though that was from the original XML to
    HTML, not from intermediate markdown)
  * Most significantly, the webpages are *huge*. Even just the dynamic
    docs are a whopping 228 MB. On average, a documentation page is
    almost 250 KB (compared to 1.2 MB the old way for all the
    applications, functions, AMI/AGI, and configs on a single webpage -
    granted it's not a fair comparison since the menu and what not isn't
    repeated that way). Taking a look at the page source of an
    application[1], much of the page is whitespace. I know that docs
    hosted on GitHub don't need to worry about disk consumption, but for
    folks building the docs themselves, I think it might be worth trying
    to clean this a little bit. Bloated webpages are obviously also
    going to be slower to load as well.

I haven't yet figured out what's introducing all the extraneous 
whitespace. The markdown files seem okay, but things seem to blow up 
somewhere in the middle. Ideally we could prevent it from happening in 
the first place, but if not, then maybe some fancy recursive 
post-processing could strip it all out.

[1] https://docs.asterisk.org/20/_Dialplan_Applications/ADSIProg/