<p>Joshua Colp <strong>merged</strong> this change.</p><p><a href="https://gerrit.asterisk.org/c/repotools/+/10941">View Change</a></p><div style="white-space:pre-wrap">Approvals:
Kevin Harwell: Looks good to me, but someone else must approve
George Joseph: Looks good to me, but someone else must approve
Joshua Colp: Looks good to me, approved; Approved for Submit
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">build: Improve handling of CHANGES and UPGRADE.txt for releases.<br><br>The release script now handles the merging of the CHANGES and<br>UPGRADE.txt files for us! When a release is being done, the script will<br>go through the staging changes in the Asterisk working directory<br>(<asterisk-home>/doc/<file>-staging) and add each change to the<br>corresponding file. A separate script (process-staging-changes) has also<br>been added that can be used when creating a new version from master. It<br>takes 3 arguments: -l/--local-root, which is optional,<br>-s/--start-version, and -e/--end-version. You can use -h for more<br>information on each option. Another script has been added that should be<br>run after process-staging-changes. This script is called<br>commit-staging-changes. It will go through all unstaged changes, add<br>them, and then commit them to the local repository. Then it will push<br>these changes to the remote. These scripts should be used for master or<br>if we need to do things manually (for some reason).<br><br>This means that there's a new way to document our major changes. All<br>changes for CHANGES will go into doc/CHANGES-staging and all changes for<br>UPGRADE.txt will go into doc/UPGRADE-staging. Each of these files should<br>have a meaningful name related to what the change is. For example, if<br>you made a change to something in pjsip, your file might be called<br>"res_pjsip_relative_title", where "relative_title" will be something a<br>little more descriptive than that. Inside of each file, you will have a<br>subject (1 or more) and corresponding headers (as of right now, the only<br>header is 'Master-Only'), with the description of the change following<br>that. You can have multiple subject lines in one file. For example, it<br>may look something like this:<br><br> Subject: res_pjsip<br> Subject: Core<br><br> This is a detailed description of what I changed.<br><br> You can have new lines in between as well, spacing is handled by the<br> release script!<br><br>The header lines (Subject:) are case sensative. One thing to note is<br>that master is different. Since changes that are master-only will only<br>go into the master branch, we add another special tag (Master-Only) that<br>will go underneath the "Subject:" header. Note that it doesn't have to<br>go under the "Subject:" headers, but it makes it easier to read. There<br>should be no space between these.<br><br> Subject: res_ari<br> Master-Only: true<br><br> This is a master-only change! You can put "True" or "true".<br><br>Note that the "Master-Only" header will only ever be true. There should<br>not be any scenarios where there will be a "Master-Only" header with the<br>value false.<br><br>For master (and possibly other releases), it will be necessary to do<br>some of this manually before running the mkrelease.py script. This is<br>because when cutting a major release (like from 16 to 17), the mainline<br>branch will never be master, so master must have these changes made<br>manually. The easiest way to do this is to make the changes before<br>cutting the new branch, so that the work does not need to be done twice.<br>Here's an example (run from the repotools dir):<br><br> ./process-staging-changes -s 16 -e 17<br> (check the working dir to make sure it looks correct)<br> ./commit-staging-changes -v 17<br><br>Note that both of these scripts take an optional parameter (-l /<br>--local-root). This works the exact same way as in the mkrelease.py<br>script. If you are not working from the default directory (/tmp), you<br>MUST specify the path with this option.<br><br>Fore more information, check out the wiki page:<br>https://wiki.asterisk.org/wiki/display/AST/CHANGES+and+UPGRADE.txt<br><br>First step for ASTERISK_28111. The directories will need to be added,<br>and all changes will need to be refactored into these directories. Also,<br>the UPGRADE.txt file will no longer branch off into different files for<br>each release of a new major version, so these should be merged together<br>as well to reduce clutter.<br><br>Change-Id: I6dc084afedaeecaf36aaec66c3cf6a5a8ed4ef3c<br>---<br>A commit-staging-changes<br>M digium_git.py<br>M mkrelease.py<br>A process-staging-changes<br>A staging_changes.py<br>5 files changed, 429 insertions(+), 6 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;"><span>diff --git a/commit-staging-changes b/commit-staging-changes</span><br><span>new file mode 100755</span><br><span>index 0000000..fc4c1f7</span><br><span>--- /dev/null</span><br><span>+++ b/commit-staging-changes</span><br><span>@@ -0,0 +1,39 @@</span><br><span style="color: hsl(120, 100%, 40%);">+#!/usr/bin/env python</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+import os</span><br><span style="color: hsl(120, 100%, 40%);">+import sys</span><br><span style="color: hsl(120, 100%, 40%);">+import logging</span><br><span style="color: hsl(120, 100%, 40%);">+from argparse import ArgumentParser</span><br><span style="color: hsl(120, 100%, 40%);">+from digium_git import get_repo,DigiumGitRepo</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+LOGGER = logging.getLogger(__name__)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+if __name__ == '__main__':</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ parser = ArgumentParser()</span><br><span style="color: hsl(120, 100%, 40%);">+ parser.add_argument("-l", "--local-root", dest="local_root",</span><br><span style="color: hsl(120, 100%, 40%);">+ help="The local root to work from", default="/tmp")</span><br><span style="color: hsl(120, 100%, 40%);">+ parser.add_argument("-v", "--version", dest="version",</span><br><span style="color: hsl(120, 100%, 40%);">+ help="The version to be released", required=True)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ (ns, args) = parser.parse_known_args()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ logging.basicConfig(level=logging.DEBUG, format="%(module)s:%(lineno)d - %(message)s")</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Default to asterisk and gerrit URL</span><br><span style="color: hsl(120, 100%, 40%);">+ repo = get_repo(local_root=ns.local_root)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGGER.debug("Adding and committing all staging changes")</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ commit_msg = "Update CHANGES and UPGRADE.txt for {0}".format(ns.version)</span><br><span style="color: hsl(120, 100%, 40%);">+ repo.add_and_commit_all_unstaged(commit_msg)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ prompt = "About to push changes to remote\n\tContinue [yes/no]?"</span><br><span style="color: hsl(120, 100%, 40%);">+ cont = raw_input(prompt)</span><br><span style="color: hsl(120, 100%, 40%);">+ if 'yes' in cont.lower() or 'y' in cont.lower():</span><br><span style="color: hsl(120, 100%, 40%);">+ repo.push_changes()</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ print "Not pushing changes to remote (HEAD needs to be reset)"</span><br><span style="color: hsl(120, 100%, 40%);">+ sys.exit(1)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ print "Done! Changes pushed to remote"</span><br><span>diff --git a/digium_git.py b/digium_git.py</span><br><span>index 0512642..d15699d 100644</span><br><span>--- a/digium_git.py</span><br><span>+++ b/digium_git.py</span><br><span>@@ -532,6 +532,23 @@</span><br><span> self.repo.index.add(files)</span><br><span> self.repo.index.commit(commit_msg)</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ def add_and_commit_all_unstaged(self, commit_msg):</span><br><span style="color: hsl(120, 100%, 40%);">+ """Add and commit all unstaged changes</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ Keyword Arguments:</span><br><span style="color: hsl(120, 100%, 40%);">+ commit_msg - Out commit message for the changes</span><br><span style="color: hsl(120, 100%, 40%);">+ """</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGGER.debug("Adding and commit all unstaged changes to branch {0}"</span><br><span style="color: hsl(120, 100%, 40%);">+ .format(self.current_branch))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ files = self.repo.index.diff(None)</span><br><span style="color: hsl(120, 100%, 40%);">+ for f in files:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.repo.git.add(f.a_path)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.repo.index.commit(commit_msg)</span><br><span style="color: hsl(120, 100%, 40%);">+ if self.current_branch not in self.updated_branches:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.updated_branches.append(self.current_branch)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> def push_changes(self):</span><br><span> """Push any changes (branches, tags, etc...) upstream"""</span><br><span> </span><br><span>diff --git a/mkrelease.py b/mkrelease.py</span><br><span>index 10e1a7e..81d4423 100755</span><br><span>--- a/mkrelease.py</span><br><span>+++ b/mkrelease.py</span><br><span>@@ -23,6 +23,7 @@</span><br><span> from release_summary import ReleaseSummary, ReleaseSummaryOptions</span><br><span> from alembic_creator import create_db_script</span><br><span> from testsuite import update_testsuite</span><br><span style="color: hsl(120, 100%, 40%);">+from staging_changes import StagingChangesExtractor</span><br><span> </span><br><span> LOGGER = logging.getLogger(__name__)</span><br><span> </span><br><span>@@ -123,6 +124,8 @@</span><br><span> global branch</span><br><span> global mainline</span><br><span> </span><br><span style="color: hsl(120, 100%, 40%);">+ project = options.project.lower()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span> if version_object.prefix:</span><br><span> mainline = '{0}/{1}.{2}'.format(version_object.prefix,</span><br><span> version_object.major,</span><br><span>@@ -133,22 +136,42 @@</span><br><span> branch = '{0}.{1}'.format(version_object.major,</span><br><span> version_object.minor)</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">- if options.project.lower() != 'asterisk':</span><br><span style="color: hsl(120, 100%, 40%);">+ if project != 'asterisk':</span><br><span> # Assume non asterisk project release branches are off of master</span><br><span> mainline = 'master'</span><br><span> </span><br><span> if repo.branch_exists(branch):</span><br><span> LOGGER.debug("Local branch {0} exists already".format(branch))</span><br><span style="color: hsl(120, 100%, 40%);">+ prompt_to_continue("Local branch {0} may not have updated CHANGES and"</span><br><span style="color: hsl(120, 100%, 40%);">+ " UPGRADE.txt files".format(branch))</span><br><span> else:</span><br><span> LOGGER.debug("Local branch {0} does not exist".format(branch))</span><br><span style="color: hsl(0, 100%, 40%);">- prompt_to_continue()</span><br><span style="color: hsl(0, 100%, 40%);">- # Switch to the mainline branch first that way the new release</span><br><span style="color: hsl(0, 100%, 40%);">- # branch is created off of that</span><br><span style="color: hsl(0, 100%, 40%);">- repo.checkout(mainline)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # We checkout mainline here anyways since we need to update CHANGES</span><br><span style="color: hsl(120, 100%, 40%);">+ # and UPGRADE.txt. If the branch does not exist, the new release will</span><br><span style="color: hsl(120, 100%, 40%);">+ # need to be created off of that anyways</span><br><span style="color: hsl(120, 100%, 40%);">+ repo.checkout(mainline)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Only update CHANGES and UPGRADE.txt if this is a full release (not beta,</span><br><span style="color: hsl(120, 100%, 40%);">+ # rc, etc.) for the Asterisk project</span><br><span style="color: hsl(120, 100%, 40%);">+ if len(version_object.modifiers) == 0 and project == 'asterisk':</span><br><span style="color: hsl(120, 100%, 40%);">+ s_version = version_object.get_previous_version()</span><br><span style="color: hsl(120, 100%, 40%);">+ start = "{0}.{1}.{2}".format(s_version.major, s_version.minor,</span><br><span style="color: hsl(120, 100%, 40%);">+ s_version.patch)</span><br><span style="color: hsl(120, 100%, 40%);">+ end = "{0}.{1}.{2}".format(version_object.major, version_object.minor,</span><br><span style="color: hsl(120, 100%, 40%);">+ version_object.patch)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ sce = StagingChangesExtractor(options.local_root, start, end)</span><br><span style="color: hsl(120, 100%, 40%);">+ ret = sce.run()</span><br><span style="color: hsl(120, 100%, 40%);">+ if ret is not 0:</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGGER.error("Failed to prepare staging changes")</span><br><span style="color: hsl(120, 100%, 40%);">+ sys.exit(1)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ repo.add_and_commit_all_unstaged("Update CHANGES and UPGRADE.txt for {0}"</span><br><span style="color: hsl(120, 100%, 40%);">+ .format(version))</span><br><span> </span><br><span> repo.checkout(branch)</span><br><span> </span><br><span style="color: hsl(0, 100%, 40%);">-</span><br><span> def extract_tags(options, repo):</span><br><span> """Extract the tags from the version to be created and the prepared repo</span><br><span> </span><br><span>diff --git a/process-staging-changes b/process-staging-changes</span><br><span>new file mode 100755</span><br><span>index 0000000..4efe98b</span><br><span>--- /dev/null</span><br><span>+++ b/process-staging-changes</span><br><span>@@ -0,0 +1,32 @@</span><br><span style="color: hsl(120, 100%, 40%);">+#!/usr/bin/env python</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+from argparse import ArgumentParser</span><br><span style="color: hsl(120, 100%, 40%);">+from staging_changes import StagingChangesExtractor</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+if __name__ == '__main__':</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ parser = ArgumentParser()</span><br><span style="color: hsl(120, 100%, 40%);">+ parser.add_argument("-l", "--local-root", dest="local_root",</span><br><span style="color: hsl(120, 100%, 40%);">+ help="The local root to work from", default="/tmp")</span><br><span style="color: hsl(120, 100%, 40%);">+ parser.add_argument("-s", "--start-version", dest="start_version",</span><br><span style="color: hsl(120, 100%, 40%);">+ help="The version to start history at", required=True)</span><br><span style="color: hsl(120, 100%, 40%);">+ parser.add_argument("-e", "--end-version", dest="end_version",</span><br><span style="color: hsl(120, 100%, 40%);">+ help="The version to release", required=True)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ (ns, args) = parser.parse_known_args()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ sce = StagingChangesExtractor(ast_path=ns.local_root, start_version=ns.start_version, end_version=ns.end_version)</span><br><span style="color: hsl(120, 100%, 40%);">+ ret = sce.run()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Only add trailing slash for printing message; it's not needed otherwise</span><br><span style="color: hsl(120, 100%, 40%);">+ if not ns.local_root.endswith("/"):</span><br><span style="color: hsl(120, 100%, 40%);">+ ns.local_root += "/"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if ret is 0:</span><br><span style="color: hsl(120, 100%, 40%);">+ used_l_opt = ""</span><br><span style="color: hsl(120, 100%, 40%);">+ if ns.local_root != "/tmp/":</span><br><span style="color: hsl(120, 100%, 40%);">+ used_l_opt = "-l {0} ".format(ns.local_root)</span><br><span style="color: hsl(120, 100%, 40%);">+ print("Done! Check {0}asterisk and make sure everything looks right. "</span><br><span style="color: hsl(120, 100%, 40%);">+ "Then run 'commit-staging-changes {1}-v {2}'.".format(ns.local_root, used_l_opt, ns.end_version))</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ print("Error! Check the logs and remember to clean up {0}asterisk".format(ns.local_root))</span><br><span>diff --git a/staging_changes.py b/staging_changes.py</span><br><span>new file mode 100644</span><br><span>index 0000000..8eceb48</span><br><span>--- /dev/null</span><br><span>+++ b/staging_changes.py</span><br><span>@@ -0,0 +1,312 @@</span><br><span style="color: hsl(120, 100%, 40%);">+#!/usr/bin/env python</span><br><span style="color: hsl(120, 100%, 40%);">+'''</span><br><span style="color: hsl(120, 100%, 40%);">+Used to extract information from <asterisk_home>/doc/<file>-staging/</span><br><span style="color: hsl(120, 100%, 40%);">+and add that information to <asterisk_home>/<file></span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+Ben Ford <bford@digium.com></span><br><span style="color: hsl(120, 100%, 40%);">+'''</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+import logging</span><br><span style="color: hsl(120, 100%, 40%);">+import os</span><br><span style="color: hsl(120, 100%, 40%);">+import fileinput</span><br><span style="color: hsl(120, 100%, 40%);">+from collections import defaultdict</span><br><span style="color: hsl(120, 100%, 40%);">+from enum import Enum</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+LOGGER = logging.getLogger(__name__)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+MODES = ["CHANGES", "UPGRADE"]</span><br><span style="color: hsl(120, 100%, 40%);">+FUNCTIONALITY_SEPARATOR_DASH_COUNT = 78</span><br><span style="color: hsl(120, 100%, 40%);">+CATEGORY_SEPARATOR_DASH_COUNT = 18</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+class HeaderFlags(Enum):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Header flags that are set based on what headers are present within each</span><br><span style="color: hsl(120, 100%, 40%);">+ subject in the change files</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ Values should be 1, 2, 4, 8, and so on</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ MASTER_ONLY = 1</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+class StagingChangesExtractor:</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Holds all the information needed to update the target files</span><br><span style="color: hsl(120, 100%, 40%);">+ (CHANGES / UPGRADE.txt)</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ class Container:</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Holds the commit subject and summary for CHANGES/UPGRADE.txt as well as</span><br><span style="color: hsl(120, 100%, 40%);">+ the timestamp of the commit</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def __init__(self, subject, message, timestamp):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Initializer</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ subject The commit subject</span><br><span style="color: hsl(120, 100%, 40%);">+ message The commit summary</span><br><span style="color: hsl(120, 100%, 40%);">+ timestamp The commit time (epoch)</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ self.subject = subject</span><br><span style="color: hsl(120, 100%, 40%);">+ self.message = message</span><br><span style="color: hsl(120, 100%, 40%);">+ self.timestamp = timestamp</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def by_timestamp(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Returns the container timestamp for sorting</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ returns timestamp (epoch)</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ return self.timestamp</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def __init__(self, ast_path, start_version, end_version):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Initializer</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ ast_path The location of the Asterisk directory we are working with</span><br><span style="color: hsl(120, 100%, 40%);">+ start_version The version we are starting from</span><br><span style="color: hsl(120, 100%, 40%);">+ end_version The version we are ending on</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ if ast_path is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception("ast_path cannot be NULL")</span><br><span style="color: hsl(120, 100%, 40%);">+ if start_version is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception("start_version cannot be NULL")</span><br><span style="color: hsl(120, 100%, 40%);">+ if end_version is None:</span><br><span style="color: hsl(120, 100%, 40%);">+ raise Exception("end_version cannot be NULL")</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ self.data = defaultdict(list)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.master_data = defaultdict(list)</span><br><span style="color: hsl(120, 100%, 40%);">+ self.ast_path = os.path.join(ast_path, "asterisk")</span><br><span style="color: hsl(120, 100%, 40%);">+ self.start_version = start_version</span><br><span style="color: hsl(120, 100%, 40%);">+ self.end_version = end_version</span><br><span style="color: hsl(120, 100%, 40%);">+ # Bits set based on what headers are present for each subject</span><br><span style="color: hsl(120, 100%, 40%);">+ self.header_flags = 0</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def gen_separator(self, count):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Generate a number of dashes to be used as a separator</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ count The number of dashes to generate</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ returns A string with <count> dash(es)</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ ret = ""</span><br><span style="color: hsl(120, 100%, 40%);">+ for num in range(count):</span><br><span style="color: hsl(120, 100%, 40%);">+ ret += "-"</span><br><span style="color: hsl(120, 100%, 40%);">+ return ret</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def gen_banner(self, master=False):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Generate the banner that will precede data in the target file</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ master True if this is a master-only change</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ returns The formatted banner to use above the changes</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ banner = ""</span><br><span style="color: hsl(120, 100%, 40%);">+ if master is True:</span><br><span style="color: hsl(120, 100%, 40%);">+ banner = "--- New functionality introduced in Asterisk {0} ".format(self.end_version)</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ banner = "--- Functionality changes from Asterisk {0} to Asterisk {1} ".format(self.start_version, self.end_version)</span><br><span style="color: hsl(120, 100%, 40%);">+ remaining = FUNCTIONALITY_SEPARATOR_DASH_COUNT - len(banner)</span><br><span style="color: hsl(120, 100%, 40%);">+ banner += self.gen_separator(remaining)</span><br><span style="color: hsl(120, 100%, 40%);">+ separator = self.gen_separator(FUNCTIONALITY_SEPARATOR_DASH_COUNT)</span><br><span style="color: hsl(120, 100%, 40%);">+ banner = separator + "\n" + banner + "\n" + separator + "\n\n"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ return banner</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def set_header_flag(self, line):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Set a header flag based on the line provided</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ line The header to parse</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ returns 0 on success</span><br><span style="color: hsl(120, 100%, 40%);">+ returns -1 on failure</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ data = line.split(":")</span><br><span style="color: hsl(120, 100%, 40%);">+ header = data[0].strip()</span><br><span style="color: hsl(120, 100%, 40%);">+ value = data[1].strip()</span><br><span style="color: hsl(120, 100%, 40%);">+ if header == "Master-Only":</span><br><span style="color: hsl(120, 100%, 40%);">+ if value == "True" or value == "true":</span><br><span style="color: hsl(120, 100%, 40%);">+ self.header_flags |= HeaderFlags.MASTER_ONLY.value</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGGER.error("Master-Only can only have a value of true!")</span><br><span style="color: hsl(120, 100%, 40%);">+ return -1</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def add_data(self, subject, message, timestamp):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Creates a container based on the parameters and adds the contents to</span><br><span style="color: hsl(120, 100%, 40%);">+ the correct dictionary, based on what flags were set during parsing</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ subject The subject of the change</span><br><span style="color: hsl(120, 100%, 40%);">+ message The change message</span><br><span style="color: hsl(120, 100%, 40%);">+ timestamp The file timestamp for sorting</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ container = self.Container(subject, message, timestamp)</span><br><span style="color: hsl(120, 100%, 40%);">+ if self.header_flags & HeaderFlags.MASTER_ONLY.value:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.master_data[subject].append(container)</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.data[subject].append(container)</span><br><span style="color: hsl(120, 100%, 40%);">+ # We should be done with header flags after adding an entry; reset</span><br><span style="color: hsl(120, 100%, 40%);">+ self.header_flags = 0</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def parse_file(self, staging_path, filename):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Given a path and filename, parse the contents and store the results</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ staging_path The directory the file is in</span><br><span style="color: hsl(120, 100%, 40%);">+ filename The name of the file to parse</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ returns 0 on success</span><br><span style="color: hsl(120, 100%, 40%);">+ returns -1 on failure</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ # Get the file timestamp (epoch)</span><br><span style="color: hsl(120, 100%, 40%);">+ timestamp = os.popen("(cd {0} && git log -1 --format=%ct {1})".format(staging_path, filename)).read().rstrip()</span><br><span style="color: hsl(120, 100%, 40%);">+ with open(os.path.join(staging_path, filename), 'r') as f:</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGGER.debug("Parsing file '{0}'".format(filename))</span><br><span style="color: hsl(120, 100%, 40%);">+ subjects = []</span><br><span style="color: hsl(120, 100%, 40%);">+ message = ""</span><br><span style="color: hsl(120, 100%, 40%);">+ line = f.readline()</span><br><span style="color: hsl(120, 100%, 40%);">+ while line != "\n":</span><br><span style="color: hsl(120, 100%, 40%);">+ # Get the subjects as they appear</span><br><span style="color: hsl(120, 100%, 40%);">+ if line.startswith("Subject:"):</span><br><span style="color: hsl(120, 100%, 40%);">+ subject = line.split("Subject:")[1].strip()</span><br><span style="color: hsl(120, 100%, 40%);">+ subjects.append(subject)</span><br><span style="color: hsl(120, 100%, 40%);">+ # If it's not a subject, it's a header, so set the appropriate flag</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ ret = self.set_header_flag(line)</span><br><span style="color: hsl(120, 100%, 40%);">+ if ret != 0:</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGGER.error("Failed to set header for line '{0}'".format(line.rstrip()))</span><br><span style="color: hsl(120, 100%, 40%);">+ return -1</span><br><span style="color: hsl(120, 100%, 40%);">+ line = f.readline()</span><br><span style="color: hsl(120, 100%, 40%);">+ line = f.readline()</span><br><span style="color: hsl(120, 100%, 40%);">+ message = " * " + line</span><br><span style="color: hsl(120, 100%, 40%);">+ line = f.readline()</span><br><span style="color: hsl(120, 100%, 40%);">+ while line != "":</span><br><span style="color: hsl(120, 100%, 40%);">+ if line == "\n":</span><br><span style="color: hsl(120, 100%, 40%);">+ # This is a special case because if there is an empty line separating some</span><br><span style="color: hsl(120, 100%, 40%);">+ # text, we don't want to add 3 spaces and a new line.</span><br><span style="color: hsl(120, 100%, 40%);">+ message += "\n"</span><br><span style="color: hsl(120, 100%, 40%);">+ else:</span><br><span style="color: hsl(120, 100%, 40%);">+ message += " " + line</span><br><span style="color: hsl(120, 100%, 40%);">+ line = f.readline()</span><br><span style="color: hsl(120, 100%, 40%);">+ for sub in subjects:</span><br><span style="color: hsl(120, 100%, 40%);">+ self.add_data(sub, message, timestamp)</span><br><span style="color: hsl(120, 100%, 40%);">+ # Once we are done with the file, remove it from the staging directory.</span><br><span style="color: hsl(120, 100%, 40%);">+ # We don't need it anymore.</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGGER.debug("Removing file {0}".format(filename))</span><br><span style="color: hsl(120, 100%, 40%);">+ os.remove(os.path.join(staging_path, filename))</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def get_staging_changes(self, mode):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Retrieve changes from <self.ast_path>/doc/<mode>-staging/</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ returns 0 on success</span><br><span style="color: hsl(120, 100%, 40%);">+ returns -1 on failure</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ if mode is not "CHANGES" and mode is not "UPGRADE":</span><br><span style="color: hsl(120, 100%, 40%);">+ return -1</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ staging_path = "{0}/doc/{1}-staging".format(self.ast_path, mode)</span><br><span style="color: hsl(120, 100%, 40%);">+ try:</span><br><span style="color: hsl(120, 100%, 40%);">+ for filename in os.listdir(staging_path):</span><br><span style="color: hsl(120, 100%, 40%);">+ # Only process text files</span><br><span style="color: hsl(120, 100%, 40%);">+ if not os.path.isfile(staging_path + "/" + filename) or not filename.endswith(".txt"):</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ ret = self.parse_file(staging_path, filename)</span><br><span style="color: hsl(120, 100%, 40%);">+ if ret != 0:</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGGER.error("Failed during parsing of file '{0}'".format(filename))</span><br><span style="color: hsl(120, 100%, 40%);">+ return -1</span><br><span style="color: hsl(120, 100%, 40%);">+ except:</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGGER.debug("Could not extract data from {0} (does it exist?)".format(staging_path))</span><br><span style="color: hsl(120, 100%, 40%);">+ return -1</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def add_staging_changes(self, mode):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Add the changes retrieved to <self.ast_path/<mode>(.txt)</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ returns 0 on success</span><br><span style="color: hsl(120, 100%, 40%);">+ returns -1 on failure</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ if mode is not "CHANGES" and mode is not "UPGRADE":</span><br><span style="color: hsl(120, 100%, 40%);">+ return -1</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ if bool(self.data) is False and bool(self.master_data) is False:</span><br><span style="color: hsl(120, 100%, 40%);">+ # There was no data to retrieve during get_staging_changes</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # If this is the master branch, we put the new stuff first</span><br><span style="color: hsl(120, 100%, 40%);">+ text_to_insert = ""</span><br><span style="color: hsl(120, 100%, 40%);">+ if bool(self.master_data) is True:</span><br><span style="color: hsl(120, 100%, 40%);">+ text_to_insert += self.gen_banner(master=True)</span><br><span style="color: hsl(120, 100%, 40%);">+ for key in sorted(self.master_data):</span><br><span style="color: hsl(120, 100%, 40%);">+ text_to_insert += key + "\n" + self.gen_separator(CATEGORY_SEPARATOR_DASH_COUNT) + "\n"</span><br><span style="color: hsl(120, 100%, 40%);">+ for obj in sorted(self.master_data[key], key=self.Container.by_timestamp):</span><br><span style="color: hsl(120, 100%, 40%);">+ text_to_insert += obj.message.rstrip() + "\n\n"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Insert the changes for the new version</span><br><span style="color: hsl(120, 100%, 40%);">+ if bool(self.data) is True:</span><br><span style="color: hsl(120, 100%, 40%);">+ text_to_insert += self.gen_banner()</span><br><span style="color: hsl(120, 100%, 40%);">+ for key in sorted(self.data):</span><br><span style="color: hsl(120, 100%, 40%);">+ text_to_insert += key + "\n" + self.gen_separator(CATEGORY_SEPARATOR_DASH_COUNT) + "\n"</span><br><span style="color: hsl(120, 100%, 40%);">+ for obj in sorted(self.data[key], key=self.Container.by_timestamp):</span><br><span style="color: hsl(120, 100%, 40%);">+ text_to_insert += obj.message.rstrip() + "\n\n"</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ file_path = "{0}/{1}".format(self.ast_path, mode)</span><br><span style="color: hsl(120, 100%, 40%);">+ if mode is "UPGRADE":</span><br><span style="color: hsl(120, 100%, 40%);">+ file_path += ".txt"</span><br><span style="color: hsl(120, 100%, 40%);">+ try:</span><br><span style="color: hsl(120, 100%, 40%);">+ # insert will be set to "1" when we reach the first new line in file</span><br><span style="color: hsl(120, 100%, 40%);">+ insert = 0</span><br><span style="color: hsl(120, 100%, 40%);">+ # This is solely used to insert in place</span><br><span style="color: hsl(120, 100%, 40%);">+ for line in fileinput.FileInput(file_path, inplace = 1):</span><br><span style="color: hsl(120, 100%, 40%);">+ if insert is 0 and line in ["\n", "\r\n"]:</span><br><span style="color: hsl(120, 100%, 40%);">+ # First new line, this is where we actually want to insert data</span><br><span style="color: hsl(120, 100%, 40%);">+ insert = 1</span><br><span style="color: hsl(120, 100%, 40%);">+ continue</span><br><span style="color: hsl(120, 100%, 40%);">+ if insert is 1:</span><br><span style="color: hsl(120, 100%, 40%);">+ # We insert here, and then just print every line back to the file</span><br><span style="color: hsl(120, 100%, 40%);">+ line = line.replace(line, "\n" + text_to_insert + line)</span><br><span style="color: hsl(120, 100%, 40%);">+ insert = 2</span><br><span style="color: hsl(120, 100%, 40%);">+ print line,</span><br><span style="color: hsl(120, 100%, 40%);">+ except:</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGGER.debug("Could not add data to {0}".format(file_path))</span><br><span style="color: hsl(120, 100%, 40%);">+ return -1</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ def run(self):</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+ Put everything together and run it</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ returns 0 on success</span><br><span style="color: hsl(120, 100%, 40%);">+ returns -1 on failure</span><br><span style="color: hsl(120, 100%, 40%);">+ '''</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ logging.basicConfig(level=logging.DEBUG, format="%(module)s:%(lineno)d - %(message)s")</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ for mode in MODES:</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGGER.debug("Extracting changes for {0}".format(mode))</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ # Clear any data we fetched previously before we do anything</span><br><span style="color: hsl(120, 100%, 40%);">+ self.data.clear()</span><br><span style="color: hsl(120, 100%, 40%);">+ self.master_data.clear()</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ ret = self.get_staging_changes(mode)</span><br><span style="color: hsl(120, 100%, 40%);">+ if ret is not 0:</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGGER.debug("Failed during {0} get_staging_changes step - aborting "</span><br><span style="color: hsl(120, 100%, 40%);">+ "(working directory probably needs to be reset!!!)".format(mode))</span><br><span style="color: hsl(120, 100%, 40%);">+ return -1</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ ret = self.add_staging_changes(mode)</span><br><span style="color: hsl(120, 100%, 40%);">+ if ret is not 0:</span><br><span style="color: hsl(120, 100%, 40%);">+ LOGGER.debug("Failed during {0} add_staging_changes step - aborting "</span><br><span style="color: hsl(120, 100%, 40%);">+ "(working directory probably needs to be reset!!!)".format(mode))</span><br><span style="color: hsl(120, 100%, 40%);">+ return -1</span><br><span style="color: hsl(120, 100%, 40%);">+</span><br><span style="color: hsl(120, 100%, 40%);">+ return 0</span><br><span></span><br></pre><p>To view, visit <a href="https://gerrit.asterisk.org/c/repotools/+/10941">change 10941</a>. To unsubscribe, or for help writing mail filters, visit <a href="https://gerrit.asterisk.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.asterisk.org/c/repotools/+/10941"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: repotools </div>
<div style="display:none"> Gerrit-Branch: master </div>
<div style="display:none"> Gerrit-Change-Id: I6dc084afedaeecaf36aaec66c3cf6a5a8ed4ef3c </div>
<div style="display:none"> Gerrit-Change-Number: 10941 </div>
<div style="display:none"> Gerrit-PatchSet: 9 </div>
<div style="display:none"> Gerrit-Owner: Benjamin Keith Ford <bford@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Benjamin Keith Ford <bford@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: George Joseph <gjoseph@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Joshua Colp <jcolp@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Kevin Harwell <kharwell@digium.com> </div>
<div style="display:none"> Gerrit-MessageType: merged </div>