[Asterisk-code-review] mkrelease: Add a Python script that creates tarballs for Ast... (repotools[master])
Matt Jordan
asteriskteam at digium.com
Wed May 6 13:44:31 CDT 2015
Matt Jordan has uploaded a new change for review.
https://gerrit.asterisk.org/383
Change subject: mkrelease: Add a Python script that creates tarballs for Asterisk
......................................................................
mkrelease: Add a Python script that creates tarballs for Asterisk
This patch adds a Python script that generates the artifacts for an
Asterisk release. This includes:
* Manipulating a local Git repository, creating the necessary branches,
tags, and pushing those changes to the upstream repo when
appropriate.
* Generating the release summaries, ChangeLog, .version file, and other
necessary files for a release.
* Creating the following archives:
- A diff archive containing the diff of the previous release with
this newly created release.
- An archive creating the newly created tag, as well as documentation
downloaded from wiki.asterisk.org, and sounds downloaded from
downloads.asterisk.org.
* Signing the archives and generating sha1, sha256, and md5 hash sums.
Note that there are a few reasons why this script was rewritten in
Python:
(1) The previous mkrelease script had a very arcane invocation. Since
the migration to Git necessitated managing the tag/branch creation
process more closely, it also meant that we had an opportunity to
simplify invocation of the script. This means some extra Asterisk
Version parsing had to occur. Once we had to add a lot of string
manipulation into the mix, Python felt like a more natural fit for
this script than bash.
(2) While Git has a natural ability to generate tarballs from a checked
out repo, much like the Asterisk ChangeLog and Release Summaries,
the Asterisk project has some extra "stuff" that it needs to do
that go beyond Git's capabilities. In this case, that included
downloading documentation, adding sound files, etc. Combined with
the version manipulation requirements, and given Python's plethora
of libraries that aid in the manipulation of files, GPG keys, and
other systems, Python felt like a natural choice.
REP-15 #close
Change-Id: Ibab7643d501085bcb2aad13915d4b0f6b24cded4
---
A mkrelease.py
1 file changed, 607 insertions(+), 0 deletions(-)
git pull ssh://gerrit.asterisk.org:29418/repotools refs/changes/83/383/1
diff --git a/mkrelease.py b/mkrelease.py
new file mode 100755
index 0000000..b3764c7
--- /dev/null
+++ b/mkrelease.py
@@ -0,0 +1,607 @@
+#!/usr/bin/env python
+"""Asterisk project make release script
+
+Lovingly extracted and converted to Python from the original mkrelease script
+
+Matt Jordan <mjordan at digium.com>
+"""
+
+import sys
+import os
+import copy
+import fnmatch
+import shutil
+import tarfile
+import gnupg
+
+from datetime import datetime
+from optparse import OptionParser
+
+from digium_git import DigiumGitRepo
+from version_parser import AsteriskVersion
+from release_summary import ReleaseSummary, ReleaseSummaryOptions
+from alembic_creator import create_db_script
+
+# The one and only Gerrit/Git server.
+GERRIT = 'ssh://gerrit.asterisk.org:29418'
+
+# The previous tag from this release.
+previous_tag = ''
+
+# String representation of the version to release.
+version = ''
+
+# String representation of the previously released version,
+prev_version = ''
+
+# String representation of the tag we should start history at.
+start_version = ''
+
+# An AsteriskVersion object representing the version to release.
+version_object = None
+
+# The full name of the release.
+release_name = ''
+
+# Whether or not we should prompt as we go along.
+interactive = False
+
+# Whether or not we should display debug information.
+debug = False
+
+
+class ExitException(Exception):
+ """Exception raised by prompt_to_continue to stop the script"""
+ pass
+
+
+def dprint(string):
+ """Print out a debug statement
+
+ Keyword Arguments:
+ string - The string to print
+ """
+ if debug:
+ print " > {0}".format(string)
+
+
+def prompt_to_continue(prompt=None):
+ """Prompt the user to see if they want to continue
+
+ Note that if options.interactive is False, this is a NoOp. If the
+ user selects No, an ExitException is raised.
+
+ Keyword Arguments:
+ prompt - The prompt to display. Can be None.
+ """
+ if not interactive:
+ return
+
+ if not prompt:
+ prompt = ''
+ prompt = "{0}\n\tContinue [yes/no]? ".format(prompt)
+
+ cont = raw_input(prompt)
+ if 'yes' in cont.lower() or 'y' in cont.lower():
+ return
+ raise ExitException()
+
+
+def setup_options(options):
+ """Set up common options based on the command line parameters
+
+ Keyword Arguments:
+ options - The parsed command line options
+ """
+ global version
+ global version_object
+ global release_name
+ global interactive
+ global debug
+
+ version = options.version
+ version_object = AsteriskVersion.create_from_string(version)
+ release_name = '{0}-{1}'.format(options.project, version.replace('/', '-'))
+ interactive = options.interactive
+ debug = options.debug
+
+
+def prepare_repo(options):
+ """Prepare the repo that the release will be made from
+
+ Keyword Arguments:
+ options - Parsed command line arguments
+
+ Returns:
+ A DigiumGitRepo object
+ """
+
+ path = os.path.join(options.local_root, options.project)
+ repo_url = '{0}/{1}'.format(GERRIT, options.project)
+ dprint("Cloning from '{0}' to '{1}'".format(repo_url, path))
+ repo = DigiumGitRepo(path, repo_url=repo_url, show_progress=debug)
+ return repo
+
+
+def prepare_branch(repo):
+ """Prepare the branch that the release will be made from
+
+ Keyword Arguments:
+ options - Parsed command line arguments
+ repo - The prepared repo
+ """
+
+ mainline = version[:version.index('.')]
+ branch = version[:version.rindex('.')]
+ if repo.remote_branch_exists(branch):
+ dprint("Remote branch {0} exists already".format(branch))
+ else:
+ dprint("Remote branch {0} does not exist".format(branch))
+ prompt_to_continue()
+ repo.create_remote_branch(branch, tracking=mainline)
+ repo.push_changes()
+ dprint("Remote branch {0} (tracking {1}) created.".format(
+ branch, mainline))
+ repo.checkout_remote_branch(branch)
+ dprint("Local branch set to {0}".format(branch))
+ return
+
+
+def extract_tags(options, repo):
+ """Extract the tags from the version to be created and the prepared repo
+
+ Keyword Arguments:
+ options - Parsed command line arguemnts
+ repo - The prepared repo
+
+ Returns:
+ A tuple of:
+ - The start tag
+ - The previous release tag
+ """
+ global prev_version
+ global start_version
+
+ previous = version_object.get_previous_version()
+ if len(version_object.modifiers) == 0 and version_object.patch == 0:
+ # If there are no modifiers and this is the first full release in a
+ # series, find the last modified version (RC, beta, etc.)
+ search = True
+ i = 1
+ while search:
+ if repo.get_tag('{0}-rc{1}'.format(version, i)):
+ start_tag = AsteriskVersion.create_from_string(
+ '{0}-rc{1}'.format(str(version), i))
+ elif repo.get_tag('{0}-beta{1}'.format(version, i)):
+ start_tag = AsteriskVersion.create_from_string(
+ '{0}-beta{1}'.format(version, i))
+ else:
+ search = False
+ i += 1
+ elif (len(version_object.modifiers) > 0 and
+ all([num == 1 for mod, num in version_object.modifiers])):
+ # If all modifiers are the first of their kind, then our start tag
+ # should be the first RC of the previous version
+ start_tag = copy.deepcopy(previous)
+ start_tag.modifiers.append(('rc', 1))
+ else:
+ start_tag = previous
+
+ prev_version = str(previous)
+ start_version = str(start_tag)
+
+ # Override with options if they were provided
+ if options.start_tag:
+ start_version = options.start_tag
+ if options.previous_tag:
+ prev_version = options.previous_tag
+
+
+def create_tag(options, repo):
+ """Create the tag
+
+ In addition to creating a new tag, this will also:
+ * Create database scripts
+ * Update the .lastclean file
+ * Update the .version file
+ * Remove old summaries and create new ones
+ * Update the ChangeLog
+
+ Keyword Arguments:
+ options - The parsed command line parameters
+ repo - Our local Git repo
+ """
+
+ def create_change_log(summary):
+ """Create the change log for the release
+
+ Keyword Arguments:
+ summary - The release summary object
+ """
+
+ change_log_path = os.path.join(options.local_root, options.project,
+ 'ChangeLog')
+ old_log_path = os.path.join(options.local_root, options.project,
+ 'OldLog')
+
+ if os.path.isfile(change_log_path):
+ dprint("Removing old ChangeLog")
+ os.unlink(change_log_path)
+
+ # Check out the start tag and copy over its ChangeLog
+ dprint("Getting previous change log from '{0}'".format(start_version))
+ repo.switch_local_branch(start_version)
+ shutil.copyfile(change_log_path, old_log_path)
+ dprint("Copied '{0}' to '{1}'".format(change_log_path, old_log_path))
+
+ # Switch back to our branch
+ repo.switch_local_branch(version[:version.rindex('.')])
+
+ # Actually create the new ChangeLog
+ with open(change_log_path, 'w') as c_file:
+ today = datetime.utcnow().strftime('%Y-%m-%d %H:%M +0000')
+ c_file.write("{0} Asterisk Development Team "
+ "<asteriskteam at digium.com>\n\n".format(today))
+ c_file.write("\t* {0} {1} Released.\n\n".format(options.project,
+ version))
+ for log_message in summary.raw_log_messages:
+ c_file.write(log_message.get_change_log())
+
+ with open(old_log_path, 'r') as old_file:
+ for line in old_file:
+ c_file.write(line)
+
+ dprint("Removing temporary OldLog file")
+ os.unlink(old_log_path)
+
+ dprint("Commiting changes to ChangeLog")
+ repo.add_and_commit([change_log_path],
+ "ChangeLog: Updated for {0}\n".format(version))
+ return
+
+ def create_release_summaries():
+ """Create new release summaries
+
+ Returns:
+ A release summary object
+ """
+
+ file_dir = os.path.join(options.local_root, options.project)
+
+ # Remove any old summaries in the directory
+ removed = False
+ for f_name in os.listdir(file_dir):
+ project = options.project
+ if fnmatch.fnmatch(f_name, '{0}-*-summary.*'.format(project)):
+ dprint("Removing old summary: '{0}'".format(f_name))
+ f_path = os.path.join(file_dir, f_name)
+ os.unlink(f_path)
+ repo.repo.git.rm(f_path)
+ removed = True
+ if removed:
+ msg = "Release summaries: Remove previous versions\n"
+ repo.repo.git.commit(message=msg)
+
+ sum_opts = ReleaseSummaryOptions()
+ sum_opts.start_tag = start_version
+ sum_opts.end_tag = 'HEAD'
+ sum_opts.previous_version = prev_version
+ sum_opts.version = version
+ sum_opts.local_root = options.local_root
+ sum_opts.project = options.project
+
+ if len(options.advisories) != 0:
+ sum_opts.release_type = ReleaseSummaryOptions.RELEASE_TYPE_SECURITY
+ sum_opts.advisories = list(options.advisories)
+ elif version_object.minor == 0 and version_object.patch == 0:
+ # New feature release of a major branch
+ sum_opts.release_type = ReleaseSummaryOptions.RELEASE_TYPE_FEATURE
+
+ file_name = '{0}-summary'.format(release_name)
+ file_path = os.path.join(file_dir, file_name)
+
+ summary = ReleaseSummary(sum_opts, debug=debug)
+ summary.to_html(out_file=file_path)
+ dprint("Release summaries created as '{0}'".format(file_path))
+
+ msg = "Release summaries: Add summaries for {0}\n".format(version)
+ repo.add_and_commit(['{0}.html'.format(file_path),
+ '{0}.txt'.format(file_path)],
+ msg)
+ return summary
+
+ def create_version_file():
+ """Create the .version file"""
+ ver_file_path = os.path.join(options.local_root,
+ options.project, '.version')
+ with open(ver_file_path, 'w') as ver_file:
+ ver_file.write(version)
+ dprint(".version file written as '{0}'".format(version))
+ repo.add_and_commit([ver_file_path],
+ ".version: Update for {0}\n".format(version))
+ return
+
+ def create_lastclean():
+ """Create the .lastclean file"""
+ src_file = os.path.join(options.local_root,
+ options.project,
+ '.cleancount')
+ dst_file = os.path.join(options.local_root,
+ options.project,
+ '.lastclean')
+ shutil.copyfile(src_file, dst_file)
+ repo.add_and_commit([dst_file],
+ ".lastclean: Update for {0}\n".format(version))
+ return
+
+ def create_database_scripts():
+ """Create the Alembic DB scripts"""
+ if version_object.major < 12:
+ return
+
+ curdir = os.getcwd()
+ work_path = os.path.join(options.local_root,
+ options.project,
+ 'contrib/ast-db-manage')
+ out_path = os.path.join(options.local_root,
+ options.project,
+ 'contrib/realtime')
+ os.chdir(work_path)
+ generated_files = []
+ for db in ['mysql', 'oracle', 'postgresql', 'mssql']:
+
+ out_db_dir = os.path.join(out_path, db)
+ if not os.path.exists(out_db_dir):
+ os.makedirs(out_db_dir)
+
+ for script in ['config', 'voicemail', 'cdr']:
+ create_db_script(database=db,
+ script=script,
+ output_location=out_db_dir)
+ generated_files.append(
+ os.path.join(out_db_dir, "{0}_{1}.sql".format(db, script)))
+
+ msg = "realtime: Add database scripts for {0}\n".format(version)
+ repo.add_and_commit(generated_files, msg)
+ os.chdir(curdir)
+ return
+
+ create_database_scripts()
+ create_lastclean()
+ create_version_file()
+ summary = create_release_summaries()
+ create_change_log(summary)
+
+ prompt_to_continue("Confirm tag creation of '{0}'".format(version))
+ repo.push_changes()
+ repo.create_tag(version)
+
+
+def create_hashes(archive, mod=None):
+ """Create hashes for an archive
+
+ Keyword Arguments:
+ archive - The archive to create hashes for
+ mod - Modifier to append to the output name. Optional.
+ """
+ full_name = release_name
+ if mod:
+ full_name = '{0}-{1}'.format(full_name, mod)
+ os.system("sha1sum {0} > {1}.sha1".format(archive, full_name))
+ os.system("sha256sum {0} > {1}.sha256".format(archive, full_name))
+ os.system("md5sum {0} > {1}.md5".format(archive, full_name))
+
+
+def sign_archive(file_name, out_file):
+ """Sign an archive
+
+ Keyword Arguments:
+ file_name - The file that we should sign.
+ out_file - The .asc file we should write to.
+ """
+ with open(file_name, 'r') as file_stream:
+ gpg = gnupg.GPG()
+ gpg.sign_file(file_stream, detach=True, output=out_file)
+
+
+def create_patch_archive(options):
+ """Create an archive of the diff between this release and the previous
+
+ Keyword Arguments:
+ options - The parsed command line parameters
+ """
+ prev_obj = version_object.get_previous_version()
+ if prev_obj.major != version_object.major:
+ # Different major version. This means this is a *big* jump that
+ # can't be expressed in a diff. Bail.
+ dprint("This version '{0}' is a new major version; bypassing "
+ "diff".format(version))
+ return
+ # Since we don't want to work with the diff, but merely dump it out,
+ # this is more easily done via the command line
+ curdir = os.getcwd()
+ os.chdir(os.path.join(options.local_root, options.project))
+ file_name = '{0}-patch'.format(release_name)
+ dprint("Generating diff from '{0}' to '{1}'".format(start_version,
+ version))
+ os.system('git diff {0}..{1} > {2}'.format(start_version,
+ version,
+ file_name))
+ os.chdir(curdir)
+ tmp_diff_file = os.path.join(options.local_root,
+ options.project,
+ file_name)
+ diff_file = os.path.join(os.getcwd(), file_name)
+ shutil.copyfile(tmp_diff_file, diff_file)
+ out_file = '{0}.tar.gz'.format(diff_file)
+ with tarfile.open(name=out_file, mode='w:gz') as tar_obj:
+ tar_obj.add(file_name)
+ sign_archive(out_file, '{0}.asc'.format(diff_file))
+
+ dprint("Created diff file '{0}'".format(out_file))
+ create_hashes(out_file, mod='patch')
+
+ os.unlink(diff_file)
+ os.unlink(tmp_diff_file)
+
+
+def create_tag_archive(repo):
+ """Create an archive from the tag
+
+ Keyword Arguments:
+ repo - Our local Git repo
+
+ Returns:
+ The file path to the archive created
+ """
+ file_name = '{0}-staging.tar.gz'.format(release_name)
+ out_file = os.path.join(os.curdir, file_name)
+ dprint("Creating staging archive '{0}'".format(out_file))
+ with open(out_file, 'w') as fs:
+ repo.repo.archive(fs, treeish=version, format='tar.gz')
+ return out_file
+
+
+def create_final_archive(archive):
+ """Take the tag archive and create a final release archive from it
+
+ Keyword Arguments:
+ archive - The full path to the tagged archive
+ """
+
+ def extract_archive():
+ """Extract the specified archive."""
+ dprint("Extract '{0}' to '{1}'".format(archive, release_name))
+ with tarfile.open(name=archive, mode='r:gz') as tar_obj:
+ tar_obj.extractall(path=release_name)
+
+ def pull_meta_files(path):
+ """Pull the various informational files out
+
+ Keyword Arguments:
+ path - The location of the extracted Asterisk release
+ """
+ shutil.copyfile(os.path.join(path, 'README'),
+ 'README-{0}'.format(version))
+ shutil.copyfile(os.path.join(path, 'ChangeLog'),
+ 'ChangeLog-{0}'.format(version))
+ summary_prefix = '{0}-summary'.format(release_name)
+ shutil.copyfile(os.path.join(path, '{0}.html'.format(summary_prefix)),
+ '{0}.html'.format(summary_prefix))
+ shutil.copyfile(os.path.join(path, '{0}.txt'.format(summary_prefix)),
+ '{0}.txt'.format(summary_prefix))
+
+ def exec_prep_tarball(path):
+ """Execute the prep_tarball script
+
+ Keyword Arguments:
+ path - The location of the extracted Asterisk release
+ """
+ branch = version[:version.index('.')]
+ curdir = os.getcwd()
+ os.chdir(path)
+ dprint("Running build_tools/prep_tarball")
+ os.system('./build_tools/prep_tarball {0}'.format(branch))
+ os.chdir(curdir)
+
+ def create_archive(path):
+ """Create the final archive
+
+ Keyword Arguments:
+ path - The location of the extracted prep Asterisk release
+ """
+ name = '{0}.tar.gz'.format(release_name)
+ dprint("Creating final archive '{0}'".format(name))
+ with tarfile.open(name=name, mode='w:gz') as tar_obj:
+ tar_obj.add(path)
+ sign_archive(name, '{0}.asc'.format(release_name))
+
+ return name
+
+ extract_archive()
+ pull_meta_files(release_name)
+ exec_prep_tarball(release_name)
+ final_archive = create_archive(release_name)
+ create_hashes(final_archive)
+
+ dprint("Cleaning up '{0}' and '{1}'".format(archive, release_name))
+ os.unlink(archive)
+ shutil.rmtree(release_name)
+ return
+
+
+def main(argv=None):
+ """Main entry point
+
+ Keyword Arguments:
+ argv - Command line parameters
+ """
+
+ if not argv:
+ argv = sys.argv
+
+ parser = OptionParser()
+ parser.add_option("-a", "--advisory", action="append", default=list(),
+ dest="advisories",
+ help="A security advisory to advertise.")
+ parser.add_option("-d", "--debug", action="store_true", default=False,
+ dest="debug", help="Provide debug detail")
+ parser.add_option("-i", "--interactive", action="store_true",
+ default=False, dest="interactive",
+ help="Provide interactive prompts")
+ parser.add_option("-l", "--local-root", action="store", type="string",
+ dest="local_root", help="The local root to work from",
+ default="/tmp")
+ parser.add_option("-p", "--project", action="store", type="string",
+ dest="project", help="The project to work from",
+ default="asterisk")
+ parser.add_option("-P", "--previous-tag", action="store", type="string",
+ dest="previous_tag", default=None,
+ help="Override the previous version tag")
+ parser.add_option("-s", "--start-tag", action="store", type="string",
+ dest="start_tag", default=None,
+ help="Override the tag to start history at")
+ parser.add_option("-v", "--version", action="store", type="string",
+ dest="version",
+ help="The version in the project to create")
+
+ (options, args) = parser.parse_args(argv)
+
+ if not options.version:
+ parser.error("A version is required.")
+
+ if options.interactive and not options.debug:
+ print "Interactive is True, setting 'debug' to True"
+ options.debug = True
+
+ # The following are all various set up steps that extract options, prepare
+ # the environment, and calculate what it is we are trying to create.
+ setup_options(options)
+ repo = prepare_repo(options)
+ prepare_branch(repo)
+ extract_tags(options, repo)
+
+ if repo.get_tag(version):
+ dprint("Tag '{0}' exists; bypassing tag creation".format(version))
+ else:
+ prompt_to_continue("Creating tag '{0}', starting history at '{1}', "
+ "with previous version {2}.".format(version,
+ start_version,
+ prev_version))
+ create_tag(options, repo)
+
+ prompt_to_continue("Tag created, proceeding to tarball.")
+ create_patch_archive(options)
+ tag_archive = create_tag_archive(repo)
+ create_final_archive(tag_archive)
+
+ print "Congratulations on the successful creation of {0} {1}!".format(
+ options.project, options.version)
+
+
+if __name__ == '__main__':
+ try:
+ sys.exit(main() or 0)
+ except ExitException:
+ print "User opted to exit"
+ sys.exit(1)
--
To view, visit https://gerrit.asterisk.org/383
To unsubscribe, visit https://gerrit.asterisk.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibab7643d501085bcb2aad13915d4b0f6b24cded4
Gerrit-PatchSet: 1
Gerrit-Project: repotools
Gerrit-Branch: master
Gerrit-Owner: Matt Jordan <mjordan at digium.com>
More information about the asterisk-code-review
mailing list