[asterisk-dev] Git Migration

[Russell Bryant]

On Tue, Sep 16, 2014 at 10:52 PM, Matthew Jordan <mjordan at digium.com> wrote:

>
>
> On Tue, Sep 16, 2014 at 5:01 PM, Russell Bryant <russell at russellbryant.net
> > wrote:
>
>> On Tue, Sep 16, 2014 at 3:48 PM, Matthew Jordan <mjordan at digium.com>
>> wrote:
>>
> So, to set what I hope are a few guidelines:
>>>
>> (1) I know this is a subject with a lot of opinions, and coming to a
> concurrence on something that is opinion based is tough. Voice opinions
> after thinking, "would I be willing to spend time working on this"?
> (2) This is a project that can easily suck several developers for a long
> time if the scope of it expands substantially. The scope should hopefully
> be kept as tight as possible, as we will have a lot of re-tooling to do
> once the git move occurs (all CI, code review, releasing, etc.)
> (3) The Asterisk project has several hard requirements, a few softer ones,
> that have to be met:
>   * Code submitted has to be assigned to a CLA. We have to verify that
> people proposing patches have acknowledged that they were licensed to
> submit said patch in some fashion.
>   * Issue tracking has to be done in JIRA (large investment, huge cost in
> moving to anything else right now).
>   * Code must be reviewed.
>   * Any substitute for Review Board, Bamboo, or other existing tools
> should have obvious benefits over the existing tools.
>

Very nice set of guidelines.


>
>> For the last few years, most of my time has been going into OpenStack
>> [1].  We use git and I have become a big fan of our workflow and
>> infrastructure.  It's all open source and reusable.
>>
>> From a high level, all patches go to a code review system.  *Every* patch
>> must be peer reviewed (usually by 2 people, but that's a policy decision).
>>  *Every* patch must also pass tests.  Once a patch passes both tests and
>> peer review, it is automatically merged into the repository.
>>
>> I *love* that workflow for several reasons.  If it's appealing, it's
>> probably much easier to do it now while you're doing a big switch anyway.
>>  If you're not sold, I'm certainly not hurt.  Like I said, I just wanted to
>> offer info.  The current plan will be less up front setup for sure.
>>
>
> I am a huge fan of this.
>
> We've been moving (slowly but surely) towards this model. Code is reviewed
> more often than not, and with Crowd integration, anyone can propose a
> patch. Tests are written for the vast majority of new work. We *do* still
> have a problem with bouncing tests - and while Bamboo may point the finger
> more at 12+ currently (due to a current spat with native RTP bridges,
> direct media, and transfers) - the bouncing is actually more pernicious in
> many ways with the 1.8/11 branches. We'd have to solve that problem if we
> move to this model.
>
> I don't consider that a bad thing at all.
>

OpenStack struggles with this problem, as well.  It can be incredibly
painful.  If a test is just failing some of the time, you get developers
confused and frustrated that seemingly unrelated failures are blocking
their patch.  Worse, you may get a mess of a backlog in the CI system as
things have to get re-run.  We keep a dashboard of current bugs being hit
sorted by frequency so we know what the biggest offenders are.
http://status.openstack.org/elastic-recheck/  (Note that running this is
another set of infrastructure that I didn't mention, but it's interesting
to see I think.)

There are things you can do to mitigate the impact while working to improve
things over time.  You can mark some CI jobs as "non-voting", meaning that
the job runs and you see the results on the patch, but it doesn't affect
the overall pass/fail result.  If you split up tests into some subsets, you
can have some jobs voting and some non-voting.


>> If you're a hands on kind of person, browse http://review.openstack.org/
>> for open code reviews.  You can also see patches going through CI pipelines
>> on http://status.openstack.org/zuul//
>>
>> The major tools involved are:
>>
>>  - gerrit for code review and repository management [2][3]
>>  - jenkins for CI [4]
>>  - Zuul, A CI job scheduler that automates running things in response to
>> events on gerrit. [5]
>>  - CGit, repo hosting [6][7]
>>
>>
> This would replace Review Board and Bamboo as well, so we'll need to
> consider the effort involved with that.
>

Yes, for sure.  I think it's worthwhile long term, and it's at least
largely automated and proven, so that helps.


>
>
>> Everything we use is managed via puppet and all of the configuration is
>> in git.  It's designed to be reusable.  The folks that run it have
>> documented how to re-use it [8] and are quite friendly.  You can find them
>> in #openstack-infra on freenode.
>>
>>
> Awesome. I'll do that - in particular, I'm interested what is involved in
> the recurring maintenance of gerrit/Jenkins/Zuul.
>
>

On IRC, jeblair is the infrastructure project lead.  He's also an Asterisk
fan, so that helps.  :-)

Depending on the depth of the question(s), it may be better to use the
openstack-infra mailing list:

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra


> -- Git hooks
>>
>> For what, exactly?  It's probably easier to discuss the problem that
>> needs to be solved.
>>
>
> In general, there are a few use cases for either git hooks or web hooks
> (or something else, such as the Gerrit event stream):
> (1) Auto-close issues in JIRA
>

It looks like there is a git plugin that supports smart commits.

https://marketplace.atlassian.com/plugins/com.xiplink.jira.git.jira_git_plugin


> (2) Auto-close reviews in Review Board (but clearly not an issue in this
> case)
> (3) Prevent pushes from private branches going to public branches. This
> would have to be done a pre-hook in some fashion.
>
> I think there are various ways to achieve all three of these, although
> admittedly the 'private branch' one is the most tricky and annoying.
>

Yeah, that is kind of tricky and annoying.  :-)

It does seem like a git hook could provide the type of validation done
today on the svn server side, though.  As long as the branch is marked as
"private" in some way (the remote branch it's tracking, some special file,
...), you could validate the push location, I suppose.


>> -- Web hooks
>>
>> Again, it's probably worth discussing the use case.  Gerrit has an event
>> stream.  That event stream includes merges.  Tools to do things in
>> responses to merges (which could be running a web hook) listen and react.
>>
>> -- Performance
>>
>> Yes.  :-)
>>
>> There's lots of stats I could dig up, but as one example, I track code
>> review stats for a subset of projects on review.openstack.org  In the
>> last year, for that subset, there have been 266,127 code reviews done (avg
>> > 700 per day).  That gives some sense of scale.  From a quick glance at
>> the CI status page (http://status.openstack.org/zuul/), it has been
>> launching about 500-600 jobs per hour today.
>>
>> -- Process Recommendation
>>
>> I discussed this a good bit above, but I'm happy to answer questions.
>>
>
> Wiki page updated!
>
> You make a lot of good points, and I think it's worth honestly exploring
> this option. It does look like it would be more work, but as you pointed
> out, now is the time to do it - and it may not be substantially more than
> what we're already going to have to do. We'll have to take a deeper look at
> Gerrit and find out.
>
> Thanks for weighing in on this!
>

You're welcome!  If nothing else, it's interesting to explore the different
approaches projects are taking for inspiration.  No matter what you end up
with, git <3.

-- 
Russell Bryant
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20140917/16b3df84/attachment.html>