Ceph » Stable releases

h3. Backport commits

h4. General guidelines

* backports should be done by cherry-picking commits that are in master, and in the most minimal way possible - see [[HOWTO_review_backport_PRs]] for more information

* it should be clear which bug the backport is fixing

* features can be backported only with permission from the relevant lead

* the tracker issue should state why the backport is necessary

h4. Detailed instructions:

* Pick one entry with the highest *Priority* field in the list of backports linked from [[HOWTO|the top level page]] (for instance "hammer backports":http://tracker.ceph.com/projects/ceph/issues?query_id=78)

* Assign the issue to yourself (or add a comment if you do not have permission) so that people know you start working on it.

* <code>git clone</code> your fork of http://github.com/ceph/ceph (for example, if your GitHub username is "foo", do: <code>git clone https://github.com/foo/ceph.git ; cd ceph</code>

* if you already have a clone of your fork, use <code>git remote -v</code> to determine the remote name. In the text that follows, we'll assume the remote name of your fork is <code>origin</code>, but that might not be the case on your system - please check

* <code>git checkout -b wip-$issue-$release origin/$release</code> to create a branch with a name that reflects the issue being fixed and the target release: this naming is convenient to select all branches containing backports for firefly or select all branches containing backports for a specific issue

* <code>git cherry-pick -x</code> the commits from the pull request that was targeting <code>master</code> (not from the pull requests targeting stable branches -- with the exception of the upcoming stable branch, which can be used because all commits to the upcoming stable branch are merged into <code>master</code> on a regular basis)

** the original issue is linked in the *Related issues* section of the issue, as a *Copied from* link

** the pull request or commit targeting master can be found by following the *Copied from* link and searching the original issue

** TIP: if the cherry-pick fails due to whitespace differences, use <code>-Xignore-all-space</code>

* <code>git push -u origin wip-$issue-$release</code> to publish the branch on github and be able to create a pull request from it

* Create the pull request, targeting the branch you are backporting to, set milestone as appropriate, prefix the PR title with <code>$release: </code> and put the backport tracker URL in the PR description

* Edit the backport tracker issue, putting the GitHub PR URL in the description - now the backport tracker and PR are interlinked

* Set the backport tracker status to "In Progress", and assign it to yourself

Do not merge the pull request yet. Merging only takes place after the backports pass [[HOWTO run integration and upgrade tests|integration and upgrade tests]] and with proper approval as described in [[HOWTO merge commits from the integration branch]].

All commits being backported must be cherry-picked from master. There are exceptions to this rule (see https://github.com/ceph/ceph/pull/4175 for instance: it fixes a regression introduced by an incorrect resolution of a backport) but they are rare. When a backport is done to fix a bug that shows in the integration tests or upgrade tests run by cron, testing the backport in an integration branch is redundant because the existing tests will keep failing.

h4. How to automate

* generate a *$github_token* by "following the GitHub instructions":https://help.github.com/articles/creating-an-access-token-for-command-line-use/

* retrieve the *$redmine_key*: it is a token that will allow you to script issue updates

** login http://tracker.ceph.com/

** visit http://tracker.ceph.com/my/account

** look for *API access key* in the page (Control-f)

** click on the *Show* link just under *API access key*

** copy the hexadecimal number it shows

** redmine_key=b586ce6a7a936e71f351c93ac0b65a588d4333

* retrieve the *$redmine_user_id* - this is an integer, not your username. It can be found by logging in at http://tracker.ceph.com and then hovering the mouse over the Redmine username at the top right of the page - where it says "Logged in as *myusername*" and examining the URL. The URL will be something like http://tracker.ceph.com/users/2544 where 2544 is the *$redmine_user_id* value.

* grab the "ceph-backport.sh" script from src/script in "master" branch of https://github.com/ceph/ceph.git

* make a "backport_common.sh" script that sets the redmine_key, redmine_user_id, and github_token variables to the right values and make sure the "source backport_common.sh" command in the script will not fail

* follow the instructions in the comment block at the top of "ceph-backport.sh"

h4. Notes on cherry-picking

The "traditional" method is to cherry-pick each commit from the master PR in turn, and resolve conflicts as you go.

If there is a conflict, explain how it was resolved in the commit message, below the Conflicts line. For instance:

<pre>

commit c60da2f3c34e7325c748d2d6e55140a0a30013fd

Author: Samuel Just <sjust@redhat.com>

Date:   Thu Nov 20 15:15:08 2014 -0800

    PGLog: include rollback_info_trimmed_to in (read|write)_log

    Fixes: #10157

    Backport: firefly, giant

    Signed-off-by: Samuel Just <sjust@redhat.com>

    (cherry picked from commit 1fe8b846641486cc294fe7e1d2450132c38d2dba)

    Conflicts:

    	src/osd/PGLog.cc

            in the context coll_t::META_COLL was replaced with META_COLL

</pre>

The difference between the original commit and the one including a conflict resolution can be displayed with

<pre>

commit=c7d0d51cb574594de6f09457c960347b11fc2474 ; picked_from=$(git show --no-patch --pretty=%b $commit  |  perl -ne 'print if(s/.*cherry picked from commit (\w+).*/$1/)') ; diff -u --ignore-matching-lines '^[^+-]' <(git show $picked_from) <(git show $commit)

</pre>

If the conflict is difficult to resolve, seek help from the author of the original commit (see "librbd: deadlock in image refresh":https://github.com/ceph/ceph/pull/4176 for instance) and assign the issue to her/him, or to a [[HOWTO#Leads|lead]]

It is also possible to cherry-pick all the commits at once. Since github generates branches for every PR, one can use the following trick:

<pre>

git fetch <upstream> pull/<prr-num>/head:pr-<prnum>

git cherry-pick -x pr-<pr-num>~<num-commits>..pr-<pr-num>

</pre>

Adding following to somewhere bash can find can make things easier

<pre>

prfetch() {

    git fetch upstream pull/${1}/head:pr-${1}

}

</pre>

h4. Notes on the ceph-backport.sh script

The ceph-backport.sh script:

* creates a pull request from the <code>wip-$issue-$release</code> branch with the same title as the issue

<pre>

account=myaccount

eval title=$(curl --silent 'http://tracker.ceph.com/issues/'$issue.json?key=$redmine_key | jq .issue.subject) ; echo $title

number=$(curl --silent --data-binary '{"title":"'"$title"'","head":"'$account':wip-'$issue-$release'","base":"'$release'","body":"http://tracker.ceph.com/issues/'$issue'"}' 'https://api.github.com/repos/ceph/ceph/pulls?access_token='$github_token | jq .number)

</pre>

* assigns the pull request to yourself (because it needs to got through integration tests before it can be approved by the original author), add the relevant labels (rgw, core, bug fix, feature...) and set the milestone to $release (requires write permission to the ceph repository)

<pre>

component=core ; curl --silent --data-binary '{"milestone":"'$release_number'","assignee":"'$account'","labels":["bug fix","'$component'"]}' 'https://api.github.com/repos/ceph/ceph/issues/'$number'?access_token='$github_token

</pre>

* Sets the issue description to only contain the URL to the pull request and change the status to *In Progress*

<pre>

redmine_status=2 # In Progress

curl --verbose -X PUT --header 'Content-type: application/json' --data-binary '{"issue":{"description":"https://github.com/ceph/ceph/pull/'$number'","status_id":'$redmine_status'}}' 'http://tracker.ceph.com/issues/'$issue.json?key=$redmine_key

</pre>

Here are the snippets above grouped together for easier copy/paste:

<pre>

set -x

redmine_key=b586c588d4333

redmine_user_id=789 # as found in the URL near "Logged in as " top right of each page

github_token=bc275830c635

github_user=dachary

issue=$1

# can't seem to extract the release number with the api

release=giant ; release_number=2

release=firefly ; release_number=3

release=hammer ; release_number=5

release=infernalis ; release_number=7

release=jewel ; release_number=8

#release_branch=$release-next

release_branch=$release

if [ $(curl --silent http://tracker.ceph.com/issues/$issue.json | jq -r .issue.tracker.name) != "Backport" ]

then

    echo "not a backport issue"

    exit 1

fi

git checkout -f -b wip-$issue-$release ceph/$release_branch

git cherry-pick -x .....

git push loic wip-$issue-$release ; sleep 2 # let github catch up

title=$(curl --silent 'http://tracker.ceph.com/issues/'$issue.json?key=$redmine_key | jq .issue.subject | tr -d '\\"')

echo "Issue title: $title"

number=$(curl --silent --data-binary '{"title":"'"$title"'","head":"'$github_user':wip-'$issue-$release'","base":"'$release_branch'","body":"http://tracker.ceph.com/issues/'$issue'"}' 'https://api.github.com/repos/ceph/ceph/pulls?access_token='$github_token | jq .number)

echo "Opened pull request $number"

component=core ; curl --silent --data-binary '{"milestone":"'$release_number'","assignee":"'$github_user'","labels":["bug fix","'$component'"]}' 'https://api.github.com/repos/ceph/ceph/issues/'$number'?access_token='$github_token

firefox https://github.com/ceph/ceph/pull/$number

redmine_status=2 # In Progress

curl --verbose -X PUT --header 'Content-type: application/json' --data-binary '{"issue":{"description":"https://github.com/ceph/ceph/pull/'$number'","status_id":'$redmine_status',"assigned_to_id":'$redmine_user_id'}}' 'http://tracker.ceph.com/issues/'$issue.json?key=$redmine_key

firefox http://tracker.ceph.com/issues/$issue

</pre>

If you would like to automate update of the tracker target version, try something like this instead of the last <code>curl</code> command:

<pre>

curl --verbose -X PUT --header 'Content-type: application/json' --data-binary '{"issue":{"description":"https://github.com/ceph/ceph/pull/'$number'","status_id":'$redmine_status',"fixed_version_id":'$redmine_fixed_version'}}' 'http://tracker.ceph.com/issues/'$issue.json?key=$redmine_key

</pre>

The values of <code>$redmine_status</code> and <code>$redmine_fixed_version</code> can be determined by setting the desired values in an issue and then peeking at the JSON using a command like this:

<pre>

curl --silent 'http://tracker.ceph.com/issues/12395.json'

</pre>