Monday, July 04, 2011

Git: Undoing

Introduction

Git is a distributed version control system.  Each git working directory has a local copy of repository.  A common Git workflow includes the following:

  1. Make changes in working directory
  2. Add changes to staging area
  3. Commit to local repository
  4. Push to origin repository on remote

In real practice, we may regret of what we have done at any stages above.  We may want to revert or undo our works.

Revert untracked files

Untracked files are simply new files that have not added to staging area nor committed before.  The most easiest way to remove untracked file is delete it from working directory.  However, if there are bunch of untracked files exist in working directory, you may remove them using “git clean” command:

$ echo "This is new file" > newfile 
... // do some work and revert the work fianlly 
$ git status –s 
?? newfile 
$ git clean -f 
Removing newfile 
$ git status 
# On branch master 
nothing to commit (working directory clean) 
$ ls -a 
.  ..  .git  install  newfile  readme

Revert modified tracked files

Assume there are 2 files in a clean git repository:

$ ls -a 
.  ..  .git  install  readme 
$ git status 
# On branch master 
nothing to commit (working directory clean)

Make some changes to the working tree:

$ echo "modified" >> install 
$ echo "modified" >> readme

And the working tree is now become

$ ls -a 
.  ..  .git  install  newfile  readme 
$ git status –s 
M install 
M readme

Both readme and install file has modified status.

Revert a single file

To revert only a readme file, run

$ cat readme 
this is a readme 
modified 
$ git checkout readme 
$ cat readme 
this is a readme 
$ git status –s 
M install 
?? newfile

Revert a working directory

To revert whole working directory, run

$ git status –s 
M install 
M readme 
?? newfile 
$ git reset --hard 
HEAD is now at 5543e79 second commit 
$ git status –s 
?? newfile

Un-staging files

By using “git add” on new or modified files will move them into staging area for commit in later stage.  To un-stage files, use “git reset HEAD <file>...":

Add 3 new files and make changes to readme file:

$ ls -a 
.  ..  .git  install  readme 
$ touch newfile1 newfile2 newfile3 
$ ls -a 
.  ..  .git  install  newfile1  newfile2  newfile3  readme 
$ cat readme 
this is a readme 
$ echo "modified" >> readme 
$ git status -s 
M readme 
?? newfile1 
?? newfile2 
?? newfile3

Add files to staging area:

$ git add * 
$ git status 
# On branch master 
# Changes to be committed: 
#   (use "git reset HEAD <file>..." to unstage) 
# 
#       new file:   newfile1 
#       new file:   newfile2 
#       new file:   newfile3 
#       modified:   readme 
#

Un-stage files:

$ git reset HEAD 
Unstaged changes after reset: 
M       readme 
$ git status 
# On branch master 
# Changed but not updated: 
#   (use "git add <file>..." to update what will be committed) 
#   (use "git checkout -- <file>..." to discard changes in working directory) 
# 
#       modified:   readme 
# 
# Untracked files: 
#   (use "git add <file>..." to include in what will be committed) 
# 
#       newfile1 
#       newfile2 
#       newfile3 
no changes added to commit (use "git add" and/or "git commit -a")

Run this to un-stage only a single file, newfile3:

$ git reset HEAD newfile3

Revert committed changes in working directory

Assume 3 commits has been done on a working directory:

$ ls -a 
.  ..  .git  install  readme 
$ git status 
# On branch master 
nothing to commit (working directory clean) 
$ echo "commit 1" >> readme 
$ git commit -a -m "commit 1" 
[master fdaa7d3] commit 1 
1 files changed, 1 insertions(+), 0 deletions(-) 
$ echo "commit 2" >> readme 
$ git commit -a -m "commit 2" 
[master 7ad2d0a] commit 2 
1 files changed, 1 insertions(+), 0 deletions(-) 
$ echo "commit 3" >> readme 
$ git commit -a -m "commit 3" 
[master 8a56cd9] commit 3 
1 files changed, 1 insertions(+), 0 deletions(-) 
$ git log --oneline 
8a56cd9 commit 3 
7ad2d0a commit 2 
fdaa7d3 commit 1 
5543e79 second commit 
f9d7ae7 this is first commit

Revert to last commit

$ git reset --hard HEAD^ 
HEAD is now at 7ad2d0a commit 2 
$ cat readme 
this is a readme 
commit 1 
commit 2 
$ git log --oneline 
7ad2d0a commit 2 
fdaa7d3 commit 1 
5543e79 second commit 
f9d7ae7 this is first commit

Revert to specific commit

We may revert to specific commit by specify sha1 hash of the commit in “git reset

$ git log --oneline 
07187ef commit 3 
ca79a77 commit 2 
37194a2 commit 1 
5543e79 second commit 
f9d7ae7 this is first commit 
$ git reset --hard 37194a2 
HEAD is now at 37194a2 commit 1 
$ git log --oneline 
37194a2 commit 1 
5543e79 second commit 
f9d7ae7 this is first commit

Revert to last few commit

Using “HEAD~n” with “git reset” allow us to revert to last n committed:

$ git log --oneline 
fdbde67 commit 3 
54d5cef commit 2 
37194a2 commit 1 
5543e79 second commit 
f9d7ae7 this is first commit 
$ git reset --hard HEAD~3 
HEAD is now at 37194a2 commit 1 
$ git log --oneline 
5543e79 second commit 
f9d7ae7 this is first commit

You may also use something like “HEAD^” to revert a file to last commit:

$ git reset –hard HEAD^

Add more caret (^) symbol to indicate revert for last few commits.  One caret represent one backward commit.  For example, to revert to last 5 commits:

$ git reset –hard HEAD^^^^^

Revert committed pushed to remote

It is not encourage to revert pushed commit to origin repository if other has pulled what you have pushed.  However, human makes mistake.  An unwanted push may be reverted too if you are aware that nobody has pulled before.

There are few ways to revert pushed commit.  This may be the most simple example:

$ git push 
Everything up-to-date 
$ git log --oneline 
f6abcfa commit 3 
0660af9 commit 2 
37194a2 commit 1 
5543e79 second commit 
f9d7ae7 this is first commit 
$ git reset --hard HEAD^^^ 
HEAD is now at 5543e79 second commit 
$ git push origin +master 
Total 0 (delta 0), reused 0 (delta 0) 
To /tmp/test.git 
+ f6abcfa...5543e79 master -> master (forced update) 
$ git pull 
Already up-to-date. 
$ git push 
Everything up-to-date 
$ git log --oneline 
5543e79 second commit 
f9d7ae7 this is first commit

You may encounter error while revert pushed commit:

$ git push origin +master
Total 0 (delta 0), reused 0 (delta 0)
remote: error: denying non-fast-forward refs/heads/master (you should pull first)
! [remote rejected] e3fce6 -> master (non-fast-forward)

This is due to the origin repository has disable non fast forward (revert) commit.  Edit .git/config in origin repository to allow non fast forward commit:

$ cat .git/config

[receive]
        denyNonFastforwards = true false

Revert again and it should work:

$ git push origin +master

Prune loose committed objects

Even if committed works has been undone, it still exists in the repository as loose object.  Run reflog command shows loose objects:

# git reflog
704b63a HEAD@{0}: checkout: moving from 05fec4a9b062d7d9883969283a3ba18e5e06aad0
05fec4a HEAD@{1}: HEAD^ --: updating HEAD
9fa2a61 HEAD@{2}: checkout: moving from 806a6fd6ae4198a20af0f301d27c13d0dd052931
806a6fd HEAD@{3}: checkout: moving from master to 806a6fd6ae4198a20af0f301d27c13
704b63a HEAD@{4}: 704b63ab353cf32b5d6a9a54b8eb2d4d52b824e8: updating HEAD
9fa2a61 HEAD@{5}: merge origin/master: Merge made by recursive.
05fec4a HEAD@{6}: HEAD^ --: updating HEAD
5f3b0a1 HEAD@{7}: rebase finished: returning to refs/heads/master
5f3b0a1 HEAD@{8}: checkout: moving from master to 5f3b0a1e2167652b32e765b9431727
704b63a HEAD@{9}: checkout: moving from test to master
5f3b0a1 HEAD@{10}: checkout: moving from 806a6fd6ae4198a20af0f301d27c13d0dd05293
806a6fd HEAD@{11}: checkout: moving from master to 806a6fd6ae4198a20af0f301d27c1
704b63a HEAD@{12}: merge 704b63ab353cf32b5d6a9a54b8eb2d4d52b824e8: Fast-forward
806a6fd HEAD@{13}: checkout: moving from 704b63ab353cf32b5d6a9a54b8eb2d4d52b824e
704b63a HEAD@{14}: checkout: moving from master to 704b63ab353cf32b5d6a9a54b8eb2
806a6fd HEAD@{15}: 806a6fd6ae4198a20af0f301d27c13d0dd052931: updating HEAD
5f3b0a1 HEAD@{16}: 5f3b0a1e2167652b32e765b94317279868faa82b: updating HEAD
4441dc5 HEAD@{17}: HEAD^ --: updating HEAD
05fec4a HEAD@{18}: HEAD^ --: updating HEAD
5f3b0a1 HEAD@{19}: checkout: moving from 5f3b0a1e2167652b32e765b94317279868faa82
5f3b0a1 HEAD@{20}: checkout: moving from master to 5f3b0a1e2167652b32e765b943172
5f3b0a1 HEAD@{21}: checkout: moving from 5f3b0a1e2167652b32e765b94317279868faa82
5f3b0a1 HEAD@{22}: checkout: moving from master to 5f3b0a1e2167652b32e765b943172
5f3b0a1 HEAD@{23}: checkout: moving from 704b63ab353cf32b5d6a9a54b8eb2d4d52b824e
704b63a HEAD@{24}: checkout: moving from master to 704b63ab353cf32b5d6a9a54b8eb2
5f3b0a1 HEAD@{25}: merge 5f3b0a1e2167652b32e765b94317279868faa82b: Fast-forward
806a6fd HEAD@{26}: merge origin/master: Fast-forward

These objects may expire and will be pruned by “git gc” after 30 days (default).  It has no harm exist in local repository except occupy some hard drive space.

It is possible to prune loose objects immediately by executing

# git reflog expire --expire=now --all
# git gc
Counting objects: 35395, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (7225/7225), done.
Writing objects: 100% (35395/35395), done.
Total 35395 (delta 26533), reused 35393 (delta 26533)
# git reflog
#

Git: Branching and Merging

Git allows us to maintain multiple branches of development.  This is common and consider a must to develop with git.  You may switch between branches to enjoy isolate development workflow.

To ease the explanation, we create a bare repository as origin and 2 cloned repository from origin.

$ mkdir project.git 
$ cd project.git/ 
$ git init --bare 
Initialized empty Git repository in /tmp/test.branch/project.git/ 
$ cd .. 
$ git clone project.git clone1 
Initialized empty Git repository in /tmp/test.branch/clone1/.git/ 
warning: You appear to have cloned an empty repository. 
$ cd clone1 
$ echo "This is readm" > readme 
$ git add readme 
$ git commit -m "first commit" 
[master (root-commit) 8b60032] first commit 
1 files changed, 1 insertions(+), 0 deletions(-) 
create mode 100644 readme 
$ git push 
No refs in common and none specified; doing nothing. 
Perhaps you should specify a branch such as 'master'. 
fatal: The remote end hung up unexpectedly 
error: failed to push some refs to '/tmp/test.branch/project.git' 
$ git push origin master 
Counting objects: 3, done. 
Writing objects: 100% (3/3), 215 bytes, done. 
Total 3 (delta 0), reused 0 (delta 0) 
Unpacking objects: 100% (3/3), done. 
To /tmp/test.branch/project.git 
* [new branch]      master –> master 
$ cd .. 
$ git clone project.git clone2 
Initialized empty Git repository in /tmp/test.branch/clone2/.git/ 
$ ls -a clone2 
.  ..  .git  readme

Create git branch

Let’s create a new branch “newfeature1” in clone1:

$ cd clone1 
$ git branch 
* master 
$ git branch newfeature1
$ git branch 
* master 
  newfeature1 
$ git checkout newfeature1 
Switched to branch 'newfeature1' 
$ git branch 
  master 
* newfeature1

The repository is now point to branch newfeature1.  Add a new file to the repository and commit changes:

$ echo "This is new feature" > feature1 
$ git add feature1 
$ git commit -m "Commit new feature" 
[newfeature 399cffe] Commit new feature 
1 files changed, 1 insertions(+), 0 deletions(-) 
create mode 100644 feature1

Switch branch

Continue from above example.  Let’s switch between branches and see what happen to the working directory:

$ git branch 
  master 
* newfeature1 
$ ls -a 
.  ..  .git  feature1  readme 
$ git checkout master 
Switched to branch 'master' 
$ ls -a 
.  ..  .git  readme

When switch branch from “newfeature1” to “master”, file “feature1” that was committed in “newfeature1” won’t appear in branch “master”.

List branches in repository

To list existing local branches:

$ git branch 
* master 
  newfeature1

Active branch is indicated by * in front.

To list local and remote tracked branch

$ git branch -a 
* master 
  newfeature1 
  remotes/origin/master

To list remote tracked branch only:

$ git branch -r 
  origin/master

Merge with Fast Forward

Assume we perform few commits into branch “newfeature1”:

$ echo "changes #1" >> feature1 
$ git commit -a -m "commit changes #1" 
[newfeature1 fc6a399] commit changes #1 
1 files changed, 1 insertions(+), 0 deletions(-) 
$ echo "changes #2" >> feature1 
$ git commit -a -m "commit changes #2" 
[newfeature1 1ea0fda] commit changes #2 
1 files changed, 1 insertions(+), 0 deletions(-) 
$ git log --oneline --graph 
* 1ea0fda commit changes #2 
* fc6a399 commit changes #1 
* 048ce34 Commit new feature 1 
* 8b60032 first commit

Perform a fast forward merge into master:

$ git checkout master 
Switched to branch 'master' 
$ git merge newfeature1 
Updating 8b60032..1ea0fda 
Fast-forward 
feature1 |    3 +++ 
1 files changed, 3 insertions(+), 0 deletions(-) 
create mode 100644 feature1 
$ git log --oneline --graph 
* 1ea0fda commit changes #2 
* fc6a399 commit changes #1 
* 048ce34 Commit new feature 1 
* 8b60032 first commit

All commit changes from newfeature1 is now merge into master.

Merge without Fast Forward

An advantage with no fast forward merge preserve the branch commit history:

$ git log --oneline --graph 
* 8b60032 first commit 
$ git merge --no-ff newfeature1 
Merge made by recursive. 
feature1 |    3 +++ 
1 files changed, 3 insertions(+), 0 deletions(-) 
create mode 100644 feature1 
$ git log --oneline --graph 
*   4725f13 Merge branch 'newfeature1' 
|\ 
| * 1ea0fda commit changes #2 
| * fc6a399 commit changes #1 
| * 048ce34 Commit new feature 1 
|/ 
* 8b60032 first commit

Push Branch to remote

So far all branches are stored in local repository only.  You may share the branch with others by pushing it to origin:

$ git branch -a 
* master 
  newfeature1 
  remotes/origin/master 
$ git checkout newfeature1 
Switched to branch 'newfeature1' 
$ git push origin HEAD 
Total 0 (delta 0), reused 0 (delta 0) 
To /tmp/test.branch/project.git 
* [new branch]      HEAD -> newfeature1 
$ git branch -r 
  origin/master 
  origin/newfeature1

Now we create more branch and push to origin:

$ git branch newfeature2 master 
$ git checkout newfeature2 
Switched to branch 'newfeature2' 
$ echo "Feature 2" > feature2 
$ git branch 
  feature3 
  master 
  newfeature1 
* newfeature2 
$ git add feature2 
$ git commit -m "commit feature 2" 
[newfeature2 ef39e45] commit feature 2 
1 files changed, 1 insertions(+), 0 deletions(-) 
create mode 100644 feature2 
$ git push origin HEAD 
Total 0 (delta 0), reused 0 (delta 0) 
To /tmp/test.branch/project.git 
* [new branch]      HEAD –> newfeature2

$ git branch newfeature3 master 
$ git checkout newfeature3 
Switched to branch 'newfeat 
$ echo "Feature 3" > feature3 
$ git add feature3 
$ git commit -m "commit feature 3" 
[newfeature3 ec1da8d] commit feature 3 
1 files changed, 1 insertions(+), 0 deletions(-) 
create mode 100644 feature3 
$ git push origin HEAD 
Counting objects: 7, done. 
Compressing objects: 100% (4/4), done. 
Writing objects: 100% (6/6), 512 bytes, done. 
Total 6 (delta 1), reused 0 (delta 0) 
Unpacking objects: 100% (6/6), done. 
To /tmp/test.branch/project.git 
* [new branch]      HEAD –> newfeature3

We have these branches now both in local and origin repositories:

$ git branch -a 
  master 
  newfeature1 
* newfeature2 
  newfeature3 
  remotes/origin/master 
  remotes/origin/newfeature1 
  remotes/origin/newfeature2 
  remotes/origin/newfeature3

Fetch Remote Branch

Let’s switch to clone2:

$ cd clone2 
$ git branch -a 
* master 
  remotes/origin/HEAD -> origin/master 
  remotes/origin/master 
$ git remote show origin 
* remote origin 
  Fetch URL: /tmp/test.branch/project.git 
  Push  URL: /tmp/test.branch/project.git 
  HEAD branch: master 
  Remote branches: 
    master      tracked 
    newfeature1 new (next fetch will store in remotes/origin) 
    newfeature2 new (next fetch will store in remotes/origin) 
    newfeature3 new (next fetch will store in remotes/origin) 
  Local branch configured for 'git pull': 
    master merges with remote master 
  Local ref configured for 'git push': 
    master pushes to master (up to date)

Notice that there are 3 new branches: newfeature1, newfeature2 and newfeature3 in origin.

Now fetch from origin,

$ git fetch 
From /tmp/test.branch/project 
* [new branch]      newfeature1 -> origin/newfeature1 
* [new branch]      newfeature2 -> origin/newfeature2 
* [new branch]      newfeature3 -> origin/newfeature3 
$ git branch -a 
* master 
  remotes/origin/HEAD -> origin/master 
  remotes/origin/master 
  remotes/origin/newfeature1 
  remotes/origin/newfeature2 
  remotes/origin/newfeature3 
$ git remote show origin 
* remote origin 
  Fetch URL: /tmp/test.branch/project.git 
  Push  URL: /tmp/test.branch/project.git 
  HEAD branch: master 
  Remote branches: 
    master      tracked 
    newfeature1 tracked 
    newfeature2 tracked 
    newfeature3 tracked 
  Local branch configured for 'git pull': 
    master merges with remote master 
  Local ref configured for 'git push': 
    master pushes to master (up to date)

To have a look on the remote branch newfeature2, use git checkout:

$ git checkout origin/newfeature2 
Note: checking out 'origin/newfeature2'.

You are in 'detached HEAD' state. You can look around, make experimental 
changes and commit them, and you can discard any commits you make in this 
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may 
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at c47ec36... This is new feature 2 
$ ls -a 
.  ..  .git  feature2  readme

The checkout origin/newfeature2 will switch local repository to “detached HEAD”.  A detached HEAD doesn’t belongs to any branch:

$ git branch -a 
* (no branch) 
  master 
  remotes/origin/master 
  remotes/origin/newfeature1 
  remotes/origin/newfeature2 
  remotes/origin/newfeature3

You shouldn’t commit any changes that doesn’t belong to a named branch or else others will have problems merging with the changes.  Instead you should name a branch or checkout local branch before commit any changes:

$ git checkout newfeature2
Branch newfeature2 set up to track remote branch newfeature2 from origin.
Switched to a new branch 'newfeature2'
$ git branch -a
  master
* newfeature2
  remotes/origin/master
  remotes/origin/newfeature1
  remotes/origin/newfeature2
  remotes/origin/newfeature3

Delete local branch

For branch that you never push to origin, you may delete it as follow:

$ git branch
  master
* newfeature1
$ git branch -d newfeature1
error: Cannot delete the branch 'newfeature1' which you are currently on.

Active branch is not allow to delete.  You may switch to other branch first

$ git checkout master
Switched to branch 'master'
$ git branch -d newfeature1
error: The branch 'newfeature1' is not fully merged.
If you are sure you want to delete it, run 'git branch -D newfeature1'.
$ git branch -D newfeature1
Deleted branch newfeature1 (was 201bb22).
$ git branch
* master

You can’t delete branch using “-d” if it is not merged with others.  To force delete, use “-D” option.

Delete remote tracked branch

Assume the repository has the following branches:

$ git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/origin/newfeature1
  remotes/origin/newfeature2
  remotes/origin/newfeature3
$ git remote show origin
* remote origin
  Fetch URL: /tmp/test.branch/project.git
  Push  URL: /tmp/test.branch/project.git
  HEAD branch: master
  Remote branches:
    master      tracked
    newfeature1 tracked
    newfeature2 tracked
    newfeature3 tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)

Delete a remote tracked branch:

$ git branch -d -r origin/newfeature3
Deleted remote branch origin/newfeature3 (was ec1da8d).
$ git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/origin/newfeature1
  remotes/origin/newfeature2
$ git remote show origin
* remote origin
  Fetch URL: /tmp/test.branch/project.git
  Push  URL: /tmp/test.branch/project.git
  HEAD branch: master
  Remote branches:
    master      tracked
    newfeature1 tracked
    newfeature2 tracked
    newfeature3 new (next fetch will store in remotes/origin)
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)

Git: Repository

Introduction

Git repository is a storage area that keep the project’s git version control information including logs, commits, version and etc. (a.k.a. repository meta data).  The repository stores in a hidden folder name .git.  There are 2 types of git repositories: bare repository and non-bare repository.

Bare repository

Bare repository is a folder that keep only git’s meta data.  It doesn’t contain project working tree.  In short, it stores all the .git folders and files directly in the folder.

Bare repository always act as a shared repository at runtime.  It has similar role in centralized version control system: the central repository in version control server.  It always act as an origin repository of many non-bare repositories of development task.

Non-bare repository

Non-bare repository or working repository is a storage folder that contain .git and working tree.  It is a repository that used in programmer’s workstation for daily work.  Most non-bare repository are usually clone from bare repository.

Create a new repository

It is common to start a repository as non bare repository:

$ git init ~/myproj
Initialized empty Git repository in /home/usr/myproj/.git/
$ cd ~/myproj
$ ls -a . .. .git $ echo my first project > readme $ git add readme $ git commit -m "My first commit" [master (root-commit) 784a3ce] My first commit Committer: root <root@revision-control.(none)> 1 files changed, 1 insertions(+), 0 deletions(-) create mode 100644 readme

A new repository has created.  Obviously, this is a non-bare repository.  It contains a .git folder a working tree.  The working tree has a file of name “readme”.

GitFaq states that

A quick rule of thumb is to never push into a repository that has a work tree attached to it, until you know what you are doing.

Thus, we shouldn’t clone the non-bare repository as origin of our working repository in development stage.  We should first make the non-bare repository to bare repository for other to clone;

$ cd ~ 
$ git clone --bare myproj myproj.git
Initialized empty Git repository in /tmp/myproj.git/ 
$ ls -a 
.   .webmin     auth    loggerhead-cache-Hpo6UL  myproj.git 
..  README.txt  config  myproj                   servers

Now, myproj.git is a bare repository of the project.  .git folder isn’t exist in bare repository.  Instead, all contents of .git stores in project folder directly.

What about myproj now?  It doesn’t know myproj.git.  It has no relation with myproj.git now.  If we continue working on myproj.  All commit works have no way to sync (push/pull) with bare repository (myproj.git).  Perform this to make myproj.git as origin of myproj.git:

$ cd ~/myproj
$ git remote add origin ~/myproj.git // for git push $ git config branch.master.remote origin // for git pull $ git config branch.master.merge refs/heads/master // for git pull

The myproj now become a clone of myproj.git.  myproj.git is origin of myproj.  You may push your commits into myproj.git as usual.  It has same working behaviour as other new clone from myproj.git.

Alternatively, you may delete myproj folder and clone from myproj.git:

$ rm –fr ~/myproj 
$ git clone ~/myproj.git ~/myproj

You may also set a default pull behaviour globally if your new project always pull from a origin branch:

$ git config --global branch.master.remote origin
$ git config --global branch.master.merge refs/heads/master

Create a bare repository

Do this If you want to start a bare repository directly:

$ git init -bare myproj.git

A bare repository is created and initialize, you may clone a new working repository from it.

Create a shared repository for team work

To create a git repository that may share for team environment accessed via SSH or other means, use something like:

$ git init --bare --shared=group project.git
Initialized empty shared Git repository in /srv/repos/git/project.git/

List the directory with attributes and notice the group write and setuid bit is on:

$ ls –l
drwxrwsr-x 7 root root 4096 Jul 7 19:27 project.git

Enable write access for developer group:

$ chgrp developer -R project.git
$ ls -l
drwxrwsr-x 7 root developer 4096 Jul 7 19:27 project.git

Share existing repository for team work

To share an existing git repository, try this:

# ls -l
drwxr-xr-x 7 root root 4096 Jul 7 15:17 project.git

Change group of repository:

# chgrp -R developer project.git
# ls -l
drwxr-xr-x 7 root developer 4096 Jul 7 15:17 project.git

Change directories permission:

# find project.git -type d |xargs chmod g+wxs
# ls -l
drwxrwsr-x 7 root developer 4096 Jul 7 15:17 project.git

Reference

  1. Stackoverflow: How do you get git to always pull from a specific branch?
  2. GitSubmoduleTutorial
  3. Git Tip of the Week: Setting up a shared repository. url: http://alblue.bandlem.com/2011/03/git-tip-of-week-setting-up-shared.html

Version Control: Git

Introduction

Git is a relatively young but powerful open source version control system compare to CVS, SVN and others.  It is famous in Linux communities.  The Linux kernel source code is using git as version control now.  More Git tools is available for Window users now.

Git doesn’t has a centralized version control server/repository like CVS or SVN.  Instead, the repository exist in each developer’s machine.  Once you have git system installed, you may start your project to enjoy git version control locally or offline.  You may carry all version control operations like commit, revert, branching, tagging and etc. in your standalone PC.  All your git work is stored in a hidden folder .git of your project.

You may however, share your repository with collaborators in your team by cloning your git repository.  To share work in team, you may push and pull with each other’s git repository.  You may also select a git repository in the team as a origin repository for everyone.  So you have a centralized control of your version control commits.  You may customize the collaboration workflow that is comfortable for your teams.  Git will suit your needs in various dimension.

Basic Git operation

init

Initialize a git repository, whether it is bare or working repository

clone

Clone repository to share repository in team work. The source repository will automatically become origin of clone repository.  You may clone local or remote repository of various source like ssh, git protocol, http or local.

add

Add changes to staging area and waiting for commit in later stage

commit

Commit all changes added to staging area

push

Push all committed snapshot to another git repository

pull

Pull all update committed snapshot from other git repository

Git Branches

Unlike other version control system, branching in Git is common and cheap operation in day to day development works.  Git doesn’t encourage developers to work with master branch directly.  Instead, all development work should work in new branch.  And merge to master once complete.

Git Best Practices

Two things you should never do in git:

  • NEVER force a push
    If you find yourself in a situation where your changes can't be pushed upstream, something is wrong. Contact another developer for help tracking down the problem.
  • NEVER rebase a branch that you pushed, or that you pulled from another person
    Rebasing published branches can lead to duplicate commits in the shared repository.

In general, the preferred workflow is:

  • create a branch from master, check it out, do your work
  • test and commit your changes
  • optionally push your branch up to the remote repository (origin) OR
  • optionally rebase your branch to master (if your changes are unpublished)
  • checkout master, make sure it's up-to-date with upstream changes
  • merge your branch into master
  • test again (and again)
  • push your local copy of master up to the remote repository master (origin/master)
  • delete your branch (and remotely, too, if you published it)

Sunday, June 19, 2011

VMware vSphere Hypervisor

Introduction

VMware vSphere Hypervisor is a bare metal hypervisor (a.k.a. type 1, native hypervisor) that run directly on host’s hardware.  It has a management interface for users to create, manage and configure virtual machines run on top of it.  VMware vSphere Hypervisor is one of the free product available to communities who like to run virtual machines natively.  A common usage for Type 1 hypervisors is deploying backend services for group or cloud applications.

An alternate virtual machine deployment strategy is running type 2 (or hosted) hypervisor under an OS.  Some examples includes KVM, VMWare Workstation and VirtualBox.  Type 2 hypervisors is used mostly in personal task that need virtual machines for testing, debugging or carry out jobs temporary without interrupting host OS.

Competitor

A direct competitor of VMware vSphere Hypervisor is XEN hypervisor.  XEN is an open source product.  There are few XEN distributors.  To get start with XEN, try Citrix XenServer.  It doesn’t involve complicated kernel compilation and installation.  It’s deployment strategy is similar to VMware vSphere Hypervisor.

After doing few performance and deployment testing for both vSphere and XEN hypervisor.  I personally few that VMware vSphere Hypervisor outperform XEN in some aspects.  The first area I test is the virtual network speed among these 2 hypervisors.  I test a host equipped with a gigabit NIC running a FreeNAS virtual machine.  I transfer a large file from CIFS share of FreeNAS.  The result is vSphere transfer at double the rate of FreeNAS.

While this doesn’t mean vSphere is simply better than XEN.  I was running XENServer out of the box. There may be options to compile an optimized version of my own XEN in order to maximised it’s performance that could perform as good as vSphere or better.  However, It is beyond my ability at the moment.  Thus, I use VMware vSphere as my first hypervisor.

Installation

Download VMware vSphere hypervisor ISO file from VMWare site.  You may need to register first before download the product.  Use one the following deployment methods that suits your environment after finish download.

Deploy via CD-ROM

This is the most easiest method.  Burn the ISO file in CD-ROM and boot the CD on the machine.  Follow instructions on screen to finish the installation.

Deploy via USB flash drive

If you want to save the world by not wasting a CD-ROM, you may create a bootable USB flash drive from the ISO by using tools like UNetbootin.  After burn the ISO into flash drive, follow the steps to create a Kickstart script or else the installation will fail in the middle.

Change file isolinux.cfg

Open isolinux.cfg in USB flash drive and add “ks=usb” as follow:

default menu.c32
menu title VMware VMvisor Boot Menu
timeout 80

label ESXi Installer
menu label ^ESXi Installer
kernel mboot.c32
append vmkboot.gz ks=usb --- vmkernel.gz --- sys.vgz --- cim.vgz --- ienviron.vgz --- install.vgz

label ^Boot from local disk
menu label ^Boot from local disk
localboot 0x80

Create a new file ks.cfg

vmaccepteula
serialnum --esx=<Serial number from VMwre>
rootpw password
autopart --firstdisk --overwritevmfs
network --bootproto=dhcp
install usb
reboot

Click here (http://www.vmware.com/pdf/vsphere4/r41/vsp_41_esxi_i_vc_setup_guide.pdf) for more information about KickStart script.

You may now boot your machine with USB boot enable in BIOS and start installation.  Please note that KickStart installation may erase your hard drive content without any warning prompt.  Use it with care.

Deploy via PXE

This is my favourite deployment strategy as all task is done via network.  You should have your PXE solution ready in your environment.

Follow the steps to prepare for PXE installation:

  1. Mount the vSphere hypervisor installation ISO file
  2. Copy all files and folders from ISO mount to TFTPBOOT folder. e.g.: /var/lib/tftpboot/vsphere.410/
  3. You may share the above folder in NFS, HTTP or FTP to use in later stage.
  4. Create a PXE entry in pxe configuration file:
    label VMware vSphere Hypervisor
          menu label VMware vSphere Hypervisor 4.1.0.update1-348481.x86_64
          kernel vsphere.410/mboot.c32
          append vsphere.410/vmkboot.gz ks=nfs://192.168.0.2/var/lib/tftpboot/ks.cfg --- vsphere.410/vmkernel.gz --- vsphere.410/sys.vgz --- vsphere.410/cim.vgz --- vsphere.410/ienviron.vgz --- vsphere.410/install.vgz
  5. The above setting use nfs to export a kickstart file ks.cfg.  You may use HTTP or FTP to export the file too.  Just find a suitable place for your KickStart file.
  6. Create a KickStart file ks.cfg:
    vmaccepteula
    serialnum --esx=<Serial number from VMwre>
    rootpw password
    autopart --firstdisk --overwritevmfs
    network --bootproto=dhcp
    install nfs --server=192.168.0.2 --dir=/var/lib/tftpboot/vsphere.410
    reboot

  7. You may use PXE boot to start the installation now.
  8. Please note that KickStart script is always required or else the installation will fail in the middle.  Again remember that KickStart installation may erase your hard drive content without any warning prompt.  Use it with care.

Configuration

Once the VMware vSphere hypervisor is up and running.  The configuration is easy.

Identify the IP address of your hypervisor and use browser to navigate to the http://<hypervisor-ip>/.  Download the Windows GUI client utility, install and start using the hypervisor.

Access vSphere hypervisor via SSH tunneling

Add forwarded port 443 and 902  to your favorite SSH client.  For example, putty SSH tunnels configuration:

1

You may then connect to your vSphere hypervisor via localhost.

The vSphere GUI client may only access to port 443, 902 and 903.  It won’t work with classic gateway port forwarding that alter the port number for client.

There is a problem with this design.  You cannot manage more than one hypervisor via SSH tunnels at the same time as the port number may conflict with each other.  A workaround solution is create a Windows machine in the same network of your hypervisors.  Install the vSphere GUI client.  Allow access the windows machine using Remote Desktop (RDP, port 3389) from remote.

Troubleshooting

Error – The Microsoft Visual J#2.0 Second Editor installer returned error code ‘4113’

You may encounter error

The Microsoft Visual J#2.0 Second Editor installer returned error code ‘4113’

when installing VMware vSphere Client 4.1.  A possible cause is you may already have a copy of Microsoft J#2.0 x86 in your system.  To overcome the problem, uninstall  Microsoft Visual J#2.0 from control panel, re-run the VMware vSphere Client 4.1 installer again.  The installer should download an updated version of Microsoft Visual J#2.0.

How to rename a .vmdk file

A vmdk file is a virtual machine disk file.  If you have created a new HDD for a virtual machine, a new .vmdk file is crated too.  vSphere doesn’t allow user to specify file name.  The file name is automatically allocated.  In common practices, we may create a new HDD device to store data purely.  We may even move the HDD to another virtual machine from time to time.  In most situations, a meaningful name is always needed to identify such HDD .vmdk file.

A solution is using console command to rename .vmdk file.  First, enable SSH from hypervisor host.  Identify .vmdk file start from folder /vmfs/volumes and use the following command to rename:

# vmfstools –E <old-name>.vmdk <new-name>.vmdk

Please note that a .vmdk HDD device consist of 2 files: <file-name>.vmdk and <file-name>-flat.vmdk

Ignore the “-flat” file and using vmfstools command to perform the rename.  Both files will be renamed accordingly.

Wednesday, November 10, 2010

Boot Linux kernel from PXE Server

It is nice and easy if we can setup an environment to perform installation of Linux via high speed LAN. The most usual method I install Linux is to burn a small boot ISO file to CDRW and use the CDRW disc to start the installation via HTTP or FTP. I learn from PXELinux and found it is a nice option to try. To setup a Linux machine via LAN, you need the following:
  1. A DHCP server that support BOOTP
  2. TFTP Server
  3. PXELinux kernel boot file
  4. Linux installation ISO file

PXELinux

You may obtain PXE Linux Boot Kernel file from http://www.kernel.org/pub/linux/utils/boot/syslinux/

Download the version of SYSLINUX you prefer and extract the tar file.  The PXE Linux Boot Kernel file name is pxelinux.0.

You should copy the pxelinux.0 to root directory of TFTP service.

DHCP Configuration

Add the following setting to dhcpd.conf to enable BOOTP

allow booting;
allow bootp;

range dynamic-bootp 192.168.0.221 192.168.0.230;
next-server 192.168.0.2;
filename "pxelinux.0";

next-server is the server of TFTP server.

TFTP Server

# Install TFTP server

yum install tftp-server

After installation,  you may proceed to TFTP configuration.  The configuration file is located at /etc/xinetd.d/tftp

TFTP Server Configuration – Enable the service

service tftp
{
        disable = no
        socket_type             = dgram
        protocol                = udp
        wait                    = yes
        user                    = root
        server                  = /usr/sbin/in.tftpd
        server_args             = -s /var/lib/tftpboot
        per_source              = 11
        cps                     = 100 2
        flags                   = IPv4
}

TFTP Server Configuration – Firewall

-A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 69 -j ACCEPT

TFTP Server Configuration – Log activities

Log activities of TFTP server Edit file /etc/xinetd.d/tftp and add "-v" option to tftpd. The flag can be specified multiple times for even higher verbosity.  You need to restart xinetd service to activate tftp logging.  The log details for TFTP may found in /var/log/messages.

service tftp
{
        disable = no
        socket_type             = dgram
        protocol                = udp
        wait                    = yes
        user                    = root
        server                  = /usr/sbin/in.tftpd
        server_args             = -v –v –s /var/lib/tftpboot
        per_source              = 11
        cps                     = 100 2
        flags                   = IPv4
}

TFTP Server – Restart service

# /sbin/service xinetd restart

A useful example of using the log is to see what files are requested by PXE client when serving Microsoft Windows boot loader. File names are always case sensitive in TFTP operation.

TFTP Server – File name remapping

Microsoft windows or DOS use backslash (\) for folder separator. It always cause problem in those TFTP server running on Linux OS where forward slash (/) is used as folder separator. When serving windows boot loader, we may encounter the situation of we keep the boot loader files in folders. To solve this problem, we may use the file name remapping option available in tftp service.

service tftp
{
        disable = no
        socket_type             = dgram
        protocol                = udp
        wait                    = yes
        user                    = root
        server                  = /usr/sbin/in.tftpd
        server_args             = -v -m /var/lib/tftpboot/rules -s /var/lib/tftpboot
        per_source              = 11
        cps                     = 100 2
        flags                   = IPv4
}

# vi /var/lib/tftpboot/rules
rg \\ /         # convert backslashes to slashes (useful for windows file names)

PXELinux boot menu

You may configure PXE boot menu in tftpboot/pxelinux.cfg

DEFAULT menu.c32
PROMPT 0

LABEL FC13.x86_64
        menu label Linux Fedora Core 13 x86_64 (ftp://bee/fc13.x86_64)
        KERNEL fc13.x86_64/vmlinuz
        APPEND initrd=fc13.x86_64/initrd.img

You should copy the kernel and initrd file from ISO images/pxeboot directory to tftpboot folder.

Boot PXE from client

Once the PXE server is up and ready, you may configure the client machine for network boot.  This is usually configure via BIOS setup while start the machine.  If the PXE server is setup properly, the PXE boot menu will load and you may start using PXE boot service.

Reference

    1. http://syslinux.zytor.com/wiki/index.php/PXELINUX