Wednesday, July 20, 2011

Git: Using SmartGit in Windows

Windows users was addicted to GUI interface.  Command line git may be hard for users to get started with.  A nice Java based GIT GUI is available to fill the space.  SmartGit is there to help manage your Git repository using nice GUI frontend.  You may choose the portable SmartGit too in this case.

Work with SmartGit

SmartGit GUI frontend is easy to use.  Most of the common Git command is available in top buttons bar:

a_thumb[1]

To make SmartGit works, you should tell SmartGit where your Git executable is.  Navigate to Edit | Preferences…

b_thumb[1]

Using external file comparator

Smart support external file comparator by navigate to Edit | Preference | Tools | File Comparators:

i

Git SSH client

SmartGit provides two SSH client access: System SSH client or Smart SSH client:

j

SmartGit as SSH Client does not possess all SSH client option.  Using System SSH client with ssh.exe or putty plink.exe may enjoy most from SSH.

Git: using msysgit in Windows

Git was originally develop to manage Linux kernel source.  It is widely use in Linux community.  It has porting to many platform too including Windows.

msysgit is windows porting of Git.  It is git command line utility running in command shell console.  I personally prefer using portable msysgit and set path in my home user’s environment variable.  By using portable, I may switch git version easily when new release is available.  No setup and installation is needed for portable release.

First configuration task for msysgit is set path variable in your local user account’s environment variable to include the <msysgit>/bin folder.  Git will then available whenever you start cmd shell.

ssh.exe: Password authentication

The most common ssh authentication is password authentication.  The following example shows a basic usage to access git repository via SSH:

C:\> git clone ssh://alice@git.example.com/srv/repos/git/project.git project
Cloning into project...
<user>@<git-server>'s password:
remote: Counting objects: 3, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.

You should supply a user name (e.g: alice) for ssh connection or else it may fail.

ssh.exe: Password authentication without user name

There are situation with Git that user name may not supply in git command clause.  For example, a git repository that consist of submodule have .gitmodules as follow:

[submodule "build"]
    path = build
    url = ssh://git.example.com/srv/repos/git/build.git
[submodule "core"]
    path = core
    url = ssh://git.example.com/srv/repos/git/core.git

As the repository is shared for team work, commit .gitmodules with user name is not practical.  The git ssh clause should not contain user name in origin repository.  Nor it should be modified to add user name in local repository.  In this case, a default user name should supply with ssh access.

To automatically supply user name to git-server, create a ~/.ssh/config file with the following entry:

Host git.example.com
  User alice

Subsequent git operation accessing ssh will use the user specify in ~/.ssh/config. This is useful when the local user name is different to ssh service user name.

ssh.exe: public key authentication

To specify the private key used for public key authentication, update ~/.ssh/config as follow:

Host git.example.com
  IdentityFile ~/.ssh/id_dsa.home
  User alice

ssh.exe: public key authentication with key agent

If a private key was protected by passphrase, each git operation accessing ssh service with the private key requires user to enter passphrase.  This is some how troublesome and will cause problem for other program like smartgit that do not prompt for passphrase.

A direct solution is using ssh key agent: ssh-agent to supply private key to ssh.  For more detail information, please refer to this article: http://chee-yang.blogspot.com/2011/07/ssh-public-key-authentication.html

msysgit: public key authentication with Putty

First, follow instructions in this article: http://chee-yang.blogspot.com/2011/07/ssh-public-key-authentication-with.html to configure putty to work with ssh.

To make msysgit work with putty as ssh client, you need a putty program: plink.exe.  Download plink.exe and keep in save location as putty.exe.

Next, create a new environment variable GIT_SSH that point to plink.exe:

h

You may start using git that use putty as ssh client.

There are 2 types of ssh connection with plink: using SSH server name or putty saved session.

Plink with SSH server name is limited and does not make use of extra options available in putty.  A saved putty session may consume options like pageant authentication and SSH user name for authentication.  To make git work with putty saved session, just name saved session same as  SSH server name.

Error – missing libconv-2.dll

In PortableGit-1.7.6-preview20110709.7z release, you may encounter the following errors when executing some Git commands::

The program can’t start because libiconv-2.dll is missing from your computer. Try installing the program to fix this problem.

git-error

To get rid of the error, locate <msysgit>\bin\libconv-2.dll and copy to <msysgit>\libexec\git-core.

SSH: Public Key Authentication with Putty

Putty is a set of Windows based SSH GUI client.  It has small footprint that may even work without installation by execute the program in standalone mode.  These are the common Putty applications:

  1. Putty.exe is SSH client program
  2. PuttyGen.exe is a program to generate public/private key pair
  3. Pageant.exe is a SSH authentication agent for Putty
  4. Plink.exe is command line interface to Putty back ends.

Generate Key Pair

Run PuttyGen.exe and press Generate button to start generating a key pair.  You may need to move mouse cursor over the blank area to generate randomness for key generating:

1_thumb3

After finish generating, you should see public key shown on screen.  Private key has generated too, but it doesn’t show on screen for safety reason:

2_thumb1

As shown on screen, the key length is 1024 bits which is consider very secure from hacking.   The public key generated is shown on top of windows.  The “Key passphrase” and “Confirm passphrase” are there to protect the private key if save the key in file.  Empty passphrase for private key will left the private key unencrypted.  If the private key has compromised, others may use the private key just as the owner has.  A passphrase provide an extra shield to protect the private key.  User will need to enter the passphrase whenever the private key is needed.

If you are ready for the key pair, copy the public key in clipboard for next stage.

Deploy Public Key to SSH service

Public key should keep in SSH server.  Copy and paste the public key text in any text editor and save the file as “authorized_key”.

3_thumb3

Find a way to copy the file to your home account in SSH server.  The file should copy in in folder ~/.ssh/.  You may use SCP or home account in samba share to copy the file into ~/.ssh/.  The folder ~/.ssh/ might not exist, create the folder to store authorized_keys.

$ cd ~/.ssh
$ chmod 0600 authorized_keys $ ls -gG total 4 -rw------- 1 226 Jul 17 13:07 authorized_keys -bash-4.1$ cat authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIBuSYSpUY3CxoLMmxnimmuC8kXRfJ8fmqDtyi6MqHLNWMGSdP/LYfc6LPatVq2tYlfDPiFrLXiMeqYBiZHLwAGDfJdm40BpclVmbovYOL4b/kFm81gk2iENSscnV6gI07aniABOJNXsWnvUeAnHpbYLR5YSc2Ko48ubjyCzBWJM/w== rsa-key-20110718

Deploy Private Key to Putty

Private key supply from SSH client end.  Private key may persist with passphrase and without passphrase.  Pass-phrased private key has extra security to protect private key.  Pass-phrased private key without private key is plain text key that may use directly for SSH authentication.  Both type of private key should keep safely from others to obtain it.

Private key may keep in any media or location or even a flash drive that carry with you anytime.  For illustration purpose of this article, the private key keep in %userprofile%/.ssh/key.ppk.  Please note that Putty requires the private key file in Putty key format.

There are numbers of SSH client application supports public key authentication.  There should be a place in setting to specify the private key file.  Just supply the private key as is to make the SSH connection works.

Access SSH with public key authentication

Try run Putty.exe and configure the SSH client for public key authentication:

4_thumb1

Click Open button to start login:

login as: <user-name>
Authenticating with public key "rsa-key-20110718"
$

The above example show login using private key without passphrase.  If SSH screen shows something like above, then the public key authentication is success.  You may have notice that the text “rsa-key-20110718” is key comment shown PuttyGen.exe.

For passphrase protected private key, the login screen looks like this:

login as: <user-name>
Authenticating with public key "rsa-key-20110718"
Passphrase for key “rsa-key-20110718”: <enter passphrase here>
$

Pageant: SSH login with key agent

Pageant is Putty key agent that cache a list of private key.  Private keys with or without passphrase may cached in Pageant.  By using Pageant, you may ignore the private key setting in Putty’s session (located in Connection | SSH | Auth, “Private key file for authentication”).  Pageant is smart enough try try try with all private keys cached for authentication.

To use Pageant.exe, double click Pageant.exe to launch pageant program.  The program will stay in Notification Areas at right end of Windows Task Bar:

c

To add a private key, right click on Pageant icon to bring out a context menu and choose “Add Key” to add a private key.  A passphrase dialog will prompt out if the private key is protected with a passphrase.

Please note that you should check “Attempt authentication using Pageant” for the SSH session to realize Pageant authentication:

d

Putty: Auto login user name

Putty session configuration allow you to specify user name to for auto login:

e

By combining Pageant and auto login user name, you may use Putty by just double click on the SSH session without supply user name and passphrase.  The SSH session is available right after you double click.

Putty: Use private key generated by OpenSSH

Windows Putty.exe can’t use the private key generated by OpenSSH directly.  Putty.exe require private key that store in putty format.  Thus, the OpenSSH private key should convert to Putty format.  Run PuttyGen.exe to perform the conversion.  Use File | Load private key function as follow:

a_thumb1

The PuttyGen will prompt a message to show you how to save the private key as putty format:

b_thumb1

Click “Save private key” button to save the private key in Putty format.  The private key file may then use for SSH public key authentication using Putty.

Monday, July 18, 2011

SSH: Public Key Authentication with OpenSSH

SSH using Unix/Linux account’s username and password for authentication by default.  Although user name and password were encrypted during authentication session, it still suffer from dictionary attack.  The usage of public key authentication in SSH is to cover the loophole.

Public key authentication uses public key cryptography scheme for encryption operation.  It perform encryption by using a key pair of private key and public key.  Public key cryptography possess very high level of security.  The secure levels increase exponentially if using larger key length.  As the name implied, public key is known to public and private key should keep by user in secure and safe place.  Public key is generated from private key but It’s very computational costly to derive private key from public key.

In SSH, the public key stores in SSH server and SSH client use private key to gain access to the SSH service.  Choosing key length of 1024 bits or higher is common practice for SSH service.  To know more about the SSH authentication protocol, read: http://www.ietf.org/rfc/rfc4252.txt.

OpenSSH: SSH authentication

OpenSSH is a free version of SSH tools.  It was designed for OpenBSD operation system.  It has been ported to other operation system including Linux and Windows.  It has became the most common SSH tools in the market.

OpenSSH doesn’t come with fancy GUI front end.  All tools are available as console program.  The most common used tool is “ssh” in Linux or “ssh.exe” in windows.  It act as SSH client to access shell account of hos

To use SSH, type

# ssh ssh-server.example.com

Some common parameters are “-l” and “-p” that supply login user name and port number respectively.

OpenSSH: Generate Key Pair

Use ssh-keygen to generate key:

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user/.ssh/id_rsa.
Your public key has been saved in /home/ccy/.ssh/id_rsa.pub.
The key fingerprint is:
30:33:d4:9c:86:2d:20:3b:8a:d3:85:30:01:d9:d0:77 user@example.com
The key's randomart image is:
+--[ RSA 2048]----+
|B*. ...= .       |
|.oo+..E *        |
|  +...=o         |
|.o o   =         |
|+ .     S        |
| .               |
|                 |
|                 |
|                 |
+-----------------+
$ ls –gG .ssh
total 12
-rw------- 1 1671 Jul 19 09:05 id_rsa
-rw-r--r-- 1  408 Jul 19 09:05 id_rsa.pub

Two file were generated:  id_rsa is private key file and id_rsa.pub is public key file.  To deploy the public key for ssh server, try this:

$ cat id_rsa.pub >> authorized_keys
$ chmod 0600 authorized_keys
$ ls -gG
total 16
-rw------- 1  408 Jul 19 09:15 authorized_keys
-rw------- 1 1671 Jul 19 09:05 id_rsa
-rw-r--r-- 1  408 Jul 19 09:05 id_rsa.pub

Copy id_rsa to any SSH client that want to connect to this server via SSH.  Keep in mind that id_rsa is store in OpenSSH file format.

OpenSSH: Deploy private key

The private key generated by OpenSSH may use directly in OpenSSH client both in Windows and Linux.  No conversion is needed.  Just copy the private key file to ~/.ssh/.folder and configure ~/ssh/config to make it works for public key authentication.

OpenSSH: Disable password authentication

Once the public key authentication scheme is ready to use in real practice, you may consider disable the classic password authentication by changing /etc/sshd_config:in SSH server:

# cat /etc/ssh/sshd_config
PubkeyAuthentication yes
PasswordAuthentication no

Remember to restart or reload sshd service to enforce changes if the configuration has updated.

Using OpenSSH in Windows

There are some tricks using OpenSSH in Windows.  OpenSSH require an environment variable “HOME” to locate the .ssh folder that keep ssh configuration file.  A common practice is set HOME to %USERPROFILE% and create a folder .ssh in %USERPROFILE% folder.  You may keep the private key file into %HOME%\.ssh folder:

F

OpenSSH: Configuration file

The configuration for OpenSSH is usually keep in ~/ssh/config.  Here is a sample OpenSSH configuration file:

Host <server1>
IdentityFile ~/.ssh/id_dsa.home User alice Host <server2> IdentityFile ~/.ssh/id_rsa.work User bob
Port 2200

The configuration specify the private key file to use for 2 server and the user name to login if the user name is different to the account’s user name.   Server2 even specify the SSH port number to connect to server2.

More detail information about the configuration file may found here: http://www.openbsd.org/cgi-bin/man.cgi?query=sshd_config

ssh-agent: SSH login with key agent

ssh-agent is a program cache private key for ssh session:

$ eval `ssh-agent`
Agent pid 8896
$ set | grep -i ssh
SSH_AGENT_PID=8896
SSH_AUTH_SOCK=/tmp/ssh-Lhbzdj8980/agent.8980
$

The above script trying to start ssh-agent.  To add a ssh private key to key agent:

$ ssh-add
Enter passphrase for /home/alice/.ssh/id_rsa: <enter-passphrase>
Identity added: /c/Users/coder/.ssh/id_rsa (/home/alice/.ssh/id_rsa)
$

ssh-agent: use in Microsoft Windows environment

I haven’t found a good way to use ssh-agent with Microsoft Windows.  Using ssh-agent in windows is not a straight solution.  Some manual steps is required to make it work.

You may ssh-agent.exe using command shell or adding an entry in windows registry: “HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run”

You should then identify the SSH_AUTH_SOCK file in %TEMP% folder and add an environment variable in your user account as follow:

g

continue with ssh-add to add private keys as usual.

Without SSH_AUTH_SOCK variable, executing ssh-add may fail with:

C:\>ssh-add
Could not open a connection to your authentication agent.

Saturday, July 16, 2011

SSH: Secure SHell basic usage

Previous days, Unix users use telnet to access server shell accounts.  Telnet is not a secure protocol.  All Telnet traffics are sent in plain text including login user name and password.  Secure Shell (SSH) was designed as replacement for Telnet and other insecure remote shells.  All traffics are encrypted.  Nowadays, telnet is no longer used by Linux community.

SSH Server

OpenSSH is a popular SSH Server in Linux community.  This server is install and run natively in most Linux distro.  it may use directly without much configuration once install.  The default port number for SSH is 22.  You may use any SSH client to access the remote shell accounts directly.

The SSH server configuration file is located in /etc/ssh/sshd_config.  After finish changing the configuration, run

# service sshd reload

to realise the new configuration for next session.

SSH Client

Linux SSH Client

OpenSSH client is default and native Linux SSH client.  Just type:

# ssh <remote-ssh-server>

to access the remote shell account.

The SSH client use your current user name as user name to login to remote ssh server , to override user name, use “-l” parameter:

# ssh -l <user> <remote-ssh-server>

Windows SSH Client: Putty

Putty support both Telnet and SSH access.  It is a free software that run under Windows and Unix platform.

Other SSH Client

Although a common usage of SSH is accessing remote shell securely, it doesn’t limit to that.  A single SSH service may act as a gateway to access surprisingly many types of intranet traffics.  This ease the network administration to allow only SSH traffics traveling though intranet in encrypted and secured manner.  Some popular usages of SSH are listed as follow:

SSH Tunneling

SSH tunneling provide a solution to transfer sensitive and unencrypted traffic over secured channel.  It is a cheap and affordable solution than VPN.  All traffics travel through SSH tunnel was encrypted.  Some popular usage of SSH tunneling are VOIP and remote access to intranet services.

SCP

Secure Copy Protocol (SCP) provide an alternate file transfer service (FTP).  WinSCP is a nice Windows SCP software to access remote server via SSH.

Version Control: CVS and Git

Both version control software CVS and GIT allow user to synchronize local repository with remote repository via SSH.

Monday, July 11, 2011

Git: Migrating from CVS

There is a git-cvsimport utility that may migrate CVS repository to Git Repository.  However, this utility is buggy and doesn’t export the CVS properly especially there are branches in the repository.  A more mature tools to use is cvs2svn. Do not confuse with the name it implied that it is CVS to SVN tool.  In fact, it contain a cvs2git utility to convert CVS repository to Git.

git-cvsimport

Using git-cvsimport is not recommended.  If you insist to use, here are some simple usage.

Install git-cvsimport utility:

# apt-get install git-cvs
To import a  CVS repository to Git, try this
# git-cvsimport -v –d <CVSROOT> –C <git.repo> <cvs-repository>

The CVSROOT may be a CVS folder that has CVSROOT folder or a CVS protocol URL.

cvs2svn

Download and install cvs2svn.

Use this command create a blob file and a dump file for next stage:

# cvs2svn-2.3.0/cvs2git --blobfile=git-blob.dat --dumpfile=git-dump.dat --username=cvs2git <cvs-repository>

The cvs-repository should be direct filesystem CVS repository.  CVS protocol URL is not allow here.

Create a Git repository:

# git init --bared --shared import.git
# cd import.git
Import the blob and dump files to git repository:
# cat ../git-blob.dat ../git-dump.dat | git fast-import

Friday, July 08, 2011

CVS: Make Repository Read Only

There are few ways to make CVS repository read only.  The most easiest so far I know of is create an empty writers file in CVSROOT folder.

Here is the initial content of CVSROOT:

# ls
checkoutlist    config,v       Emptydir   modules,v  taginfo
checkoutlist,v  cvswrappers    history    notify     taginfo,v
commitinfo      cvswrappers,v  loginfo    notify,v   val-tags
commitinfo,v    editinfo       loginfo,v  rcsinfo    verifymsg
config          editinfo,v     modules    rcsinfo,v  verifymsg,v

Create an empty writers file:

# touch writers
# ls
checkoutlist    config,v       Emptydir   modules,v  taginfo      writers,v
checkoutlist,v  cvswrappers    history    notify     taginfo,v
commitinfo      cvswrappers,v  loginfo    notify,v   val-tags
commitinfo,v    editinfo       loginfo,v  rcsinfo    verifymsg
config          editinfo,v     modules    rcsinfo,v  verifymsg,v

An empty writers means nobody may write into repository, thus the repository is read only now.

Thursday, July 07, 2011

Using TurnKey Linux Appliances

Introduction

TurnKey Linux (TKL) appliances provides an easy turn key solution to deploy essential and common network services for your organization or team on LAN or WAN.  It is deployed as an appliance.  All configurations is done via web browser.  It save your time to study and install related software packages by packing everything in a single 130~200MB ISO file.  You may install these software appliance in bare machine, virtual machine or even Amazon EC2 could server.

At the time this article is written, there are 45+ ready TurnKey appliances ranging from content management, web development, issue tracking and messaging.  These appliances are based on Ubuntu Linux – one of the most friendly and easy to use Linux distributor in the market.

Once install, the console will show you how to access the server.  The configuration may perform in Webmin GUI via HTTP web browser.  This save your time and efforts compare to command line configuration practice in used by Linux community.  User with little or no knowledge of Linux may perform the job well too.

However, TurnKey Linux appliance is not perfect.  This article is not about using TurnKey Linux appliance but attempt to illustrate some features that you may want in TKL appliance.  Some of these features need to configure in command line console.

Upgrade Webmin1

Webmin is widely used in TurnKey appliance as front end configuration control panel.  Both Webmin and TurnKey Linux are independent to each others.  You may attempt to upgrade Webmin to latest version to get more updated features.

To upgrade Webmin, simply use “Upgrade Webmin” icon in Webmin configuration page.

To ease the installation and save your time and bandwidth if you have many TKL appliances deployed, try download the latest Webmin .tar.gz file and use the upload option to upgrade Webmin.

Add or Upgrade Webmin modules

Use “Webmin Modules” icon in Webmin configuration page to manage Webmin modules.  You may install or upgrade new Webmin module that you need but missing in TKL appliance.

Add or Upgrade Software Packages

New features or updates may available in updated software packages.   Using System | Software Packages to upgrade to latest updates.

Webmin: Time

Webmin Time module is missing in TKL appliance.  Install a time module and you are ready to configure time zone and date time of TKL appliance instance.

LDAP Authentication

If LDAP directory services is available to serve the account information, you may configure TKL to support account authentication via LDAP.

First, install these packages:

  1. libpam-ldap
  2. libnss-ldap
  3. nss-updatedb
  4. libnss-db

You may use apt-get in console:

# apt-get install libpam-ldap libnss-ldap nss-updatedb libnss-db

or using the Webmin’s Software Packages to install those packages.

Configure LDAP client via:

# dpkg-reconfigure ldap-auth-config

Follow the screen instruction to fill in relevant LDAP parameters.2

LDAP client configuration is available in Webmin too.  Install “ldap-client” to get this feature in Webmin.

Edit /etc/nsswitch.conf to add ldap support:

# vi /etc/nsswitch.conf
passwd:         compat ldap
group:          compat ldap

This may also configured by “Services using LDAP” of Webmin LDAP client:

To verify the LDAP configuration,  run “getent passwd” to check if the LDAP accounts appears in the list.

If LDAP accounts are alias accounts.  You may add “DEREF always” in /etc/ldap.conf to make dereference works.

Create home directory

A common usage of LDAP account authentication is provide a secure shell login option for users.  Home directories will not be created for LDAP accounts by default.  There is a feature in pam that may automatically create home directory for user when login.

Edit “/etc/auth-client-config//profile.d/acc-default” to add pam_mkhomedir.so as follow:

# cat /etc/auth-client-config//profile.d/acc-default
...
#
# this example is for using ldap to authenticate and authorize.  This is only
# an example, and you will most likely have to create your own profiles to
# authenticate with your system. Note that this example requires the
# libpam-cracklib package to be installed.
#
[ldap_example]
nss_passwd=passwd: files ldap
nss_group=group: files ldap
nss_shadow=shadow: files ldap
nss_netgroup=netgroup: nis
pam_auth=auth       required     pam_env.so
        auth       sufficient   pam_unix.so likeauth nullok
        auth       sufficient   pam_ldap.so use_first_pass
        auth       required     pam_deny.so
pam_account=account    sufficient   pam_unix.so
        account    sufficient   pam_ldap.so
        account    required     pam_deny.so
pam_password=password   required     pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3
        password   sufficient   pam_unix.so nullok md5 shadow use_authtok
        password   sufficient   pam_ldap.so use_first_pass
        password   required     pam_deny.so
pam_session=session    required     pam_limits.so
        session    required     pam_unix.so
        session    required     pam_mkhomedir.so skel=/etc/skel/
        session    optional     pam_ldap.so

Next execute this to update pam configuration:

# auth-client-config -a -p ldap_example

Now, user login to via ssh may have the home directory created automatically.

Monday, July 04, 2011

Git: Submodule

Introduction

For those who familiar with CVS ampersand modules mechanism, git submodule is some how similar to it.

Prepare a submodule bare repository

Create 3 repositories that may use as submodules:

$ mkdir ~/project
$ cd ~/project
$ mkdir sub1.git sub2.git sub3.git
$ git init --bare sub1.git
Initialized empty Git repository in /tmp/test.sub/sub1.git/
$ git init --bare sub2.git
Initialized empty Git repository in /tmp/test.sub/sub2.git/
$ git init --bare sub3.git
Initialized empty Git repository in /tmp/test.sub/sub3.git/

Three bare repositories now exists in your home folder:

$ dir *.git
sub1.git:
HEAD  branches  config  description  hooks  info  objects  refs

sub2.git:
HEAD  branches  config  description  hooks  info  objects  refs

sub3.git:
HEAD  branches  config  description  hooks  info  objects  refs

Prepare super git repository

The super git repository is an example of a git repository that hold some submodules.

$ mkdir super
$ cd ~/super
$ git init
$ echo “This is super” > readme
$ git add readme
$ git commit –m “Super commit”

Add submodules

Next, we add 3 sub modules into the super git repository:

$ git submodule add /tmp/test.sub/sub1.git
Initialized empty Git repository in /tmp/test.sub/super/sub1/.git/
warning: You appear to have cloned an empty repository.
fatal: You are on a branch yet to be born
Unable to checkout submodule 'sub1'
$ git submodule add /tmp/test.sub/sub2.git
Initialized empty Git repository in /tmp/test.sub/super/sub2/.git/
warning: You appear to have cloned an empty repository.
fatal: You are on a branch yet to be born
Unable to checkout submodule 'sub2'
$ git submodule add /tmp/test.sub/sub3.git
Initialized empty Git repository in /tmp/test.sub/super/sub3/.git/
warning: You appear to have cloned an empty repository.
fatal: You are on a branch yet to be born
Unable to checkout submodule 'sub3'
$ ls -a
.  ..  .git  sub1  sub2  sub3

You may have noticed there are warning and error messages when execute “git submodule add” command.  This is due to the sub modules are empty in the example.  You won’t get the messages if submodule is not empty.

We should further re-execute “git submodule add” again to complete the task:

$ git submodule add /tmp/test.sub/sub1.git
Adding existing repo at 'sub1' to the index
$ git submodule add /tmp/test.sub/sub2.git
Adding existing repo at 'sub2' to the index
$ git submodule add /tmp/test.sub/sub3.git
Adding existing repo at 'sub3' to the index
$ ls -a
.  ..  .git  .gitmodules  sub1  sub2  sub3

A new file .gitmodules was created:

$ cat .gitmodules
[submodule "sub1"]
        path = sub1
        url = /home/user/sub1.git
[submodule "sub2"]
        path = sub2
        url = /home/user/sub2.git
[submodule "sub3"]
        path = sub3
        url = /home/user/sub3.git

We should now add some changes to all empty sub module directory:

$ cd sub1
$ touch README
$ git add README
$ git commit -m "commit #1"
[master (root-commit) 3a8d946] commit #1
0 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 README
$ git push origin master
Counting objects: 3, done.
Writing objects: 100% (3/3), 201 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
To /tmp/test.sub/sub1.git
* [new branch]      master -> master
$ cd ../sub2
$ touch README
$ git add README
$ git commit -m "commit #1"
[master (root-commit) 63b5d68] commit #1
0 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 README
$ git push origin master
Counting objects: 3, done.
Writing objects: 100% (3/3), 200 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
To /tmp/test.sub/sub2.git
* [new branch]      master -> master
$ cd ../sub3
$ touch README
$ git add README
$ git commit -m "commit #1"
[master (root-commit) 46b2a53] commit #1
0 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 README
$ git push origin master
Counting objects: 3, done.
Writing objects: 100% (3/3), 201 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
To /tmp/test.sub/sub3.git
* [new branch]      master –> master
$ cd ..

The git repository status now become

$ git status
# On branch master
#
# Initial commit
#
# Changes to be committed:
#   (use "git rm --cached <file>..." to unstage)
#
#       new file:   .gitmodules
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       sub1/
#       sub2/
#       sub3/

Add and commit submodule folder

$ git add sub1 sub2 sub3
$ git status
# On branch master
#
# Initial commit
#
# Changes to be committed:
#   (use "git rm --cached <file>..." to unstage)
#
#       new file:   .gitmodules
#       new file:   sub1
#       new file:   sub2
#       new file:   sub3
#

Commit changes:

$ git commit -m "super commit"
[master (root-commit) 7029640] super commit
4 files changed, 12 insertions(+), 0 deletions(-)
create mode 100644 .gitmodules
create mode 160000 sub1
create mode 160000 sub2
create mode 160000 sub3
$ git status
# On branch master
nothing to commit (working directory clean)

To further make a super git repository as bare repository, run

$ git clone –bare ~/super ~/super.git
$ git remote add origin ~/super.git
$ git config branch.master.remote origin
$ git config branch.master.merge refs/heads/master

Clone a git repository with submodules

This example shows how to work with clone of a super git repository that contain submodules:

$ cd ~/project
$ git clone super.git super.clone
$ cd ~/super.clone
$ ls –a *

readme

sub1:
.  ..

sub2:
.  ..

sub3:
.  ..

All sub module’s directory is empty at this moment.  Run this to get updated commits

$ git submodule init
$ git submodule update
Initialized empty Git repository in /home/user/super.clone2/sub1/.git/
Submodule path 'sub1': checked out '8acdbdd86119c4a9777bbefe13bd8f80c96a8b7a'
Initialized empty Git repository in /home/user/super.clone2/sub2/.git/
Submodule path 'sub2': checked out 'e390c3e3114fde7643a3544c0a262eff5a52e09c'
Initialized empty Git repository in /home/user/super.clone2/sub3/.git/
Submodule path 'sub3': checked out '001355362c4ed758f17206a3fbbb2f7637983434'
$ ls -a *
readme

sub1:
.  ..  .git  readme

sub2:
.  ..  .git  readme

sub3:
.  ..  .git  readme

An alternate way to clone repository is run everything in a single command:

$ git clone --recursive ~/super.git ~/super.clone
Initialized empty Git repository in /tmp/submodules/super.clone2/.git/
Submodule 'sub1' (/home/user/sub1.git) registered for path 'sub1'
Submodule 'sub2' (/home/user/sub2.git) registered for path 'sub2'
Submodule 'sub3' (/home/user/sub3.git) registered for path 'sub3'
Initialized empty Git repository in /home/user/super.clone2/sub1/.git/
Submodule path 'sub1': checked out '8acdbdd86119c4a9777bbefe13bd8f80c96a8b7a'
Initialized empty Git repository in /home/user/super.clone2/sub2/.git/
Submodule path 'sub2': checked out 'e390c3e3114fde7643a3544c0a262eff5a52e09c'
Initialized empty Git repository in /home/user/super.clone2/sub3/.git/
Submodule path 'sub3': checked out '001355362c4ed758f17206a3fbbb2f7637983434'
$ ls -a *
sub1:
.  ..  .git  README

sub2:
.  ..  .git  README

sub3:
.  ..  .git  README

Add changes to submodule

The following example shows a submodule workflow to add, commit, push new changes:

Check out a branch first

$ cd ~/project/super/sub1
super/sub1$ git branch
* (no branch)
  master
super/sub1$ git checkout master
super/sub1$ git branch
* master

Make a changes

$ cat readme
This is submodule
super/sub1$ echo "New changes" >> README
super/sub1$ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   readme
#
no changes added to commit (use "git add" and/or "git commit -a")

Commit and push changes

$ git commit -a -m "New changes"
[master 6a26854] New changes
1 files changed, 1 insertions(+), 0 deletions(-)
$ git push
Counting objects: 5, done.
Writing objects: 100% (3/3), 246 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
To /tmp/test.sub/sub1.git
   3a8d946..6a26854  master –> master

Check status of submodule directory

$ cd ..
$ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   sub1
#
no changes added to commit (use "git add" and/or "git commit -a")

The submodule “sub1” status is modified now.

Commit and push changes of submodule directory

$ git commit -a -m "commit sub1"
[master 6039eee] commit sub1
1 files changed, 1 insertions(+), 1 deletions(-)
$ git status
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#
nothing to commit (working directory clean)
$ git push
Counting objects: 3, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 309 bytes, done.
Total 2 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (2/2), done.
To /tmp/test.sub/super.git
   7029640..6039eee  master –> master

Update a submodule

Try this to pull changes from origin repository to update local repository:

Pull changes

$ cd ../super.clone
super.clone$ git pull
remote: Counting objects: 3, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 2 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (2/2), done.
Updating 7029640..6039eee
Fast-forward
sub1 |    2 +-
1 files changed, 1 insertions(+), 1 deletions(-)

$ cat sub1/README
$ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   sub1
#
no changes added to commit (use "git add" and/or "git commit -a")

Up to this stage, the file has not updated to latest HEAD version yet.  After pull from origin, the status of sub1 show modified.  Continue to next topic to complete submodule update task.

Update submodule

To pull latest commit snapshot from origin, run “git submodule update” to checkout latest committed snapshot:

$ git submodule update
Submodule path 'sub1': checked out '6a26854c7985d6ba516dcca8fc8cb42908bcb639'
$ git status
# On branch master
nothing to commit (working directory clean)
$ cat sub1/README
New Changes

The repository folder is in clean state now.  sub1 folder also updated to latest committed snapshot.  Let’s continue to check the status of sub1 by continue to next topic.

Detached HEAD in submodule

Let’s check the status of sub1 now:

$ cd sub1
$ cat README
New Changes
$ git branch
* (no branch)
  master

The current position of sub1 point to a commit stage without any label.  This is known as detached HEAD commit or headless commit.  Commit detached HEAD changes is not encounrage.  It may be problems for other collaborators to retrieve the changes.

Instead, we should checkout the master branch, work, commit changes and merge commits to master branch:

$ git checkout master
Previous HEAD position was 6a26854... New changes
Switched to branch 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
$ cat README
$

The master branch’s README file is in last committed snapshot.  Let’s check the status of super git repository after checkout:

$ cd ..
$ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   sub1
#
no changes added to commit (use "git add" and/or "git commit -a")

It’s parent directory has changed to modified.  That simply means sub1 doesn’t match with latest committed snapshot.

$ git merge origin
Updating 3a8d946..6a26854
Fast-forward
README |    1 +
1 files changed, 1 insertions(+), 0 deletions(-)
$ cat README
New Changes

You may also use “git pull” to replace the “git merge origin” .  Let’s check parent status again:

$ cd ..
$ git status
# On branch master
nothing to commit (working directory clean)

It is in clean state now.

Git: Undoing

Introduction

Git is a distributed version control system.  Each git working directory has a local copy of repository.  A common Git workflow includes the following:

  1. Make changes in working directory
  2. Add changes to staging area
  3. Commit to local repository
  4. Push to origin repository on remote

In real practice, we may regret of what we have done at any stages above.  We may want to revert or undo our works.

Revert untracked files

Untracked files are simply new files that have not added to staging area nor committed before.  The most easiest way to remove untracked file is delete it from working directory.  However, if there are bunch of untracked files exist in working directory, you may remove them using “git clean” command:

$ echo "This is new file" > newfile 
... // do some work and revert the work fianlly 
$ git status –s 
?? newfile 
$ git clean -f 
Removing newfile 
$ git status 
# On branch master 
nothing to commit (working directory clean) 
$ ls -a 
.  ..  .git  install  newfile  readme

Revert modified tracked files

Assume there are 2 files in a clean git repository:

$ ls -a 
.  ..  .git  install  readme 
$ git status 
# On branch master 
nothing to commit (working directory clean)

Make some changes to the working tree:

$ echo "modified" >> install 
$ echo "modified" >> readme

And the working tree is now become

$ ls -a 
.  ..  .git  install  newfile  readme 
$ git status –s 
M install 
M readme

Both readme and install file has modified status.

Revert a single file

To revert only a readme file, run

$ cat readme 
this is a readme 
modified 
$ git checkout readme 
$ cat readme 
this is a readme 
$ git status –s 
M install 
?? newfile

Revert a working directory

To revert whole working directory, run

$ git status –s 
M install 
M readme 
?? newfile 
$ git reset --hard 
HEAD is now at 5543e79 second commit 
$ git status –s 
?? newfile

Un-staging files

By using “git add” on new or modified files will move them into staging area for commit in later stage.  To un-stage files, use “git reset HEAD <file>...":

Add 3 new files and make changes to readme file:

$ ls -a 
.  ..  .git  install  readme 
$ touch newfile1 newfile2 newfile3 
$ ls -a 
.  ..  .git  install  newfile1  newfile2  newfile3  readme 
$ cat readme 
this is a readme 
$ echo "modified" >> readme 
$ git status -s 
M readme 
?? newfile1 
?? newfile2 
?? newfile3

Add files to staging area:

$ git add * 
$ git status 
# On branch master 
# Changes to be committed: 
#   (use "git reset HEAD <file>..." to unstage) 
# 
#       new file:   newfile1 
#       new file:   newfile2 
#       new file:   newfile3 
#       modified:   readme 
#

Un-stage files:

$ git reset HEAD 
Unstaged changes after reset: 
M       readme 
$ git status 
# On branch master 
# Changed but not updated: 
#   (use "git add <file>..." to update what will be committed) 
#   (use "git checkout -- <file>..." to discard changes in working directory) 
# 
#       modified:   readme 
# 
# Untracked files: 
#   (use "git add <file>..." to include in what will be committed) 
# 
#       newfile1 
#       newfile2 
#       newfile3 
no changes added to commit (use "git add" and/or "git commit -a")

Run this to un-stage only a single file, newfile3:

$ git reset HEAD newfile3

Revert committed changes in working directory

Assume 3 commits has been done on a working directory:

$ ls -a 
.  ..  .git  install  readme 
$ git status 
# On branch master 
nothing to commit (working directory clean) 
$ echo "commit 1" >> readme 
$ git commit -a -m "commit 1" 
[master fdaa7d3] commit 1 
1 files changed, 1 insertions(+), 0 deletions(-) 
$ echo "commit 2" >> readme 
$ git commit -a -m "commit 2" 
[master 7ad2d0a] commit 2 
1 files changed, 1 insertions(+), 0 deletions(-) 
$ echo "commit 3" >> readme 
$ git commit -a -m "commit 3" 
[master 8a56cd9] commit 3 
1 files changed, 1 insertions(+), 0 deletions(-) 
$ git log --oneline 
8a56cd9 commit 3 
7ad2d0a commit 2 
fdaa7d3 commit 1 
5543e79 second commit 
f9d7ae7 this is first commit

Revert to last commit

$ git reset --hard HEAD^ 
HEAD is now at 7ad2d0a commit 2 
$ cat readme 
this is a readme 
commit 1 
commit 2 
$ git log --oneline 
7ad2d0a commit 2 
fdaa7d3 commit 1 
5543e79 second commit 
f9d7ae7 this is first commit

Revert to specific commit

We may revert to specific commit by specify sha1 hash of the commit in “git reset

$ git log --oneline 
07187ef commit 3 
ca79a77 commit 2 
37194a2 commit 1 
5543e79 second commit 
f9d7ae7 this is first commit 
$ git reset --hard 37194a2 
HEAD is now at 37194a2 commit 1 
$ git log --oneline 
37194a2 commit 1 
5543e79 second commit 
f9d7ae7 this is first commit

Revert to last few commit

Using “HEAD~n” with “git reset” allow us to revert to last n committed:

$ git log --oneline 
fdbde67 commit 3 
54d5cef commit 2 
37194a2 commit 1 
5543e79 second commit 
f9d7ae7 this is first commit 
$ git reset --hard HEAD~3 
HEAD is now at 37194a2 commit 1 
$ git log --oneline 
5543e79 second commit 
f9d7ae7 this is first commit

You may also use something like “HEAD^” to revert a file to last commit:

$ git reset –hard HEAD^

Add more caret (^) symbol to indicate revert for last few commits.  One caret represent one backward commit.  For example, to revert to last 5 commits:

$ git reset –hard HEAD^^^^^

Revert committed pushed to remote

It is not encourage to revert pushed commit to origin repository if other has pulled what you have pushed.  However, human makes mistake.  An unwanted push may be reverted too if you are aware that nobody has pulled before.

There are few ways to revert pushed commit.  This may be the most simple example:

$ git push 
Everything up-to-date 
$ git log --oneline 
f6abcfa commit 3 
0660af9 commit 2 
37194a2 commit 1 
5543e79 second commit 
f9d7ae7 this is first commit 
$ git reset --hard HEAD^^^ 
HEAD is now at 5543e79 second commit 
$ git push origin +master 
Total 0 (delta 0), reused 0 (delta 0) 
To /tmp/test.git 
+ f6abcfa...5543e79 master -> master (forced update) 
$ git pull 
Already up-to-date. 
$ git push 
Everything up-to-date 
$ git log --oneline 
5543e79 second commit 
f9d7ae7 this is first commit

You may encounter error while revert pushed commit:

$ git push origin +master
Total 0 (delta 0), reused 0 (delta 0)
remote: error: denying non-fast-forward refs/heads/master (you should pull first)
! [remote rejected] e3fce6 -> master (non-fast-forward)

This is due to the origin repository has disable non fast forward (revert) commit.  Edit .git/config in origin repository to allow non fast forward commit:

$ cat .git/config

[receive]
        denyNonFastforwards = true false

Revert again and it should work:

$ git push origin +master

Prune loose committed objects

Even if committed works has been undone, it still exists in the repository as loose object.  Run reflog command shows loose objects:

# git reflog
704b63a HEAD@{0}: checkout: moving from 05fec4a9b062d7d9883969283a3ba18e5e06aad0
05fec4a HEAD@{1}: HEAD^ --: updating HEAD
9fa2a61 HEAD@{2}: checkout: moving from 806a6fd6ae4198a20af0f301d27c13d0dd052931
806a6fd HEAD@{3}: checkout: moving from master to 806a6fd6ae4198a20af0f301d27c13
704b63a HEAD@{4}: 704b63ab353cf32b5d6a9a54b8eb2d4d52b824e8: updating HEAD
9fa2a61 HEAD@{5}: merge origin/master: Merge made by recursive.
05fec4a HEAD@{6}: HEAD^ --: updating HEAD
5f3b0a1 HEAD@{7}: rebase finished: returning to refs/heads/master
5f3b0a1 HEAD@{8}: checkout: moving from master to 5f3b0a1e2167652b32e765b9431727
704b63a HEAD@{9}: checkout: moving from test to master
5f3b0a1 HEAD@{10}: checkout: moving from 806a6fd6ae4198a20af0f301d27c13d0dd05293
806a6fd HEAD@{11}: checkout: moving from master to 806a6fd6ae4198a20af0f301d27c1
704b63a HEAD@{12}: merge 704b63ab353cf32b5d6a9a54b8eb2d4d52b824e8: Fast-forward
806a6fd HEAD@{13}: checkout: moving from 704b63ab353cf32b5d6a9a54b8eb2d4d52b824e
704b63a HEAD@{14}: checkout: moving from master to 704b63ab353cf32b5d6a9a54b8eb2
806a6fd HEAD@{15}: 806a6fd6ae4198a20af0f301d27c13d0dd052931: updating HEAD
5f3b0a1 HEAD@{16}: 5f3b0a1e2167652b32e765b94317279868faa82b: updating HEAD
4441dc5 HEAD@{17}: HEAD^ --: updating HEAD
05fec4a HEAD@{18}: HEAD^ --: updating HEAD
5f3b0a1 HEAD@{19}: checkout: moving from 5f3b0a1e2167652b32e765b94317279868faa82
5f3b0a1 HEAD@{20}: checkout: moving from master to 5f3b0a1e2167652b32e765b943172
5f3b0a1 HEAD@{21}: checkout: moving from 5f3b0a1e2167652b32e765b94317279868faa82
5f3b0a1 HEAD@{22}: checkout: moving from master to 5f3b0a1e2167652b32e765b943172
5f3b0a1 HEAD@{23}: checkout: moving from 704b63ab353cf32b5d6a9a54b8eb2d4d52b824e
704b63a HEAD@{24}: checkout: moving from master to 704b63ab353cf32b5d6a9a54b8eb2
5f3b0a1 HEAD@{25}: merge 5f3b0a1e2167652b32e765b94317279868faa82b: Fast-forward
806a6fd HEAD@{26}: merge origin/master: Fast-forward

These objects may expire and will be pruned by “git gc” after 30 days (default).  It has no harm exist in local repository except occupy some hard drive space.

It is possible to prune loose objects immediately by executing

# git reflog expire --expire=now --all
# git gc
Counting objects: 35395, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (7225/7225), done.
Writing objects: 100% (35395/35395), done.
Total 35395 (delta 26533), reused 35393 (delta 26533)
# git reflog
#