RAID Techniques in ZFS (RAID-1) !!!

Like in UFS, we can use RAID techniques in ZFS also....Let us create few RAID volumes.

The standard RAID technique RAID-1 (mirroring) and we also have raidz2,3 techniques which includes parity bit.

Need to know :

root@mysrv1 # zpool create mirp1 raidz1 c5d3
invalid vdev specification: raidz1 requires at least 2 devices
root@mysrv1 #
root@mysrv1 # zpool create mirp1 raidz2 c5d3 c5d4
invalid vdev specification: raidz2 requires at least 3 devices
root@mysrv1 #
root@mysrv1 # zpool create mirp1 raidz3 c5d3 c5d4 c5d5
invalid vdev specification: raidz3 requires at least 4 devices
root@mysrv1 #

raidz1 -- even if one disk got failed, data can be retrieved using the other disk and parity.
raidz2 -- even if two disks got failed, data can be retrieved using the other disk and parity.
raidz3 -- even if three disks got failed, data can be retrieved using the other disk and parity.

root@mysrv1 #
root@mysrv1 # zpool list
NAME    SIZE  ALLOC   FREE  CAP  HEALTH  ALTROOT
rpool  9.94G  6.13G  3.81G  61%  ONLINE  -
root@mysrv1 #

To create a mirror pool we need to give mirror keyword.

root@mysrv1 #
root@mysrv1 # zpool create mirp mirror c5d1 c5d2 
root@mysrv1 #
root@mysrv1 # zpool status mirp
  pool: mirp
 state: ONLINE
 scan: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        mirp        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c5d1    ONLINE       0     0     0
            c5d2    ONLINE       0     0     0
errors: No known data errors
root@mysrv1 #

We can add a log volume to a mirror pool....

root@mysrv1 # zpool add mirp log c5d3
root@mysrv1 #
root@mysrv1 #
root@mysrv1 # zpool status mirp
  pool: mirp
 state: ONLINE
 scan: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        mirp        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c5d1    ONLINE       0     0     0
            c5d2    ONLINE       0     0     0
        logs
          c5d3      ONLINE       0     0     0
errors: No known data errors
root@mysrv1 #
root@mysrv1 # zpool remove mirp c5d3         ---- removing a log vol from pool..
root@mysrv1 #

NOTE : We cannot remove c#d#: only inactive hot spares, cache, top-level, or log devices can be removed

root@mysrv1 #
root@mysrv1 # zpool status mirp                   ---- since this is a log vol it got removed. 
 pool: mirp
 state: ONLINE
 scan: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        mirp        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c5d1    ONLINE       0     0     0
            c5d2    ONLINE       0     0     0
errors: No known data errors
root@mysrv1 #

Similarly we can add or remove cache to a mirror pool...

root@mysrv1 # zpool add mirp cache c5d3
root@mysrv1 #
root@mysrv1 #
root@mysrv1 # zpool status mirp
  pool: mirp
 state: ONLINE
 scan: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        mirp        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c5d1    ONLINE       0     0     0
            c5d2    ONLINE       0     0     0
         cache
           c5d3      ONLINE       0     0     0

errors: No known data errors
root@mysrv1 #
root@mysrv1 #
root@mysrv1 # zpool remove mirp c5d3
root@mysrv1 #

Considering a hard disk failure we can allocate a spare disk in advance , so that we can simply replace the faulted disk with the spare disk when a disk is failed.

root@mysrv1 # zpool add mirp spare c5d3
root@mysrv1 #
root@mysrv1 #
root@mysrv1 # zpool status mirp
  pool: mirp
 state: ONLINE
 scan: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        mirp        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c5d1    ONLINE       0     0     0
            c5d2    ONLINE       0     0     0
        spares
          c5d3      AVAIL
errors: No known data errors
root@mysrv1 #
root@mysrv1 #

Assume c5d2 disk failed and we need to replace it with hot spare....

root@mysrv1 # zpool replace mirp c5d2 c5d3
root@mysrv1 #
root@mysrv1 #
root@mysrv1 # zpool status mirp
  pool: mirp
 state: ONLINE
 scan: resilvered 104K in 0h0m with 0 errors on Mon Apr 14 07:22:03 2014
config:
        NAME         STATE     READ WRITE CKSUM
        mirp         ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            c5d1     ONLINE       0     0     0
            spare-1  ONLINE       0     0     0
              c5d2   ONLINE       0     0     0
              c5d3   ONLINE       0     0     0
        spares
          c5d3       INUSE     currently in use
errors: No known data errors
root@mysrv1 #
root@mysrv1 #

Once the spare disk is resilvered successfully , we can detach the failed disk.....

root@mysrv1 # zpool status mirp
  pool: mirp
 state: ONLINE
 scan: resilvered 104K in 0h0m with 0 errors on Mon Apr 14 07:22:03 2014
config:
        NAME         STATE     READ WRITE CKSUM
        mirp         ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            c5d1     ONLINE       0     0     0
            spare-1  ONLINE       0     0     0
              c5d2   ONLINE       0     0     0
              c5d3   ONLINE       0     0     0
        spares
          c5d3       INUSE     currently in use
errors: No known data errors
root@mysrv1 #
root@mysrv1 #
root@mysrv1 # zpool detach mirp c5d2         --- detaching failed disk...
root@mysrv1 #
root@mysrv1 #
root@mysrv1 # zpool status mirp
  pool: mirp
 state: ONLINE
 scan: resilvered 104K in 0h0m with 0 errors on Mon Apr 14 07:22:03 2014
config:
        NAME        STATE     READ WRITE CKSUM
        mirp        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c5d1    ONLINE       0     0     0
            c5d3    ONLINE       0     0     0
errors: No known data errorsroot@mysrv1 #

##################################################################################

ZFS (Zetabyte Filesystem) and zpool creation !!!

ZFS is a combined file system and logical volume manager. ZFS provides protection against data corruption, Hence ZFS is self-healing. ZFS includes the concepts of filesystem and volume management, snapshots and clones.

It supports high storage capacities, maximum file size can be 16 exbibytes (2^64 bytes) and maximum volume size of 256 zebibytes (2^78 bytes).

In ZFS we create zpools and on zpool we create filesystems. Compared to UFS, ZFS administration is much more easy and simple.

Creation of filesystem is so easy no need of partitions and mount points.
Simply we can create zpool by adding disks to pool. Since this is 1st post of ZFS let's go with simple tasks, just creation of pools.

root@mysrv1 #
root@mysrv1 # echo |format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
       0. c0d0 <SUN-DiskSlice-10GB cyl 2062 alt 2 hd 24 sec 424>
          /virtual-devices@100/channel-devices@200/disk@0
Specify disk (enter its number): Specify disk (enter its number):
root@mysrv1 #
root@mysrv1 #
root@mysrv1 # mkfile 100m c5d{1..5}
root@mysrv1 #
root@mysrv1 #
root@mysrv1 #
root@mysrv1 # ls -lrth /dev/dsk/c5*
-rw------T   1 root     root        100M Apr 14 03:54 /dev/dsk/c5d1
-rw------T   1 root     root        100M Apr 14 03:54 /dev/dsk/c5d2
-rw------T   1 root     root        100M Apr 14 03:54 /dev/dsk/c5d3
-rw------T   1 root     root        100M Apr 14 03:54 /dev/dsk/c5d4
-rw------T   1 root     root        100M Apr 14 03:54 /dev/dsk/c5d5
root@mysrv1 #
root@mysrv1 #
root@mysrv1 # zpool list       ------ to view all zpools (Here rpool means root pool, default pool comes with zfs)NAME    SIZE  ALLOC   FREE  CAP  HEALTH  ALTROOT
rpool  9.94G  6.12G  3.81G  61%  ONLINE  -
root@mysrv1 #

Now let us create a new zpool ,

root@mysrv1 # zpool create neo c5d1 c5d2
root@mysrv1 #
root@mysrv1 # zpool list
NAME    SIZE  ALLOC   FREE  CAP  HEALTH  ALTROOT
neo     190M    71K   190M   0%  ONLINE  -
rpool  9.94G  6.12G  3.81G  61%  ONLINE  -
root@mysrv1 #
root@mysrv1 # df -kh /neo          ------ default mount point will be created as per pool name..
Filesystem             size   used  avail capacity  Mounted on
neo                    158M    31K   158M     1%    /neo
root@mysrv1 # root@mysrv1 #
root@mysrv1 #
root@mysrv1 # zpool status neo            -------- To view status of neo pool 
 pool: neo
 state: ONLINE
 scan: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        neo         ONLINE       0     0     0
          c5d1      ONLINE       0     0     0
          c5d2      ONLINE       0     0     0
errors: No known data errors
root@mysrv1 #

Adding a new disk to your zpool ,

root@mysrv1 # zpool add neo c5d3
root@mysrv1 #
root@mysrv1 #
root@mysrv1 # zpool status neo
  pool: neo
 state: ONLINE
 scan: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        neo         ONLINE       0     0     0
          c5d1      ONLINE       0     0     0
          c5d2      ONLINE       0     0     0
          c5d3      ONLINE       0     0     0
errors: No known data errors
root@mysrv1 #
root@mysrv1 #

If you want to create a pool with your own mount point , we can mention with -m option.

root@mysrv1 #
root@mysrv1 # zpool create -m /pl1 pool c5d4
root@mysrv1 #
root@mysrv1 # df -kh
Filesystem             size   used  avail capacity  Mounted on
rpool/ROOT/s10s_u11wos_24a
                       9.8G   6.1G   3.7G    63%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   5.0G   504K   5.0G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
/platform/SUNW,Sun-Fire-T200/lib/libc_psr/libc_psr_hwcap1.so.1
                       9.8G   6.1G   3.7G    63%    /platform/sun4v/lib/libc_psr.so.1
/platform/SUNW,Sun-Fire-T200/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
                       9.8G   6.1G   3.7G    63%    /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd                       0K     0K     0K     0%    /dev/fd
swap                   5.0G    32K   5.0G     1%    /tmp
swap                   5.0G    48K   5.0G     1%    /var/run
rpool/export           9.8G    32K   3.7G     1%    /export
rpool/export/home      9.8G    31K   3.7G     1%    /export/home
rpool                  9.8G   106K   3.7G     1%    /rpool
neo                    253M    31K   253M     1%    /neo
pool                    63M    31K    63M     1%    /pl1
root@mysrv1 #
root@mysrv1 #

In the above df -kh output , we can see that our poolname is pool and the mount point is /pl1.

root@mysrv1 #
root@mysrv1 # zpool list
NAME    SIZE  ALLOC   FREE  CAP  HEALTH  ALTROOT
neo     285M  95.5K   285M   0%  ONLINE  -
pool     95M   116K  94.9M   0%  ONLINE  -
rpool  9.94G  6.12G  3.81G  61%  ONLINE  -
root@mysrv1 #
root@mysrv1 #
root@mysrv1 # zpool status
  pool: neo state: ONLINE
 scan: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        neo         ONLINE       0     0     0
          c5d1      ONLINE       0     0     0
          c5d2      ONLINE       0     0     0
          c5d3      ONLINE       0     0     0
errors: No known data errors
  pool: pool state: ONLINE
 scan: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        pool        ONLINE       0     0     0
          c5d4      ONLINE       0     0     0
errors: No known data errors
  pool: rpool state: ONLINE
 scan: none requested
config:
        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c0d0s0    ONLINE       0     0     0
errors: No known data errors
root@mysrv1 #
root@mysrv1 #

To destroy a pool ,

root@mysrv1 #
root@mysrv1 # zpool list
NAME    SIZE  ALLOC   FREE  CAP  HEALTH  ALTROOT
neo     285M  95.5K   285M   0%  ONLINE  -
pool     95M    94K  94.9M   0%  ONLINE  -
rpool  9.94G  6.12G  3.81G  61%  ONLINE  -
root@mysrv1 #
root@mysrv1 # zpool destroy pool
root@mysrv1 #                                                              --- "pool" is destroyed...
root@mysrv1 #
root@mysrv1 # zpool list
NAME    SIZE  ALLOC   FREE  CAP  HEALTH  ALTROOT
neo     285M  95.5K   285M   0%  ONLINE  -
rpool  9.94G  6.12G  3.81G  61%  ONLINE  -
root@mysrv1 #
root@mysrv1 #

###################################################################################

Configuring bash shell !!!

Let us play with bash shell, Usually admins like us prefer working with bash shell. It is more comfortable shell to make things simple.
Now let us configure bash shell in such a way that it provides much more flexibility.

Below are the main files for configuring bash shell...

[root@mysrv1 ~]# ls -lrth .bash*
-rw-r--r--. 1 root root  176 May 20  2009 .bash_profile
-rw-r--r--. 1 root root   17 Apr  6 04:30 .bash_logout
-rw-------. 1 root root 1.1K Apr  6 04:36 .bash_history
-rw-r--r--. 1 root root  211 Apr  6 04:43 .bashrc
[root@mysrv1 ~]#

.bash_profile    --- When we invoke the bash shell, it first attempts to read and execute commands                                 from this file.
.bash_logout     --- When we exits from bash shell, it read and execute commands from this file.
.bash_history    --- This file maintains the command history in bash shell.
.bashrc              --- In this file we can specify our own aliases and functions.

Usually for temporary login prompt we can give ,

[root@mysrv1 ~]# PS1='MY SERVER# '
MY SERVER#
MY SERVER#
MY SERVER# exit
logout
[root@mysrv1 ~]#
[root@mysrv1 ~]#              --- soon after exiting from the shell,changes in prompt will be reverted

But to maintain a particular prompt for a user we need to edit his .bashrc file so that whenever he invokes a shell mentioned prompt will be displayed.

[root@mysrv1 ~]# vi .bashrc
[root@mysrv1 ~]#
[root@mysrv1 ~]#
[root@mysrv1 ~]# cat .bashrc
# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
PS1='\h@root #'

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
[root@mysrv1 ~]#

Now to observe the change take a new session,

login as: root
root@192.168.18.133's password:
Last login: Sun Apr  6 04:40:30 2014 from 192.168.18.1
mysrv1@root #
mysrv1@root #
mysrv1@root #

So from now onwards root user's login prompt will be the displayed output.

Now let us see the other use of .bashrc file.This time I will create new alias for different commands for ease purpose.

[root@mysrv1 ~]# alias                           --- cmd to see defined aliases
alias cp='cp -i'
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias mv='mv -i'
alias rm='rm -i'
alias which='alias | /usr/bin/which --tty-only --read-alias --show-dot --show-tilde'
[root@mysrv1 ~]#

[root@mysrv1 ~]#
[root@mysrv1 ~]# alias c=clear
[root@mysrv1 ~]#
[root@mysrv1 ~]# c

[root@mysrv1 ~]#
[root@mysrv1 ~]#                    ------ Now the screen is cleared
[root@mysrv1 ~]#

[root@mysrv1 ~]# exit            ------- exit from current session
exit
[root@mysrv1 ~]#
[root@mysrv1 ~]#
[root@mysrv1 ~]# c
-bash: c: command not found      ------- this time my alias won't work, since it is set temporary.
[root@mysrv1 ~]#

So now to set a alias permanently we need to give alias in .bashrc file.

[root@mysrv1 ~]# bash
[root@mysrv1 ~]#
[root@mysrv1 ~]#
[root@mysrv1 ~]# vi .bashrc
[root@mysrv1 ~]#
[root@mysrv1 ~]#
[root@mysrv1 ~]# cat .bashrc
# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
alias c='clear'                 --- c is for clear screen
alias i='ifconfig'              --- i is for ifconfig

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
[root@mysrv1 ~]#

Let us try checking it...

[root@mysrv1 ~]# su -
[root@mysrv1 ~]#
[root@mysrv1 ~]# c            --- for c screen is cleared...
[root@mysrv1 ~]#
[root@mysrv1 ~]#
[root@mysrv1 ~]#

[root@mysrv1 ~]# i            --- for i ifconfig is executed...
eth0      Link encap:Ethernet  HWaddr 00:0C:29:4F:66:DD
          inet addr:192.168.18.133  Bcast:192.168.18.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe4f:66dd/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4624 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2877 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:436891 (426.6 KiB)  TX bytes:457876 (447.1 KiB)

eth1      Link encap:Ethernet  HWaddr 00:0C:29:4F:66:E7
          inet addr:192.168.18.134  Bcast:192.168.18.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe4f:66e7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:514 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:49306 (48.1 KiB)  TX bytes:2518 (2.4 KiB)

eth2      Link encap:Ethernet  HWaddr 00:0C:29:4F:66:F1
          inet addr:192.168.18.132  Bcast:192.168.18.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe4f:66f1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:513 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:49246 (48.0 KiB)  TX bytes:2333 (2.2 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:14 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:822 (822.0 b)  TX bytes:822 (822.0 b)

[root@mysrv1 ~]#

I will exit from bash and again I will invoke bash shell to check....

[root@mysrv1 ~]# exit
logout
[root@mysrv1 ~]#
[root@mysrv1 ~]#
[root@mysrv1 ~]#
[root@mysrv1 ~]# i
bash: i: command not found
[root@mysrv1 ~]# bash
[root@mysrv1 ~]#
[root@mysrv1 ~]# i
eth0      Link encap:Ethernet  HWaddr 00:0C:29:4F:66:DD
          inet addr:192.168.18.133  Bcast:192.168.18.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe4f:66dd/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4666 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2907 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:440409 (430.0 KiB)  TX bytes:463064 (452.2 KiB)

eth1      Link encap:Ethernet  HWaddr 00:0C:29:4F:66:E7
          inet addr:192.168.18.134  Bcast:192.168.18.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe4f:66e7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:515 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:49366 (48.2 KiB)  TX bytes:2518 (2.4 KiB)

eth2      Link encap:Ethernet  HWaddr 00:0C:29:4F:66:F1
          inet addr:192.168.18.132  Bcast:192.168.18.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe4f:66f1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:514 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:49306 (48.1 KiB)  TX bytes:2333 (2.2 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:14 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:822 (822.0 b)  TX bytes:822 (822.0 b)

[root@mysrv1 ~]#
[root@mysrv1 ~]# c
[root@mysrv1 ~]#
[root@mysrv1 ~]#
[root@mysrv1 ~]#   --- So even after logging into a new session of bash we can able to use alias....
[root@mysrv1 ~]#

.bash_profile is starting file which it will read and execute when we invoke bash shell.
Observe the content of .bash_profile file , it internally executes .bashrc file and its 
commands like PATH...

[root@mysrv1 ~]# cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin                ---- Here we can export EDITOR by giving EDITOR=vi

export PATH
[root@mysrv1 ~]#

.bash_history maintains history of commands that are used in bash shell...

[root@mysrv1 ~]# tail .bash_history
bash
vi .bash_logout
bash
cd /etc/
ls -lrth isuue*
ls -lrth issue*
c
bash
[root@mysrv1 ~]#

.bash_logout file is useful utility of bash shell with the which help of which we can take 
backup of some files and output of some commands automatically before exiting from 
bash shell.This file gets executed when we exit from bash shell.

[root@mysrv1 ~]#
[root@mysrv1 ~]# cat .bash_logout
# ~/.bash_logout
[root@mysrv1 ~]#
[root@mysrv1 ~]# vi .bash_logout
[root@mysrv1 ~]# 
[root@mysrv1 ~]#
[root@mysrv1 ~]# cat .bash_logout
# ~/.bash_logout
cp /etc/passwd /etc/passwd_old                 --- file which I need backup.
ls -lrth >>/tmp/oo                                       --- command which I need output.
[root@mysrv1 ~]#
[root@mysrv1 ~]# ls -lrth
total 96K
-rw-r--r--. 1 root root 9.0K Mar 15 14:37 install.log.syslog
-rw-r--r--. 1 root root  39K Mar 15 14:41 install.log
-rw-------. 1 root root 2.3K Mar 15 14:41 anaconda-ks.cfg
drwxr-xr-x. 3 root root 4.0K Apr  4 03:34 test
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Templates
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Public
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Downloads
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Videos
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Pictures
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Music
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Documents
drwxr-xr-x. 2 root root 4.0K Apr  5 05:35 Desktop
[root@mysrv1 ~]#
[root@mysrv1 ~]#
[root@mysrv1 ~]# ls -lrth /etc/passwd*                      --- now there is no passwd_old file
-rw-r--r--. 1 root root 1.4K Mar 15 14:42 /etc/passwd-
-rw-r--r--. 1 root root 1.4K Mar 15 14:42 /etc/passwd.OLD
-rw-r--r--. 1 root root 1.4K Mar 15 14:42 /etc/passwd
[root@mysrv1 ~]#
[root@mysrv1 ~]#
[root@mysrv1 ~]# cat /tmp/oo                                     --- now there is no /tmp/oo file
cat: /tmp/oo: No such file or directory
[root@mysrv1 ~]#

Now I will exit from my bash shell to check my new .bash_logout file

[root@mysrv1 ~]# exit
[root@mysrv1 ~]#
[root@mysrv1 ~]# logout
[root@localhost ~]#
[root@localhost ~]#
[root@localhost ~]# su -
[root@mysrv1 ~]#
[root@mysrv1 ~]# cat /tmp/oo               ----- /tmp/oo is created and grabbed the output of ls -lrth
total 96K
-rw-r--r--. 1 root root 9.0K Mar 15 14:37 install.log.syslog
-rw-r--r--. 1 root root  39K Mar 15 14:41 install.log
-rw-------. 1 root root 2.3K Mar 15 14:41 anaconda-ks.cfg
drwxr-xr-x. 3 root root 4.0K Apr  4 03:34 test
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Templates
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Public
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Downloads
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Videos
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Pictures
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Music
drwxr-xr-x. 2 root root 4.0K Apr  4 03:57 Documents
drwxr-xr-x. 2 root root 4.0K Apr  5 05:35 Desktop
[root@mysrv1 ~]#
[root@mysrv1 ~]#
[root@mysrv1 ~]# ls -lrth /etc/passwd*
-rw-r--r--. 1 root root 1.4K Mar 15 14:42 /etc/passwd-
-rw-r--r--. 1 root root 1.4K Mar 15 14:42 /etc/passwd.OLD
-rw-r--r--. 1 root root 1.4K Mar 15 14:42 /etc/passwd
-rw-r--r--. 1 root root 1.4K Apr  6 05:42 /etc/passwd_old    ----- backup of passwd file is taken
[root@mysrv1 ~]#
[root@mysrv1 ~]#

Thus We can configure a bash shell as per our requirement to make our login shell much more convenient and tasks can be performed much more easier.

################################################################################

awk and sed !!!

awk is a acronym. It came from its authors Alfred Aho, Peter Weinberger, and Brian Kernighan

A  -- Aho
W -- Weinberger
K -- Kernighan

In scripting , it useful to extract data by a command in a required format.

NF= number of fiels
NR= number of records

\t --- tab space
\n --- new line

options,
  -F to specify delimiter

Let us learn this with some examples.

root@mysrv1:~# cd /test
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# touch {awk,sed}_{ex1,ex2,ex3}
root@mysrv1:/test#
root@mysrv1:/test# ls -lrth
total 10
-rw-r--r--   1 root     root         164 Apr  5 03:20 awk1
-rw-r--r--   1 root     root         164 Apr  5 03:20 cpawk
-rw-r--r--   1 root     root           0 Apr  5 03:21 awk_ex1
-rw-r--r--   1 root     root           0 Apr  5 03:21 awk_ex2
-rw-r--r--   1 root     root           0 Apr  5 03:21 awk_ex3
-rw-r--r--   1 root     root           0 Apr  5 03:21 sed_ex1
-rw-r--r--   1 root     root           0 Apr  5 03:21 sed_ex2
-rw-r--r--   1 root     root           0 Apr  5 03:21 sed_ex3
root@mysrv1:/test#

To print a particular field from the output of a command.

root@mysrv1:/test# ls -lrth |awk '{print $4}'              ----- $4 specifies 4rth filed.
root
root
root
root
root
root
root
root
root@mysrv1:/test#
root@mysrv1:/test#

root@mysrv1:/test# ls -lrth |awk '{print $9}'              ----- $9 specifies 9th filed.
awk1
cpawk
awk_ex1
awk_ex2
awk_ex3
sed_ex1
sed_ex2
sed_ex3
root@mysrv1:/test#
root@mysrv1:/test#

If we need some text between required fields,

root@mysrv1:/test#
root@mysrv1:/test# ls -lrth |awk '{print $4,"is the owner of file",$9}'
root is the owner of file awk1
root is the owner of file cpawk
root is the owner of file awk_ex1
root is the owner of file awk_ex2
root is the owner of file awk_ex3
root is the owner of file sed_ex1
root is the owner of file sed_ex2
root is the owner of file sed_ex3
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# ls -lrth |awk '{print "file",$9,"is created on",$6,$7}'
file  is created on
file awk1 is created on Apr 5
file cpawk is created on Apr 5
file awk_ex1 is created on Apr 5
file awk_ex2 is created on Apr 5
file awk_ex3 is created on Apr 5
file sed_ex1 is created on Apr 5
file sed_ex2 is created on Apr 5
file sed_ex3 is created on Apr 5
root@mysrv1:/test#

Let us create a file in which data contains some delimiters and different fileds and rows.
So that we can discuss awk briefly.... 

root@mysrv1:/test# ls
awk_ex1  awk_ex2  awk_ex3  awk1     cpawk    sed_ex1  sed_ex2  sed_ex3
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# cat awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#
root@mysrv1:/test#

To print required fileds by ignoring delimiter..

root@mysrv1:/test#
root@mysrv1:/test# cat awk1 | awk -F: '{print $1,$5}'       -F to specify delimiter to be ignored
oraz10 /oracle/Z10
sapadm /home/sapadm
daaadm /home/daaadm
test /home/test
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# awk -F: '{print $1,$5}' awk1
oraz10 /oracle/Z10
sapadm /home/sapadm
daaadm /home/daaadm
test /home/test
root@mysrv1:/test#

To print a specific record containing a specific text.

root@mysrv1:/test# cat awk1 | awk '/oraz10/'                 ----my pattern is oraz10
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# awk '/oraz10/' awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# awk -F: '$3 >102' awk1           ----to print records with filed 3 value >102
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# awk -F: '$4 ==100' awk1        ----to print records with filed 4 value =100
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
root@mysrv1:/test#

Let us go more in detail by specifying entire requirements at once using BEGIN and END.

root@mysrv1:/test# cat awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#

root@mysrv1:/test# awk -F: 'BEGIN {print "user\tuid\tgid\thome";}
{print $1"\t"$3"\t"$4"\t"$5;}
END{print;}' awk1
user        uid      gid        home
oraz10   104      102       /oracle/Z10
sapadm  102      100      /home/sapadm
daaadm  103      100      /home/daaadm
test         60004  10        /home/test

                        #####%%%@@@%%%@@@%%%@@@%%%#####

SED ---- STREAM EDITOR

sed is helpful to edit content of a file. It is a temporary effect , if we are ok with the output of sed command then we can save the file accordingly.

Usually sed is more helpful in scripting , to edit or view a particular file's output as per requirement.

The main and most useful utility from sed is to insert/append data after a particular line and to substitute a particular word in the file...

sed options :
/g - Global        --- usually we  use while substitution.
/p - Print
-n --- squeezes the errors...
-d --- to delete a record. Now let us see with some examples 

root@mysrv1:/test#
root@mysrv1:/test# cat awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#

root@mysrv1:/test#
root@mysrv1:/test# sed -n '1p' awk1                            ----- to print 1st record
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# sed -n '1,3p' awk1                         ----- to print 1st to 3rd record
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
root@mysrv1:/test#
root@mysrv1:/test# sed -ne '1p' -e '3p' awk1               ----- to print 1st and 3rd record
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh

daaadm:x:103:100:/home/daaadm:/usr/bin/csh
root@mysrv1:/test#

To substitute a word ,

root@mysrv1:/test#
root@mysrv1:/test# sed 's/sapadm/oraadm/' awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
oraadm:x:102:100:/home/sapadm:/usr/bin/csh              --- here I substituted sapadm with oraadm
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#

Observe difference when we give /g for globally change....

root@mysrv1:/test#
root@mysrv1:/test# sed 's/sapadm/oraadm/g' awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
oraadm:x:102:100:/home/oraadm:/usr/bin/csh     --- here I substituted sapadm with oraadm globally
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#
root@mysrv1:/test#

Appending and Inserting data using sed...

root@mysrv1:/test#
root@mysrv1:/test# sed '2a\
> d10:23:gg:vim' awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
d10:23:gg:vim                                                          ------- Appended after 2nd line (since 2a)
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#

root@mysrv1:/test#
root@mysrv1:/test# sed '2i\
> orarep:x:105:102:/ora/file:/usr/bin/csh' awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
orarep:x:105:102:/ora/file:/usr/bin/csh                  ------- Inserting after 2nd line (since 2a)
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#

As I told sed is only a temporary effect , now observe below output the original awk1 file is not changed even after inserting a line.....

root@mysrv1:/test#
root@mysrv1:/test# cat awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#

Scripting Basics !!!

Basics which are helpful in Shell Scripting :

First and most common thing which is more helpful in scripting is ,

1) echo $?      -----  It validates any particular command or a step in a script. If return code is 0 then
statement or command is valid or else it throws a big value for invalid command.

Mostly used in "if" loops.

Let me explain with following example :

root@mysrv1:~# 
root@mysrv1:~#  i+-                        ----- It is a invalid command so echo $? throws error
bash : i+-: command not found
root@mysrv1:~# 
root@mysrv1:~#  echo $?
127
root@mysrv1:~#  i=0                       ----- Since this is valid echo $? throws 0 return code.
root@mysrv1:~# 
root@mysrv1:~# echo $?
0
root@mysrv1:~# 
root@mysrv1:~# 

In "if" loops we use this to say if condition is true so perform the task , if not got to else statement. So the command or the statement in the script which is before " if " condition is valid then if [$?==0] is true so script goes with statement 1 otherwise statement 2.

if [$?==0]
statement 1
else
statement 2

2) Always true while loop,

In some cases we need a task to be done and so we require a true condition so that the task will be given in loop to run in some time intervals..

Example for a always true condition in while loop :     while 2>1

root@mysrv1:~# 

root@mysrv1:~# while 2>1
>do
>metastat -c
>sleep 5
>done
d20              m   30GB d21 d22 (resync-21%)
    d21          s   30GB c1t0d0s1
    d22          s   30GB c1t1d0s1
d10              m   14GB d12 d11 (resync-35%)
    d11          s   14GB c1t0d0s0
    d12          s   14GB c1t1d0s0
d20              m   30GB d21 d22 (resync-21%)
    d21          s   30GB c1t0d0s1
    d22          s   30GB c1t1d0s1
d10              m   14GB d12 d11 (resync-36%)
    d11          s   14GB c1t0d0s0
    d12          s   14GB c1t1d0s0
d20              m   30GB d21 d22 (resync-21%)
    d21          s   30GB c1t0d0s1
    d22          s   30GB c1t1d0s1
d10              m   14GB d12 d11 (resync-36%)
    d11          s   14GB c1t0d0s0
    d12          s   14GB c1t1d0s0
d20              m   30GB d21 d22 (resync-22%)
    d21          s   30GB c1t0d0s1
    d22          s   30GB c1t1d0s1
d10              m   14GB d12 d11 (resync-37%)
    d11          s   14GB c1t0d0s0
    d12          s   14GB c1t1d0s0
^C

root@mysrv1:~# 

root@mysrv1:~# 

Since this condition is always true , this loop executes repeatedly for every 5 secs interval until we interrupt it.

3) Powerful and most helpful in scripts : for loop.

For iterations we need "for" loop. To do same particular thing, like to execute same command for "n" different files or in "n" different servers we use for loops.

For loops in scripting is mainly to reduce burden of performing same task "n" number of times.

We need uname of 20 different servers, we need to list the ip's in a file. Lets say "ips" is file.

root@mysrv1:~# 

root@mysrv1:~# for i in `cat ips`
>do
>ssh $i ;uname -a;
>exit
>done

################################################################################