awk and sed !!!

awk is a acronym. It came from its authors Alfred Aho, Peter Weinberger, and Brian Kernighan

A  -- Aho
W -- Weinberger
K -- Kernighan

In scripting , it useful to extract data by a command in a required format.

NF= number of fiels
NR= number of records

\t --- tab space
\n --- new line

options,
  -F to specify delimiter

Let us learn this with some examples.

root@mysrv1:~# cd /test
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# touch {awk,sed}_{ex1,ex2,ex3}
root@mysrv1:/test#
root@mysrv1:/test# ls -lrth
total 10
-rw-r--r--   1 root     root         164 Apr  5 03:20 awk1
-rw-r--r--   1 root     root         164 Apr  5 03:20 cpawk
-rw-r--r--   1 root     root           0 Apr  5 03:21 awk_ex1
-rw-r--r--   1 root     root           0 Apr  5 03:21 awk_ex2
-rw-r--r--   1 root     root           0 Apr  5 03:21 awk_ex3
-rw-r--r--   1 root     root           0 Apr  5 03:21 sed_ex1
-rw-r--r--   1 root     root           0 Apr  5 03:21 sed_ex2
-rw-r--r--   1 root     root           0 Apr  5 03:21 sed_ex3
root@mysrv1:/test#

To print a particular field from the output of a command.

root@mysrv1:/test# ls -lrth |awk '{print $4}'              ----- $4 specifies 4rth filed.
root
root
root
root
root
root
root
root
root@mysrv1:/test#
root@mysrv1:/test#

root@mysrv1:/test# ls -lrth |awk '{print $9}'              ----- $9 specifies 9th filed.
awk1
cpawk
awk_ex1
awk_ex2
awk_ex3
sed_ex1
sed_ex2
sed_ex3
root@mysrv1:/test#
root@mysrv1:/test#

If we need some text between required fields,

root@mysrv1:/test#
root@mysrv1:/test# ls -lrth |awk '{print $4,"is the owner of file",$9}'
root is the owner of file awk1
root is the owner of file cpawk
root is the owner of file awk_ex1
root is the owner of file awk_ex2
root is the owner of file awk_ex3
root is the owner of file sed_ex1
root is the owner of file sed_ex2
root is the owner of file sed_ex3
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# ls -lrth |awk '{print "file",$9,"is created on",$6,$7}'
file  is created on
file awk1 is created on Apr 5
file cpawk is created on Apr 5
file awk_ex1 is created on Apr 5
file awk_ex2 is created on Apr 5
file awk_ex3 is created on Apr 5
file sed_ex1 is created on Apr 5
file sed_ex2 is created on Apr 5
file sed_ex3 is created on Apr 5
root@mysrv1:/test#

Let us create a file in which data contains some delimiters and different fileds and rows.
So that we can discuss awk briefly.... 

root@mysrv1:/test# ls
awk_ex1  awk_ex2  awk_ex3  awk1     cpawk    sed_ex1  sed_ex2  sed_ex3
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# cat awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#
root@mysrv1:/test#

To print required fileds by ignoring delimiter..

root@mysrv1:/test#
root@mysrv1:/test# cat awk1 | awk -F: '{print $1,$5}'       -F to specify delimiter to be ignored
oraz10 /oracle/Z10
sapadm /home/sapadm
daaadm /home/daaadm
test /home/test
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# awk -F: '{print $1,$5}' awk1
oraz10 /oracle/Z10
sapadm /home/sapadm
daaadm /home/daaadm
test /home/test
root@mysrv1:/test#

To print a specific record containing a specific text.

root@mysrv1:/test# cat awk1 | awk '/oraz10/'                 ----my pattern is oraz10
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# awk '/oraz10/' awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# awk -F: '$3 >102' awk1           ----to print records with filed 3 value >102
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# awk -F: '$4 ==100' awk1        ----to print records with filed 4 value =100
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
root@mysrv1:/test#

Let us go more in detail by specifying entire requirements at once using BEGIN and END.

root@mysrv1:/test# cat awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#

root@mysrv1:/test# awk -F: 'BEGIN {print "user\tuid\tgid\thome";}
{print $1"\t"$3"\t"$4"\t"$5;}
END{print;}' awk1
user        uid      gid        home
oraz10   104      102       /oracle/Z10
sapadm  102      100      /home/sapadm
daaadm  103      100      /home/daaadm
test         60004  10        /home/test

                        #####%%%@@@%%%@@@%%%@@@%%%#####

SED ---- STREAM EDITOR

sed is helpful to edit content of a file. It is a temporary effect , if we are ok with the output of sed command then we can save the file accordingly.

Usually sed is more helpful in scripting , to edit or view a particular file's output as per requirement.

The main and most useful utility from sed is to insert/append data after a particular line and to substitute a particular word in the file...

sed options :
/g - Global        --- usually we  use while substitution.
/p - Print
-n --- squeezes the errors...
-d --- to delete a record. Now let us see with some examples 

root@mysrv1:/test#
root@mysrv1:/test# cat awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#

root@mysrv1:/test#
root@mysrv1:/test# sed -n '1p' awk1                            ----- to print 1st record
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
root@mysrv1:/test#
root@mysrv1:/test#
root@mysrv1:/test# sed -n '1,3p' awk1                         ----- to print 1st to 3rd record
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
root@mysrv1:/test#
root@mysrv1:/test# sed -ne '1p' -e '3p' awk1               ----- to print 1st and 3rd record
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh

daaadm:x:103:100:/home/daaadm:/usr/bin/csh
root@mysrv1:/test#

To substitute a word ,

root@mysrv1:/test#
root@mysrv1:/test# sed 's/sapadm/oraadm/' awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
oraadm:x:102:100:/home/sapadm:/usr/bin/csh              --- here I substituted sapadm with oraadm
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#

Observe difference when we give /g for globally change....

root@mysrv1:/test#
root@mysrv1:/test# sed 's/sapadm/oraadm/g' awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
oraadm:x:102:100:/home/oraadm:/usr/bin/csh     --- here I substituted sapadm with oraadm globally
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#
root@mysrv1:/test#

Appending and Inserting data using sed...

root@mysrv1:/test#
root@mysrv1:/test# sed '2a\
> d10:23:gg:vim' awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
d10:23:gg:vim                                                          ------- Appended after 2nd line (since 2a)
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#

root@mysrv1:/test#
root@mysrv1:/test# sed '2i\
> orarep:x:105:102:/ora/file:/usr/bin/csh' awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
orarep:x:105:102:/ora/file:/usr/bin/csh                  ------- Inserting after 2nd line (since 2a)
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#

As I told sed is only a temporary effect , now observe below output the original awk1 file is not changed even after inserting a line.....

root@mysrv1:/test#
root@mysrv1:/test# cat awk1
oraz10:x:104:102:/oracle/Z10:/usr/bin/csh
sapadm:x:102:100:/home/sapadm:/usr/bin/csh
daaadm:x:103:100:/home/daaadm:/usr/bin/csh
test:x:60004:10:/home/test:/bin/csh
root@mysrv1:/test#