Difference between revisions of "Linux-AWK/C2/Loops-in-awk/English"

From Script | Spoken-Tutorial
Jump to: navigation, search
(Created page with "'''Title of script: Loops in Awk''' '''Author: Antara Roy Choudhury''' '''Keywords: for, while, do-while, next, nextfile''' {| style="border-spacing:0;" | style="backgrou...")
 
Line 1: Line 1:
'''Title of script: Loops in Awk'''
+
'''Title of script:''' Built-in variables and awk Script
  
'''Author: Antara Roy Choudhury'''
+
'''Author:''' Antara Roy Choudhury
  
'''Keywords: for, while, do-while, next, nextfile'''
+
'''Keywords:''' Built-in variables, RS, ORS, NR, NF, FS, OFS, FILENAME
  
  
Line 13: Line 13:
 
|-
 
|-
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 1: Introduction
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 1: Introduction
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Welcome to this spoken tutorial on '''Loops''' in '''awk'''.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Welcome to the spoken tutorial on '''awk built-in variables''' and '''awk script.'''
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 2: Learning Objective
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 2: Learning Objectives
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| In this tutorial we will learn about-
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| In this tutorial we will learn about
  
  
* '''while'''
+
* '''Built-in variables '''
* '''do-while'''
+
* '''awk script'''
* '''for '''
+
* and more '''looping constructs '''in''' awk'''
+
  
 
We will do this through some examples.
 
We will do this through some examples.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 3: System requirement
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 3a: System requirement
 
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| To record this tutorial, I am using  
 
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| To record this tutorial, I am using  
  
 
* '''Ubuntu Linux 16.04 OS '''and''' '''
 
* '''Ubuntu Linux 16.04 OS '''and''' '''
* '''gedit text editor 3.20.1'''
+
* '''gedit text editor version 3.20.1'''
 
+
* You can use any text editor of your choice.
+
  
  
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 4: Pre-requisite
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 3b: Code Files
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| * To practice this tutorial, you should have gone through the previous '''awk''' tutorials on our website.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| The files used in this tutorial are available in the '''Code Files''' link on this tutorial page.
* You should have familiarity with any '''programming language''' like C or C++
+
* If not, then please go through the corresponding tutorials on our website.
+
  
  
 +
Please download and use them.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 5: Code Files
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 4: Prerequisite
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| The files used in this tutorial are available in the '''Code Files''' link on this tutorial page.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| To practice this tutorial, you should have gone through the earlier '''awk tutorials''' on this website.
  
  
Please download and extract them.
+
If not, then please go through the corresponding tutorials on this website.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 6: Loops
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 5: awk built-in variables
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| A '''loop''' allows us to perform one or more actions repeatedly.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| First, let us see some of the '''built-in variables '''in '''awk'''.  
  
  
'''while, do-while '''and''' for '''are the available '''loops''' in '''awk'''.  
+
* Capital '''RS''' specifies the '''record separator''' in an '''input''' file. By default, it is '''newline'''.
 +
* Capital '''FS''' specifies the '''field separator '''in an '''input''' file.
 +
 
 +
By default, the value of '''FS''' is a '''whitespace'''.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 7: While loop
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 5: awk built-in variables
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| The syntax of '''while''' '''loop''' can be seen here.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| * Capital '''ORS''' defines the '''output record separator'''.
  
 +
By default, it is '''newline'''.
  
'''While loop '''first checks whether the specified '''condition''' is true.  
+
* Capital '''OFS''' defines the '''output field separator'''.
  
If yes, then it executes the code within the '''body.'''
+
By default, it is '''whitespace.'''
  
  
This '''loop '''will be repeated as long as the specified '''while condition '''is '''true'''.
+
Let us understand the meaning of each of these.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show awkdemo.txt file in gedit
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show awkdemo.txt in Gedit
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| We will use the same '''awkdemo.txt '''file, that we have used earlier.  
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Let us have a look at the '''awkdemo''' file now.
 +
 
 +
 
 +
When we are processing this '''awkdemo''' file with '''awk''' command, this becomes our '''input '''file.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| show while_loop.awk in Gedit
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight appropriately
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| I have already written a script named '''while_loop.awk'''
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Observe that all the records are separated from each other by a '''newline character'''.
  
  
The same is available in the '''Code Files''' link of this tutorial.
+
'''newline''' is the default value for '''record separator RS variable. '''
  
|-
+
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Point to the pipe symbol
+
So, there is no need to do anything else.
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Here we have set the '''field separator''' as '''Pipe symbol.'''
+
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight i = 1
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight vertical bar | character
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Initially, we have to the set the value of the '''loop variable i '''as 1.  
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Notice that all the fields are separated by the '''pipe symbol'''.  
  
  
 +
How can we inform '''awk '''about it?
  
 +
 +
Let us see.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight f = 1
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 6: How to reset value of FS variable?
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Here, we have taken one more variable '''f''' and initialized it to 1.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| By default, any number of '''spaces''' or a '''tabs''' separate the fields.
  
  
Variable '''f''' represents the '''field counter '''or the position of the '''fields''' for each record.
+
We can reset this with the help of '''hyphen capital F''' option as learnt in our earlier tutorials.
  
|-
+
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| <nowiki>Highlight while (i<=3)</nowiki>
+
Or else, we can reset this in the '''BEGIN section '''with the use of '''FS''' '''variable'''.  
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Now, in the '''while condition''', we check if '''i''' is less than or equal to 3.
+
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight Print statement
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"|  
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| If yes, then it will print the value in the '''f'''<sup>th</sup>''' field, '''for that record in '''awkdemo.txt '''file.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Let us do this through an example.
 +
 
 +
 
 +
Suppose, I want to find out the name of students who are getting a stipend of more than Rs.5000.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight f++
+
| style="border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Open the terminal
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Then we will increase the '''field counter f''' by 1.
+
| style="border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Open the '''terminal''' by pressing '''CTRL, ALT''' and '''T''' keys.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight i++
+
| style="border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| <nowiki>cd /<saved folder></nowiki>
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| After that, we will also increment the value of '''loop''' variable '''i''' by 1.
+
| style="border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Go to the folder in which you downloaded and extracted the '''Code Files''' using '''cd command.'''
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight printf("\n")
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| awk 'BEGIN{FS="|"} $6>5000 {print $2,$6}' awkdemo.txt
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| This '''printf''' is for printing a '''newline character''' at the end of each row.
+
| style="border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Type the command as shown here.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"|  
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight {FS="|"} area
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| This '''loop''' will be executed for all the records in the '''awkdemo.txt''' file.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Here in the '''BEGIN''' section, we have assigned the value of '''FS''' as a '''pipe symbol.'''
  
  
Which means the first 3 fields will get printed for each record.
+
Similarly, we can modify '''RS variable.'''
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"|  
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| <nowiki>[Enter]</nowiki>
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Let’s execute this code now.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Press '''Enter''' to execute the command.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Open the terminal
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Open the '''terminal''' by pressing '''Ctrl, Alt''' and '''T''' keys.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| The output shows the list of students who are receiving more than Rs.5000 as a stipend.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| <nowiki>cd /<saved folder></nowiki>
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight appropriately
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Go to the folder in which you downloaded and extracted the '''Code Files''' using '''cd command'''
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Here the '''name '''field and the '''stipend '''field is separated by a blank '''space'''.
 +
 
 +
 
 +
Also, all the records are separated by a '''newline character.'''
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Type in terminal:
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide
  
awk -f while_loop.awk awkdemo.txt
 
  
<nowiki>[Enter]</nowiki>
 
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Now type:
 
  
'''awk space minus small f while_loop.awk space awkdemo.txt'''
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Suppose we want '''colon '''as the '''output field separator.'''
  
  
Press '''Enter'''
+
And double '''newline '''as '''output record separator'''.
  
|-
+
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output
+
How can we do this? Let us see.
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Observe that we get the first three '''fields''' of all the rows in the output.
+
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"|  
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| In terminal
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Let us do the same with the '''do-while loop'''.  
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| In the '''terminal''', press the '''up arrow '''key to get the previously executed command.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 8: Do While Loop
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Modify the previous command
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| The syntax of '''do-while loop''' can be seen here.
+
  
  
The&nbsp;'''do-while&nbsp;loop''' always executes the&nbsp;code inside the '''body''' once.
+
awk 'BEGIN{FS="|";OFS=":"; ORS="\n\n"} $6>5000 {print $2, $6}' awkdemo.txt
  
 +
<nowiki>[Enter]</nowiki>
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Modify the command as shown here.
  
Then it checks the specified '''condition.'''
 
  
 +
And then press '''Enter.'''
  
And repeats the&nbsp;code inside the '''body''',&nbsp;as long as the specified&nbsp;'''condition'''&nbsp;is '''true'''.
+
|-
 +
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| We get the output in the desired format.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| show do_loop.awk in Gedit
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show sample.txt
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| I have already written a script and named it as '''do_loop.awk'''
+
  
  
The same is available in the '''Code Files '''link.  
+
Point or highlight
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Now, suppose our new input file is '''sample.txt.'''
 +
 
 +
 
 +
Observe that the '''field separator '''here is '''newline '''and '''record separator '''is double '''newline.'''
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the contents and highlight appropriately
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"|  
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| In this code, these are the '''statements '''within the '''do loop, '''which will be executed first.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| How can we extract the roll no. and name information from this file?
  
  
This is the '''condition '''that will be checked.
+
Yes, you have guessed correctly.
  
  
After that, the '''statements '''inside the '''loop '''will be executed repeatedly as long as the '''condition '''is '''true'''.
+
We have to modify both '''FS''' and '''RS''' '''variables'''.
 +
 
 +
 
 +
Pause this tutorial and do this as an assignment.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"|  
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 7:
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| This '''loop''' will iterate for all the records in the '''awkdemo.txt''' file
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Next, let us see other '''built-in variables.'''
  
  
Which means the first 3 fields will get printed for all the records.
+
* Capital '''NR''' gives us the '''Number of Records''' processed by '''awk'''
 +
* Capital '''NF''' gives the '''Number of Fields '''in the current record
  
|-
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Type:
 
  
awk -f do_loop.awk awkdemo.txt
 
  
<nowiki>[Enter]</nowiki>
+
|-
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Let’s switch to the '''terminal. '''Let me clear the''' terminal.'''
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Let us see one example of this.  
  
Now type:
 
  
 +
Suppose, we want to find incomplete lines in the file.
  
'''awk space -f do_loop.awk awkdemo.txt'''
 
  
and press '''Enter'''
+
Here, incomplete line means it has less than the normal 6 fields.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Scroll up and show outputs of while loop and do loop
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Switch to the terminal and clear it
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| We get the same output.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Switch to the '''terminal'''.
  
  
Then why do we have both '''while '''and '''do-while loops'''?
+
Let me clear the terminal using '''Ctrl''' and '''L''' keys
 
+
Let us understand the difference.
+
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| In while_loop.awk, assign i=4
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Type
 +
 
 +
awk 'BEGIN{FS="|"} NF !=6 {print NR, $0}' awkdemo.txt
 +
 
 +
<nowiki>[Enter]</nowiki>
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Type the command as shown.
  
<nowiki>[code uploaded as while_loop_mod.awk]</nowiki>
 
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Switch to the file '''while_loop.awk'''
 
  
  
Now, change the value of '''loop counter i '''from 1 to 4.
 
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the command and highlight appropriate areas as per narration
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| As the fields are separated by '''pipe '''symbol, set the '''FS''' value to '''pipe''' symbol in the '''BEGIN section.'''
  
<nowiki>(i<=3)</nowiki>
 
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| This will make the specified '''condition false''' from the beginning.
 
  
 +
Next we have written '''NF!=6'''.
  
So this means, we should not get any output.
 
  
|-
+
This checks whether the number of fields in the current line, is not equal to 6.
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Press Ctrl+S
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Save the file and switch to '''terminal'''.
+
  
|-
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| awk -f while_loop.awk awkdemo.txt
 
  
<nowiki>[Enter]</nowiki>
+
If true, then
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Press up key until you get the command for executing the '''while loop.'''
+
  
 +
* '''print section '''will print the record’s line number '''NR''',
 +
* along with the entire line denoted by '''$0'''.
  
Now press '''Enter.'''
+
Press '''Enter'''.
  
 
|-
 
|-
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| See, we are not getting any output apart from blank lines.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| In the output, we can see that record number 16 is the incomplete record.  
  
  
For each record in the '''awkdemo.txt''' file, blank lines are getting printed in the output.
+
It has only 5 '''fields '''instead of 6.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"|  
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Retain the same screen
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Now, let us make the same changes in the '''do loop '''file.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Let us see one more example.
 +
 
 +
 
 +
How can we print the first and last '''field '''for each student regardless of how many '''fields''' there are?
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| In do_loop.awk, assign i=4
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Type:
  
<nowiki>[code uploaded as do_loop_mod.awk]</nowiki>
 
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Switch to the file '''do_loop.awk'''
 
  
 +
awk -F"|" '{print $1,$NF}' awkdemo.txt
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Type the command as shown here on the '''terminal'''.
  
Change the value of '''i '''from 1 to 4
 
  
  
Save the file and switch to the '''terminal.'''
 
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| awk -f do_loop.awk awkdemo.txt
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight -F
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Here we have used '''hyphen capital F''' option instead of setting '''FS''' '''variable'''.
  
<nowiki>[Enter]</nowiki>
 
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Press the Up arrow key until you get the command for '''do loop.'''
 
  
 +
Press '''Enter'''.
  
Now press '''Enter.'''
+
|-
 +
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the Output
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| We get only the first and the last '''fields''' for each record in the file.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 8:
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Let’s try something else now.
  
  
 +
Suppose, the student records are distributed across two files '''demo1.txt''' and '''demo2.txt'''.
  
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| In the output, only the first field for each row is printed.
 
  
 +
We want to print the first 3 lines from each of these two files.
  
What is the reason?
+
 
 +
We can do this using '''NR variable'''.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight in code file
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show demo1.txt & demo2.txt in gedit
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| For each row, '''awk''' first prints the value at the first''' field, '''because value of variable''' f '''is initialized to '''1.'''
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Here are the contents of the two files.
  
 +
|-
 +
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Type:
  
Then the '''condition '''is checked.
+
<nowiki>awk 'NR<=3 {print NR, $0}' demo1.txt demo2.txt</nowiki>
  
  
Since the value of the '''loop counter i '''is 4, the '''condition '''is '''false'''.
+
<nowiki>[Enter]</nowiki>
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Now to display the first 3 lines from each file, type the following command on the '''terminal.'''
  
  
Hence, the '''loop '''is terminated there only, for that record.
+
Press '''Enter.'''
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight in code file
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| This '''loop''' will iterate for all the records in the '''awkdemo.txt''' file.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| The output shows only the first 3 records of '''demo1.txt''' file.
  
  
Which means the first '''field''' for each record will get printed.
+
How can we print the same for the second file also?
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight output
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 8:
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| We are getting the output at least once for each record.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| The solution is to use '''FNR''' instead of '''NR'''.
  
  
Use the '''do-while loop, '''for a job to be executed at least once irrespective of any other '''condition'''.  
+
'''FNR''' is the '''current record number '''in the current file.  
  
|-
+
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Retain same screen
+
'''FNR''' is incremented each time a new record is read.
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| We can do the same with the '''for loop''' also.
+
 
 +
 
 +
It is reinitialized to zero each time a new input file is started.
  
 
|-
 
|-
| style="border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 9: for loop
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 9:
| style="border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| The syntax of '''for loop '''can be seen here.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| But '''NR''' is the number of input records '''awk''' has processed since the starting of the program's execution.
  
  
The&nbsp;'''for&nbsp;statement''' starts by executing&nbsp;'''''initialization'''''.  
+
It does to reset to zero with a new file.
  
 +
|-
 +
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| In Terminal
  
Then, as long as the'''&nbsp;''condition'''''&nbsp;is true, it repeatedly executes&nbsp;the '''''statements''' ''within&nbsp;and then&nbsp;'''''increments''.&nbsp;'''
 
  
 +
Press up key
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Switch to '''terminal.'''
  
Assuming your familiarity with a language like '''C''' or '''C++''', I am not explaining the syntax in detail.
+
 
 +
Press the '''up arrow''' key to get the previous command.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| show for_loop.awk in Gedit
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| <nowiki>awk ‘FNR<=3 {print NR,FNR, $0}’ demo1.txt demo2.txt</nowiki>
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| This is how the '''for loop '''for this '''condition''' looks like.
+
 
 +
<nowiki>[Enter]</nowiki>
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Modify the previous command as follows.
  
  
Here, '''initialization, condition''' checking and variable incrementation are done in the same line.
+
Type '''FNR''' instead of '''NR.'''
  
  
Try this out by yourself.
+
In the '''Print section,''' next to '''NR,''' type '''FNR''' and press '''Enter.'''
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 10: looping constructs
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output and highlight appropriately
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| There are some more''' looping constructs '''
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| See, we get the correct output now.
  
* '''break'''
 
* '''continue'''
 
* '''exit'''
 
  
We will see some relevant examples on these in further tutorials.
+
'''FNR''' is set to zero with new file but '''NR''' keeps on increasing.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show awkdemo_mod.txt
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 10:
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| We may have a single and multiline '''comments''' in our file.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Let us now look at some other '''built-in variables.'''
  
  
Here notice that the single line '''comments''' are declared with single '''hash '''(#) symbol'''.'''
+
'''FILENAME''' variable gives the name of the file being read.
  
  
The multiline '''comments''' are declared with the help of double '''hash '''(##) symbol.
+
'''ARGC''' specifies the number of '''arguments''' provided at the '''command line'''.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 11
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Now, there is no point of checking and printing these '''comments''' in the output.  
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| '''ARGV''' represents an '''array '''that stores the '''command line arguments.'''
  
  
We have to skip the lines starting with '''hash '''(##) symbol.
+
'''ENVIRON''' specifies the '''array '''of the '''shell environment variables '''and corresponding values.
  
  
How can we do this?
+
As '''ARGV''' and '''ENVIRON''' use '''array''' in '''awk''', we will look at those in subsequent tutorials.
  
 
|-
 
|-
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"|  
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"|  
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Recall the case of giving 50% increment in the stipend for those who are getting more than 8000.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Let us have a look at the variable '''FILENAME''' now.
  
  
We will use the same example for skipping the '''comments'''.
+
How can we print the name of the current file being processed?
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| show next.awk
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Type:
  
 +
awk '{print "We are processing input file " FILENAME}' awkdemo.txt
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Switch to the '''terminal''' and type the command as shown.
  
 +
|-
 +
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight the space here
  
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| I have created a file named''' next.awk '''as shown here for this execution.
+
awk '{print "We are processing input file " FILENAME}' awkdemo.txt
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Here we have used '''space '''as a '''string concatenation operator.'''
  
|-
+
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Highlight''' $0~/^#/ {next;}'''
+
Press '''Enter''' to execute the command.
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Now, what does this command mean?
+
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 11: next
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output.
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| '''awk''' will search for the pattern, '''caret sign hash symbol(^#) '''at the beginning of each line.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| The output shows the '''input filename '''multiple times.
  
  
If the pattern is found, the keyword '''next '''tells '''awk''' to skip the current line immediately.
+
This is because, this command prints the filename once for each row in the '''awkdemo.txt '''file.
  
  
Then '''awk''' will start processing from the next line in the file.
+
How can we print this only once?
  
 +
|-
 +
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Press Up arrow key
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Clear the '''terminal'''
  
This will save the processing time.
+
 
 +
Press the '''up arrow '''key to get the previously executed command.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Type
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Modify the command to become:
  
awk -F "|" -f next.awk awkdemo_mod.txt
+
awk 'END{print "We are processing input file " FILENAME}' awkdemo.txt
  
 
<nowiki>[Enter]</nowiki>
 
<nowiki>[Enter]</nowiki>
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Switch to the '''terminal''' and type the command as shown and
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Modify the previous command as shown here.
  
press '''Enter.'''
+
 
 +
Press '''Enter.'''
  
 
|-
 
|-
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| We get the output without any '''comments.'''
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| We get the filename only once.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"|  
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Retain same screen
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Suppose, we have the students’ records in multiple files with the same format.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| There are some other '''built-in variables '''in '''awk'''.
  
  
Say in '''awkdemo_mod.txt''' and '''awkdemo2.txt'''
+
Please browse the internet to know more on them.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show awkdemo2.txt
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 12
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| See, it is similar to our previous file.
+
  
  
It also has '''comments''' preceeded with '''hash <nowiki>#</nowiki> '''sysmbol.
 
  
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Suppose, we want to
  
And it has large text at the end with double '''hash #<nowiki>#</nowiki>''' symbol.
+
* find the students who have passed and have stipend more than Rs.8000
 +
* use '''comma '''as the '''output field separator'''
 +
* and print “'''The data is shown for file'''” and the name of file in the '''footer section'''.  
 +
 
 +
How can we do this?
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 12: nextfile
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| awk 'BEGIN{FS="|"; OFS=","} $5=="Pass" && $6>8000 {print NR, $2, $5, $6} END{print "The data is shown for file " FILENAME }' awkdemo.txt
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| So our data is in two different files.
+
  
  
So, '''awk''' should process both the files to give an increment to all the students.
+
<nowiki>[Enter]</nowiki>
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| In the '''terminal''' type the following command
  
  
Once we reach double '''hash'''(##) symbol of the first file,
+
Press '''Enter'''.
  
'''awk''' should stop processing that file entirely.
+
|-
 +
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output and highlight appropriately
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| We can see that only one student has passed and gets stipend more than Rs.8000.
  
  
Then it has to start the execution from the next file.
+
And the record number is 2.
  
  
This will save the processing time.
+
We can also see the name of the file in the '''footer''', as desired.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Type '''$0~/^#/ {next;} '''after the begin statement.
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 13
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| We can use '''awk''' for more and more complex tasks.
  
  
Highlight '''$0~/^#/ {nextfile;}'''
+
In that case, it becomes more difficult to write the '''commands''' every time on the '''terminal'''.
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Modify the '''next.awk '''as shown here.
+
  
  
I have added '''dollar zero tilde slash caret symbol double hash slash within curly brackets nextfile semicolon '''below the '''begin statement'''.
+
We can instead write the '''awk''' program in a separate file.
  
  
This will search for double '''hash #''' symbol at the beginning of a line.
+
To be executable, that file should have the '''dot awk '''extension.
  
 +
|-
 +
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 14
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| While executing, we can just specify this '''awk''' program filename with the''' awk command.'''
  
If found, '''awk''' will skip the current file to process the next file.
+
 
 +
For doing so, we need to use '''hyphen small f''' option.
 +
 
 +
 
 +
Let us see an example.  
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Press Ctrl+s
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show prog1.awk in gedit
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Save this file.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| I have already written an '''awk '''program and saved it as '''prog1 dot awk.'''
 +
 
 +
 
 +
This code is also available in the '''Code Files''' link.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Type
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| In the terminal show the last executed command
  
awk -F "|" -f next.awk awkdemo_mod.txt awkdemo2.txt
 
  
<nowiki>[Enter]</nowiki>
+
Highlight the portion in the last executed command and also in this program
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Switch to the the '''terminal '''and type the following command.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Switch to the '''terminal'''.
  
  
'''awk space -F "|" space -f space next.awk space awkdemo_mod.txt space awkdemo2.txt'''
+
See what have we written inside '''single quotes''' of the '''command '''last executed?
  
  
Press''' Enter'''
+
Content of '''prog1.awk''' file is exactly the same.
 +
 
 +
 
 +
The only difference is that in the '''awk''' file, we have not written inside the '''single quotes.'''
 +
 
 +
|-
 +
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Type:
 +
 
 +
<nowiki>awk -f prog1.awk awkdemo.txt [enter]</nowiki>
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| To execute the file, type the following on the '''terminal-'''
 +
 
 +
 
 +
'''awk space hyphen small f space prog1.awk space awkdemo.txt '''and pres'''s Enter'''
  
 
|-
 
|-
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output
 
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Show the output
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| See, we are getting the output from both the files.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| We are getting exactly the same output as we have seen before.
 +
 
 +
 
 +
So, this way you can write '''awk''' programs and use it multiple times.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 13: Summary
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"|  
 
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| This brings us to the end of this tutorial.
 
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| This brings us to the end of this tutorial.
  
Line 504: Line 564:
 
Let us summarize.
 
Let us summarize.
  
 +
|-
 +
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 15
  
In this tutorial we learnt about-
+
Summary
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| In this tutorial we learnt about-
  
* '''while'''
+
* '''Built-in variables'''
* '''do… while'''
+
* '''awk script'''
* '''for'''
+
* '''next'''
+
* '''nextfile '''in''' awk'''
+
  
 +
using various examples.
  
 +
|-
 +
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 16
 +
 +
Assignment 1
 +
 +
 +
 +
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| As an assignment-
 +
 +
 +
1. Write an '''awk''' script to print the last field of the 5th line in '''awkdemo.txt '''file.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 14: Assignment
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 17
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| As an assignment for the student records of '''awkdemo2.txt''',
+
  
* print only even '''fields'''(i.e. '''field''' 2, '''field''' 4 etc.
+
Assignment 2
* irrespective of how many '''fields''' are there in the '''input file'''.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| 1. Open the system file '''/etc/passwd '''on the '''terminal. '''
  
  
 +
2. Identify all the '''separators '''therein.
 +
 +
 +
3. Now write a '''script '''to process the file from the 20th line onwards.
 +
 +
 +
4. That too, only for the lines that contain more than 6 fields.
 +
 +
 +
5. You should print the '''line number''', entire line and count of '''fields '''in that particular line.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 13:
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 18
  
 
About Spoken Tutorial project
 
About Spoken Tutorial project
Line 534: Line 615:
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 14:
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 19
  
 
Spoken Tutorial workshops
 
Spoken Tutorial workshops
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| The&nbsp;Spoken Tutorial Project&nbsp;team conducts workshops using spoken tutorials and gives certificates on passing online tests.
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| The&nbsp;Spoken Tutorial Project&nbsp;team conducts workshops using spoken tutorials and gives certificates.
  
  
Line 543: Line 624:
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 15:
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 20
  
 
Forum for specific questions:
 
Forum for specific questions:
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Do you have questions in THIS Spoken Tutorial?
+
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Pls post your timed queries in this Forum.
  
*
+
|-
 +
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 21
  
Please visit this site
+
Acknowledgement
 
+
|-
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 16: Acknowledgement
+
 
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Spoken Tutorial Project is funded by NMEICT, MHRD, Government of India.
 
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Spoken Tutorial Project is funded by NMEICT, MHRD, Government of India.
  
*
 
  
More information on this mission is available at this link.
+
More information on this mission is available at
 +
 
 +
this link.
  
 
|-
 
|-
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| Slide 17: Thanks
+
| style="background-color:#ffffff;border-top:0.035cm solid #000001;border-bottom:0.035cm solid #000001;border-left:0.035cm solid #000001;border-right:none;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"|  
 
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| The script has been contributed by Antara.
 
| style="background-color:#ffffff;border:0.035cm solid #000001;padding-top:0cm;padding-bottom:0cm;padding-left:0.191cm;padding-right:0.191cm;"| The script has been contributed by Antara.
  

Revision as of 06:41, 31 January 2018

Title of script: Built-in variables and awk Script

Author: Antara Roy Choudhury

Keywords: Built-in variables, RS, ORS, NR, NF, FS, OFS, FILENAME


Visual Cue
Narration
Slide 1: Introduction Welcome to the spoken tutorial on awk built-in variables and awk script.
Slide 2: Learning Objectives In this tutorial we will learn about


  • Built-in variables
  • awk script

We will do this through some examples.

Slide 3a: System requirement To record this tutorial, I am using
  • Ubuntu Linux 16.04 OS and
  • gedit text editor version 3.20.1


Slide 3b: Code Files The files used in this tutorial are available in the Code Files link on this tutorial page.


Please download and use them.

Slide 4: Prerequisite To practice this tutorial, you should have gone through the earlier awk tutorials on this website.


If not, then please go through the corresponding tutorials on this website.

Slide 5: awk built-in variables First, let us see some of the built-in variables in awk.


  • Capital RS specifies the record separator in an input file. By default, it is newline.
  • Capital FS specifies the field separator in an input file.

By default, the value of FS is a whitespace.

Slide 5: awk built-in variables * Capital ORS defines the output record separator.

By default, it is newline.

  • Capital OFS defines the output field separator.

By default, it is whitespace.


Let us understand the meaning of each of these.

Show awkdemo.txt in Gedit Let us have a look at the awkdemo file now.


When we are processing this awkdemo file with awk command, this becomes our input file.

Highlight appropriately Observe that all the records are separated from each other by a newline character.


newline is the default value for record separator RS variable.


So, there is no need to do anything else.

Highlight vertical bar | character Notice that all the fields are separated by the pipe symbol.


How can we inform awk about it?


Let us see.

Slide 6: How to reset value of FS variable? By default, any number of spaces or a tabs separate the fields.


We can reset this with the help of hyphen capital F option as learnt in our earlier tutorials.


Or else, we can reset this in the BEGIN section with the use of FS variable.

Let us do this through an example.


Suppose, I want to find out the name of students who are getting a stipend of more than Rs.5000.

Open the terminal Open the terminal by pressing CTRL, ALT and T keys.
cd /<saved folder> Go to the folder in which you downloaded and extracted the Code Files using cd command.
awk 'BEGIN{FS="|"} $6>5000 {print $2,$6}' awkdemo.txt Type the command as shown here.
Highlight {FS="|"} area Here in the BEGIN section, we have assigned the value of FS as a pipe symbol.


Similarly, we can modify RS variable.

[Enter] Press Enter to execute the command.
Show the output The output shows the list of students who are receiving more than Rs.5000 as a stipend.
Highlight appropriately Here the name field and the stipend field is separated by a blank space.


Also, all the records are separated by a newline character.

Slide


Suppose we want colon as the output field separator.


And double newline as output record separator.


How can we do this? Let us see.

In terminal In the terminal, press the up arrow key to get the previously executed command.
Modify the previous command


awk 'BEGIN{FS="|";OFS=":"; ORS="\n\n"} $6>5000 {print $2, $6}' awkdemo.txt

[Enter]

Modify the command as shown here.


And then press Enter.

Show the output We get the output in the desired format.
Show sample.txt


Point or highlight

Now, suppose our new input file is sample.txt.


Observe that the field separator here is newline and record separator is double newline.

How can we extract the roll no. and name information from this file?


Yes, you have guessed correctly.


We have to modify both FS and RS variables.


Pause this tutorial and do this as an assignment.

Slide 7: Next, let us see other built-in variables.


  • Capital NR gives us the Number of Records processed by awk
  • Capital NF gives the Number of Fields in the current record


Slide Let us see one example of this.


Suppose, we want to find incomplete lines in the file.


Here, incomplete line means it has less than the normal 6 fields.

Switch to the terminal and clear it Switch to the terminal.


Let me clear the terminal using Ctrl and L keys

Type

awk 'BEGIN{FS="|"} NF !=6 {print NR, $0}' awkdemo.txt

[Enter]

Type the command as shown.



Show the command and highlight appropriate areas as per narration As the fields are separated by pipe symbol, set the FS value to pipe symbol in the BEGIN section.


Next we have written NF!=6.


This checks whether the number of fields in the current line, is not equal to 6.


If true, then

  • print section will print the record’s line number NR,
  • along with the entire line denoted by $0.

Press Enter.

Show the output In the output, we can see that record number 16 is the incomplete record.


It has only 5 fields instead of 6.

Retain the same screen Let us see one more example.


How can we print the first and last field for each student regardless of how many fields there are?

Type:


awk -F"|" '{print $1,$NF}' awkdemo.txt

Type the command as shown here on the terminal.



Highlight -F Here we have used hyphen capital F option instead of setting FS variable.


Press Enter.

Show the Output We get only the first and the last fields for each record in the file.
Slide 8: Let’s try something else now.


Suppose, the student records are distributed across two files demo1.txt and demo2.txt.


We want to print the first 3 lines from each of these two files.


We can do this using NR variable.

Show demo1.txt & demo2.txt in gedit Here are the contents of the two files.
Type:

awk 'NR<=3 {print NR, $0}' demo1.txt demo2.txt


[Enter]

Now to display the first 3 lines from each file, type the following command on the terminal.


Press Enter.

Show the output The output shows only the first 3 records of demo1.txt file.


How can we print the same for the second file also?

Slide 8: The solution is to use FNR instead of NR.


FNR is the current record number in the current file.


FNR is incremented each time a new record is read.


It is reinitialized to zero each time a new input file is started.

Slide 9: But NR is the number of input records awk has processed since the starting of the program's execution.


It does to reset to zero with a new file.

In Terminal


Press up key

Switch to terminal.


Press the up arrow key to get the previous command.

awk ‘FNR<=3 {print NR,FNR, $0}’ demo1.txt demo2.txt

[Enter]

Modify the previous command as follows.


Type FNR instead of NR.


In the Print section, next to NR, type FNR and press Enter.

Show the output and highlight appropriately See, we get the correct output now.


FNR is set to zero with new file but NR keeps on increasing.

Slide 10: Let us now look at some other built-in variables.


FILENAME variable gives the name of the file being read.


ARGC specifies the number of arguments provided at the command line.

Slide 11 ARGV represents an array that stores the command line arguments.


ENVIRON specifies the array of the shell environment variables and corresponding values.


As ARGV and ENVIRON use array in awk, we will look at those in subsequent tutorials.

Let us have a look at the variable FILENAME now.


How can we print the name of the current file being processed?

Type:

awk '{print "We are processing input file " FILENAME}' awkdemo.txt

Switch to the terminal and type the command as shown.
Highlight the space here

awk '{print "We are processing input file " FILENAME}' awkdemo.txt

Here we have used space as a string concatenation operator.


Press Enter to execute the command.

Show the output. The output shows the input filename multiple times.


This is because, this command prints the filename once for each row in the awkdemo.txt file.


How can we print this only once?

Press Up arrow key Clear the terminal


Press the up arrow key to get the previously executed command.

Modify the command to become:

awk 'END{print "We are processing input file " FILENAME}' awkdemo.txt

[Enter]

Modify the previous command as shown here.


Press Enter.

Show the output We get the filename only once.
Retain same screen There are some other built-in variables in awk.


Please browse the internet to know more on them.

Slide 12


Suppose, we want to
  • find the students who have passed and have stipend more than Rs.8000
  • use comma as the output field separator
  • and print “The data is shown for file” and the name of file in the footer section.

How can we do this?

awk 'BEGIN{FS="|"; OFS=","} $5=="Pass" && $6>8000 {print NR, $2, $5, $6} END{print "The data is shown for file " FILENAME }' awkdemo.txt


[Enter]

In the terminal type the following command


Press Enter.

Show the output and highlight appropriately We can see that only one student has passed and gets stipend more than Rs.8000.


And the record number is 2.


We can also see the name of the file in the footer, as desired.

Slide 13 We can use awk for more and more complex tasks.


In that case, it becomes more difficult to write the commands every time on the terminal.


We can instead write the awk program in a separate file.


To be executable, that file should have the dot awk extension.

Slide 14 While executing, we can just specify this awk program filename with the awk command.


For doing so, we need to use hyphen small f option.


Let us see an example.

Show prog1.awk in gedit I have already written an awk program and saved it as prog1 dot awk.


This code is also available in the Code Files link.

In the terminal show the last executed command


Highlight the portion in the last executed command and also in this program

Switch to the terminal.


See what have we written inside single quotes of the command last executed?


Content of prog1.awk file is exactly the same.


The only difference is that in the awk file, we have not written inside the single quotes.

Type:

awk -f prog1.awk awkdemo.txt [enter]

To execute the file, type the following on the terminal-


awk space hyphen small f space prog1.awk space awkdemo.txt and press Enter

Show the output We are getting exactly the same output as we have seen before.


So, this way you can write awk programs and use it multiple times.

This brings us to the end of this tutorial.


Let us summarize.

Slide 15

Summary

In this tutorial we learnt about-
  • Built-in variables
  • awk script

using various examples.

Slide 16

Assignment 1


As an assignment-


1. Write an awk script to print the last field of the 5th line in awkdemo.txt file.

Slide 17

Assignment 2

1. Open the system file /etc/passwd on the terminal.


2. Identify all the separators therein.


3. Now write a script to process the file from the 20th line onwards.


4. That too, only for the lines that contain more than 6 fields.


5. You should print the line number, entire line and count of fields in that particular line.

Slide 18

About Spoken Tutorial project

The video at the following link summarises the Spoken Tutorial project.


Please download and watch it.

Slide 19

Spoken Tutorial workshops

The Spoken Tutorial Project team conducts workshops using spoken tutorials and gives certificates.


For more details, please write to us.

Slide 20

Forum for specific questions:

Pls post your timed queries in this Forum.
Slide 21

Acknowledgement

Spoken Tutorial Project is funded by NMEICT, MHRD, Government of India.


More information on this mission is available at

this link.

The script has been contributed by Antara.


And this is Praveen from IIT Bombay signing off.


Thanks for joining.

Contributors and Content Editors

Antarade, Nancyvarkey