Python/C3/Statistics/English

From Script | Spoken-Tutorial
Revision as of 11:37, 29 November 2012 by Chandrika (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search
Visual Cue Narration
Show Slide 1

Containing title, name of the production team along with the logo of MHRD

Hello friends and welcome to the tutorial on 'Statistics' using Python.
Show Slide 2

Learning objectives

At the end of this tutorial,you will be able to,
  1. Do statistical operations in Python
  2. Sum a set of numbers
  3. Find their mean,median and standard deviation


Show Slide 3

Pre-requisite

Before beginning this tutorial,we would suggest you to complete the tutorial on

Unexpected indentation.

"Loading Data from files" "Getting started with Lists" "Accessing Pieces of Arrays".

Let us invoke our ipython interpreter with pylab loaded. ipython -pylab
Open the file sslc2.txt and show For this tutorial, we will use data file that is at the path /home/fossee/sslc2.txt. It contains record of students and their performance in one of the State Secondary Board Examination. It has 180,000 lines of record. We are going to read it and process this data. We can see the content of file by double clicking on it. It might take some time to open since it is quite a large file. Please don't edit the data since it has a particular structure.
cat /home/fossee/sslc2.txt To check the contents of the file, we use the cat command.
Show Slide 4

Data structure

Each line in the file is a set of 11 fields separated by semi-colons. Consider a sample line from this file. A;015163;JOSEPH RAJ S;083;042;47;00;72;244;;;

The following are the fields in any given line. * Region Code which is 'A' * Roll Number 015163 * Name JOSEPH RAJ S * Marks of 5 subjects: ** English 083 ** Hindi 042 ** Maths 47 **

Unexpected indentation.

Science 35 ** Social 72

Block quote ends without a blank line; unexpected unindent.

  • Total marks 244


L=loadtxt('/home/fossee/sslc2.txt',usecols=(3,4,5,6,7,),delimiter=';')
L
Lets load this data as an array and then run various functions on it.

To get the data as an array, we use the loadtxt command

L.shape We get our output in the form of an array. loadtxt function loads data from an external file. Delimiter specifies the kind of character, that the fields of data separated by. usecols specifies the columns to be used. So (3,4,5,6,7) loads those columns. The 'comma' is added because usecols is a sequence.

As we can see L is an array. We can get the shape of this array using

We get a tuple stating the numbers of rows and columns respectively. Lets start applying statistical operations on these. We will start with the most basic, summing. How do you find the sum of marks of all subjects for the first student.
L[0,:] As we know from our knowledge of accessing pieces of arrays, to access the first row, we will do
totalmarks=sum(L[0,:])
totalmarks
Now to sum this we can say
totalmarks/len(L[0,:]) Now to get the mean we can divide the totalmarks by the length
mean(L[0,:]) or simply use the function mean.
mean? But we have such a large data set and calculating the mean for each student one by one is impossible. Is there a way to reduce the work.

For this we will look into the documentation of mean

mean(L,1) As we know L is a two dimensional array. We can calculate the mean across each of the axis of the array. The axis of rows is referred by number 0 and columns by 1. So to calculate mean across all columns, we will pass extra parameter 1 for the axis.
mean(L,0) L here, is a two dimensional array.

Similarly to calculate average marks scored by all the students for each subject can be calculated using

L[:,0] Next, let us calculate the median of English marks for the all the students. We can access English marks of all students using
median(L[:,0]) To get the median we will simply use the function median.
median(L,0) For all the subjects we can use the same syntax as mean and calculate median across all rows using median
std(L[:,0]) Similarly to calculate standard deviation for English we will use the function std
std(L,0) and for all rows, we do,
Pause the video here, try out the following exercise and resume the video.
Show Slide 5

Assignment 1

In the given file football.txt at path /home/fossee/football.txt , one column is player name,second is goals at home and third goals away.

Unexpected indentation.

1.Find the total goals for each player 2.Mean of home and away goals 3.Standard deviation of home and away goals

Open the file football.txt and keep open for some time This is the required data.
Show Slide 6

Solution 1

The solution is on your screen.
Show Slide 7

Summary slide

This brings us to the end of the tutorial. In this tutorial,we have learnt to,
  1. Do the standard statistical operations sum , mean median and standard deviation in Python.
  2. Combine text loading and the statistical operation to solve real world problems.


Show Slide 8

Self assessment questions slide

Here are some self assessment questions for you to solve
  1. Given a two dimensional list, two_dimensional_list=[[3,5,8,2,1],[4,3,6,2,1]] how do we calculate the mean of each row?
  2. Calculate the median of the given list? student_marks=[74,78,56,87,91,82]
  3. Suppose there is a file with 6 columns but we wish to load text only in columns 2,3,4,5. How do we specify that?


Show Slide 9

Solution of self assessment questions on slide

And the answers,

1. To get the mean of each row, we just pass 1 as the second parameter to the function mean.

mean(two_dimensional_list,1)

2. We use the function median to calculate the median of the list

median(student_marks)

3. To specify the particular columns of a file, we use the parameter usecols=(2,3,4,5)

Show Slide 10

Acknowledgment slide

Hope you have enjoyed this tutorial and found it useful. Thank you!

Contributors and Content Editors

Chandrika