Snehalathak: Created page with " {| style="border-spacing:0;" ! Visual Cue ! Narration |- | style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;..."

2015-09-02T05:19:40Z

Created page with " {| style="border-spacing:0;" ! <center>Visual Cue</center> ! <center>Narration</center> |- | style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;..."

New page

{| style="border-spacing:0;"
! <center>Visual Cue</center>
! <center>Narration</center>

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Slide Number 1'''

'''Title Slide'''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Hello everyone.

Welcome to this tutorial on '''Parsing Data.'''

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Slide Number 2'''

'''Learning Objectives'''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| In this tutorial, we will learn to,

* Download '''FASTA''' and '''GenBank''' files from '''NCBI''' database website.<br/>

* And '''Parse''' data files using functions in '''Sequence Input/Output''' module.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Slide Number 3'''

'''Pre-requisites'''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| To follow this tutorial you should be familiar with,

* Undergraduate Biochemistry or Bioinformatics<br/>

* And basic '''Python''' programming

Refer to the '''Python''' tutorials at the given link.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Slide Number 4'''

'''System Requirement'''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| To record this tutorial I am using

* '''Ubuntu''' OS version. 14.10
* '''Python''' version 2.7.8
* '''Ipython interpretor''' version 2.3.0
* '''Biopython''' 1.64
* And '''Mozilla Firefox '''browser 35.0

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Slide Number 5'''

'''Data files'''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Scientific data in biology is generally stored in text files such as '''FASTA''', '''GenBank''', '''EMBL''', '''Swiss-Prot''' etc

Data files can be download from the database websites.

Open the website link given below in any web browser.

'''http://www.ncbi.nlm.nih.gov/nucleotide'''

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Cursor on the web-page.

Download FASTA files
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| A web-page opens.

Let us download '''FASTA''' and '''GenBank''' files for human '''insulin''' gene.

In the search box type, '''human insulin'''.

Click on search button.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Scroll down the page.

Click on check box.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| The web-page shows many files for human insulin gene.

For demonstration, I will select 4 files with the name “'''Homo sapiens Insulin mRNA”. '''

I will choose files that have less than 500 base pairs.

Click on the check box to select the file to download.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Bring the cursor to the “Send to” option. (Located on the top right hand corner.)

Click on the selection button.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Bring the cursor to the “'''Send to'''” option, located at the top right corner of the page.

Click on the small selection button with a down arrow present next to the “'''Send to'''” button.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Under the heading “Choose file destination” Click on “File” option.

Click on “format” drop down list box.

Choose “fasta” and click on “Create file” option.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Under the heading “'''Choose destination'''” Click on “'''File'''” option.

You can save this file in any file format listed under “'''format'''” drop down list box.

Choose “'''FASTA'''” from the given options.

Then click on “'''Create file'''” option.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| A dialog box appears on screen. Click on “Save file” option.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| A dialog box appears on the screen.

Select “'''Open with'''” click on '''OK .'''

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Cursor on the text editor.

Cursor on the first line.

Scroll down.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| The file opens in a text editor.

The file shows 4 records, since we had selected four files to download.

The first line in each record is an '''identifier''' line,

It starts with a “greater than (>) symbol”.

This is followed by a '''sequence'''.

Save the file in your home folder as “'''sequence.fasta'''”.

Close the text editor.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Cursor on the web-page.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Follow the same steps as above to download the files in '''GenBank''' format:

for the same files selected earlier.

Select the file format as G'''enBank.'''

'''Create file''', open with a text editor.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Scroll down.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Notice that the sequence file in '''GenBank''' format has more features than a '''FASTA''' file.

Save the file as '''sequence.gb '''in your home folder'''.'''

Close the text editor'''.'''

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Click on the check boxes.

Select '''Human insulin gene complete cds,'''

click on the check box.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| For demonstration purpose we need a FASTA file with a single record.

For this, clear the earlier selection by again clicking on the check boxes.

Now select the file “'''Human insulin gene complete cds'''”.

Click on the check box.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Save the file as '''insulin.fasta.'''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| And Follow the same steps shown earlier to save the file in the home folder.

Save the file as '''insulin.fasta.'''

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Cursor on the text editor.

Close the text editor.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Biological data stored in these files can be extracted and modified using '''Biopython''' libraries.

Close the text editor.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Slide Number 6'''

'''Parsing'''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Extracting data from data files is called as '''Parsing'''.

Most file formats can be parsed using functions available in '''SeqIO''' module.

Most commonly used functions of '''SeqIO''' module are, '''parse''', '''read''', '''write''', and '''convert'''.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Slide number 6'''

Open the terminal using ctrl, alt and t keys.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Open the terminal by pressing '''ctrl''', '''alt''' and '''t''' keys simultaneously.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Type '''ipython''' at the prompt.

>>>ipython
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Start '''Ipython''' by typing '''ipython''' at the prompt.

Press enter.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Cursor on the terminal.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Next import '''SeqIO''' module from '''Bio''' package.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Type,

>>> '''from Bio import SeqIO'''

Press enter
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| At the prompt type,

'''from Bio import SeqIO'''

Press enter

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| (Open the file in text editor and scroll down)
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| We will start with the most important function “'''parse'''”.

For demonstration, I will use a '''FASTA''' file that has many records.

Which we had downloaded earlier from the database.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Type,

'''or seq_record in SeqIO.parse("sequence.fasta", "fasta"):'''

'''print(seq_record.id)'''

'''print(repr(seq_record.seq))'''

'''print(len(seq_record))'''

Highlight all the lines.

Press enter key twice to get the output.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| For simple '''FASTA''' parsing, type the following at the prompt.

Here we are using the '''parse''' function to read contents of '''sequence.fasta''' file.

For the output print, '''record id''', sequence present in the record and also the length of the sequence.

Also notice that the '''parse''' function is used to read sequence data as '''Sequence record objects'''.

It is generally used with a '''for''' loop.

It can accept two '''arguments''', the first one is the file name to read the data.

The second specifies the file format.

Press enter key twice to get the output.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Highlight the first line.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| The output shows the''' identifier line, '''followed by the sequence contained in the file.

Also the length of the sequence for all the records in the file.

Notice that the '''FASTA''' format does not specify the alphabet.

So, the output does not specifies it as as a '''DNA sequence'''.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Type,

for seq_record in SeqIO.parse("sequence.gb", "genbank"):

| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| The same steps can be repeated for parsing '''GenBank''' file.

For Demonstration we will use the '''GenBank''' file which we have download earlier from the database.

Press up arrow key to get the lines of code which we had used earlier.

Change the file name to '''sequence.gb '''

Change the file format to '''genbank.'''

The rest of the code remains same.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| from Bio import SeqIO

for seq_record in SeqIO.parse("insulin.gb", "genbank"):

print(seq_record.id)

print(seq_record.seq)

'''print(len(seq_record))'''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Press enter key twice to get the output.

Here too the output shows the '''record id''', sequence and the length of the sequence for all the records in the file.

Notice that the '''GenBank''' format specifies the sequence as DNA sequence.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Cursor on the terminal.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Similarly '''Swiss-prot''' and '''EMBL''' files can be parsed using same code as above.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Cursor on the terminal.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| If your file contains a single record then type the following lines for '''parsing'''.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Type,'''

'''>>> from Bio import SeqIO '''

'''>>> record = SeqIO.read("insulin.fasta", "fasta") '''

'''>>> record'''

'''Press enter '''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Here we will use the previously saved FASTA file with a single record, that is '''insulin.fasta '''as an example'''.'''

Notice that we have used '''read''' function instead of parse function.

Press enter.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Cursor on the terminal.
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| The output shows the contents for the file '''insulin.fasta'''.

It shows the sequence as sequence record object.

And other attributes such as '''GI, accession number '''and '''description'''.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| At the prompt type,

>>> record.seq

press enter
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| We can also view the individual attributes of this record as follows.

At the prompt type, '''record dot seq '''

press enter

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| Cursor on the terminal'''.'''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| The output shows the sequence present in the file.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| type,

>>> record.id

press enter

| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| To view the identifiers for this record, type, '''record dot id.'''

press enter

The output shows the GI number and accession number etc.

You can use the function described above to parse the data files of your choice.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Slide Number 9'''

Summary
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Now Let's summarize,

In this tutorial we have learnt to,

Download '''FASTA''' and '''GenBank''' files from '''NCBI''' database website.

And use '''parse''' and '''read''' functions from the '''SeqIO''' module:

To extract data such as record ids, description and sequences, from '''FASTA''' and '''GenBank''' files.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Slide Number 10'''

'''Assignment'''

| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Now for the assignment,

Download '''FASTA''' files for nucleotide sequence of your choice from '''NCBI''' database.

Convert the file of sequences to their reverse complements.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Type at the prompt,'''

'''>>> from Bio import SeqIO '''

'''>>> for record in SeqIO.parse("sequence.fasta", "fasta"): '''

'''... '''

'''print(record.id) '''

'''... '''

'''print(record.seq.reverse_complement())'''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Your completed assignment should have the following lines of code.

Use '''parse''' function to load nucleotide sequences from the '''FASTA''' file.

Next print reverse complements using the Seq object’s built in reverse complement method.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Slide Number 11'''

'''Acknowledgement '''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| Video at the following link, summarizes the spoken-tutorial project.

Please download and watch it.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Slide Number 12'''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| The Spoken Tutorial Project Team:

Conducts workshops and gives certificates to those who pass an on-line test.

For more details, please write to us.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"| '''Slide number 13'''
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| The Spoken Tutorial Project is funded by NMEICT, MHRD, Government of India.

More information on this Mission is available at the link shown.

|-
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:none;padding:0.097cm;"|
| style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;border-left:1pt solid #000000;border-right:1pt solid #000000;padding:0.097cm;"| This is Snehalatha from IIT Bombay signing off. Thank you for joining.

|}

Biopython/C2/Parsing-Data/English - Revision history

Snehalathak: Created page with " {| style="border-spacing:0;" ! Visual Cue ! Narration |- | style="background-color:#ffffff;border-top:none;border-bottom:1pt solid #000000;..."