Difference between revisions of "Python-for-Automation/C3/Web-Scraping/English"

Welcome

Learning Objectives

• Scrape data from any website
• Extract it to a CSV file
• Perform basic data analysis
• Generate visualizations

System Requirements

• Ubuntu Linux OS 22.04
• Python 3.12.3

• Ubuntu Linux OS version 22.04
• Python 3.12.3

https://www.spoken-tutorial.org

• You must have basic knowledge of using Linux Terminal and Python
• For pre-requisite Linux and Python Tutorials, please visit this website
• Python libraries required for automation must be installed

• The files used in this tutorial are provided in the Code files link.
• Please download and extract the files.
• Make a copy and then use them while practicing.

Web Scraping

We will automate extracting data and information from web pages and parsing HTML content.

• Requests library to fetch HTML content from a web page
• BeautifulSoup library to parse and extract information from the HTML content
• Matplotlib library to create static, animated, and interactive visualizations
• Pandas library to provide data structures and data analysis tools
• Xlsxwriter library is used for creating and formatting Excel files
• Datetime library handles date operations like parsing strings into date objects

Web Scraping - Example

Data analysis is done with workshops conducted between 2022 and 2023 on certain software.

Data such as State, City, Institution, Department, Organizer, Date and Participants are handled.

Open the Text Editor with the source file

Now, we will go through the source code in the text editor.

from bs4 import BeautifulSoup

import pandas as pd

from datetime import datetime

import matplotlib.pyplot as plt

from mpl_toolkits.mplot3d import Axes3D

Then, we parse it with BeautifulSoup to return a soup object for further analysis.

Then, we find the table with the provided class name in the HTML.

An empty list is returned if no table is found.

Then, an empty list is initialized to store the extracted data.

For each row in the table, we extract all cells and strip the text content from each cell.

Then, we store the values in a list.

Then, we convert the date column to a datetime object.

This is to verify if it falls within the specified time range.

If both conditions are met, the row's data is added to the list.

Pagination is the process of dividing content into discrete pages.

If page numbers are found, it returns the last one as the total number of pages.

If no pagination is found, it returns 1, assuming there's only one page.

First, it fetches the initial page’s content and then determines the number of pages.

Finally we extract the relevant data based on the filters.

First, we modify the url to request each specific page, fetch content and extract data.

The data from each page is then appended to the overall dataset and returned.

piechart_visualization function generates a pie chart showing the FOSS categories.

Additionally, the FOSS counts are saved to an Excel sheet with the specified sheet name.

It then writes the data to an excel sheet with the specified sheet name.

Then counting and retrieving unique FOSS types, departments, and organizers.

If no data is found for the given college, it returns None.

Otherwise, it returns the unique FOSS categories, departments, and organizers.

Then we group the data by city and FOSS type and calculate the total number of participants.

Unique city names and FOSS types are extracted, and numeric mappings are created to represent them.

The axes are labeled, and ticks are mapped to cities and FOSS values with a title.

If the name is longer than 31 characters, it truncates the name to 28 characters with ellipsis.

We also set a date range from January 1 2022 to January 1 2023.

We then convert the start and end dates from string format to datetime objects for comparison.

The DataFrame is then created with specified columns, and duplicates are removed.

We also analyze and save workshop data for a specific college if available.

The data is saved and a confirmation message is printed.

Start Virtual Environment

Type

> source Automation/bin/activate

We will open the virtual environment we created for the Automation series.

Type source space Automation forward slash bin forward slash activate.

Then press enter.

Type

> cd Downloads

> python3 webscraping.py

Then type python3 webscraping.py and press Enter

The pie chart displays the distribution of workshops by FOSS category.

This shows the proportion of each FOSS type as a percentage of the total workshops.

Close the window to see the next graph.

This shows the number of workshops conducted in each city.

Close the window to go to the next graph.

This graph displays cities, FOSS types and participant counts on the three axes.

Files App > Downloads > st_data.xlsx

Zoom and show the data

We can see the 10 columns here which contain all the data the website had.

This is the data of the pie chart we have seen earlier.

This is the data of the bar graph we have seen earlier.

Type

> deactivate

Type deactivate.

• Price Monitoring - E-commerce websites scrape their Competitor websites.
• They monitor their prices and adjust theirs accordingly.

• Academic Research - Researchers can scrape data from academic journals and websites.
• They collect information for studies and research.
• Financial Data Analysis - Analysts use web scraping to collect data from financial websites.
• They analyze stock prices, market trends etc.

In this tutorial, we have learnt to

• Extract data from websites
• Save data to a CSV file
• Perform basic data analysis
• Generate visualizations

Assignment

• Extract the workshop data using different foss filters, start and end date.
• Write the data to an Excel sheet.

Spoken Tutorial Workshops

For more details, please write to us.

FOSSEE Forum

Thanks for joining.