Difference between revisions of "Python-for-Automation/C3/Log-Monitor/English"

From Script | Spoken-Tutorial
Jump to: navigation, search
(Created page with " <div style="margin-left:1.27cm;margin-right:0cm;"></div> {| border="1" |- || '''Visual Cue''' || '''Narration''' |- |- style="border:1pt solid #000000;padding-top:0cm;paddin...")
 
(No difference)

Latest revision as of 19:30, 26 November 2024


Visual Cue Narration
Show slide:

Welcome

Welcome to the Spoken Tutorial on "Log Monitoring ".
Show slide:

Learning Objectives

In this tutorial, we will learn about
  • System logs and
  • How to monitor logs
Show slide: To record this tutorial, I am using
  • Ubuntu Linux os version 22.04
  • Python version 3.12.3
Show slide:

Prerequisite

https://spoken-tutorial.org

To follow this tutorial
  • You must have basic knowledge of using Linux Terminal and Python.
  • For pre-requisite Linux and Python Tutorials, please visit this website.
  • Python libraries required for automation must be installed
Show slide:

Code files

  • The files used in this tutorial are provided in the Code files link.
  • Please download and extract the files.
  • Make a copy and then use them while practicing.
Show slide:

Log monitoring

Log Monitoring in automation :
  • Collects and analyzes log data for insights.
  • Record events, errors, and system messages.
  • Helps to detect issues before they become critical.
Show slide: Libraries used
  • Logging library is used to log events during the execution of the program.
  • collections.Counter is used for counting elements in an iterable and for summarizing log data.
Show slide: There are different types of log files.
  • Application Logs will record events within a running application.
  • System Logs captures the operating system events.
  • Security Logs will track access control and authorization events.
  • Audit Logs will analyze what is happening on your system in great detail.
Only narration: In this tutorial we’ll see how to monitor syslog files using Python code.
Open terminal(Ctrl + Alt + T) Open the terminal by pressing Control + Alt + T keys simultaneously.

Let us see how to access the log file in our system.

Type cd /var/log Type cd space forward slash var forward slash log.

Press enter.

Type ls Let us list the files in this directory.Type ls and press enter.
Highlight the output We can see that all the log files stored in the system are displayed here.
Only narration: For demonstration purpose we will work with the Syslog files.
Open syslog:Type nano syslog Type nano syslog to open the syslog file.
Show Syslog The syslog is a central log file in Ubuntu.

It records system messages, user activities, and errors.

Highlight: Timestamp Each log entry starts with a timestamp of when the event occurred.
Highlight: Host Next to the timestamp is the Host name of the machine where the event was recorded.
Highlight: Service/Program After the Host , is the Service/Program name of the system service generating the log.
Highlight: Message Finally we get the message - which is the details of the event or error.
Only narration In this tutorial, we will read this file and identify any abnormalities in the file.We will generate a report based on the analysis.
Press ctrl + x Press ctrl + x to exit the file.
Download logMonitor.py Let us now look into the code file.

Download logMonitor.py from code files.

Open logMonitor.py Open logMonitor.py file using text editor.
Highlight First we import necessary libraries.
Highlight:

logger = logging.getLogger(__name__)

logger.setLevel(logging.DEBUG)

A logger instance is created using logging.getLogger.

The log level is set to DEBUG.This is to capture all messages, including lower-severity ones.

Highlight:

log_file = '/var/log/syslog'

Then we define the path to the log file.Here I set it as the location of my syslog file. You can change it according to your file path.
Highlight:

file_handler = logging.FileHandler(log_file)

A file handler is created using logging.FileHandler.

This handler writes log messages to the log file.

Highlight:

file_handler.setLevel(logging.DEBUG)

Next we set the file handler to capture DEBUG level messages and above.
Highlight:

file_handler.setFormatter(formatter)

We then define a format for log messages.

Each message includes time, the logger’s name, log level, and message itself.

Highlight:

logger.addHandler(file_handler)

Finally, the handler is added to the logger using logger.addHandler.
Highlight: The different logging levels are demonstrated here.

logger.debug() logs detailed info for debugging purposes.

logger.info() logs general operational events in the program.

logger.warning(), logger.error(), and logger.critical() log issues with increasing severity.

Highlight: Now define a function to check for errors or critical messages in the log.
Highlight:

abnormalities = []

Initialize an empty list to store abnormal log lines.
Highlight:

with open(log_file, 'r') as file:

Open the log file in read mode to analyze its content.
Using a for loop, iterate through each line.Check if the line contains 'ERROR' or 'CRITICAL' log levels.

These lines are considered abnormalities and are stored in a list.

Highlight:

return abnormalities

It returns the list of abnormal log entries
Highlight:

def summarize_log_file(log_file):

Next we define a function to summarize log messages by their levels.
Highlight: Initialize a dictionary to count occurrences of each log level.
Highlight :

with open(log_file, 'r') as file:

Open the log file in read mode for processing.
Highlight: Using a for loop, iterate through each line and count the occurrences of each log level.

The results are stored in a dictionary.

Highlight:

return log_data

Finally we return the dictionary summarizing log level counts.
Now we define a function to find the entity causing the most abnormalities.
Extract and store the entity name from each abnormal log entry.

Counter library is used to count the occurrences of each entity.

Next we will find the entity that appears the most frequently in abnormalities.

Highlight:

return most_common_entity

Finally return the most frequent abnormal entity and its count.
Next we define a function to find the period with the most abnormalities.
Extract the date from each abnormality entry and get only the year and month from the dates.
Highlight:

month_counts = Counter(months)

Using Counter we count occurrences of each month.
Then find the month with the most abnormalities using the most_common function.
Next we define a function to plot the log level data.
In this function we extract the log levels as a list from the log data dictionary.
Extract the counts for each log level.
Then using plt.figure create a figure for the plot with a specified size.
We plot a bar chart for the log levels with corresponding counts.You can specify your own color palette here.
Set the x-axis label to 'Log Level' and y-axis label to 'Count'
Highlight:

plt.title('Log Level Distribution')

Also set the title of the plot as 'Log Level Distribution'.
Highlight:

plt.show()

Then we display the plot using plt.show() .
Highlight:

abnormalities = check_abnormalities(log_file)

Next call the check_abnormalities function to find abnormal log entries.
Highlight:

abnormality_count = len(abnormalities)

Then calculate the total number of abnormalities found.

This is done by finding the length of the abnormalities dictionary.

Finally, print the total number of abnormalities found in the log file.
Next we call summarize_log_file to summarize the log data by log levels.
Using a for loop, iterate through the log summary data.

Print the count for each log level.

We now call find_most_abnormal_entity to identify the most abnormal entity and print it.
Highlight: Then call the find_max_abnormalities_period to identify the period with the most abnormalities and print it.
Highlight:

plot_log_data(log_summary)

Finally we call plot_log_data to plot the summarized log data.
Only narration Now let us execute the code.
Only narration Save the code as logMonitor.py in the Downloads folder.
Open terminal(Ctrl + Alt + T) Open the terminal by pressing Control + Alt + T keys simultaneously.
>Type source Automation/bin/activate

Then press enter.

We’ll open the virtual environment for the Automation series.Type source space Automation forward slash bin forward slash activate.

Then press enter.

>Type cd Downloads Now type, cd Downloads.

Press Enter.

>Type sudo python3 logMonitor.py Let us run the code.

Type sudo space python3 space logMonitor dot py Press Enter.

Highlight :Total abnormalities found in log file: 179 We can see that the count of total abnormalities found in my log file is shown.
Highlight: Next the log summary is also printed which gives me the count of each log level.
Highlight : The entity or activity that causes the most abnormalities is printed.
Highlight : Followed by the period with the maximum amount of abnormalities is also printed.
Show the plot: Here we can see the boxplot of my syslog summary.
>Type: deactivate Type deactivate to exit the virtual environment.
Show slide:

Summary

This brings us to the end of the tutorial. Let us summarize.

In this tutorial, we have learnt about

  • System logs
  • How to monitor logs
Show slide:

Assignment

As an assignment, please do the following:
  • Scan the log file for occurrences of the word "ERROR" and "CRITICAL" .
  • Extract the timestamp of each occurrence.
  • Plot the frequency of errors by time using matplotlib.
Show slide:

About the Spoken Tutorial Project

The video at the following link summarizes the Spoken Tutorial project.

Please download and watch it.

Show Slide:

Spoken Tutorial Workshops

The Spoken Tutorial Project team conducts workshops and gives certificates.

For more details, please write to us.

Show Slide:

Answers for THIS Spoken Tutorial

Please post your timed queries in this forum.
Show Slide:

FOSSEE Forum

For any general or technical questions on Python for

Automation, visit the FOSSEE forum and post your question.

Show slide

Acknowledgement

Spoken Tutorial Project was established by the Ministry of Education, Government of India.
Show slide:

Thank You

This is Dhaarani Pushpam S, a FOSSEE Semester long intern 2024, IIT Bombay signing off.


Thanks for joining.

Contributors and Content Editors

Nirmala Venkat

Retrieved from "https://script.spoken-tutorial.org/index.php?title=Python-for-Automation/C3/Log-Monitor/English&oldid=56708"