Difference between revisions of "Python-for-Automation/C3/Log-Monitor/English"

Welcome

Learning Objectives

• System logs and
• How to monitor logs

• Ubuntu Linux OS version 22.04
• Python version 3.12.3

Prerequisite

https://spoken-tutorial.org

• You must have basic knowledge of using Linux Terminal and Python.
• For pre-requisite Linux and Python Tutorials, please visit this website.
• Python libraries required for automation must be installed.

Code files

• The files used in this tutorial are provided in the Code files link.
• Please download and extract the files.
• Make a copy and then use them while practicing.

Log monitoring

• Collects and analyzes log data for insights.
• Record events, errors, and system messages.
• Helps to detect issues before they become critical.

Libraries used

• Logging library is used to log events during the execution of the program.
• collections.Counter is used for counting elements in an iterable and for summarizing log data.

Types of logs

• Application Logs will record events within a running application.
• System Logs captures the operating system events.
• Security Logs will track access control and authorization events.
• Audit Logs will analyze what is happening on your system in great detail.

Let us see how to access the log file in our system.

Press Enter.

It records system messages, user activities, and errors.

We will generate a report based on the analysis.

Download logMonitor.py from Code files.

logger = logging.getLogger(__name__)

logger.setLevel(logging.DEBUG)

The log level is set to DEBUG.

This is to capture all messages, including lower-severity ones.

log_file = '/var/log/syslog'

Here I set it as the location of my syslog file.

You can change it according to your file path.

file_handler = logging.FileHandler(log_file)

This handler writes log messages to the log file.

file_handler.setLevel(logging.DEBUG)

file_handler.setFormatter(formatter)

Each message includes time, the logger’s name, log level, and message itself.

logger.addHandler(file_handler)

logger.debug() logs detailed info for debugging purposes.

logger.info() logs general operational events in the program.

logger.warning(), logger.error(), and logger.critical() log issues with increasing severity.

abnormalities = []

with open(log_file, 'r') as file:

Check if the line contains 'ERROR' or 'CRITICAL' log levels.

These lines are considered abnormalities and are stored in a list.

return abnormalities

def summarize_log_file(log_file):

with open(log_file, 'r') as file:

The results are stored in a dictionary.

return log_data

Counter library is used to count the occurrences of each entity.

Next we will find the entity that appears the most frequently in abnormalities.

return most_common_entity

Abnormalities_period.

Months

month_counts = Counter(months)

You can specify your own color palette here.

plt.title('Log Level Distribution')

plt.show()

abnormalities = check_abnormalities(log_file)

abnormality_count = len(abnormalities)

This is done by finding the length of the abnormalities dictionary.

Print the count for each log level.

plot_log_data(log_summary)

Then press enter.

Type source space Automation forward slash bin forward slash activate.

Then press Enter.

Press Enter.

Type sudo space python3 space logMonitor dot py Press Enter.

Summary

Let us summarize.

In this tutorial, we have learnt about

• System logs
• How to monitor logs

Assignment

• Scan the log file for occurrences of the word "ERROR" and "CRITICAL" .
• Extract the timestamp of each occurrence.
• Plot the frequency of errors by time using matplotlib.

About the Spoken Tutorial Project

Please download and watch it.

Spoken Tutorial Workshops

For more details, please write to us.

Answers for THIS Spoken Tutorial

FOSSEE Forum

visit the FOSSEE forum and post your question.

Acknowledgement

Thank You

Thanks for joining.