Python-for-Automation/C3/Log-Monitor/English

Visual Cue	Narration
Show slide: Welcome	Welcome to the Spoken Tutorial on "Log Monitoring ".
Show slide: Learning Objectives	In this tutorial, we will learn about System logs and How to monitor logs
Show slide:	To record this tutorial, I am using Ubuntu Linux os version 22.04 Python version 3.12.3
Show slide: Prerequisite https://spoken-tutorial.org	To follow this tutorial You must have basic knowledge of using Linux Terminal and Python. For pre-requisite Linux and Python Tutorials, please visit this website. Python libraries required for automation must be installed
Show slide: Code files	The files used in this tutorial are provided in the Code files link. Please download and extract the files. Make a copy and then use them while practicing.
Show slide: Log monitoring	Log Monitoring in automation : Collects and analyzes log data for insights. Record events, errors, and system messages. Helps to detect issues before they become critical.
Show slide: Libraries used	Logging library is used to log events during the execution of the program. collections.Counter is used for counting elements in an iterable and for summarizing log data.
Show slide:	There are different types of log files. Application Logs will record events within a running application. System Logs captures the operating system events. Security Logs will track access control and authorization events. Audit Logs will analyze what is happening on your system in great detail.
Only narration:	In this tutorial we’ll see how to monitor syslog files using Python code.
Open terminal(Ctrl + Alt + T)	Open the terminal by pressing Control + Alt + T keys simultaneously. Let us see how to access the log file in our system.
Type cd /var/log	Type cd space forward slash var forward slash log. Press enter.
Type ls	Let us list the files in this directory.Type ls and press enter.
Highlight the output	We can see that all the log files stored in the system are displayed here.
Only narration:	For demonstration purpose we will work with the Syslog files.
Open syslog:Type nano syslog	Type nano syslog to open the syslog file.
Show Syslog	The syslog is a central log file in Ubuntu. It records system messages, user activities, and errors.
Highlight: Timestamp	Each log entry starts with a timestamp of when the event occurred.
Highlight: Host	Next to the timestamp is the Host name of the machine where the event was recorded.
Highlight: Service/Program	After the Host , is the Service/Program name of the system service generating the log.
Highlight: Message	Finally we get the message - which is the details of the event or error.
Only narration	In this tutorial, we will read this file and identify any abnormalities in the file.We will generate a report based on the analysis.
Press ctrl + x	Press ctrl + x to exit the file.
Download logMonitor.py	Let us now look into the code file. Download logMonitor.py from code files.
Open logMonitor.py	Open logMonitor.py file using text editor.
Highlight	First we import necessary libraries.
Highlight: logger = logging.getLogger(__name__) logger.setLevel(logging.DEBUG)	A logger instance is created using logging.getLogger. The log level is set to DEBUG.This is to capture all messages, including lower-severity ones.
Highlight: log_file = '/var/log/syslog'	Then we define the path to the log file.Here I set it as the location of my syslog file. You can change it according to your file path.
Highlight: file_handler = logging.FileHandler(log_file)	A file handler is created using logging.FileHandler. This handler writes log messages to the log file.
Highlight: file_handler.setLevel(logging.DEBUG)	Next we set the file handler to capture DEBUG level messages and above.
Highlight: file_handler.setFormatter(formatter)	We then define a format for log messages. Each message includes time, the logger’s name, log level, and message itself.
Highlight: logger.addHandler(file_handler)	Finally, the handler is added to the logger using logger.addHandler.
Highlight:	The different logging levels are demonstrated here. logger.debug() logs detailed info for debugging purposes. logger.info() logs general operational events in the program. logger.warning(), logger.error(), and logger.critical() log issues with increasing severity.
Highlight:	Now define a function to check for errors or critical messages in the log.
Highlight: abnormalities = []	Initialize an empty list to store abnormal log lines.
Highlight: with open(log_file, 'r') as file:	Open the log file in read mode to analyze its content.
	Using a for loop, iterate through each line.Check if the line contains 'ERROR' or 'CRITICAL' log levels. These lines are considered abnormalities and are stored in a list.
Highlight: return abnormalities	It returns the list of abnormal log entries
Highlight: def summarize_log_file(log_file):	Next we define a function to summarize log messages by their levels.
Highlight:	Initialize a dictionary to count occurrences of each log level.
Highlight : with open(log_file, 'r') as file:	Open the log file in read mode for processing.
Highlight:	Using a for loop, iterate through each line and count the occurrences of each log level. The results are stored in a dictionary.
Highlight: return log_data	Finally we return the dictionary summarizing log level counts.
	Now we define a function to find the entity causing the most abnormalities.
	Extract and store the entity name from each abnormal log entry. Counter library is used to count the occurrences of each entity. Next we will find the entity that appears the most frequently in abnormalities.
Highlight: return most_common_entity	Finally return the most frequent abnormal entity and its count.
	Next we define a function to find the period with the most abnormalities.
	Extract the date from each abnormality entry and get only the year and month from the dates.
Highlight: month_counts = Counter(months)	Using Counter we count occurrences of each month.
	Then find the month with the most abnormalities using the most_common function.
	Next we define a function to plot the log level data.
	In this function we extract the log levels as a list from the log data dictionary.
	Extract the counts for each log level.
	Then using plt.figure create a figure for the plot with a specified size.
	We plot a bar chart for the log levels with corresponding counts.You can specify your own color palette here.
	Set the x-axis label to 'Log Level' and y-axis label to 'Count'
Highlight: plt.title('Log Level Distribution')	Also set the title of the plot as 'Log Level Distribution'.
Highlight: plt.show()	Then we display the plot using plt.show() .
Highlight: abnormalities = check_abnormalities(log_file)	Next call the check_abnormalities function to find abnormal log entries.
Highlight: abnormality_count = len(abnormalities)	Then calculate the total number of abnormalities found. This is done by finding the length of the abnormalities dictionary.
	Finally, print the total number of abnormalities found in the log file.
	Next we call summarize_log_file to summarize the log data by log levels.
	Using a for loop, iterate through the log summary data. Print the count for each log level.
	We now call find_most_abnormal_entity to identify the most abnormal entity and print it.
Highlight:	Then call the find_max_abnormalities_period to identify the period with the most abnormalities and print it.
Highlight: plot_log_data(log_summary)	Finally we call plot_log_data to plot the summarized log data.
Only narration	Now let us execute the code.
Only narration	Save the code as logMonitor.py in the Downloads folder.
Open terminal(Ctrl + Alt + T)	Open the terminal by pressing Control + Alt + T keys simultaneously.
>Type source Automation/bin/activate Then press enter.	We’ll open the virtual environment for the Automation series.Type source space Automation forward slash bin forward slash activate. Then press enter.
>Type cd Downloads	Now type, cd Downloads. Press Enter.
>Type sudo python3 logMonitor.py	Let us run the code. Type sudo space python3 space logMonitor dot py Press Enter.
Highlight :Total abnormalities found in log file: 179	We can see that the count of total abnormalities found in my log file is shown.
Highlight:	Next the log summary is also printed which gives me the count of each log level.
Highlight :	The entity or activity that causes the most abnormalities is printed.
Highlight :	Followed by the period with the maximum amount of abnormalities is also printed.
Show the plot:	Here we can see the boxplot of my syslog summary.
>Type: deactivate	Type deactivate to exit the virtual environment.
Show slide: Summary	This brings us to the end of the tutorial. Let us summarize. In this tutorial, we have learnt about System logs How to monitor logs
Show slide: Assignment	As an assignment, please do the following: Scan the log file for occurrences of the word "ERROR" and "CRITICAL" . Extract the timestamp of each occurrence. Plot the frequency of errors by time using matplotlib.
Show slide: About the Spoken Tutorial Project	The video at the following link summarizes the Spoken Tutorial project. Please download and watch it.
Show Slide: Spoken Tutorial Workshops	The Spoken Tutorial Project team conducts workshops and gives certificates. For more details, please write to us.
Show Slide: Answers for THIS Spoken Tutorial	Please post your timed queries in this forum.
Show Slide: FOSSEE Forum	For any general or technical questions on Python for Automation, visit the FOSSEE forum and post your question.
Show slide Acknowledgement	Spoken Tutorial Project was established by the Ministry of Education, Government of India.
Show slide: Thank You	This is Dhaarani Pushpam S, a FOSSEE Semester long intern 2024, IIT Bombay signing off. Thanks for joining.

Contributors and Content Editors

Madhurig, Nirmala Venkat