Difference between revisions of "Machine-Learning-using-R/C2/Quadratic-Discriminant-Analysis-in-R/English"

Opening Slide

Learning Objectives

• Quadratic Discriminant Analysis (QDA).
• Comparison between QDA and LDA.
• Assumptions for QDA.
• Applications of QDA
• Implementation of QDA using Raisin Dataset.
• Visualization of the QDA separator
• Limitations of QDA

System Specifications

• Windows 11
• R version 4.3.0
• RStudio version 2023.06.1

It is recommended to install R version 4.2.0 or higher.

Prerequisites

https://spoken-tutorial.org

• Basic programming in R.
• Basics of Machine Learning.

If not, please access the relevant tutorials on this website.

Quadratic Discriminant Analysis

• Quadratic discriminant analysis is a statistical method used for classification.
• QDA constructs a data-driven non-linear separator between two classes.
• The covariance matrix for different classes is not necessarily equal.
• A quadratic function describes the decision boundary between each pair of classes.

Differences between LDA and QDA

• LDA assumes that each class has the same covariance matrix.
• QDA relaxes the assumption of an equal covariance matrix for all the classes.
• LDA constructs a linear boundary, while QDA constructs a non-linear boundary.
• When the covariance matrices of different classes are the same, QDA reduces to LDA.

Assumptions for QDA

QDA is primarily used when data is multivariate Gaussian.

QDA assumes that each class has its own covariance matrix.

QDA is used when data is multivariate Gaussian and each class has its own covariance matrix.

Applications of QDA

• Medical Diagnosis.
• Bio-Imaging classification.
• Fraud Detection.

Implementation Of QDA

For more information on Raisin data please see the Additional Reading material on this tutorial page.

Download Files

Please download these files from the Code files link of this tutorial.

Make a copy and then use them while practising.

point to QDA.R and the folder QDA.

Point to the MLProject folder on the Desktop.

This folder is located in the MLProject folder on my Desktop.

I have also set the QDA folder as my working Directory.

In this tutorial, we will create a QDA classifier model on the raisin dataset.

Point to QDA.R in RStudio.

For this, click on the script QDA.R.

Script QDA.R opens in RStudio.

Highlight the command library(readxl)

Highlight the command library(MASS)

Highlight the command library(caret)

Highlight the command library(ggplot2)

library(dplyr)

#install.packages(“package_name”)

Point to the command.

We will use the readxl package to load the excel file of our Raisin Dataset.

The MASS package contains the qda() function to create our classifier.

We will use the caret package to create the confusion matrix.

The ggplot2 package will be used to create the decision boundary plot.

We will use the dplyr package to aid the visualisation of the confusion matrix.

Please ensure that all the packages are installed correctly.

As I have already installed the packages.

I have directly imported them.

data<- read_xlsx("Raisint.xlsx")

Drag boundary to see the Environment tab clearly.

In the Environment tab below Data, you will see the data variable.

Then click on data to load the dataset in the Source window.

data$class <- factor(data$class)

data<-data[c("minorAL",ecc,"class")]

data$class <- factor(data$class)

Select the commands and click the Run button

Select and run the commands.

Click on data.

set.seed(1)

index_split<- sample(1:nrow(data),size=0.7*nrow(data),replace=FALSE)

In the Source window type these commands

set.seed(1)

Highlight the command

index_split<- sample(1:nrow(data),size=0.7*nrow(data),replace=FALSE)

We will create a vector of indices using sample() function.

It will be 70% of the total number of rows for training and 30% for testing.

The training data is chosen using simple random sampling without replacement.

Select the commands and run them.

train_data <- data[index_split, ]

test_data <- data[-c(index_split), ]

train_data <- data[index_split, ]

Highlight the command

test_data <- data[-c(index_split), ]

This creates testing data, consisting of 270 unique rows.

Point to the sets in the Environment Tab

Click the train_data and test_data

The data sets are shown in the Environment tab.

Click on train_data and test_data to load them in the Source window.

QDA_model <- qda(class~.,data=train_data)

In the Source window type these commands

QDA_model <- qda(class~.,data=train_data)

Highlight the command

QDA_model

Click Save and Click Run buttons.

We pass two parameters to the qda() function.

1. formula
2. data on which the model should train.

Click Save.

Select and run the commands.

The output is shown in the console window.

Highlight the command Prior probabilities of group

Highlight the command Group means

This indicates the composition of classes in the training data.

These indicate the mean values of the predictor variables of each class.

predicted_values <- predict(QDA_model, test_data)

predicted_values

predicted_values <- predict(model, test)

Highlight the command

predicted_values

Click on Save and Run buttons.

This predicts the class and posterior probability for the testing data.

Select and run the commands.

Point the output in the console

Highlight the command class

Highlight the command posterior

This shows us that our predicted variable has two components.

class contains the predicted classes of the testing data.

Posterior contains the posterior probability of an observation belonging to each class.

In the Source window type these commands

Point to the confusion in the Environment Tab

Highlight the attribute

table

The list is created from the actual and predicted class labels of testing data and it is stored in the confusion variable.

It helps to assess the classification model's performance and accuracy.

Select and run the command.

The confusion matrix list is shown in the Environment tab.

Click confusion to load it in the Source window.

confusion list contains a component table containing the required confusion matrix.

tab <- confusion_matrix$table

tab = as.data.frame(tab)

tab$Prediction <- factor(tab$Prediction, levels = rev(levels(tab$Prediction)))

tab <- tab %>%

rename(Actual = Reference) %>%

mutate(cor = if_else(Actual == Prediction, 1,0))

tab$cor <- as.factor(tab$cor)

ggplot(tab, aes(Actual,Prediction)) +

geom_tile(aes(fill= cor),alpha = 0.4) + geom_text(aes(label=Freq)) +

scale_fill_manual(values = c("red","green")) +

theme_light() +

theme(legend.position = "None",

line = element_blank()) +

scale_x_discrete(position = "top")

}

In the Source window type these commands

Highlight the command

tab <- confusion_matrix$table

Highlight the command

tab <- confusion_matrix$table

tab = as.data.frame(tab)

tab$Prediction <- factor(tab$Prediction, levels = rev(levels(tab$Prediction)))

tab <- tab %>%

rename(Actual = Reference) %>%

mutate(cor = if_else(Actual == Prediction, 1,0))

tab$cor <- as.factor(tab$cor)

Highlight the command

ggplot(tab, aes(Actual,Prediction)) +

geom_tile(aes(fill= cor),alpha = 0.4) + geom_text(aes(label=Freq)) +

scale_fill_manual(values = c("red","green")) +

theme_light() +

theme(legend.position = "None",

line = element_blank()) +

scale_x_discrete(position = "top")

}

It fetches the confusion matrix table from the list.

It creates a data frame from the table which is suitable for plotting using GGPlot2.

It plots the confusion matrix using the data frame created.

It represents correct and incorrect predictions using different colors.

Select and run the commands.

plot_confusion_matrix(confusion)

plot_confusion_matrix(confusion)

Click on Save and Run buttons.

Select and run the command.

The output is seen in the plot window.

Observe that:

22 samples of class Kecimen have been incorrectly classified.

11 samples of class Besni have been incorrectly classified.

Overall, the model has misclassified only 33 out of 270 samples.

grid <- expand.grid(minorAL = seq(min(data$minorAL), max(data$minorAL), length = 500),

ecc = seq(min(data$ecc), max(data$ecc), length = 500))

grid$class = predict(QDA_model, newdata = grid)$class

grid$classnum <- as.numeric(grid$class)

grid <- expand.grid(minorAL = seq(min(data$minorAL), max(data$minorAL), length = 500),

ecc = seq(min(data$ecc), max(data$ecc), length = 500))

Highlight the command

grid$class = predict(QDA_model, newdata = grid)$class

grid$classnum <- as.numeric(grid$class)

grid$classnum <- as.numeric(grid$class)

It stores it in a variable 'grid'.

Then, it uses the QDA model to predict the class of each point in this grid.

It stores these predictions as a new column 'classin the grid dataframe.

The as.numeric function encodes the predicted classes string labels into numeric values.

The resulting grid of points and their predicted classes will be used to visualize the decision boundaries of the QDA model.

Select and run these commands.

Click grid on the Environment tab to load the grid dataframe in the source window.

ggplot() +

geom_raster(data = grid, aes(x = minorAL, y = ecc, fill = class), alpha = 0.4) +

geom_point(data = train_data, aes(x = minorAL, y = ecc, color = class)) +

geom_contour(data = grid, aes(x = minorAL, y = ecc, z = classnum),

colour = "black", linewidth = 0.7) +

scale_fill_manual(values = c("#ffff46", "#FF46e9")) +

scale_color_manual(values = c("red", "blue")) +

labs(x = "MinorAL", y = "ecc", title = "QDA Decision Boundary") +

theme_minimal()

In the Source window type these commands

ggplot() +

geom_raster(data = grid, aes(x = var, y = kurt, fill = class), alpha = 0.3) +

geom_point(data = train_data, aes(x = var, y = kurt, color = class)) +

geom_contour(data = grid, aes(x = var, y = kurt, z = classnum),

colour = "black", linewidth = 1.2) +

scale_fill_manual(values = c("#ffff46", "#FF46e9")) +

scale_color_manual(values = c("red", "blue")) +

labs(x = "Variance", y = "Kurtosis", title = "QDA Decision Boundary") +

theme_minimal()

)

It plots the grid points with colors indicating the predicted classes.

geom_raster creates a colour map indicating the predicted classes of the grid points.

geom_point plots the training data points in the plot.

geom_contour creates the decision boundary of the QDA.

The scale_fill_manual function assigns specific colors to the classes and so does scale_color_manual function.

The overall plot provides a visual representation of the decision boundary.

And the distribution of training data points of the model.

Select and run these commands.

Drag boundaries to see the plot window clearly.

And our model has separated most of the data points clearly.

Limitations of QDA

• Multicollinearity among predictors may lead to poor performance.
• The presence of outliers in data may also lead to poor performance.

Let us summarize.

Summary

• Quadratic Discriminant Analysis (QDA).
• Comparison between QDA and LDA.
• Assumptions for QDA.
• Applications of QDA
• Implementation Of QDA using Raisin Dataset.
• Visualization of the QDA separator
• Limitations of QDA

Assignment

• Apply QDA on the wine dataset.
• Measure the accuracy of the model.

This dataset can be found in the HDclassif package.

Install the package and import the dataset using the data() command

About the Spoken Tutorial Project

Please download and watch it.

Spoken Tutorial Workshops

Please contact us.

Spoken Tutorial Forum to answer questions

Do you have questions in THIS Spoken Tutorial?

Choose the minute and second where you have the question.

Explain your question briefly.

Someone from the FOSSEE team will answer them.

Please visit this site.

Forum to answer questions

Please visit the forum given in the link.

Textbook Companion

We give certificates to those who do this.

For more details, please visit these sites.

Acknowledgment

Thank You

Thank you for joining.