Machine-Learning-using-R/C2/Quadratic-Discriminant-Analysis-in-R/English

From Script | Spoken-Tutorial
Jump to: navigation, search

Title of the script: Quadratic Discriminant Analysis in R

Author: Yate Asseke Ronald Olivera and Debatosh Chakraborty

Keywords: R, RStudio, machine learning, supervised, unsupervised, QDA, quadratic discriminant analysis, video tutorial.


Visual Cue Narration
Show slide

Opening Slide

Welcome to this spoken tutorial on Quadratic Discriminant Analysis in R
Show slide

Learning Objectives

In this tutorial, we will learn about:
  • Quadratic Discriminant Analysis (QDA).
  • Comparison between QDA and LDA.
  • Assumptions for QDA.
  • Applications of QDA
  • Implementation of QDA using Raisin Dataset.
  • Visualization of the QDA separator
  • Limitations of QDA
Show slide

System Specifications

This tutorial is recorded using,
  • Windows 11
  • R version 4.3.0
  • RStudio version 2023.06.1

It is recommended to install R version 4.2.0 or higher.

Show slide

Prerequisites

https://spoken-tutorial.org

To follow this tutorial, the learner should know
  • Basic programming in R.
  • Basics of Machine Learning.

If not, please access the relevant tutorials on this website.

Show slide

Quadratic Discriminant Analysis

  • Quadratic discriminant analysis is a statistical method used for classification.
  • QDA constructs a data-driven non-linear separator between two classes.
  • The covariance matrix for different classes is not necessarily equal.
  • A quadratic function describes the decision boundary between each pair of classes.
Show Slide

Differences between LDA and QDA

Now let’s see the differences between QDA and LDA


  • LDA assumes that each class has the same covariance matrix.
  • QDA relaxes the assumption of an equal covariance matrix for all the classes.
  • LDA constructs a linear boundary, while QDA constructs a non-linear boundary.
  • When the covariance matrices of different classes are the same, QDA reduces to LDA.
Show Slides

Assumptions for QDA


QDA is primarily used when data is multivariate Gaussian.

QDA assumes that each class has its own covariance matrix.


Now let us see the assumption of QDA

QDA is used when data is multivariate Gaussian and each class has its own covariance matrix.

Show slide.

Applications of QDA

  • Medical Diagnosis.
  • Bio-Imaging classification.
  • Fraud Detection.
QDA technique is used in several applications.
Show Slide

Implementation Of QDA

Let us implement QDA on the Raisin dataset with two chosen variables.

For more information on Raisin data please see the Additional Reading material on this tutorial page.

Show slide

Download Files

We will use a script file QDA.R and Raisin Dataset ‘raisin.xlsx’

Please download these files from the Code files link of this tutorial.

Make a copy and then use them while practising.

[Computer screen]

point to QDA.R and the folder QDA.

Point to the MLProject folder on the Desktop.

I have downloaded and moved these files to the QDA folder.

This folder is located in the MLProject folder on my Desktop.

I have also set the QDA folder as my working Directory.

In this tutorial, we will create a QDA classifier model on the raisin dataset.

Let us switch to RStudio.
Click QDA.R in RStudio

Point to QDA.R in RStudio.

Let us open the script QDA.R in RStudio.

For this, click on the script QDA.R.

Script QDA.R opens in RStudio.

[RStudio]

Highlight the command library(readxl)

Highlight the command library(MASS)

Highlight the command library(caret)

Highlight the command library(ggplot2)

library(dplyr)

#install.packages(“package_name”)

Point to the command.

Select and run these commands to import the packages.


We will use the readxl package to load the excel file of our Raisin Dataset.

The MASS package contains the qda() function to create our classifier.

We will use the caret package to create the confusion matrix.

The ggplot2 package will be used to create the decision boundary plot.

We will use the dplyr package to aid the visualisation of the confusion matrix.

Please ensure that all the packages are installed correctly.

As I have already installed the packages.

I have directly imported them.

[RStudio]

data<- read_xlsx("Raisint.xlsx")

Highlight the command data<- read_xlsx("Raisin.xlsx") Run this command to load the Raisin dataset.

Drag boundary to see the Environment tab clearly.

In the Environment tab below Data, you will see the data variable.

Then click on data to load the dataset in the Source window.

[Rstudio]

data$class <- factor(data$class)

Click on QDA.R in the Source window and close the tab.
Highlight the command.

data<-data[c("minorAL",ecc,"class")]

data$class <- factor(data$class)

Select the commands and click the Run button

We now select three columns from data and convert the variable data$class to a factor.

Select and run the commands.

Click on the Environment tab.

Click on data.

Click on data to load the modified data in the Source window.
Point to the data. Now let us split our data into training and testing data.
[RStudio]

set.seed(1)

index_split<- sample(1:nrow(data),size=0.7*nrow(data),replace=FALSE)

Click on QDA.R in the Source window.

In the Source window type these commands

Highlight the command

set.seed(1)

Highlight the command

index_split<- sample(1:nrow(data),size=0.7*nrow(data),replace=FALSE)

First we set a seed for reproducible results.


We will create a vector of indices using sample() function.


It will be 70% of the total number of rows for training and 30% for testing.

The training data is chosen using simple random sampling without replacement.

Select the commands and run them.

[RStudio]

train_data <- data[index_split, ]

test_data <- data[-c(index_split), ]

In the Source window type these commands.
Highlight the command

train_data <- data[index_split, ]

Highlight the command

test_data <- data[-c(index_split), ]

This creates training data, consisting of 630 unique rows.

This creates testing data, consisting of 270 unique rows.

Select the commands and click the Run button.


Point to the sets in the Environment Tab

Click the train_data and test_data

Select the commands and run them.

The data sets are shown in the Environment tab.

Click on train_data and test_data to load them in the Source window.

Let’s perform QDA on the training dataset.
[Rstudio]


QDA_model <- qda(class~.,data=train_data)

Click on QDA.R in the Source window.

In the Source window type these commands

Highlight the command

QDA_model <- qda(class~.,data=train_data)

Highlight the command

QDA_model

Click Save and Click Run buttons.

We use this command to create QDA Model

We pass two parameters to the qda() function.

  1. formula
  2. data on which the model should train.

Click Save.

Select and run the commands.

The output is shown in the console window.

Point the output in the console

Highlight the command Prior probabilities of group

Highlight the command Group means

These are the parameters of our model.

This indicates the composition of classes in the training data.

These indicate the mean values of the predictor variables of each class.

Drag boundary to see the Source window. Drag boundary to see the Source window.
Let us now use our model to make predictions on test data.
[RStudio]

predicted_values <- predict(QDA_model, test_data)

predicted_values

In the Source window type these commands
Highlight the command

predicted_values <- predict(model, test)

Highlight the command

predicted_values

Click on Save and Run buttons.

Let’s use this command to predict the class variable from the test data using the trained QDA model.

This predicts the class and posterior probability for the testing data.

Select and run the commands.

Click on predicted_values in the Environment tab.

Point the output in the console

Highlight the command class


Highlight the command posterior

Click on predicted_values in the Environment tab

This shows us that our predicted variable has two components.

class contains the predicted classes of the testing data.

Posterior contains the posterior probability of an observation belonging to each class.

Let us compute the accuracy of our model.
confusion <- confusionMatrix(test_data$class,predicted_values$class) Click on QDA.R in the source window.

In the Source window type these commands

Highlight the command confusionMatrix(test_data$class,predicted_values$class)

Point to the confusion in the Environment Tab

Highlight the attribute

table

This command creates a confusion matrix list.

The list is created from the actual and predicted class labels of testing data and it is stored in the confusion variable.

It helps to assess the classification model's performance and accuracy.

Select and run the command.

The confusion matrix list is shown in the Environment tab.

Click confusion to load it in the Source window.

confusion list contains a component table containing the required confusion matrix.

plot_confusion_matrix <- function(confusion_matrix){

tab <- confusion_matrix$table

tab = as.data.frame(tab)

tab$Prediction <- factor(tab$Prediction, levels = rev(levels(tab$Prediction)))

tab <- tab %>%

rename(Actual = Reference) %>%

mutate(cor = if_else(Actual == Prediction, 1,0))

tab$cor <- as.factor(tab$cor)

ggplot(tab, aes(Actual,Prediction)) +

geom_tile(aes(fill= cor),alpha = 0.4) + geom_text(aes(label=Freq)) +

scale_fill_manual(values = c("red","green")) +

theme_light() +

theme(legend.position = "None",

line = element_blank()) +

scale_x_discrete(position = "top")

}

Now let’s plot the confusion matrix from the table.

In the Source window type these commands

Highlight the command

tab <- confusion_matrix$table


Highlight the command

tab <- confusion_matrix$table

tab = as.data.frame(tab)

tab$Prediction <- factor(tab$Prediction, levels = rev(levels(tab$Prediction)))

tab <- tab %>%

rename(Actual = Reference) %>%

mutate(cor = if_else(Actual == Prediction, 1,0))

tab$cor <- as.factor(tab$cor)

Highlight the command

ggplot(tab, aes(Actual,Prediction)) +

geom_tile(aes(fill= cor),alpha = 0.4) + geom_text(aes(label=Freq)) +

scale_fill_manual(values = c("red","green")) +

theme_light() +

theme(legend.position = "None",

line = element_blank()) +

scale_x_discrete(position = "top")

}

These commands create a function plot_confusion_matrix to display the confusion matrix from the confusion matrix list created.

It fetches the confusion matrix table from the list.

It creates a data frame from the table which is suitable for plotting using GGPlot2.

It plots the confusion matrix using the data frame created.

It represents correct and incorrect predictions using different colors.

Select and run the commands.

[RStudio]

plot_confusion_matrix(confusion)

In the Source window type these commands
Highlight the command

plot_confusion_matrix(confusion)

Click on Save and Run buttons.

We are using the created plot_confusion_matrix() function to generate the visual plot of the confusion matrix in confusion variable.

Select and run the command.

The output is seen in the plot window.

Point the output in the plot window Drag boundary to see the plot window clearly.


Observe that:

22 samples of class Kecimen have been incorrectly classified.

11 samples of class Besni have been incorrectly classified.

Overall, the model has misclassified only 33 out of 270 samples.

[RStudio]

grid <- expand.grid(minorAL = seq(min(data$minorAL), max(data$minorAL), length = 500),

ecc = seq(min(data$ecc), max(data$ecc), length = 500))

grid$class = predict(QDA_model, newdata = grid)$class

grid$classnum <- as.numeric(grid$class)

In the Source window type these commands.
Highlight the command

grid <- expand.grid(minorAL = seq(min(data$minorAL), max(data$minorAL), length = 500),

ecc = seq(min(data$ecc), max(data$ecc), length = 500))

Highlight the command

grid$class = predict(QDA_model, newdata = grid)$class

grid$classnum <- as.numeric(grid$class)

grid$classnum <- as.numeric(grid$class)

This block of code first creates a grid of points spanning the range of minorAL and ecc features in the dataset.

It stores it in a variable 'grid'.

Then, it uses the QDA model to predict the class of each point in this grid.

It stores these predictions as a new column 'classin the grid dataframe.

The as.numeric function encodes the predicted classes string labels into numeric values.

The resulting grid of points and their predicted classes will be used to visualize the decision boundaries of the QDA model.

Select and run these commands.

Click grid on the Environment tab to load the grid dataframe in the source window.

[RStudio]

ggplot() +

geom_raster(data = grid, aes(x = minorAL, y = ecc, fill = class), alpha = 0.4) +

geom_point(data = train_data, aes(x = minorAL, y = ecc, color = class)) +

geom_contour(data = grid, aes(x = minorAL, y = ecc, z = classnum),

colour = "black", linewidth = 0.7) +

scale_fill_manual(values = c("#ffff46", "#FF46e9")) +

scale_color_manual(values = c("red", "blue")) +

labs(x = "MinorAL", y = "ecc", title = "QDA Decision Boundary") +

theme_minimal()

Click on QDA.R in the Source window.

In the Source window type these commands

Highlight the command


ggplot() +

geom_raster(data = grid, aes(x = var, y = kurt, fill = class), alpha = 0.3) +

geom_point(data = train_data, aes(x = var, y = kurt, color = class)) +

geom_contour(data = grid, aes(x = var, y = kurt, z = classnum),

colour = "black", linewidth = 1.2) +

scale_fill_manual(values = c("#ffff46", "#FF46e9")) +

scale_color_manual(values = c("red", "blue")) +

labs(x = "Variance", y = "Kurtosis", title = "QDA Decision Boundary") +

theme_minimal()

)

We are creating the decision boundary plot using ggplot2.

It plots the grid points with colors indicating the predicted classes.

geom_raster creates a colour map indicating the predicted classes of the grid points.

geom_point plots the training data points in the plot.

geom_contour creates the decision boundary of the QDA.

The scale_fill_manual function assigns specific colors to the classes and so does scale_color_manual function.

The overall plot provides a visual representation of the decision boundary.

And the distribution of training data points of the model.

Select and run these commands.

Drag boundaries to see the plot window clearly.

Point to the plot. We can see that the decision boundary of our model is non-linear.

And our model has separated most of the data points clearly.

Show slide.

Limitations of QDA

  • Multicollinearity among predictors may lead to poor performance.
  • The presence of outliers in data may also lead to poor performance.
These are the limitations of QDA
With this, we come to the end of this tutorial.

Let us summarize.

Show Slide

Summary

In this tutorial we have learned about:
  • Quadratic Discriminant Analysis (QDA).
  • Comparison between QDA and LDA.
  • Assumptions for QDA.
  • Applications of QDA
  • Implementation Of QDA using Raisin Dataset.
  • Visualization of the QDA separator
  • Limitations of QDA
Here is an assignment for you.
Show Slide

Assignment

  • Apply QDA on the wine dataset.
  • Measure the accuracy of the model.

This dataset can be found in the HDclassif package.

Install the package and import the dataset using the data() command

Show slide

About the Spoken Tutorial Project

The video at the following link summarizes the Spoken Tutorial project.

Please download and watch it.

Show slide

Spoken Tutorial Workshops

We conduct workshops using Spoken Tutorials and give certificates.


Please contact us.

Show Slide

Spoken Tutorial Forum to answer questions

Do you have questions in THIS Spoken Tutorial?

Choose the minute and second where you have the question.

Explain your question briefly.

Someone from the FOSSEE team will answer them.

Please visit this site.


Please post your timed queries in this forum.
Show Slide

Forum to answer questions

Do you have any general/technical questions?

Please visit the forum given in the link.

Show Slide

Textbook Companion

The FOSSEE team coordinates the coding of solved examples of popular books and case study projects.

We give certificates to those who do this.

For more details, please visit these sites.

Show Slide

Acknowledgment

The Spoken Tutorial project was established by the Ministry of Education Govt of India.
Show Slide

Thank You

This tutorial is contributed by Yate Asseke Ronald and Debatosh Chakraborty from IIT Bombay.

Thank you for joining.

Contributors and Content Editors

Madhurig, Ushav

Retrieved from "https://script.spoken-tutorial.org/index.php?title=Machine-Learning-using-R/C2/Quadratic-Discriminant-Analysis-in-R/English&oldid=56597"