Introduction

In the realm of machine learning, linear regression and logistic regression are foundational techniques used for prediction and classification tasks, respectively. These methods form the basis for understanding how machine learning models predict outcomes based on input features. In this chapter, we’ll delve into the roles of linear regression and logistic regression in the context of prediction, exploring their applications, mathematical formulations, and real-world examples.

Linear Regression

Linear regression is a powerful tool for predicting continuous numerical values. Its fundamental principle is to find the best-fitting line that represents the relationship between input features and the target variable. The equation of a simple linear regression model can be expressed as:

$$y=m x+b$$

Where:

• $y$ is the predicted output
• $m$ is the slope of the line
• $x$ is the input feature
• $b$ is the $y$-intercept

For instance, in predicting housing prices, linear regression can help determine how features like square footage, number of bedrooms, and location influence the price. The model estimates the coefficients $m$ and $b$ by minimizing the sum of squared differences between the predicted values and actual target values. This allows us to make predictions for new data points.

Logistic Regression

Logistic regression is a classification technique used to predict binary outcomes (0 or 1), often representing probabilities. Despite its name, it’s not a regression algorithm; rather, it estimates the probability that an instance belongs to a certain class. The logistic function (sigmoid) is used to squash the output into the range [0, 1], making it suitable for probability estimation.

The logistic regression model is defined as:

$$P(y=1)=\frac{1}{1+e^{-\left(b_0+b_1 x_1+b_2 x_2+\ldots+b_n x_n\right)}}$$

Where:

• $P(y=1)$ is the probability of the positive class
• $e$ is the base of the natural logarithm
• $b_0, b_1, b_2, \ldots, b_n$ are the coefficients of the model
• $x_1, x_2, \ldots, x_n$ are the input features

For instance, in email classification as spam or not spam, logistic regression can calculate the probability that an email is spam based on features like the frequency of certain words.