Multiple Linear Regression

If $Y$ is linearly dependent only on $X$, then we can use the ordinary least square regression line, $\hat{Y}=a+bX$.

However, if $Y$ shows linear dependency on $m$ variables $X_1$, $X_2$, ..., $X_m$, then we need to find the values of $a$ and $m$ other constants ($b_1$, $b_2$, ..., $b_m$). We can then write the regression equation as:

$$\hat{Y}=a+\sum_{i=1}^{m}b_iX_i$$

Matrix Form of the Regression Equation

Let's consider that $Y$ depends on two variables, $X_1$ and $X_2$. We write the regression relation as $\hat{Y}=a+b_1X_1+b_2X_2$. Consider the following matrix operation:

$$\begin{bmatrix} 1 & X_1 & X_2\\ \end{bmatrix}\cdot\begin{bmatrix} a \\ b_1\\ b_2\\ \end{bmatrix}=a+b_1X_1+b_2X_2$$

We define two matrices, $X$ and $B$ as:

$$X=\begin{bmatrix}1 & X_1 & X_2\\\end{bmatrix}$$

$$B=\begin{bmatrix}a \\b_1\\b_2\\\end{bmatrix}$$

Now, we rewrite the regression relation as $\hat{Y}=X\cdot B$. This transforms the regression relation into matrix form.

Generalized Matrix Form

We will consider that $Y$ shows a linear relationship with $m$ variables, $X_1$, $X_2$, ..., $X_m$. Let's say that we made $n$ observations on different tuples $(x_1, x_2, ..., x_m)$:

$y_1=a+b_1\cdot x_{1,1} + b_2\cdot x_{2,1} + ... + b_m\cdot x_{m,1}$
$y_2=a+b_2\cdot x_{1,2} + b_2\cdot x_{2,2} + ... + b_m\cdot x_{m,2}$
$...$
$y_n=a+b_n\cdot x_{1,n} + b_2\cdot x_{2,n} + ... + b_m\cdot x_{m,n}$

Now, we can find the matrices:

$$X=\begin{bmatrix}1 & x_{1,1} & x_{2,1} & x_{3,1} & ... & x_{m,1} \\1 & x_{1,2} & x_{2,2} & x_{3,2} & ... & x_{m,2} \\1 & x_{1,3} & x_{2,3} & x_{3,3} & ... & x_{m,3} \\... & ... & ... & ... & ... & ... \\1 & x_{1,n} & x_{2,n} & x_{3,n} & ... & x_{m,n} \\\end{bmatrix}$$

$$Y=\begin{bmatrix}y_1 \\y_2\\y_3\\...\\y_n\\\end{bmatrix}$$

Finding the Matrix B

We know that $Y=X\cdot B$

$$\Rightarrow X^T\cdot Y=X^T\cdot X \cdot B$$

$$\Rightarrow (X^T\cdot X)^{-1}\cdot X^T \cdot Y=I\cdot B$$

$$\Rightarrow B= (X^T\cdot X)^{-1}\cdot X^T \cdot Y$$

Finding the Value of Y

Suppose we want to find the value of for some tuple $Y$, then $(x_1, x_2, ..., x_m)$,

$$Y=\begin{bmatrix} 1 & x_1 & x_2 & ... & x_m\\ \end{bmatrix}\cdot B$$

Multiple Regression in Python

We can use the fit function in the sklearn.linear_model.LinearRegression class.

from sklearn import linear_model
x = [[5, 7], [6, 6], [7, 4], [8, 5], [9, 6]]
y = [10, 20, 60, 40, 50]
lm = linear_model.LinearRegression()
lm.fit(x, y)
a = lm.intercept_
b = lm.coef_
print(f"Linear regression coefficients between Y and X : a={a}, b_0={b[0]}, b_1={b[1]}")

Linear regression coefficients between Y and X : a=51.953488372092984, b_0=6.65116279069768, b_1=-11.162790697674419