This chapter explores fundamental matrix concepts, including operations, applications in summarizing data, and their role in multiple linear regression.
The dot product of two vectors is the sum of the element-wise multiplications of their corresponding entries.
For matrix multiplication, to compute the element at row \(i\), column \(j\) of \(AB\), we take:
1. Row \(i\) from \(A\)
2. Column \(j\) from \(B\)
3. Multiply corresponding elements and sum them:
\[
AB_{i,j} = A_{i,1}B_{1,j} + A_{i,2}B_{2,j} + \dots + A_{i,n}B_{n,j}
\]
Let:
\[
A =
\begin{bmatrix}
1 & 2 & 3 \\
4 & 5 & 6
\end{bmatrix}
\] (\(2 \times 3\) matrix) and
\[
B =
\begin{bmatrix}
7 & 8 \\
9 & 10 \\
11 & 12
\end{bmatrix}
\] (\(3 \times 2\) matrix)
Since \(A\) has 3 columns and \(B\) has 3 rows, multiplication is valid, and the resulting matrix \(AB\) has dimension \(2 \times 2\).
To compute the element in row 1, column 1 of \(AB\):
\[
AB_{1,1} = (1 \times 7) + (2 \times 9) + (3 \times 11) = 7 + 18 + 33 = 58
\]
To compute the element in row 1, column 2:
\[
AB_{1,2} = (1 \times 8) + (2 \times 10) + (3 \times 12) = 8 + 20 + 36 = 64
\]
To compute the element in row 2, column 1:
\[
AB_{2,1} = (4 \times 7) + (5 \times 9) + (6 \times 11) = 28 + 45 + 66 = 139
\]
To compute the element in row 2, column 2:
\[
AB_{2,2} = (4 \times 8) + (5 \times 10) + (6 \times 12) = 32 + 50 + 72 = 154
\]
Thus, the final matrix product is:
\[
AB =
\begin{bmatrix}
58 & 64 \\
139 & 154
\end{bmatrix}
\]
Let \(C_{n \times 1}\) be a column vector of chosen numbers, and let \(Y_{n \times 1}\) be a column vector representing a sample of data. The computation \(C' Y\) (the transpose of \(C\) multiplied by \(Y\)) allows us to compute:
If all elements of \(C\) are equal to \(\frac{1}{n}\), then the matrix multiplication: \[ C' Y = \begin{bmatrix} \frac{1}{n} & \frac{1}{n} & \dots & \frac{1}{n} \end{bmatrix} \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} = \frac{1}{n} (y_1 + y_2 + \dots + y_n) = \frac{1}{n} \sum_{i=1}^{n} y_i = \bar{Y} \] This shows that \(C' Y\) computes the sample mean \(\bar{Y}\) using matrix multiplication.
If \(C\) contains weights that sum to 1 and are all positive, then the matrix multiplication: \[ C' Y = \begin{bmatrix} w_1 & w_2 & \dots & w_n \end{bmatrix} \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} = w_1 y_1 + w_2 y_2 + \dots + w_n y_n = \sum_{i=1}^{n} w_i Y_i = \bar{Y}_w \] This shows that \(C' Y\) computes the weighted mean \(\bar{Y}_w\) using matrix multiplication.
If all elements of \(C\) are set to 1, then the matrix multiplication: \[ C' Y = \begin{bmatrix} 1 & 1 & \dots & 1 \end{bmatrix} \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} = y_1 + y_2 + \dots + y_n = \sum_{i=1}^{n} y_i \] This shows that \(C' Y\) computes the sum of all values in \(Y\) using matrix multiplication.
If \(C\) is not a vector but a matrix, then multiple computations can be performed simultaneously using matrix multiplication: \[ C' Y = \begin{bmatrix} \frac{1}{n} & \frac{1}{n} & \dots & \frac{1}{n} \\ 1 & 1 & \dots & 1 \end{bmatrix} \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} = \begin{bmatrix} \bar{Y} \\ \sum Y \end{bmatrix} \] This shows that \(C' Y\) stores the mean in the first row and the sum in the second, efficiently computing both.
If \(Y\) is multiplied by its own transpose, then the matrix multiplication: \[ Y' Y = \begin{bmatrix} y_1 & y_2 & \dots & y_n \end{bmatrix} \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} = y_1^2 + y_2^2 + \dots + y_n^2 = \sum_{i=1}^{n} Y_i^2 \] This shows that \(Y' Y\) computes the sum of squares of \(Y\) using matrix multiplication, which is useful for variance and regression calculations.
This section describes how to summarize numerical data when working with multiple variables, leading up to the multivariate normal distribution (MVN) and how to estimate its parameters.
Basic summaries for individual variables:
When working with multiple variables:
For two variables \(X_1\) and \(X_2\):
\(X_1\)
\(X_2\)
In addition to individual means and standard deviations, we also measure covariance (\(\sigma_{12}\)), which describes how the two variables vary together. Covariance is directly related to correlation.
Matrices provide a compact way to represent the relationships between multiple variables.
\(X_1\) and \(X_2\) follow a multivariate normal distribution (MVN):
A multivariate normal distribution (MVN) extends the normal distribution to multiple variables, describing their relationships through both their individual distributions and their dependencies. It is defined by:
In practice, we estimate MVN parameters from a sample:
Sample Mean Vector: \[ \bar{X} = \begin{bmatrix} \bar{X}_1 \\ \bar{X}_2 \end{bmatrix} \] Sample Covariance Matrix: \[ S = \begin{bmatrix} s_1^2 & s_{12} \\ s_{12} & s_2^2 \end{bmatrix} \]
where:
To check if data follows an MVN distribution:
mqqnorm(dataset)) should be linear with slight tail deviations.The multivariate normal distribution is widely used in statistics and machine learning:
How does linear regression use matrix operations and MVN assumptions?
To estimate and test coefficients in MLR:
M = matrix(c(0,1,2,3), 2, 2) # 2x2 (r x c) matrix, filled by column
N = matrix(c(4,3,2,1), 2, 2)
t(M) # Transpose
M * N # Elementwise multiplication
M %*% N # Matrix multiplication
Minv = solve(M) # Matrix inverse
M %*% Minv # Should return identity matrix c(1,0,0,1): confirms inverse# Predict y from x using matrix algebra
x = c(1,2,3,4,5)
y = c(6.5, 10.8, 14, 21.2, 26.8) # Response vector
# Design matrix: column of 1s for intercept, then x
bigX = cbind(rep(1, 5), x)
# Estimate beta using matrix multiplication
beta_hat = solve(t(bigX) %*% bigX) %*% t(bigX) %*% y
beta_hat
# Compare to lm()
fit = lm(y ~ x)
coef(summary(fit))[,1] # Estimated coefficients
summary(fit)
# Estimate standard errors manually
sigma2 = 1.261^2 # From residual standard error in model summary
cov_beta = sigma2 * solve(t(bigX) %*% bigX)
sqrt(diag(cov_beta)) # Standard errors
# Compare to lm() standard errors
coef(summary(fit))[,2]