Linear Regression

23 Jun 2026 · Stone Liu

The Slope of the Regression Line

Lets consider a table of values here, where is called the independent variable and is called the dependent variable.

If we collected these data points out in the wild, then the easiest thing to do is to assume that these data points have a Linear Relationship. If we assume that the data points have a linear relationship then we have our very simple linear regression model:

Notice that we have a to mark off each of the and values. This is because that it is expected that we can only approximate the relationship between the two variables, since it is very rare to gather data in the real world with a true linear relationship. We can approximate with the following formula

Jason does a really nice job visualizing what this formula is describing, consider the horizontal line and the vertical line . These are the averages of all the and data points. For our current sample that would be

If we were to directly calculate for we would see that

data_points = [(11, 10), (17, 14), (18, 20), (22, 27), (31, 33), (50, 82)]
x = [point[0] for point in data_points]
y = [point[1] for point in data_points]
beta_hat, beta_hat_intercept = np.polyfit(x, y, deg=1) # degree 1 polynomial

So the regression line is of the form .

Linear Transformations

Let’s take a slight tangent and explore the concept of the Linear Transformation in Linear Algebra. Visualizing this in 2D space, all linear transformations must preserve the original basis vectors. in we have

For example, take the vector , it can be written as . Now say we that apply some linear transformation, call it . I really love how 3Blue1Brown describes the point of linear transformations. We don’t really have to “watch” where the vector lands in the transformation , we just have to observe where its basis vectors and land.

Say that is a function that maps

Notice that we did not calculate explicitly, we simply calculated where sent the basis vectors for and then deduced that must be where is going to land. Now we can describe how matricies can be served as a linear transformation. Say we put the column vectors of and into a matrix, then we get

What happens when we multiply this matrix with our ? We get

So literally the is the transformation since we can multiply with any vector in and that will describe the linear transformation .

The Normal Equation

Consider the same data points as last time.

We could represent this as a series of linear equations as such

Simply eye-balling the graph above, we know that clearly there is no singular line that passes through all data points. But we can approximately get the value of and that will get us the closest. Lets convert this system of equations into matrix form by taking the coefficients on each side.

What happens when we multiply both sides by the transpose on both sides? Well that means that we are effectively applying a Linear Transformation where the columns of are indeed made up of the transformations of the basis vectors in .

The transformation we have applied will send the column vectors of which live in . If we carry out all of these multiplications and inverses (Done with the help of numpy) then we can see that we get

X = np.array(
    [
        [11, 1],
        [17, 1],
        [18, 1],
        [22, 1],
        [31, 1],
        [50, 1],
    ]
)
y = np.array(y)
# Normal Equation
XTX = X.T @ X
XTY = X.T @ y
b = np.linalg.inv(XTX) @ XTY

=> [1.84505364, -14.81883194]

Which is precisely what we get from computing the slope of the line and intercepts beforehand.