NICEFA Logo
nicefa.
Library
←Module

Derivation of the Ordinary Least Squares (OLS) Estimators for Simple Linear Regression

Exploring the cinematic intuition of Derivation of the Ordinary Least Squares (OLS) Estimators for Simple Linear Regression.

Visualizing...

Our institutional research engineers are currently mapping the formal proof for Derivation of the Ordinary Least Squares (OLS) Estimators for Simple Linear Regression.

Apply for Institutional Early Access →

The Formal Theorem

Let the simple linear regression model be defined by Yi=β0+β1Xi+ϵi Y_i = \beta_0 + \beta_1 X_i + \epsilon_i for i=1,…,n i = 1, \dots, n , where ϵi \epsilon_i are independent and identically distributed errors with E[ϵi]=0 E[\epsilon_i] = 0 and Var(ϵi)=σ2 Var(\epsilon_i) = \sigma^2 . The Ordinary Least Squares (OLS) estimators β^0 \hat{\beta}_0 and β^1 \hat{\beta}_1 are the values that minimize the Residual Sum of Squares (RSS):
S(β0,β1)=∑i=1n(Yi−β0−β1Xi)2 S(\beta_0, \beta_1) = \sum_{i=1}^n (Y_i - \beta_0 - \beta_1 X_i)^2
The resulting point estimators are:
β^1=∑i=1n(Xi−Xˉ)(Yi−Yˉ)∑i=1n(Xi−Xˉ)2andβ^0=Yˉ−β^1Xˉ \hat{\beta}_1 = \frac{\sum_{i=1}^n (X_i - \bar{X})(Y_i - \bar{Y})}{\sum_{i=1}^n (X_i - \bar{X})^2} \quad \text{and} \quad \hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{X}

Analytical Intuition.

Picture a galaxy of data points scattered across a Cartesian void. Each point (Xi,Yi) (X_i, Y_i) exerts a gravitational pull on a rigid, infinite glass rod—the regression line. The OLS method is the mathematical engine that finds the equilibrium position where the rod experiences the least total 'stress.' This stress is defined as the sum of the areas of squares built upon the vertical distances between the points and the rod. By squaring these deviations, we treat positive and negative errors equally and disproportionately penalize distant outliers, forcing the line to respect the collective trend rather than being hijacked by a single anomaly. Through the lens of multivariable calculus, we are navigating a high-dimensional parabolic landscape, seeking the unique global minimum where the partial derivatives with respect to the intercept β0 \beta_0 and slope β1 \beta_1 vanish. At this stationary point, the 'Normal Equations' emerge, revealing that the optimal line must pass through the centroid (Xˉ,Yˉ) (\bar{X}, \bar{Y}) , balancing the data with surgical precision. It is the most efficient linear unbiased estimator, a foundational pillar of statistical inference.
CAUTION

Institutional Warning.

Students frequently mistake the vertical residuals minimized in OLS for the perpendicular distances used in orthogonal regression. Additionally, many overlook the requirement that the sum of residuals must equal zero at the OLS solution, which is a direct consequence of the first-order condition for the intercept.

Academic Inquiries.

01

Why do we minimize the sum of squares rather than the sum of absolute values?

Squaring makes the objective function S S continuously differentiable everywhere, allowing for a unique analytical solution via calculus, and it aligns with the Maximum Likelihood Estimator under Gaussian assumptions.

02

Does the OLS line always pass through the mean of the data?

Yes, the first-order condition ∂S∂β0=0 \frac{\partial S}{\partial \beta_0} = 0 ensures that the line passes exactly through the centroid (Xˉ,Yˉ) (\bar{X}, \bar{Y}) , provided an intercept is included in the model.

03

What happens to the estimators if the independent variables are centered?

If the Xi X_i values are centered such that Xˉ=0 \bar{X} = 0 , the intercept estimator simplifies directly to β^0=Yˉ \hat{\beta}_0 = \bar{Y} , and the two estimators become uncorrelated.

Standardized References.

  • Definitive Institutional SourceMontgomery, D. C., Peck, E. A., & Vining, G. G., Introduction to Linear Regression Analysis.

Related Proofs Cluster.

Intermediate

Proof of Chebyshev's Inequality

Exploring the cinematic intuition of Proof of Chebyshev's Inequality.

Intermediate

Derivation of the Mean and Variance of the Binomial Distribution

Exploring the cinematic intuition of Derivation of the Mean and Variance of the Binomial Distribution.

Intermediate

Derivation of the Mean and Variance of the Poisson Distribution

Exploring the cinematic intuition of Derivation of the Mean and Variance of the Poisson Distribution.

Advanced

The Conceptual Proof of the Central Limit Theorem (CLT)

Exploring the cinematic intuition of The Conceptual Proof of the Central Limit Theorem (CLT).

Institutional Citation

Reference this proof in your academic research or publications.

NICEFA Visual Mathematics. (2026). Derivation of the Ordinary Least Squares (OLS) Estimators for Simple Linear Regression: Visual Proof & Intuition. Retrieved from https://nicefa.org/library/applied-statistics/derivation-of-the-ordinary-least-squares--ols--estimators-for-simple-linear-regression

Dominate the Logic.

"Abstract theory is just a movement we haven't seen yet."

Subscribe for Full ProofsEarly Access