Derive the least-squares solution for linear regression

Derive the closed-form solution of ordinary least-squares linear regression from the loss function, and explain each step. Then discuss when the closed form is a bad idea (cost, conditioning, a singular matrix), what regularisation changes in the formula, and how gradient descent compares.