Regression Analysis with PHP

Introduction:

Regression analysis is one of the core tools of statistics that helps investigate relationships between variables. If there is only a single explanatory variable, it is termed “Simple Regression” – It is often difficult to come across such situations in real life. Computing a simple regression involves fitting a straight line through the scatter graph obtained from the sample. From coordinate geometry basics, recall that a straight line has the equation:

y=c + mx

where m = slope of the line, and c is the y intercept.

There can of course be many straight lines through a scatter plot.. Regression analysis picks ONE line in particular – The criteria used is to minimize the sum of squares of the errors (distance from the points in the scatter plot to the line constructed).

If there is more than one variable on which the prediction depends, it is called “Multiple Regression” – It accounts for multiple additional factors (separately) so that the effect of each (independent) variable on the dependent variable can be assessed.

The equation for a (linear) multiple regression can be expressed as:

y = a+bx1+cx2+…

Mathematically, it is identical to “Simple Regression”. For example, in solving a 2 parameter regression, we need 3 dimensions – so, instead of estimating a straight line, we select a single “plane” – again such that the sum of squares of errors is minimum.

We can in fact, extrapolate this to an arbitrarily large number of independent variables. Fortunately, Computers have no problems crunching the numbers using matrix algebra to solve numerous simultaneous equations to arrive at the results.

Using Matrix Algebra to Solve Multiple Regression:

Let X be the data matrix of the predictor (independent) variables.
Let Y be the data vector representing the criterion (dependent) variable.
and, Let ‘b’ be the data vector representing the regression coefficients.

The formula for computing b (coefficient matrix) using Matrix algebra is given by the following formula:

b = (X’X)^-1X’Y

The proof of this equation is quite simple:

1. The simplified equation for a (simple) linear regression in matrix terms is Y=Xb+e
2. Assume that the average error (e) will equal 0. The equation becomes Y=Xb and we need to find the value of ‘b’
3. multiply both sides of the equation by X’ (transpose of X) : X’Y = X’Xb
4. We are trying to get rid of X’X on the RHS.. so multiply both sides of the equation with (X’X)^-1 – the inverse of X’X:
(X’X)^-1X’Y = (X’X)^-1(X’X)b
5. Any matrix multiplied by its inverse is the identity matrix I:
(X’X)^-1X’Y = Ib
6. Ib = b = (X’X)^-1X’Y

[Note: the above equation contains the Moore-Penrose pseudoinverse matrix which ensures that inverses take place on SQUARE matrices. A ^pseudo-1=(A’A)^-1 *A’

To solve equation Y = Xb, the naive solution would be to multiply both sides by X^-1
X^-1Y = X^-1Xb, and thus
X^-1Y = b

However, recall that matrix inverses are only defined on Square matrices. We cannot guarantee that the matrix of independent variables will be a square matrix..in fact, it never is! the sample size needs to be reasonable to ensure correct results. The use of a pseudoinverse matrix solves this problem nicely.]

Example:

A typical workflow involving regression analysis goes like this:

1. Formulate a hypothesis about the relationship between variables of interest
2. Gather data from a representative sample
3. Compute regression parameters and prove (or disprove) the hypothesis

A common example of multiple regression analysis is in college Student Admissions. The admissions committee bases its decision on numerous factors that it believes will influence the students GPA.

For example, a college could hypothesize that the GPA that will be attained by the admitted student is dependent on his/her Highschool GPA, SAT score (V+Q) and Letters of recommendation (The strength of recommendation letters will of course need to be quantified)

A sample of students currently attending the college is then surveyed for their current college GPA, high school GPA, SAT scores and Recommendation letter strength.

The predictor equation for college GPA (dependent variable) is :

Y’ = a+b1X1+b2X2+b3X3

where Y’ = Predicted GPA
{b1, b2, b3} = Regression coefficients
a = Intercept
X1 = HS GPA, X2 = SAT Score and X3 = Strength of Recommendation letters

So, once the coefficients and intercept are determined, it is quite easy to plug in values for X1, X2, X3 and “predict” the future GPA of a student.

Implementation:

Although there are numerical statistical packages that can easily compute regression parameters, they are all geared towards “offline” computation. That is, data is collected first, and then the numbers are crunched to arrive at the result. I was recently involved in a project that required online regression arithmetic – i.e., display the relevant regression parameters at the end of an online survey. I decided to write my own PHP library for this purpose (complete source code is attached at the end of this blog).

Each observation can be written as an equation as follows:

For explanatory purposes, consider a simple linear regression. In matrix terms, the equation looks like:

We can then solve for {b0,b1}… Note how the X matrix needs to be filled with 1’s in the first column. The exact same approach works for multiple regression too – We just use larger matrices!

‘n’ is the sample size (number of observations)

Please refer to the word doc attached in the download for a detailed derivation on why solving the above matrix equations gives us the sum of least square errors.

The various formulae used are:

1. b = (X’X)^-1X’Y  (Regression coefficients. This is an nX1 array)
2. SSR = b’X’Y – (1/n) (Y’UU’Y) (sum of squares due to regression – this is a scalar. U is a unit vector of dimensions nX1)
3. SSE = Y’Y-b’X’Y (Sum of squares due to errors – scalar)
4. SSTO = SSR+SSE (Total sum of squares – scalar)
5. dfTotal = sample_size – 1 (Total degrees of freedom)
6. dfModel = num_independent – 1 (Model degrees of freedom)
7. dfResidual = dfTotal – dfModel (Residual degrees of freedom)
8. MSE = SSE/dfResidual  (Mean square error – scalar)
9. SE = (X’X)^-1*(MSE) then take Square Root of elements on the diagonal
10. t-stat = b[i][j]/SE[i][j]
11. R²= SSR/SSTO
12. F = (SSR/dfModel)/(SSE/dfResidual)

The heart of the library is the Lib_Matrix class. It handles all required matrix manipulation operations. A fluent interface has been provided where possible to make code more intuitive and readable. The Regression computation is performed in the “Lib_Regression” class. I have also attached the complete Unit test suite (100% code coverage) to help you better understand the API.

File Name	Purpose
Matrix.php	Core Matrix Manipulation
Regression.php	Compute various regression parameters
MatrixTest.php	Complete unit test suite for both matrix and regression classes
Excel Regression.xlsx	Regression performed on the test case testRegressionPassingArrays() using MSexcel
MyReg.csv	Sample CSV file used by function testRegressionUsingCSV()

An example API use scenario is described below:

//independent variables.
$x = array(
            array(1, 8, 2),
            array(1, 40.5, 24.5),
            array(1, 4.5, .5),
            array(1, .5, 2),
            array(1, 4.5, 4.5),
            array(1, 7, 8),
            array(1, 24.5, 40.5),
            array(1, 4.5, 2),
            array(1, 32, 24.5),
            array(1, .5, 4.5),
        );
//dependent variables
 $y = array(array(4.5),
            array(22.5),
            array(2),
            array(.5),
            array(18),
            array(2),
            array(32),
            array(4.5),
            array(40.5),
            array(2));

$reg = new Lib_Regression();
$reg->setX($x);
$reg->setY($y);

//NOTE: passing true to the compute method generates standardized coefficients
$reg->Compute();    //go!

var_dump($reg->getSSE());
var_dump($reg->getSSR());
var_dump($reg->getSSTO());
var_dump($reg->getRSQUARE());
var_dump($reg->getF());
var_dump($reg->getRSQUAREPValue());
var_dump($reg->getCoefficients());
var_dump($reg->getStandardError());
var_dump($reg->getTStats());
var_dump($reg->getPValues());

Click here to download the regression library(Github)

References:

http://marketing.byu.edu/htmlpages/books/pcmds/REGRESS.html

http://davidmlane.com/hyperstat/prediction.html

http://luna.cas.usf.edu/~mbrannic/files/regression/regma.htm

http://www.geog.ucsb.edu/~joel/g210_w07/lecture_notes/lect09/oh07_09_2.html

http://home.ubalt.edu/ntsbarsh/Business-stat/otherapplets/pvalues.htm#rtdist