Introduction

In OpenSourceConnections blog Learning to Rank 101 – Linear Models, Search Relevancy expert Doug Turnbull asks the following question in his closing remarks:

What measures can we use to evaluate how good the model is?

As possible “measures” of a good fit, we suggest below in R to inspect the following linear regression model’s properties:

Training data

The following very limited training data was extracted from Doug’s blog:

# Load training data set
trainingDat <- read.csv(file = 'rocky.csv', header = TRUE)

# Remove the comment column
colCount <- length(colnames(trainingDat))
trainingDat <- trainingDat[,1:(colCount-1)]

# Show data
trainingDat
##    grade titleScore overviewScore ratingScore
## 1      4      10.65          8.41         7.4
## 2      3       0.00          6.75         7.0
## 3      3       8.22          9.72         6.6
## 4      3       8.22          8.41         0.0
## 5      3       8.22          7.68         6.9
## 6      3       8.22          7.15         0.0
## 7      3       8.22          5.28         0.0
## 8      2       0.00          0.00         7.6
## 9      2       0.00          0.00         7.1
## 10     2       0.00          0.00         6.7
## 11     2       0.00          0.00         0.0
## 12     2       0.00          0.00         0.0
## 13     2       0.00          0.00         0.0

Linear Regression Model and Prediction

A linear regression model grade ~ . (All other variables) is built below, and a couple of predictions are made.

# Build regression model
lm1 <- lm(grade ~ ., data = trainingDat)

# Predict the two samples used in Doug's plot
# titleScore,overviewScore,movieRating,comment
# 12.28,9.82,6.40,# 7555    rambo@Rambo
# 0.00,10.76,7.10,# 1368    rambo@First Blood
predict(lm1, data.frame(titleScore=12.28, overviewScore=9.82, ratingScore=6.40))
##        1 
## 3.636923
predict(lm1, data.frame(titleScore=0, overviewScore=10.76, ratingScore=7.10))
##        1 
## 3.053139

Let’s note first that our model’s predictions differ from Doug’s since our training data is only a small excerpt from Doug’s. More interestingly, as a suggested methodology, we suggest to examine the predictors’ P-values as shown in the next section.

Linear Model’s P-Values

summary(lm1)
## 
## Call:
## lm(formula = grade ~ ., data = trainingDat)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.40664 -0.10290 -0.04746  0.04665  0.55966 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept)    1.95335    0.13610  14.352 1.65e-07 ***
## titleScore     0.05549    0.03702   1.499   0.1681    
## overviewScore  0.08923    0.04133   2.159   0.0592 .  
## ratingScore    0.01968    0.02324   0.847   0.4192    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.2791 on 9 degrees of freedom
## Multiple R-squared:  0.862,  Adjusted R-squared:  0.8159 
## F-statistic: 18.73 on 3 and 9 DF,  p-value: 0.0003287

The above P-values are above 0.05, which rejects \(H_{0}\) using a 95% confidence interval, and treats all three predictors as not statistically signicant. Of course, the training dataset is too small to jump to any conclusion, and we are simply illustrating the P-value examination step as a way to assess the statistical significance of each predictor.

Linear Model’s Diagnostic Residuals Plots

Finally in this short essay, another technique to assess a linear model is to visualize the model’s “diagnostic residuals plots” (Residual = Observed value - Predicted value). We show two plots:

# Plot the "diagnostic residuals plots":
par(mfrow = c(1, 2))
plot(lm1, which = 2)
plot(lm1, which = 3)

A model’s residuals plots show how poorly a model represents data.

The “Normal Q-Q” plot shows the model’s “iid (Independent and Identically Distributed random variables) normality”. A so-called “thick crayon line” test shows that most values are aligned, with the exception of 3 outliers. In this contrived exercise, the Normal Q-Q plot looks pretty good.

The Scale-Location plot shows how the residuals spread along the range of predictors. The rule of thumb is that it is good to see a horizontal line with equally (randomly) spread points above and below the line. As can be seen in the State-Location plot above, we are not seeing a good spread, hence pointing to problems in the model. Again, the dataset is too small in this exercise to expect good residuals plots, and we are simply illustrating the step of examining the residuals plots as part of assessing a linear regression model.