Simple Linear Regression Solutions to Exercises

In the exercise the usual linear regression model is used

Yi = β0 + β1xi + εi,

εi ∼ N(0, σ2

ε ),

i = 1, . . . , 5

a) Can a significant relationship between yield and temperature be docu-

mented on the usual significance level α = 0.05?

Chapter 5

5.3 YIELD OF CHEMICAL PROCESS

9

Solution

It could most easily be solved by running the regression in R as:

D <- data.frame(x=c(0,25,50,75,100),

y=c(14,38,54,76,95))

fit <- lm(y ~ x, data=D)

summary(fit)

Call:

lm(formula = y ~ x, data = D)

Residuals:

1

5

-1.4 2.6 -1.4 0.6 -0.4

4

2

3

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) 15.4000

x

0.8000

---

Signif. codes:

1.4967

0.0244

10.3

32.7 0.000063 *

0.002 **

0 ’’ 0.001 ’’ 0.01 ’’ 0.05 ’.’ 0.1 ’ ’ 1

Residual standard error: 1.93 on 3 degrees of freedom

Multiple R-squared:

F-statistic: 1.07e+03 on 1 and 3 DF,

0.997, Adjusted R-squared:

p-value: 0.0000627

0.996

Alternatively one could use hand calculations and use the formula in Theorem 5.12

for the t-test of the null hypothesis: H0 : β1 = 0.

The relevant test statistic and p-value can be read off in the R output as 32.7 and

0.000063. So the answer is:

Yes, as the relevant test statistic and p-value are resp. 32.7 and 0.00006 < 0.05 = α.

b) Give the 95% confidence interval of the expected yield at a temperature of

xnew = 80 ◦C.

Chapter 5

5.3 YIELD OF CHEMICAL PROCESS

10

Solution

We use the formula in Equation (5-59) for the confidence limit of the line (the ex-

pected value of Yi for a value xnew):

ˆβ0 + ˆβ1xnew ± t1−α/2 ˆσ

(cid:115)

1

n

+

(xnew − ¯x)2

Sxx

,

and we have to compute ˆβ0, ˆβ1 and se either by hand OR in R as above:

ˆβ0 = 15.4, ˆβ1 = 0.8, ˆσ = 1.932.

So the confidence interval becomes

(15.4 + 0.8 · 80) ± 3.182 · 1.932

(cid:114)

1

5

+

(80 − 50)2

6250

,

since

sx =

1

n − 1

n

∑

i=1

(xi − ¯x)2 =

1

n − 1

Sxx ⇔

Sxx = (n − 1)sx = 4 · 39.5282 = 6250.

79.40 ± 3.61 = [75.79, 83.01].

Thus the answer is

In R this could be by:

predict(fit, newdata=data.frame(x=80), interval="confidence",

level=0.95)

fit

upr

1 79.4 75.79 83.01

lwr

c) What is the upper quartile of the residuals?

Solution

The five residuals become: -1.4, 2.6, -1.4, 0.6 og -0.4.

We use the basic definition of finding a quantile (from Definition 1.7) and the upper

quartile is q0.75 (see Definition 1.8). We set n = 5, p = 0.75, so

np = 3.75

Chapter 5

5.3 YIELD OF CHEMICAL PROCESS

11

So the upper quartile is the 4th observation in the ordered sequence:

−1.4, −1.4, −0.4, 0.6, 2.6.

This is also found in the summary() output above under

Residuals:

1

2

5

-1.4 2.6 -1.4 0.6 -0.4

4

3

So the answer is: 0.6.

Chapter 5

5.4 PLASTIC MATERIAL

12

5.4 Plastic material

Exercise 5.4

Plastic material

In the manufacturing of a plastic material, it is believed that the cooling time has

an influence on the impact strength. Therefore a study is carried out in which

plastic material impact strength is determined for 4 different cooling times. The

results of this experiment are shown in the following table:

Cooling times in seconds (x)

Impact strength in kJ/m2 (y)

15

42.1

25

36.0

35

31.8

40

28.7

The following statistics may be used:

¯x = 28.75, ¯y = 34.65, Sxx = 368.75.

a) What is the 95% confidence interval for the slope of the regression model,

expressing the impact strength as a linear function of the cooling time?

Chapter 5

5.4 PLASTIC MATERIAL

13

Solution

The easiest way to get to the confidence interval is to use the standard error for the

slope ( ˆσβ1 or denoted with SEβ1) given in the R output:

x <- c(15,25,35,40)

y <- c(42.1,36.0,31.8,28.7)

summary(lm(y ~ x))

Call:

lm(formula = y ~ x)

Residuals:

1

2

0.2814 -0.6051

3

4

0.4085 -0.0847

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept)

x

---

Signif. codes:

49.639

-0.521

0.878

0.029

56.5 0.00031 *

-18.0 0.00308 **

0 ’’ 0.001 ’’ 0.01 ’’ 0.05 ’.’ 0.1 ’ ’ 1

Residual standard error: 0.556 on 2 degrees of freedom

0.991

Multiple R-squared:

F-statistic: 324 on 1 and 2 DF,

0.994, Adjusted R-squared:

p-value: 0.00308

the standard error for the slope is ˆσβ1 = 0.029 (also known as the sampling distribu-

tion standard deviation for ˆβ1). Finding the relevant t-quantile (with ν = 2 degrees

of freedom (either of):

c(qt(0.025, df=2), qt(0.975, df=2))

[1] -4.303

4.303

|t0.025| = 4.303, which using Theorem 5.15 gives

giving

−0.521 ± 4.303 · 0.029,

−0.521 ± 0.125,

or, that we say with high confidence that the true parameter value is in the interval,

i.e.

−0.646 ≤ β1 ≤ −0.396.

Chapter 5

5.4 PLASTIC MATERIAL

14

b) Can you conclude that there is a relation between the impact strength and

the cooling time at significance level α = 5%?

Solution

The relevant p-value can be read off directly from the summary output: 0.00308, and

we can conclude: Yes, as the relevant p-value is 0.00308, which is smaller than 0.05.

c) For a similar plastic material the tabulated value for the linear relation

between temperature and impact strength (i.e the slope) is −0.30. If the

following hypothesis is tested (at level α = 0.05)

H0 : β1 = −0.30

H1 : β1 (cid:54)= −0.30

with the usual t-test statistic for such a test, what is the range (for t) within

which the hypothesis is accepted?

Solution

The so-called critical values for the t-statistic with ν = 2 degrees of freedom is found

as (or at least the negative one of the two): t0.025 = −4.303 - in R: qt(0.975,2)). So

the answer becomes:

[−4.303, 4.303].

Chapter 5

5.5 WATER POLUTION

15

5.5 Water polution

Exercise 5.5

Water polution

In a study of pollution in a water stream, the concentration of pollution is mea-

sured at 5 different locations. The locations are at different distances to the

pollution source. In the table below, these distances and the average pollution

are given:

Distance to the pollution source (in km)

Average concentration

2

11.5

4

10.2

6

10.3

8

9.68

10

9.32

a) What are the parameter estimates for the three unknown parameters in

the usual linear regression model: 1) The intercept (β0), 2) the slope (β1)

and 3) error standard deviation (σ)?

Chapter 5

5.5 WATER POLUTION

16

Solution

The question is solved by considering the following R-output:

D <- data.frame(concentration=c(11.5, 10.2, 10.3, 9.68, 9.32),

distance=c(2, 4, 6, 8, 10))

fit <- lm(concentration ~ distance, data=D)

summary(fit)

Call:

lm(formula = concentration ~ distance, data = D)

Residuals:

1

2

0.324 -0.488

Coefficients:

3

4

0.100 -0.032

5

0.096

Estimate Std. Error t value Pr(>|t|)

(Intercept)

distance

---

Signif. codes:

11.664

-0.244

0.365

0.055

31.96 0.000067 *

-4.43

0.021 *

0 ’’ 0.001 ’’ 0.01 ’’ 0.05 ’.’ 0.1 ’ ’ 1

Residual standard error: 0.348 on 3 degrees of freedom

0.823

Multiple R-squared:

F-statistic: 19.7 on 1 and 3 DF, p-value: 0.0213

0.868, Adjusted R-squared:

Given the knowledge of the R-output structure, the three values can be read off

directly from the output.

So the correct answer is: ˆβ0 = 11.7, ˆβ1 = −0.244 and SE ˆσ = ˆσ = 0.348.

b) How large a part of the variation in concentration can be explained by the

distance?

Solution

The amount of variation in the model output (Y) explained by the variable input

(x) can be found from the squared correlation, that can be read off directly from the

Chapter 5

5.5 WATER POLUTION

17

output as "Multiple R-squared". So the correct answer is: R2 = 86.8% (it is actually

an estimate of the variation in concentration which can be explained by distance,

since it is what we found with the particular data at hand. If the sample was taken

again, then this value would vary. We should actually calculate a confidence interval

for R2 to understand how accurate this estimate is!).

c) What is a 95%-confidence interval for the expected pollution concentra-

tion 7 km from the pollution source?

Solution

The wanted number is estimated by the point on the line (using xnew = 7)

and the confidence interval is given by

−0.244 · 7 + 11.664 = 9.96,

9.96 ± t0.025(3) · ˆσ

(cid:115)

1

5

+

(7 − 6)2

Sxx

,

where Sxx = 42 + 22 + 02 + 22 + 42 = 40 and t0.025(3) = 3.182 (in R: qt(0.975,3))

we have that

(cid:114)

1

5

+

1

40

= 0.525,

3.182 · 0.348

where sx is:

sd(D$distance)

[1] 3.162

and thus

This could also have been found by

Sxx = (n − 1) · s2

x = 4 · 3.1622 = 40.

predict(fit, newdata=data.frame(distance=7), interval="confidence",

level=0.95)

fit

upr

1 9.956 9.431 10.48

lwr

So the correct answer is:

9.96 ± 0.525 = [9.43, 10.5].

Chapter 5

5.6 MEMBRANE PRESSURE DROP

18

5.6 Membrane pressure drop

Exercise 5.6

Membrane pressure drop

When purifying drinking water you can use a so-called membrane filtration.

In an experiment one wishes to examine the relationship between the pressure

drop across a membrane and the flux (flow per area) through the membrane.

We observe the following 10 related values of pressure (x) and flux (y):

Pressure (x)

Flux (y)

1

1.02

1.15

2

2.08

0.85

3

2.89

1.56

4

4.01

1.72

5

6

5.32 5.83

4.32 5.07

7

7.26

5.00

8

7.96

5.31

9

9.11

6.17

10

9.99

7.04

Copy this into R to avoid typing in the data:

D <- data.frame(

pressure=c(1.02,2.08,2.89,4.01,5.32,5.83,7.26,7.96,9.11,9.99),

flux=c(1.15,0.85,1.56,1.72,4.32,5.07,5.00,5.31,6.17,7.04)

)

a) What is the empirical correlation between pressure and flux estimated to?

Give also an interpretation of the correlation.

Chapter 5

5.6 MEMBRANE PRESSURE DROP

19

Solution

The questions are most easily solved by using lm in R:

D <- data.frame(

pressure=c(1.02,2.08,2.89,4.01,5.32,5.83,7.26,7.96,9.11,9.99),

flux=c(1.15,0.85,1.56,1.72,4.32,5.07,5.00,5.31,6.17,7.04)

)

fit <- lm(flux ~ pressure, data=D)

summary(fit)

Call:

lm(formula = flux ~ pressure, data = D)

Residuals:

Min

3Q

-0.989 -0.318 -0.140 0.454

1Q Median

Max

1.046

Coefficients:

Estimate Std. Error t value

-0.43

10.23 0.0000072 *

Pr(>|t|)

0.68

0.4417

0.0706

(Intercept) -0.1886

0.7225

pressure

---

Signif. codes:

0 ’’ 0.001 ’’ 0.01 ’’ 0.05 ’.’ 0.1 ’ ’ 1

Residual standard error: 0.645 on 8 degrees of freedom

Multiple R-squared:

F-statistic: 105 on 1 and 8 DF,

0.92

p-value: 0.00000718

0.929, Adjusted R-squared:

The found coefficient of determination (see Theorem 5.25) can be read off the R out-

put to be 0.929. The sign of the correlation is the same as the sign of the slope, which

can be read off to be positive ( ˆβ1 = 0.7225), so the correlation is

√

ˆρ = r =

0.929 = 0.964.

So the empirical correlation is 0.964, and thus flux is found to increase with increas-

ing pressure.

b) What is a 90% confidence interval for the slope β1 in the usual regression

model?

Chapter 5

5.6 MEMBRANE PRESSURE DROP

20

Solution

We use the formula for the slope (β1, see Method 5.15) confidence interval, and can

actually just realize that the correct t-quantile to use is the t1−0.05(8) = 1.860 (in R:

qt(0.95,8)), and the other values we read of the summary output.

So the confidence interval is: 0.7225 ± 1.860 · 0.0706.

c) How large a part of the flux-variation (∑10

i=1(yi − ¯y)2) is not explained by

pressure differences?

Solution

The squared correlation, r2 = 0.929 express the explained variation, this means that

1 − 0.929 = 0.071 express the unexplained variation by the model.

d) Can you at significance level α = 0.05 reject the hypothesis that the line

passes through (0, 0)?

Solution

The hypothesis is the same as:

H0 : β1 = 0

which is the hypothesis results provided in the output in the "intercept" row of

summary, so: No, since the relevant p-value is 0.68, which is larger than α

e) A confidence interval for the line at three different pressure levels: xA

new =

3.5, xB

new = 5.0 and xC

new = 9.5 will look as follows:

ˆβ0 + ˆβ1 · xU

new ± CU

where U then is either A, B or C. Write the constants CU in increasing

order.

Chapter 5

5.6 MEMBRANE PRESSURE DROP

21

Solution

The formula for the Confidence limits of α + βxnew includes the following term:

(xnew − ¯x)2

Sxx

and this is the ONLY term in CU that makes CU different between the three Us. And

since ¯x = 5.547 it is clear that

(5.0 − 5.547)2 < (3.5 − 5.547)2 < (9.5 − 5.547)2

and hence

(xB

new − 5.547)2 < (xA

new − 5.547)2 < (xC

new − 5.547)2

So CB < CA < CC

Chapter 5

5.7 MEMBRANE PRESSURE DROP (MATRIX FORM)

22

5.7 Membrane pressure drop (matrix form)

Exercise 5.7

Membrane pressure drop (matrix form)

This exercise uses the data presented in Exercise 6 above.

a) Find parameters values, standard errors, t-test statistics, and p-values for

the standard hypotheses tests.

Copy this into R to avoid typing in the data:

D <- data.frame(

pressure=c(1.02,2.08,2.89,4.01,5.32,5.83,7.26,7.96,9.11,9.99),

flux=c(1.15,0.85,1.56,1.72,4.32,5.07,5.00,5.31,6.17,7.04)

)

Solution

Chapter 5

5.7 MEMBRANE PRESSURE DROP (MATRIX FORM)

23

D <- data.frame(

pressure=c(1.02,2.08,2.89,4.01,5.32,5.83,7.26,7.96,9.11,9.99),

flux=c(1.15,0.85,1.56,1.72,4.32,5.07,5.00,5.31,6.17,7.04)

)

fit <- lm(flux ~ pressure, data=D)

summary(fit)

Call:

lm(formula = flux ~ pressure, data = D)

Residuals:

Min

3Q

-0.989 -0.318 -0.140 0.454

1Q Median

Max

1.046

Coefficients:

Estimate Std. Error t value

-0.43

10.23 0.0000072 *

Pr(>|t|)

0.68

0.4417

0.0706

(Intercept) -0.1886

0.7225

pressure

---

Signif. codes:

0 ’’ 0.001 ’’ 0.01 ’’ 0.05 ’.’ 0.1 ’ ’ 1

Residual standard error: 0.645 on 8 degrees of freedom

Multiple R-squared:

F-statistic: 105 on 1 and 8 DF,

0.92

p-value: 0.00000718

0.929, Adjusted R-squared:

The parameter estimates are given in the first column, the standard errors in the

second column, the t-test statistics are given in the third column and the p-values of

the standard hypothesis are given in the last column.

b) Reproduce the above numbers by matrix vector calculations. You will

need some matrix notation in R:

– Matrix multiplication (XY): X%*%Y

– Matrix transpose (XT): t(X)

– Matrix inverse (X−1): solve(X)

– Make a matrix from vectors (X = [xT

1 ; xT

2 ]): cbind(x1,x2)

See also Example 5.24.

Chapter 5

5.7 MEMBRANE PRESSURE DROP (MATRIX FORM)

24

Solution

X <- cbind(1, D$pressure)

y <- D$flux

n <- length(y)

beta <- solve(t(X) %%X ) %% t(X) %*% y

beta

[,1]

[1,] -0.1886

[2,] 0.7225

e <- y - X %*% beta

s <- sqrt(sum(e^2)/(n-2))

Vbeta <- s^2 solve(t(X) %%X )

se.beta <- sqrt(diag(Vbeta))

t.obs <- beta / se.beta

p.value <- 2 * (1 - pt(abs(t.obs), df = n-2))

Collection in a table

analasis.table <- cbind(beta, se.beta, t.obs, p.value)

analasis.table

se.beta

[1,] -0.1886 0.44171 -0.4269 0.680696710

[2,] 0.7225 0.07064 10.2269 0.000007177

Put some names on our table

colnames(analasis.table) <- c("Estimates","Std.Error","t.obs","p.value")

rownames(analasis.table) <- c("beta1","beta2")

analasis.table

Estimates Std.Error

t.obs

p.value

0.44171 -0.4269 0.680696710

0.07064 10.2269 0.000007177

beta1

beta2

-0.1886

0.7225

Done!!

Chapter 5

5.8 INDEPENDENCE AND CORRELATION

25

5.8 Independence and correlation

Exercise 5.8

Independence and correlation

Consider the layout of independent variable in Example 5.11,

a) Show that Sxx = n·(n+1)

12·(n−1) .

Hint: you can use the following relations

n

∑

i=1

n

∑

i=1

i =

n(n + 1)

2

,

i2 =

n(n + 1)(2n + 1)

6

.

Solution

¯x becomes

¯x =

=

i − 1

n − 1

1

n

n

∑

i=1

1

n(n − 1)

=

1

n(n − 1)

(cid:18) n(n + 1)

2

− n

n

∑

i=1

(cid:19)

(i − 1)

=

1

2

,

and Sxx becomes

Sxx =

(cid:18) i − 1

n − 1

(cid:19)2

−

1

2

n

∑

i=1

n

4

= −

= −

n

4

+

+

1

(n − 1)2

1

(n − 1)2

(i2 + 1 − 2i)

n

∑

i=1

(cid:18) n(n + 1)(2n + 1) − 6n2

6

(cid:19)

=

=

n

(n − 1)2

n

(n − 1)2

(cid:18) 4n2 + 6n + 2 − 12n − 3(n − 1)2

12

n(n + 1)

12(n − 1)

(cid:18) n2 − 1

12

=

(cid:19)

.

(cid:19)

Chapter 5

5.8 INDEPENDENCE AND CORRELATION

26

b) Show that the asymptotic correlation between ˆβ0 and ˆβ1 is

n→∞ ρn( ˆβ0, ˆβ1) = −

lim

√

3

2

.

Solution

The correlation between ˆβ0 and ˆβ0 is

ρn( ˆβ0, ˆβ1) =

Cov( ˆβ0, ˆβ1)

(cid:113)

V( ˆβ0) V( ˆβ1)

σ2 ¯x/Sxx

(cid:16) 1

n + ¯x2

Sxx

¯x/Sxx

(cid:114)(cid:16) Sxx

σ4

(cid:114)

1

Sxx

= −

= −

(cid:17)

n + ¯x2

(cid:17) 1

Sxx

= −

¯x

(cid:113) Sxx

n + ¯x2

.

Notice that the correlation is not a function of the variance (σ2), but only a function

of the independent variables. Now insert the values of ¯x and Sxx

ρn( ˆβ0, ˆβ1) = −

= −

= −

.

2

2

1

2

1

(cid:113) n+1

4

12(n−1) + 1

1

(cid:113) 2n−1

6(n−1)

= −

(cid:115)

6(n − 1)

2(n − 1/2)

= −

1

(cid:113) n+1+3(n−1)

12(n−1)

2

(cid:112)

2

6(n − 1)

√

2n − 1

√

(cid:114)

= −

3

2

n − 1

n − 1/2

which converges to −

√

3

2 for n → ∞.

Consider a layout of the independent variable where n = 2k and xi = 0 for i ≤ k

and xi = 1 for k < i ≤ n.

c) Find Sxx for the new layout of x.

Chapter 5

5.8 INDEPENDENCE AND CORRELATION

27

Solution

and

Snew

xx =

=

¯x =

1

2

,

(cid:18)

0 −

(cid:19)2

1

2

+

2k

∑

i=k+1

(cid:18)

1 −

(cid:19)2

1

2

k

∑

i=1

k

4

+

k

4

=

k

2

=

n

4

.

d) Compare Sxx for the two layouts of x.

Solution

Sxx

Snew

xx

=

n(n + 1)

12(n − 1)

4

n

=

(n + 1)

3(n − 1)

< 1;

f or n > 2

which imply that Snew

xx > Sxx for all n > 2.

e) What is the consequence for the parameter variance in the two layouts?

Solution

The larger Sxx for the new layout imply that the parameter variance is smaller for

the new layout (given that data comes from the same model).

f) Discuss pro’s and cons for the two layouts.

Chapter 5

5.8 INDEPENDENCE AND CORRELATION

28

Solution

The smaller parameter variance for the new layout would suggest that we should

use this layout. However, we would not be able to check that data is in fact generated

by a linear model. Consider e.g. data generated by the model

yi = β0 + β1x2

i + εi,

εi ∼ N(0, σ2),

if we only look at xi = 0 or xi = 1 we will not be able to detect that the relationship

is in fact non-linear.