Rb at 20:22, 7 August 2015

2015-08-07T20:22:26Z

Rb at 20:19, 7 August 2015

2015-08-07T20:19:31Z

Rb: Created page with "= Introduction and Data Upload = Set your working directory and load data

setwd("C:/Users/Ralph Becker/Dropbox/R")   # This sets the working directo..."

2015-08-07T20:15:57Z

Created page with "= Introduction and Data Upload = Set your working directory and load data <pre class="r">setwd("C:/Users/Ralph Becker/Dropbox/R") # This sets the working directo..."

New page

= Introduction and Data Upload =

Set your working directory and load data

<pre class="r">setwd("C:/Users/Ralph Becker/Dropbox/R") # This sets the working directory, ensure data file is in here
mydata <- read.csv("Mroz.csv",na.strings = ".") # Opens mroz.csv from working directory</pre>
Let's look at the <code>wage</code> variable ([http://eclr.humanities.manchester.ac.uk/index.php/MROZ_Variable_Description Variable Descriptions]):

<pre class="r">summary(mydata$wage)</pre>
<pre>## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's
## 0.1282 2.2630 3.4820 4.1780 4.9710 25.0000 325</pre>
In our dataset we have 325 observations that have no wage information. You will have to be clear what the reason for this is. Are they not working, or are these genuinly missing observations? The clue here is in the <code>inlf</code> variable which indicates whether a particular women is in the labour force (inlf = 1) or not (inlf = 0). All those observations for which the wage is missing coincide with (inlf = 0). You can see that from the spreadsheet or by checking

<pre class="r">mydata$inlf[is.na(mydata$wage)]</pre>
<pre>## [1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
## [36] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
## [71] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
## [106] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
## [141] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
## [176] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
## [211] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
## [246] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
## [281] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
## [316] 0 0 0 0 0 0 0 0 0 0</pre>
The <code>is.na( )</code> function is a very useful one. It checks whether the elemets in <code>mydata$wage</code> are missing or not. It returns values of either "TRUE" if they are missing or "FALSE" for observations where the wage is not missing. When you put this function into square brackets after <code>mydata$inlf</code> then R will only return those values of the <code>inlf</code> variable where the corresponding value for the <code>wage</code> was missing (as this would produce a "TRUE" from the <code>is.na( )</code> function.)

What we are about to do is the following. We shall first estimate the following regression model:

<math>wage_i = \alpha_0 + \alpha_1 age_i + \alpha_2 educ_i + eps_i</math>

on the basis of all data have wage information. Then, subsequently we shall use the estimated model to predict the wages of those who are not currently working. I.e. we'd be answering the question of how much they should expect to earn (on the basis of their age and education).

Before we do this, there are two reasons of why we should look at this analysis with a pinch of salt

# Including the education variable in such an analysis is likely to be problematic as it is likely to be correlated with the error terms. This will produce biased coefficient estimates (also see the section on [http://eclr.humanities.manchester.ac.uk/index.php/IV_in_R Instrumental variables estimation])
# The reason why the women is most likely not a random decision and therefore they are not a randomly selected subset. Therefore it may well be that the realtionship we estimated between wage and the explanatory variables on the basis of those women who do work may not be the same for women who do not work.

= Regression and Prediction =

Despite these reservations we shall use this example to demonstrate how we use R to produce predictions. Let's start by splitting our dataframe into the two subgroups (those for whom we do have wages and those for whom we dont't). and we shall again use the <code>is.na( )</code> function to gether with the <code>subset</code> function.

<pre class="r">mydata_est <- subset(mydata, is.na(wage) == FALSE)
mydata_pred <- subset(mydata, is.na(wage) == TRUE)</pre>
Those observations for which wage is missing go into the <code>mydata_pred</code> dataframe and all others into <code>mydata_est</code>.

We now run our regresison on the data in the <code>mydata_est</code> dataframe

<pre class="r">lm1 <- lm(wage~age+educ, data=mydata_est)</pre>
All the results are in <code>lm1</code>. we now use this estimated model to produce predicted wages for all those observations that are in <code>mydata_pred</code>. To do this we use the <code>predict) _</code>function. We need to hand in the estimated model <code>lm1</code> and then the dataframe to which we want to apply the estimated model to produce forecasts (<code>mydata_pred</code>). This, of course, requires that this dataframe has a <code>age</code> and <code>educ</code> variable.

<pre class="r">pr1 <- predict(lm1,mydata_pred)</pre>
Check out the new object <code>pr1</code>. It is a vector with 325 predictions for the 325 observations in <code>mydata_pred</code>.

== New data ==

Sometimes you don't have actual observations for which you want to produce forecast, but rather you have a set of conditioning values at which you want to clculate predictions.

Now we will show how you would set this up. You would basically want to produce a new dataframe which contains, at the least, two variables named <code>age</code> and <code>educ</code>. You could do this as follows (try and understand what age/education combinations are being produced here:

<pre class="r">age = seq(20, 55, 1) # creates ages 20, 21, 22, ..., 54, 55 , 36 observations
educ = rep(c(8,10,12,14),9) # creates 9 repetions of 8,10,12,14, altogether 36 observations
new_df <- data.frame(age,educ)</pre>
And now we apply the estimated model <code>lm1</code> to this new dataset.

<pre class="r">pr2 <- predict(lm1,new_df)</pre>
Of course you could have asked for forecasts for very different age/education combinations. The <code>predict( )</code> function has a few extra bells and whistles you could use. You know well (or at least you should) that when we make predictions using a linear regresison model we get the conditional expectation, but there wil be uncertainlty around this expectation. One of the options if the <code>predict</code> function allows you to get confidence intervals.

Let's try the following:

<pre class="r">pr3 <- predict(lm1,new_df, level = 0.99, interval = "prediction")</pre>
here we set the confidence <code>level</code> to 0.99 (if you don't set it explicitly R will use a default of 0.95) and ask R to produce prediction intervals. Look at the resulting <code>pr3</code> object, it now has three columns. Let's loook at the 1st row of <code>new_df</code> and <code>pr3</code>

<pre class="r">new_df[1,]</pre>
<pre>## age educ
## 1 20 8</pre>
<pre class="r">pr3[1,]</pre>
<pre>## fit lwr upr
## 1.397456 -6.789651 9.584564</pre>
The 1st of our new cases is for someone with age 20 and 8 years of education. The forecast we produced is 1.40 with lower confidence interval boundary at -3.82 and the upper boundary at 6.61. So the model tells us that there is a 99% probability that a women aged 20 with 8 years of education would earn something between -3.82 and 6.61 dollars per hour. Clearly this is a huge confidence interval and in fact it reaches into negative hourly wages which don't really make sense. There are three reasons for this.

# We set a high level of confidence, resulting in a very wide interval
# Our conditioning values were at the lower range of observed ages and education levels. If we had used conditioning values closer to the respective averages we would get a narrower interval (try it!)
# We asked R to get us an interval for an individual ()by using <code>interval = "prediction"</code> which means that R also factors is the observed level of error variance.

The last point is important as it allows us to distinguish this from a slightly different type of confidence interval:

<pre class="r">pr4 <- predict(lm1,new_df, level = 0.99, interval = "confidence")</pre>
All that has changed is the type of interval we asked for, <code>interval = "confidence"</code>. Before discussing what the difference is, let's see how the result changes

<pre class="r">new_df[1,]</pre>
<pre>## age educ
## 1 20 8</pre>
<pre class="r">pr4[1,]</pre>
<pre>## fit lwr upr
## 1.39745636 -0.05580818 2.85072091</pre>
We can see that the confidence interval is now much narrower, although we still have the same expected value of 1.40. The question this type of interval asks is the following. Assume you have many women aged 20 and with 8 years of education. What should you expect the average wage of these many women to be. Answer: With 99% probability, the model tells us, this value should be between -0.06 and 2.85.

← Older revision		Revision as of 20:22, 7 August 2015
Line 1:		Line 1:
		+	Here we will be using the Women's Wages dataset which is made available [[R\|here]].
		+
	= Introduction and Data Upload =		= Introduction and Data Upload =

Predicting from Regression in R - Revision history

Rb at 20:22, 7 August 2015

Rb at 20:19, 7 August 2015

Rb: Created page with "= Introduction and Data Upload = Set your working directory and load data
setwd("C:/Users/Ralph Becker/Dropbox/R") # This sets the working directo..."

← Older revision		Revision as of 20:19, 7 August 2015
Line 95:		Line 95:
	## 1.39745636 -0.05580818 2.85072091</pre>		## 1.39745636 -0.05580818 2.85072091</pre>
	We can see that the confidence interval is now much narrower, although we still have the same expected value of 1.40. The question this type of interval asks is the following. Assume you have many women aged 20 and with 8 years of education. What should you expect the average wage of these many women to be. Answer: With 99% probability, the model tells us, this value should be between -0.06 and 2.85.		We can see that the confidence interval is now much narrower, although we still have the same expected value of 1.40. The question this type of interval asks is the following. Assume you have many women aged 20 and with 8 years of education. What should you expect the average wage of these many women to be. Answer: With 99% probability, the model tells us, this value should be between -0.06 and 2.85.
		+
		+	We can see that the confidence interval is now much narrower, although we still have the same expected value of 1.40. The question this type of interval asks is the following. Assume you have many women aged 20 and with 8 years of education. What should you expect the average wage of these many women to be. Answer: With 99% probability, the model tells us, this value should be between -0.06 and 2.85.
		+
		+	Why does predicting an average produce such a narrower prediction interval. As you are calculating averages you are averaging out individual variation and hence we don't need to take that into account. This tends to massively reduce the width of the confidence interval.