One Minute Paper Results
What was the most important thing you learned during this class?
What important question remains unanswered for you?
Types of Data
- Numerical (quantitative)
- Continuous
- Discrete
- Categorical (qualitative)
- Regular categorical
- Ordinal
Data Types in R
Variance
Population Variance: S2=Σ(xi−ˉx)2N Consider a dataset with five values (black points in the figure). For the largest value, the deviance is represented by the blue line ( xi−ˉx ).
See also:
https://shiny.rit.albany.edu/stat/visualizess/
https://github.com/jbryer/VisualStats/
Variance (cont.)
Population Variance: S2=Σ(xi−ˉx)2N In the numerator, we square each of these deviances. We can conceptualize this as a square. Here, we add the deviance in the y direction.
Variance (cont.)
Population Variance: S2=Σ(xi−ˉx)2N
We end up with a square.
Variance (cont.)
Population Variance: S2=Σ(xi−ˉx)2N We can plot the squared deviance for all the data points. That is, each component in the numerator is the area of each of these squares.
Variance (cont.)
Population Variance: S2=Σ(xi−ˉx)2N The variance is therefore the average of the area of all these squares, here represented by the orange square.
Population versus Sample Variance
Typically we want the sample variance. The difference is we divide by n−1 to calculate the sample variance. This results in a slightly larger area (variance) then if we divide by n.
Population Variance (yellow): S2=Σ(xi−ˉx)2N
Sample Variance (green): s2=Σ(xi−ˉx)2n−1
Robust Statistics
Consider the following data randomly selected from the normal distribution:
set.seed(41)x <- rnorm(30, mean = 100, sd = 15)mean(x); sd(x)## [1] 103.1934## [1] 16.8945median(x); IQR(x)## [1] 103.9947## [1] 25.68004
Robust Statistics
Robust Statistics
Let's add an extreme value:
x <- c(x, 1000)Robust Statistics
Let's add an extreme value:
x <- c(x, 1000)
About legosets 
To install the brickset package:
remotes::install_github('jbryer/brickset')To load the load the legosets dataset.
data('legosets', package = 'brickset')The legosets data has 16355 observations of 34 variables.
names(legosets)## [1] "setID" "name" "year" ## [4] "theme" "themeGroup" "subtheme" ## [7] "category" "released" "pieces" ## [10] "minifigs" "bricksetURL" "rating" ## [13] "reviewCount" "packagingType" "availability" ## [16] "agerange_min" "US_retailPrice" "US_dateFirstAvailable"## [19] "US_dateLastAvailable" "UK_retailPrice" "UK_dateFirstAvailable"## [22] "UK_dateLastAvailable" "CA_retailPrice" "CA_dateFirstAvailable"## [25] "CA_dateLastAvailable" "DE_retailPrice" "DE_dateFirstAvailable"## [28] "DE_dateLastAvailable" "height" "width" ## [31] "depth" "weight" "thumbnailURL" ## [34] "imageURL"Structure (str)
str(legosets)## 'data.frame': 16355 obs. of 34 variables:## $ setID : int 7693 7695 7697 7698 25534 7418 7419 6020 22704 7421 ...## $ name : chr "Small house set" "Medium house set" "Medium house set" "Large house set" ...## $ year : int 1970 1970 1970 1970 1970 1970 1970 1970 1970 1970 ...## $ theme : chr "Minitalia" "Minitalia" "Minitalia" "Minitalia" ...## $ themeGroup : chr "Vintage" "Vintage" "Vintage" "Vintage" ...## $ subtheme : chr NA NA NA NA ...## $ category : chr "Normal" "Normal" "Normal" "Normal" ...## $ released : logi TRUE TRUE TRUE TRUE TRUE TRUE ...## $ pieces : int 67 109 158 233 NA 1 1 60 65 NA ...## $ minifigs : int NA NA NA NA NA NA NA NA NA NA ...## $ bricksetURL : chr "https://brickset.com/sets/1-8" "https://brickset.com/sets/2-8" "https://brickset.com/sets/3-6" "https://brickset.com/sets/4-4" ...## $ rating : num 0 0 0 0 0 0 0 0 0 0 ...## $ reviewCount : int 0 0 1 0 0 0 0 1 0 0 ...## $ packagingType : chr "{Not specified}" "{Not specified}" "{Not specified}" "{Not specified}" ...## $ availability : chr "{Not specified}" "{Not specified}" "{Not specified}" "{Not specified}" ...## $ agerange_min : int NA NA NA NA NA NA NA NA NA NA ...## $ US_retailPrice : num NA NA NA NA NA 1.99 NA NA 4.99 NA ...## $ US_dateFirstAvailable: Date, format: NA NA ...## $ US_dateLastAvailable : Date, format: NA NA ...## $ UK_retailPrice : num NA NA NA NA NA NA NA NA NA NA ...## $ UK_dateFirstAvailable: Date, format: NA NA ...## $ UK_dateLastAvailable : Date, format: NA NA ...## $ CA_retailPrice : num NA NA NA NA NA NA NA NA NA NA ...## $ CA_dateFirstAvailable: Date, format: NA NA ...## $ CA_dateLastAvailable : Date, format: NA NA ...## $ DE_retailPrice : num NA NA NA NA NA NA NA NA NA NA ...## $ DE_dateFirstAvailable: Date, format: NA NA ...## $ DE_dateLastAvailable : Date, format: NA NA ...## $ height : num NA NA NA NA NA ...## $ width : num NA NA NA NA NA ...## $ depth : num NA NA NA NA NA NA NA NA 5.08 NA ...## $ weight : num NA NA NA NA NA NA NA NA NA NA ...## $ thumbnailURL : chr "https://images.brickset.com/sets/small/1-8.jpg" "https://images.brickset.com/sets/small/2-8.jpg" "https://images.brickset.com/sets/small/3-6.jpg" "https://images.brickset.com/sets/small/4-4.jpg" ...## $ imageURL : chr "https://images.brickset.com/sets/images/1-8.jpg" "https://images.brickset.com/sets/images/2-8.jpg" "https://images.brickset.com/sets/images/3-6.jpg" "https://images.brickset.com/sets/images/4-4.jpg" ...RStudio Eenvironment tab can help 
Data Wrangling Cheat Sheet 
Tidyverse vs Base R 
Pipes %>% and |> 
The pipe operator (%>%) introduced with the magrittr R package allows for the chaining of R operations. Base R has now added their own pipe operator (|>). They take the output from the left-hand side and passes it as the first parameter to the function on the right-hand side.
You can do this in two steps:
tab_out <- table(legosets$category)prop.table(tab_out)Or as nested function calls.
prop.table(table(legosets$category))Using the pipe (|>) operator we can chain these calls in a what is arguably a more readable format:
table(legosets$category) |> prop.table()## ## Book Collection Extended Gear Normal Other ## 0.028798533 0.032100275 0.025191073 0.143564659 0.713420972 0.054050749 ## Random ## 0.002873739Filter
Filter
dplyr
mylego <- legosets %>% filter(themeGroup == 'Educational' & year > 2015)Base R
mylego <- legosets[legosets$themeGroups == 'Educaitonal' & legosets$year > 2015,]nrow(mylego)## [1] 61Select
dplyr
mylego <- mylego %>% select(setID, pieces, theme, availability, US_retailPrice, minifigs)Base R
mylego <- mylego[,c('setID', 'pieces', 'theme', 'availability', 'US_retailPrice', 'minifigs')]head(mylego, n = 4)## setID pieces theme availability US_retailPrice minifigs## 1 26803 103 Education {Not specified} NA 6## 2 26689 142 Education {Not specified} NA 4## 3 26804 98 Education {Not specified} NA 6## 4 26277 188 Education Educational 78.95 NARelocate
Relocate
dplyr
mylego %>% relocate(where(is.numeric), .after = where(is.character)) %>% head(n = 3)## theme availability setID pieces US_retailPrice minifigs## 1 Education {Not specified} 26803 103 NA 6## 2 Education {Not specified} 26689 142 NA 4## 3 Education {Not specified} 26804 98 NA 6Base R
mylego2 <- mylego[,c('theme', 'availability', 'setID', 'pieces', 'US_retailPrice', 'minifigs')]head(mylego2, n = 3)## theme availability setID pieces US_retailPrice minifigs## 1 Education {Not specified} 26803 103 NA 6## 2 Education {Not specified} 26689 142 NA 4## 3 Education {Not specified} 26804 98 NA 6Rename
Rename
dplyr
mylego %>% dplyr::rename(USD = US_retailPrice) %>% head(n = 3)## setID pieces theme availability USD minifigs## 1 26803 103 Education {Not specified} NA 6## 2 26689 142 Education {Not specified} NA 4## 3 26804 98 Education {Not specified} NA 6Base R
names(mylego2)[5] <- 'USD'head(mylego2, n = 3)## theme availability setID pieces USD minifigs## 1 Education {Not specified} 26803 103 NA 6## 2 Education {Not specified} 26689 142 NA 4## 3 Education {Not specified} 26804 98 NA 6Mutate
Mutate
dplyr
mylego %>% filter(!is.na(pieces) & !is.na(US_retailPrice)) %>% mutate(Price_per_piece = US_retailPrice / pieces) %>% head(n = 3)## setID pieces theme availability US_retailPrice minifigs Price_per_piece## 1 26277 188 Education Educational 78.95 NA 0.4199468## 2 25949 280 Education Educational 224.95 NA 0.8033929## 3 25954 1 Education Educational 14.95 NA 14.9500000Base R
mylego2 <- mylego[!is.na(mylego$US_retailPrice) & !is.na(mylego$Price_per_piece),]mylego2$Price_per_piece <- mylego2$Price_per_piece / mylego2$US_retailPricehead(mylego2, n = 3)## [1] setID pieces theme availability ## [5] US_retailPrice minifigs Price_per_piece## <0 rows> (or 0-length row.names)Group By and Summarize
legosets %>% group_by(themeGroup) %>% summarize(mean_price = mean(US_retailPrice, na.rm = TRUE), sd_price = sd(US_retailPrice, na.rm = TRUE), median_price = median(US_retailPrice, na.rm = TRUE), n = n(), missing = sum(is.na(US_retailPrice)))## # A tibble: 15 × 6## themeGroup mean_price sd_price median_price n missing## <chr> <dbl> <dbl> <dbl> <int> <int>## 1 Action/Adventure 31.3 29.9 20.0 1280 462## 2 Basic 13.1 12.8 7.99 843 473## 3 Constraction 15.1 14.0 9.99 501 125## 4 Educational 89.0 107. 59.7 452 294## 5 Girls 23.4 22.6 15.0 677 225## 6 Historical 25.5 27.7 15.0 473 125## 7 Junior 18.6 13.2 17.8 228 93## 8 Licensed 42.9 58.3 25.0 2060 467## 9 Miscellaneous 14.3 20.8 6.99 4925 2117## 10 Model making 52.8 65.1 30.0 582 166## 11 Modern day 31.2 33.7 20.0 1723 763## 12 Pre-school 23.8 19.4 20.0 1487 699## 13 Racing 24.8 30.2 10 270 59## 14 Technical 60.8 68.1 40.0 550 137## 15 Vintage 9.71 9.56 7.50 304 264