Scalable self-paced e‑learning for code with automated feedback
23rd October 2024 @ WOMBAT 2024
Mitchell O’Hara-Wild, Monash University
Tools for teaching
I create lots of tools to support my teaching.
Lately I’ve been working on tools to give students targeted feedback at scale.
Why moodle?
Creating a quiz in Moodle
Literate programming quizzes
Literate programming
An approach for creating documents or code using both writing and code in the same place
Widely used literate programming tools include:
- Quarto
- Jupyter notebooks
- R Markdown
Statistics education ❤️ Literate programming
Literate programming for statistical quizzes is a great match! We work closely with data and code, and want to test students on these skills
Literate programming quizzes
💥 This isn’t a new idea…
The exams package for R has been on CRAN for over 15 years!
Lots of information at https://www.r-exams.org/
Literate programming quizzes
Why re-invent the R/exams wheel?
Literate programming quizzes
Using R/exams
boxhist.Rmd [106 lines]
```{r data generation, echo = FALSE, results = "hide"}
## DATA
n <- sample(30:50, 1)
m <- sample(1:4, 1)
s <- runif(1, 0.5, 2)
delta <- ifelse(runif(1) < 0.2, sample(8:12), 0)
p2 <- runif(1, 0.45, 0.55)
skewed <- left <- FALSE
if(!delta) {
skewed <- runif(1) < 0.6
left <- runif(1) < 0.3
}
dgpBoxhist <- function(n = 40, mean = 0, sd = 1, delta = 0,
p2 = 0.5, skewed = FALSE, left = FALSE)
{
SK <- function(x) abs(diff(diff(fivenum(x)[2:4]))/diff(fivenum(x)[c(2, 4)]))
sim <- function(x){
x <- rnorm(n)
if(skewed) exp(x) else x
}
x <- sim()
if(skewed) while(SK(x) < 0.7) x <- sim() else while(SK(x) > 0.15) x <- sim()
if(left) x <- -x
x <- mean + sd * scale(x)
k <- sample(1:n, round(p2 * n))
x[k] <- x[k] + delta
as.vector(sample(x))
}
x <- round(dgpBoxhist(n = n, mean = m, sd = s, delta = delta,
p2 = p2, skewed = skewed, left = left), digits = 2)
b <- boxplot(x, plot = FALSE)
spread <- tol <- signif(diff(range(c(b$stats, b$out)))/25, 1)
write.csv(data.frame(x), file = "boxhist.csv", quote = FALSE, row.names = FALSE)
## QUESTION/SOLUTION
questions <- solutions <- explanations <- rep(list(""), 6)
type <- rep(list("schoice"), 6)
questions[[1]] <- paste("The distribution is ", c("", "_not_ "), "unimodal.", sep = "")
solutions[[1]] <- c(delta < 1, delta > 1)
questions[[2]] <- paste("The distribution is", c("symmetric.", "right-skewed.", "left-skewed."))
solutions[[2]] <- c(!skewed, skewed & !left, skewed & left)
questions[[3]] <- paste("The boxplot shows ", c("", "_no_ "), "outliers.", sep = "")
solutions[[3]] <- c(length(b$out) > 0, length(b$out) < 1)
questions[[4]] <- "A quarter of the observations is smaller than which value?"
solutions[[4]] <- explanations[[4]] <- signif(b$stats[[2]], 3)
type[[4]] <- "num"
questions[[5]] <- "A quarter of the observations is greater than which value?"
solutions[[5]] <- explanations[[5]] <- signif(b$stats[[4]], 3)
type[[5]] <- "num"
questions[[6]] <- paste("Half of the observations are",
sample(c("smaller", "greater"), 1), "than which value?")
solutions[[6]] <- explanations[[6]] <- signif(b$stats[[3]], 3)
type[[6]] <- "num"
explanations[1:3] <- lapply(solutions[1:3], function(x) ifelse(x, "True", "False"))
solutions[1:3] <- lapply(solutions[1:3], mchoice2string)
if(any(explanations[4:6] < 0)) explanations[4:6] <- lapply(solutions[4:6], function(x) paste("$", x, "$", sep = ""))
```
Question
========
For the `r n` observations of the variable `x` in the data file
[boxhist.csv](boxhist.csv) draw a histogram, a boxplot and a stripchart.
Based on the graphics, answer the following questions or check the correct
statements, respectively. _(Comment: The tolerance for numeric answers is
$\pm`r tol`$, the true/false statements are either about correct or clearly wrong.)_
```{r questionlist, echo = FALSE, results = "asis"}
answerlist(unlist(questions), markup = "markdown")
```
Solution
========
\
```{r boxplot_hist, echo = FALSE, results = "hide", fig.height = 4.5, fig.width = 9, fig.path = "", fig.cap = ""}
par(mfrow = c(1, 2))
boxplot(x, axes = FALSE)
axis(2, at = signif(b$stats, 3), las = 1)
box()
hist(x, freq = FALSE, main = "")
rug(x)
```
```{r solutionlist, echo = FALSE, results = "asis"}
answerlist(unlist(explanations), markup = "markdown")
```
Meta-information
================
extype: cloze
exsolution: `r paste(solutions, collapse = "|")`
exclozetype: `r paste(type, collapse = "|")`
exname: Boxplot and histogram
extol: `r tol`Literate programming quizzes
Using R/exams
Introducing moodlequiz!
Introducing moodlequiz!
How it works
demo.Rmd [88 lines]
More moodlequiz to come
Sustainable and scalable?
Rewriting learnr.numbat.space
Early work in progress (ready S1 2025)
Ingredients of code assessment
Assessment using Outputs
Assessment using the Environment
Assessment using Inputs
Assessment of Inputs (code style)
Quarto Live
Quarto Live
Thanks for your time!
Final remarks
- Statistics assessments should test theory AND code.
- Use small automated quizzes to evaluate student performance and give feedback at scale.
- Literate programming makes creating quizzes and exercises easy.
- Using webr to run R in the browser is great for teaching.
