Preparation exploration and visualization

The aim: make data usable, inspect it, then explain it

The lecture moves through preparation → exploration → visualization. These stages feed back into each other. A plot can reveal an impossible value, leading you back to cleaning; cleaning can reveal the need for a new feature.

“Garbage in, garbage out” means sophisticated models do not fix fundamentally wrong measurements, units or records. Preparation often takes much of the work. Standardize formats and units, ensure consistent types/scales, organize records and split complex fields where needed.

Measurement scales determine what calculations mean

A category encoded as a number can still be nominal or ordinal. Assigning bronze 1, silver 2 and gold 3 does not make the medals an interval quantity. The choice of scale affects summaries, graphs and modelling.

Six data-quality dimensions

The slide's biological consistency example is a prompt to investigate conflicting records, not a rule for invalidating someone based solely on a category label. Always interpret the definition and context of the recorded variables.

An extreme value is not automatically an error. The Olympics demonstrations show a very light young gymnast, a very tall basketball player and an old participant connected to historical art competitions. Context explains values that initially look suspicious. Investigate before deleting.

FAIR data

Findable: data and metadata can be located, ideally using a persistent identifier and meaningful indexing.

Accessible: users can retrieve data through a defined procedure. Access can be restricted for privacy; accessible does not mean public to everyone.

Interoperable: data use shared, understandable formats/vocabularies so they can work with other data.

Reusable: sufficiently clear descriptions, provenance and use conditions allow appropriate reuse.

Illustration: height = 180 without units or protocol is hard to reuse. A data dictionary saying “standing height in cm, measured without shoes using protocol X” supplies meaning. FAIR supports reproducibility but does not by itself prove the measurements are correct.

Wrangling with dplyr

A data frame stores observations in rows and variables in columns. The following examples use a small invented athlete table so every operation is easy to follow.

library(dplyr)
athletes <- data.frame(
  id = 1:4,
  sport = c("row", "run", "row", "run"),
  height_cm = c(180, 170, 190, 175),
  weight_kg = c(81, 60, 90, 70),
  age = c(24, 22, NA, 28)
)

arrange(desc(age)) reverses the sort. filter(year == 2008 | medal == "Gold") uses or: either condition is enough. & means both must hold. select() is about columns; filter() is about rows.

The pipe %>% passes the result on its left to the next function. Read a pipeline from top to bottom as sequential instructions.

athletes %>%
  mutate(height_m = height_cm / 100,
         BMI = weight_kg / height_m^2) %>%
  arrange(BMI) %>%
  select(id, sport, BMI)

BMI uses height in metres. A centimetre/metre error changes the squared denominator enormously. After creating a feature, inspect whether the values make sense.

athletes %>%
  group_by(sport) %>%
  summarise(records = n(),
            mean_age = mean(age, na.rm = TRUE),
            .groups = "drop")
# sport records mean_age
# row         2       24
# run         2       25

n() counts rows, including rows with missing age. The rowers' mean age uses only the one observed age. There are two records but not two observed ages. In the historical Olympics data, a row can be an athlete-event entry, so counts are not necessarily counts of unique people. Check the unit of observation.

The slide exercises select Games city, sort years, convert a height column to numeric, convert cm to m, calculate BMI, filter year/medal, and summarize count and mean age per medal. Converting a factor directly with as.numeric() can produce level codes; inspect and convert the intended labels carefully.

Joining tables through keys

Suppose table A has athletes with IDs 1, 2, 3, and table B has test values for IDs 2, 3, 4.

left_join(athletes, tests, by = "id")
# If the names differ:
left_join(athletes, tests, by = c("id" = "athlete_id"))

Keys can include several variables, such as athlete ID plus date. Software can infer shared names, but you should check that the inferred columns really define a match. Repeated keys can multiply rows: if an athlete has three test records, a join may produce three rows for that athlete. That can be appropriate or accidental depending on the question.

Duplicates and missing values

duplicated(x) marks repeated elements/rows. unique(x) returns unique values/rows. distinct(data, ...) keeps distinct combinations of specified columns. Removing duplicate athletes without considering event or date can destroy valid repeated observations.

colSums(is.na(athletes))

This counts missing cells in each column. It does not explain why they are missing. Use is.na(x), not x == NA: comparing an unknown value does not yield a usable true/false decision. Lecture 7 develops imputation; here, first understand the pattern and its implications.

Exploratory data analysis: inspect before assuming

1. Size and structure: dim(), nrow(), colnames(), head(), str().
2. Simple summaries: summary(), means, medians, SD, ranges and missingness.
3. Distributions: histograms, boxplots and violin plots.
4. Relationships: scatterplots and correlation plots.
5. Investigate surprises: use context and return to preparation when needed.

EDA aims to expose underlying structure, important variables and potential problems, and generate hypotheses. It is iterative; it does not merely certify that the data are ready.

The Olympics examples explore changing participation, age distributions and champions, height/weight, and missingness. Early female-athlete distributions may look unstable because participation was small. Differences between graphs can reflect sample size or changing composition, not just a real shift in the phenomenon.

Read the charts

Boxplot: the box spans Q1 to Q3, the central 50% of observations. The line marks the median. IQR = Q3 − Q1. Under the common rule, whiskers extend to the most extreme observations within 1.5 IQR of the box; more distant points are flagged. Whiskers are not invariably minimum/maximum. A flagged value may be real.

Violin: greater width means greater estimated density around that value, not a larger numerical measurement. A violin and a boxplot can complement each other: one shows distribution shape, the other compact summaries.

Correlation: positive values mean the variables tend to increase together; negative values mean one tends to decrease as the other increases. A coefficient near zero means little linear association; nonlinear patterns may still exist. Correlation does not establish causation. The slide's humorous correlations show how unrelated trends and many comparisons can produce impressive-looking relationships.

For a simple ordinary linear regression with an intercept, R² = r²; r is not itself explained variance. That relationship does not apply indiscriminately to every model. The slides explicitly correct a mistaken explained-variance label.

Build a ggplot from layers

library(ggplot2)
ggplot(mpg, aes(x = displ, y = hwy)) +
  geom_point()

mpg is the example dataset; displ is engine displacement and hwy highway mileage. aes() maps variables to visual properties; geom_point() draws points. + adds layers, unlike the dplyr pipe's sequential data processing.

ggplot(mpg, aes(displ, hwy, colour = class)) +
  geom_point()

# Alternative: one panel per class
ggplot(mpg, aes(displ, hwy)) +
  geom_point() +
  facet_wrap(~ class)

aes(colour = class) assigns colours according to categories. geom_point(colour = "blue") sets one fixed colour. Putting a literal colour inside aes() usually maps a category label instead of simply setting the colour.

Scales and coordinate systems are related but distinct. A theme makes the plot readable; it does not change the observations. Choose a chart for the question, label units, simplify distractions, use deliberate accessible colour and make the message interpretable without an oral explanation.

Dashboards and communicating results

A dashboard combines relevant indicators, filters and visualizations. The transcript's volleyball example uses jump measurements, distributions and trends, with athlete/date filtering and annotations for coaches. An R Shiny interface can make these interactive. The purpose is to support understanding and decisions, rather than add interaction for its own sake.