Correlation Analysis
Hypothesis testing (Notebook 40) asked "do these groups differ?" Correlation analysis asks a different question: "do these two variables move together — and how strongly?" It is the quickest way to find relationships in a dataset, the backbone of feature selection, and the setup for regression (Section 6). It is also the single most misused idea in statistics, because a correlation is constantly mistaken for a cause.
Notebook
About 150 minutes to complete
Enrol to read the notebook and download it to run.
What this module covers
Hypothesis testing (Notebook 40) asked "do these groups differ?" Correlation analysis asks a different question: "do these two variables move together — and how strongly?" It is the quickest way to find relationships in a dataset, the backbone of feature selection, and the setup for regression (Section 6). It is also the single most misused idea in statistics, because a correlation is constantly mistaken for a cause.
By the end of it
- Define correlation — direction and strength of association — and read a coefficient between −1 and +1
- Compute and choose between Pearson (linear), Spearman (monotonic, rank-based), and Kendall (rank concordance) correlations
- Test a correlation for statistical significance and build a confidence interval (Fisher z)
- Produce and read correlation matrices, heatmaps, and pair plots
- Compute a partial correlation to control for a lurking third variable
- Explain — with worked examples — why correlation is not causation (confounders, spurious correlation, Simpson's paradox)