Editing Simpson's paradox (section)

== Simpson's second paradox ==
A second, less well-known paradox was also discussed in Simpson's 1951 paper. It can occur when the "sensible interpretation" is not necessarily found in the separated data, like in the Kidney Stone example, but can instead reside in the combined data. Whether the partitioned or combined form of the data should be used hinges on the process giving rise to the data, meaning the correct interpretation of the data cannot always be determined by simply observing the tables.<ref>{{Cite journal|last1=Norton|first1=H. James|last2=Divine|first2=George|date=August 2015|title=Simpson's paradox ... and how to avoid it|journal=Significance|volume=12|issue=4|pages=40–43|doi=10.1111/j.1740-9713.2015.00844.x|doi-access=free}}</ref>

[[Judea Pearl]] has shown that, in order for the partitioned data to represent the correct causal relationships between any two variables, <math>X</math> and <math>Y</math>, the partitioning variables must satisfy a graphical condition called "back-door criterion":<ref>{{cite journal|last1=Pearl|first1=Judea|year=2014|title=Understanding Simpson's Paradox|journal=The American Statistician|volume=68|issue=1|pages=8–13|doi=10.2139/ssrn.2343788|s2cid=2626833}}</ref><ref>{{cite journal|last1=Pearl|first1=Judea|year=1993|title=Graphical Models, Causality, and Intervention|journal=Statistical Science|volume=8|issue=3|pages=266–269|doi=10.1214/ss/1177010894|doi-access=free}}</ref>
# They must block all spurious paths between <math>X</math> and <math>Y</math>
# No variable can be affected by <math>X</math>
This criterion provides an algorithmic solution to Simpson's second paradox, and explains why the correct interpretation cannot be determined by data alone; two different graphs, both compatible with the data, may dictate two different back-door criteria.

When the back-door criterion is satisfied by a set ''Z'' of covariates, the adjustment formula (see [[Confounding]]) gives the correct causal effect of ''X'' on ''Y''. If no such set exists, Pearl's ''do''-calculus can be invoked to discover other ways of estimating the causal effect.<ref name="pearl" /><ref name="pearl-bow">{{cite book |last1=Pearl |first1=J. |last2=Mackenzie |first2=D. |title=The Book of Why: The New Science of Cause and Effect |date=2018 |publisher=Basic Books |location=New York, NY}}</ref> The completeness of ''do''-calculus <ref>{{cite journal |last1=Shpitser |first1=I. |last2=Pearl |first2=J. |editor1-last=Dechter |editor1-first=R. |editor2-last=Richardson |editor2-first=T.S. |title=Identification of Conditional Interventional Distributions |journal=Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence |date=2006 |pages=437–444 |publisher=AUAI Press |location=Corvallis, OR}}</ref><ref name="pearl-bow" /> can be viewed as offering a complete resolution of the Simpson's paradox.