Editing Scientific method (section)

===Certainty, probabilities, and statistical inference===

Claims of scientific truth can be opposed in three ways: by falsifying them, by questioning their certainty, or by asserting the claim itself to be incoherent.{{efn| ...simplified and (post-modern) philosophy notwithstanding.{{harvp|Gauch Jr|2002|p=33}}}} Incoherence, here, means internal errors in logic, like stating opposites to be true; falsification is what Popper would have called the honest work of conjecture and refutation<ref name= trialAndErr/> — certainty, perhaps, is where difficulties in telling truths from non-truths arise most easily.

Measurements in scientific work are usually accompanied by estimates of their [[uncertainty]].<ref name="conjugatePairs" /> The uncertainty is often estimated by making repeated measurements of the desired quantity. Uncertainties may also be calculated by consideration of the uncertainties of the individual underlying quantities used. Counts of things, such as the number of people in a nation at a particular time, may also have an uncertainty due to [[data collection]] limitations. Or counts may represent a sample of desired quantities, with an uncertainty that depends upon the [[sampling method]] used and the number of samples taken.

In the case of measurement imprecision, there will simply be a 'probable deviation' expressing itself in a study's conclusions. Statistics are different. [[Inductive reasoning#Statistical generalisation|Inductive statistical generalisation]] will take sample data and extrapolate more general conclusions, which has to be justified — and scrutinised. It can even be said that statistical models are only ever useful, [[All models are wrong|but never a complete representation of circumstances]].

In statistical analysis, expected and unexpected bias is a large factor.<ref name="Welsby Weatherall 2022 pp. 793–798">{{cite journal | last1=Welsby | first1=Philip D | last2=Weatherall | first2=Mark | title=Statistics: an introduction to basic principles | journal=Postgraduate Medical Journal | volume=98 | issue=1164 | date=2022-10-01 | issn=0032-5473 | doi=10.1136/postgradmedj-2020-139446 | pages=793–798| pmid=34039698 }}</ref> [[Research question]]s, the collection of data, or the interpretation of results, all are subject to larger amounts of scrutiny than in comfortably logical environments. Statistical models go through a [[Statistical model validation|process for validation]], for which one could even say that awareness of potential biases is more important than the hard logic; errors in logic are easier to find in [[peer review]], after all.{{efn|... and [[John Ioannidis]], in 2005,<ref name="mostRwrong" /> has shown that not everybody respects the principles of statistical analysis; whether they be the principles of inference or otherwise.{{Broader|#Relationship with statistics}}}} More general, claims to rational knowledge, and especially statistics, have to be put into their appropriate context.<ref name="Gauch Jr 2002 p30/ch4">{{harvp|Gauch Jr|2002|loc=Quotes from p. 30, expanded on in ch. 4}}: Gauch gives two simplified statements on what he calls "rational-knowledge claim". It is either "I hold belief X for reasons R with level of confidence C, where inquiry into X is within the domain of competence of method M that accesses the relevant aspects of reality" (inductive reasoning) or "I hold belief X because of presuppositions P." (deductive reasoning)</ref> Simple statements such as '9 out of 10 doctors recommend' are therefore of unknown quality because they do not justify their methodology.

Lack of familiarity with statistical methodologies can result in erroneous conclusions. Foregoing the easy example,{{efn|For instance, extrapolating from a single scientific observation, such as "This experiment yielded these results, so it should apply broadly," exemplifies inductive wishful thinking. [[inductive reasoning#statistical generalization|Statistical generalisation]] is a form of inductive reasoning. Conversely, assuming that a specific outcome will occur based on general trends observed across multiple experiments, as in "Most experiments have shown this pattern, so it will likely occur in this case as well," illustrates faulty [[Deductive reasoning#Probability logic|deductive probability logic]].}} multiple probabilities interacting is where, for example medical professionals,<!--justification: medical professional = authoritative science communicator--><ref name="Gigerenzer 2015">{{cite book | last=Gigerenzer | first=Gerd | title=Risk Savvy | publisher=Penguin | publication-place=New York, New York | date=2015-03-31 | isbn=978-0-14-312710-9 | page=}} leads: (n=1000) only 21% of [[gynaecologist]]s got an example question on [[Bayes' theorem]] right. Book, including the assertion, introduced in {{cite web | last=Kremer | first=William | title=Do doctors understand test results? | website=BBC News | date=6 July 2014 | url=https://www.bbc.com/news/magazine-28166019 | access-date=24 April 2024}}</ref> have shown a lack of proper understanding. [[Bayes' theorem]] is the mathematical principle lining out how standing probabilities are adjusted given new information. The [[boy or girl paradox]] is a common example. In knowledge representation, [[mutual information#Bayesian estimation of mutual information|Bayesian estimation of mutual information]] between [[random variable]]s is a way to measure dependence, independence, or interdependence of the information under scrutiny.<ref name= prml >{{cite book |first1=Christopher M. |last1=Bishop |url=https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf |date=2006 |title=Pattern Recognition and Machine Learning |pages=21, 30, 55, 152, 161, 277, 360, 448, 580 |publisher=Springer Science+Business Media |via=Microsoft }}</ref>

Beyond commonly associated [[survey methodology]] of [[field research]], the concept together with [[probabilistic reasoning]] is used to advance fields of science where research objects have no definitive states of being. For example, in [[statistical mechanics]].