Editing Reinforcement learning (section)

== Statistical comparison of reinforcement learning algorithms ==
Efficient comparison of RL algorithms is essential for research, deployment and monitoring of RL systems. To compare different algorithms on a given environment, an agent can be trained for each algorithm. Since the performance is sensitive to implementation details, all algorithms should be implemented as closely as possible to each other.<ref>{{Cite journal |last1=Engstrom |first1=Logan |last2=Ilyas |first2=Andrew |last3=Santurkar |first3=Shibani |last4=Tsipras |first4=Dimitris |last5=Janoos |first5=Firdaus |last6=Rudolph |first6=Larry |last7=Madry |first7=Aleksander |date=2019-09-25 |title=Implementation Matters in Deep RL: A Case Study on PPO and TRPO |url=https://openreview.net/forum?id=r1etN1rtPB |journal=ICLR |language=en}}</ref> After the training is finished, the agents can be run on a sample of test episodes, and their scores (returns) can be compared. Since episodes are typically assumed to be [[i.i.d]], standard statistical tools can be used for hypothesis testing, such as [[Student's t-test|T-test]] and [[permutation test]].<ref>{{Cite journal |last=Colas |first=Cédric |date=2019-03-06 |title=A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms |url=https://openreview.net/forum?id=ryx0N3IaIV |journal=International Conference on Learning Representations |arxiv=1904.06979 |language=en}}</ref> This requires to accumulate all the rewards within an episode into a single number—the episodic return. However, this causes a loss of information, as different time-steps are averaged together, possibly with different levels of noise. Whenever the noise level varies across the episode, the statistical power can be improved significantly, by weighting the rewards according to their estimated noise.<ref>{{Cite journal |last1=Greenberg |first1=Ido |last2=Mannor |first2=Shie |date=2021-07-01 |title=Detecting Rewards Deterioration in Episodic Reinforcement Learning |url=https://proceedings.mlr.press/v139/greenberg21a.html |journal=Proceedings of the 38th International Conference on Machine Learning |language=en |publisher=PMLR |pages=3842–3853|arxiv=2010.11660 }}</ref>