Trustworthy online controlled experiments : ǂa ǂpractical guide to A B testing

Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests. "A/B testing is the gold standard of creating verifiable and repeatable experiments, and this book is its definitive text" -- Steve Blank, father of modern entrepreneurship, author of The Startup Owner's Manual and The Four Steps to the Epiphany "This book is a great resource for executives, leaders, researchers or engineers looking to use online controlled experiments" -- Harry Shum, Executive Vice President, Microsoft Artificial Intelligence and Research Group "A great book that is both rigorous and accessible. Readers will learn how to bring trustworthy controlled experiments, which have revolutionized internet product development, to their organizations" -- Adam D'Angelo, Co-founder and CEO of Quora and prior CTO of Facebook "Kohavi, Tang and Xu have a wealth of experience and excellent advice to convey, so the book has lots of practical real world examples and lessons learned over many years of the application of these techniques at scale." -- Jeff Dean, Google Senior Fellow, and SVP, Google Research "The secret sauce for a successful online business is experimentation. But it is a secret no longer. Here three masters of the art describe the ABCs of A/B testing so that you too can continuously improve your online services." -- Hal Varian, Chief Economist, Google, and author of Intermediate Microeconomics: A Modern Approach "This is the new bible of how to get from data to decisions in the digital age." -- Scott Cook, Intuit Co-founder & Chairman of the executive committee Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions. Learn how to • Use the scientific method to evaluate hypotheses using controlled experiments. • Define key metrics and ideally an Overall Evaluation Criterion. • Test for trustworthiness of the results and alert experimentersto violated assumptions. • Build a scalable platform that lowers the marginal cost of experiments close to zero. • Avoid pitfalls like carryover effects and Twyman's law \* Understand how statistical issues play out in practice. Presents The Essential Concepts In Thirty-four Brief Stories. Drawing On His Experience As A Medical Researcher, Vickers Blends Explanations And Humor With Minimal Math, To Help Readers Understand And Interpret The Statistics They Read Every Day. Introduction: I Tell A Friend That My Job Is More Fun That You'd Think : What Is Statistics? -- Describing Data: So Bill Gates Walks Into A Diner : On Means And Medians ; Bill Gates Goes Back To The Diner : Standard Deviation And Interquartile Range ; A Skewed Shot, A Biased Referee ; You Can't Have 2.6 Children : On Different Types Of Data ; Why Your High School Math Teacher Was Right : How To Draw A Graph -- Daily Distributions: Chutes-and-ladders And Serum Hemoglobin Levels : Thoughts On The Normal Distribution ; If The Normal Distribution Is So Normal, How Come My Data Never Are? ; But I Like That Sweater : What Amount Of Fit Is A Good Enough Fit? -- Variation Of Study Results: Confidence Intervals: Long Hair : A Standard Error Of The Older Male ; How To Avoid A Rainy Wedding : Variation And Confidence Intervals ; Statistical Ties, And Why You Shouldn't Wear One : More On Confidence Intervals --^ Hypothesis Testing: Choosing A Route To Cycle Home : What P-values Do For Us ; The Probability Of A Dry Toothbrush : What Is A P-value Anyway? ; Michael Jordan Won't Accept The Null Hypothesis : How To Interpret High P-values ; The Difference Between Sports And Business : Thoughts On The T Test And The Wilcoxon Test ; Meeting Up With Friends : On Sample Size, Precision And Statistical Power -- Regression And Decision Making: When To Visit Chicago : About Linear And Logistic Regression ; My Assistant Turns Up For Work With Shorter Hair : About Regression And Confounding ; I Ignore My Child's Cough, My Wife Panics : About Specificity And Sensitivity ; Avoid The Sales : Statistics To Help Make Decisions --^ Some Common Statistical Errors, And What They Teach Us: One Better Than Tommy John : Four Statistical Errors, Some Of Which Are Totally Trivial, But All Of Which Matter A Great Deal ; Weed Control For P-values : A Single Scientific Question Should Be Addressed By A Single Statistical Test ; How To Shoot A Tv Episode : Statistical Analyses That Don't Provide Meaningful Numbers ; Sam, 93 Years Old, 700 Pound Florida Super-granddad : Two Common Errors In Regression ; Regression To The Mike : A Statistical Explanation Of Why An Eligible Friend Of Mine Is Still Single ; Oj Simpson, Sally Clark, George And Me : About Conditional Probability ; Boy Meets Girl, Girl Rejects Boy,^ Boy Starts Multiple Testing ; Some Things That Have Never Happened To Me : Why You Shouldn't Compare P-values ; How To Win The Marathon : Avoiding Errors When Measuring Things That Happen Over Time ; The Difference Between Bad Statistics And A Bacon Sandwich : Are There Rules In Statistics? ; Look At Your Garbage Bin : It May Be The Only Thing You Need To Know About Statistics ; Numbers That Mean Something : Linking Math And Science ; Statistics Is About People, Even If You Can't See The Tears. Andrew Vickers. Includes Bibliographical References (p. 209-210) And Index. "This authoritative, expanded and updated second edition of Encyclopedia of Machine Learning and Data Mining provides easy access to core information for those seeking entry into any aspect within the broad field of Machine Learning and Data Mining. A paramount work, its 800 entries - about 150 of them newly updated or added - are filled with valuable literature references, providing the reader with a portal to more detailed information on any given topic. Topics for the Encyclopedia of Machine Learning and Data Mining include Learning and Logic, Data Mining, Applications, Text Mining, Statistical Learning, Reinforcement Learning, Pattern Mining, Graph Mining, Relational Mining, Evolutionary Computation, Information Theory, Behavior Cloning, and many others. Topics were selected by a distinguished international advisory board. Each peer-reviewed, highly-structured entry includes a definition, key words, an illustration, applications, a bibliography, and links to related literature. The entries are expository and tutorial, making this reference a practical resource for students, academics, or professionals who employ machine learning and data mining methods in their projects. Machine learning and data mining techniques have countless applications, including data science applications, and this reference is essential for anyone seeking quick access to vital information on the topic" -- Provided by publisher's website Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests. Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions. Learn how to - Use the scientific method to evaluate hypotheses using controlled experiments - Define key metrics and ideally an Overall Evaluation Criterion - Test for trustworthiness of the results and alert experimenters to violated assumptions - Build a scalable platform that lowers the marginal cost of experiments close to zero - Avoid pitfalls like carryover effects and Twyman's law - Understand how statistical issues play out in practice. "Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests. Based on practical experiences at companies that each runs more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for experienced practitioners who want to improve the way they and their organizations make data-driven decisions"-- Provided by publisher A brief, authoritative introduction to field experimentation in the social sciences. Written by two leading experts on experimental methods, this concise text covers the major aspects of experiment design, analysis, and interpretation in clear language. Students learn how to design randomized experiments, analyze the data, and interpret the findings. Beyond the authoritative coverage of the basic methodology, the authors include numerous features to help students achieve a deeper understanding of field experimentation, including rich examples from the social science literature, problem sets and discussions, data sets, and further readings. What is a p-value Anyway? offers a fun introduction to the fundamental principles of statistics, presenting the essential concepts in thirty-four brief, enjoyable stories. Drawing on his experience as a medical researcher, Vickers blends insightful explanations and humor, with minimal math, to help readers understand and interpret the statistics they read every day. KEY TOPICS: Describing data; Data distributions; Variation of study results: confidence intervals; Hypothesis testing; Regression and decision making; Some common statistical errors, and what they teach us MARKET: For all readers interested in statistics. What is a p -value Anyway? offers a fun introduction to the fundamental principles of statistics, presenting the essential concepts in thirty-four brief, enjoyable stories. Drawing on his experience as a medical researcher, Vickers blends insightful explanations and humor, with minimal math, to help readers understand and interpret the statistics they read every day. Describing data; Data distributions; Variation of study results: confidence intervals; Hypothesis testing; Regression and decision making; Some common statistical errors, and what they teach us For all readers interested in statistics. This monograph will provide an in-depth mathematical treatment of modern multiple test procedures controlling the false discovery rate (FDR) and related error measures, particularly addressing applications to fields such as genetics, proteomics, neuroscience and general biology. The book will also include a detailed description how to implement these methods in practice. Moreover new developments focusing on non-standard assumptions are also included, especially multiple tests for discrete data. The book primarily addresses researchers and practitioners but will also be beneficial for graduate students. "Written by two leading experts on experimental methods, this concise text covers the major aspects of experiment design, analysis, and interpretation in clear language. Students learn how to design randomized experiments, analyze the data, and interpret the findings. Beyond the authoritative coverage of the basic methodology, Field Experiments includes numerous features to help students achieve a deeper understanding of field experimentation, including rich examples from the social science literature, problem sets and discussions, datasets, and lists of further readings"--Back cover Introduction -- Causal Inference And Experimentation -- Sampling Distributions, Statistical Inference, And Hypothesis Testing -- Using Covariates In Experimental Design And Analysis -- One-sided Noncompliance -- Two-sided Noncompliance -- Attrition -- Interference Between Experimental Units -- Mediation -- Integration Of Research Findings -- Writing An Experimental Prospectus, Research Report, And Journal Article -- Experimental Challenges And Opportunities. Alan S. Gerber, Donald P. Green. Includes Bibliographical References And Index.