PROBABILITY THEORY – THE LOGIC OF SCIENCE

PROBABILITY THEORY – THE LOGIC OF SCIENCE

E. T. Jaynes - Wayman Crow Professor of Physics

EDITORS FORWARD

 Jaynes died April 30, 1998. Before his death he asked me to finish and publish his book on probability theory. I struggled with this for some time, because there is no doubt in my mind that Jaynes wanted this book finished. Unfortunately, most of the later Chapters, Jaynes’ intended volume 2 on applications, were either missing or incomplete and some of the early also Chapters had missing pieces. I could have written these latter Chapters and filled the missing pieces, but if I did so, the work would no longer belong to Jaynes; rather, it would be a Jaynes-Bretthorst hybrid with no way to tell which material came from which author. In the end, I decided that the missing Chapters would have to stay missing—the work would remain Jaynes’.

There were a number of missing pieces of varying length that Jaynes had marked by inserting the phrase “MUCH MORE COMING.” I could have left these comments in the text, but they were ugly and they made the book looks very incomplete. Jaynes intended this book to serve as both a reference and a text book. Consequently, there are question boxes scattered throughout most Chapters. In the end, I decided to replace the “MUCH MORE COMING” comments by introducing an “editors” question box. If you answer these questions, you will have filled in the missing material. You will be able to identify these questions because I used a shaded box for the editors questions, while Jaynes’ question boxes are not shaded.

Jaynes’ wanted to include a series of computer programs that implemented some of the calcu- lations in this book. I had originally intended to include these programs. But as time went on, it became increasingly obvious that many of the programs were not available and the ones that were, were written in a particularly obscure form of BASIC (it was the programs that were obscure, not the BASIC). Consequently, I removed references to these programs and, where necessary, inserted a few sentences to direct people to the necessary software tools to implement the calculations.

Finally, while I am the most obvious person who has worked on getting this book into pub- lication, I am not the only person to do so. Some of Jaynes’ closest friends have assisted me in completing this work. These include Tom Grandy, Ray Smith, Tom Loredo, Myron Tribus and John Skilling, and I would like to thank them for their assistance. I would also like to thank Joe Ackerman for allowing me to take the time necessary to get this work published.

PREFACE The following material is addressed to readers who are already familiar with applied mathematics at the advanced undergraduate level or preferably higher; and with some field, such as physics, chemistry, biology, geology, medicine, economics, sociology, engineering, operations research, etc., where inference is needed.† A previous acquaintance with probability and statistics is not necessary;

indeed, a certain amount of innocence in this area may be desirable, because there will be less to unlearn.

We are concerned with probability theory and all of its conventional mathematics, but now viewed in a wider context than that of the standard textbooks. Every Chapter after the first has “new” (i.e. not previously published) results that we think will be found interesting and useful.

Many of our applications lie outside the scope of conventional probability theory as currently taught. But we think that the results will speak for themselves, and that something like the theory expounded here will become the conventional probability theory of the future.

History: The present form of this work is the result of an evolutionary growth over many years. My interest in probability theory was stimulated first by reading the work of Harold Jeffreys (1939) and realizing that his viewpoint makes all the problems of theoretical physics appear in a very different light. But then in quick succession discovery of the work of R. T. Cox (1946), C. E. Shannon (1948) and G. Pólya (1954) opened up new worlds of thought, whose exploration has occupied my mind for some forty years. In this much larger and permanent world of rational thinking in general, the current problems of theoretical physics appeared as only details of temporary interest.

The actual writing started as notes for a series of lectures given at Stanford University in 1956, expounding the then new and exciting work of George Pólya on “Mathematics and Plausible Rea- soning.” He dissected our intuitive “common sense” into a set of elementary qualitative desiderata and showed that mathematicians had been using them all along to guide the early stages of discov- ery, which necessarily precede the finding of a rigorous proof. The results were much like those of James Bernoulli’s “Art of Conjecture” (1713), developed analytically by Laplace in the late 18’th century; but Pólya thought the resemblance to be only qualitative.

However, Pólya demonstrated this qualitative agreement in such complete, exhaustive detail as to suggest that there must be more to it. Fortunately, the consistency theorems of R. T. Cox were enough to clinch matters; when one added Pólya’s qualitative conditions to them the result was a proof that, if degrees of plausibility are represented by real numbers, then there is a uniquely determined set of quantitative rules for conducting inference. That is, any other rules whose results conflict with them will necessarily violate an elementary—and nearly inescapable—desideratum of rationality or consistency.

But the final result was just the standard rules of probability theory, given already by Bernoulli and Laplace; so why all the fuss? The important new feature was that these rules were now seen as uniquely valid principles of logic in general, making no reference to “chance” or “random variables”;

so their range of application is vastly greater than had been supposed in the conventional probability theory that was developed in the early twentieth century. As a result, the imaginary distinction between “probability theory” and “statistical inference” disappears, and the field achieves not only logical unity and simplicity, but far greater technical power and flexibility in applications.

By “inference” we mean simply: deductive reasoning whenever enough information is at hand to permit it; inductive or plausible reasoning when—as is almost invariably the case in real problems—the necessary information is not available. But if a problem can be solved by deductive reasoning, probability theory is not needed for it; thus our topic is the optimal processing of incomplete information.

In the writer’s lectures, the emphasis was therefore on the quantitative formulation of Pólya’s viewpoint, so it could be used for general problems of scientific inference, almost all of which arise out of incomplete information rather than “randomness.” Some personal reminiscences about George Pólya and this start of the work are in Chapter 5.

But once the development of applications started, the work of Harold Jeffreys, who had seen so much of it intuitively and seemed to anticipate every problem I would encounter, became again the central focus of attention. My debt to him is only partially indicated by the dedication of this book to his memory. Further comments about his work and its influence on mine are scattered about in several Chapters.

In the years 1957-1970 the lectures were repeated, with steadily increasing content, at many other Universities and research laboratories. In this growth it became clear gradually that the outstanding difficulties of conventional “statistical inference” are easily understood and overcome.

But the rules which now took their place were quite subtle conceptually, and it required some deep thinking to see how to apply them correctly. Past difficulties which had led to rejection of Laplace’s work, were seen finally as only misapplications, arising usually from failure to define the problem unambiguously or to appreciate the cogency of seemingly trivial side information, and easy to correct once this is recognized. The various relations between our “extended logic” approach and the usual “random variable” one appear in almost every Chapter, in many different forms.

Eventually, the material grew to far more than could be presented in a short series of lec- tures, and the work evolved out of the pedagogical phase; with the clearing up of old difficulties accomplished, we found ourselves in possession of a powerful tool for dealing with new problems.

Since about 1970 the accretion has continued at the same pace, but fed instead by the research activity of the writer and his colleagues. We hope that the final result has retained enough of its hybrid origins to be usable either as a textbook or as a reference work; indeed, several generations of students have carried away earlier versions of our notes, and in turn taught it to their students.

In view of the above, we repeat the sentence that Charles Darwin wrote in the Introduction to his Origin of Species: “I hope that I may be excused for entering on these personal details, as I give them to show that I have not been hasty in coming to a decision.” But it might be thought that work done thirty years ago would be obsolete today. Fortunately, the work of Jeffreys, Pólya and Cox was of a fundamental, timeless character whose truth does not change and whose importance grows with time. Their perception about the nature of inference, which was merely curious thirty years ago, is very important in a half-dozen different areas of science today; and it will be crucially important in all areas 100 years hence.

Foundations: From many years of experience with its applications in hundreds of real problems, our views on the foundations of probability theory have evolved into something quite complex, which cannot be described in any such simplistic terms as “pro-this” or “anti-that.” For exam- ple, our system of probability could hardly be more different from that of Kolmogorov, in style, philosophy, and purpose. What we consider to be fully half of probability theory as it is needed in current applications—the principles for assigning probabilities by logical analysis of incomplete information—is not present at all in the Kolmogorov system.

Yet when all is said and done we find ourselves, to our own surprise, in agreement with Kolmogorov and in disagreement with his critics, on nearly all technical issues. As noted in Ap- pendix A, each of his axioms turns out to be, for all practical purposes, derivable from the Pólya-Cox  Some of the material in the early Chapters was issued in 1958 by the Socony-Mobil Oil Company as Number 4 in their series “Colloquium Lectures in Pure and Applied Science.”

G. Larry Bretthorst

VOLUME I – PRINCIPLES AND ELEMENTARY APPLICATIONS

Chapter 1 Plausible Reasoning

1.  Deductive and Plausible Reasoning
2.  Analogies with Physical Theories
3.  The Thinking Computer
4.  Introducing the Robot
5.  Boolean Algebra
6.  Adequate Sets of Operations
7.  The Basic Desiderata
8.  Comments
9.  Common Language vs. Formal Logic
10.  Nitpicking

 Chapter 2 The Quantitative Rules

1.  The Product Rule
2.  The Sum Rule
3.  Qualitative Properties
4.  Numerical Values
5.  Notation and Finite Sets Policy
6.  Comments
7.  “Subjective” vs. “Objective”
8.  Gödel’s Theorem
9.  Venn Diagrams
10.  The “Kolmogorov Axioms”
11.  Chapter 3 Elementary Sampling Theory
12.  Sampling Without Replacement
13.  Logic Versus Propensity
14.  Reasoning from Less Precise Information
15.  Expectations
16.  Other Forms and Extensions
17.  Probability as a Mathematical Tool
18.  The Binomial Distribution
19.  Sampling With Replacement
20.  Digression: A Sermon on Reality vs. Models
21.  Correction for Correlations
22.  Simplification
23.  Comments
24. Look Ahead

 Chapter 4 Elementary Hypothesis Testing

1.  Prior Probabilities
2.  Testing Binary Hypotheses with Binary Data
3.  Non-Extensibility Beyond the Binary Case
4.  Multiple Hypothesis Testing
5. Continuous Probability Distribution Functions (pdf’s)
6.  Testing an Infinite Number of Hypotheses
7.  Simple and Compound (or Composite) Hypotheses
8.  Comments
9.  Etymology
10.  What Have We Accomplished?

 Chapter 5 Queer Uses For Probability Theory

1.  Extrasensory Perception
2.  Mrs. Stewart’s Telepathic Powers
3.  Digression on the Normal Approximation
4.  Back to Mrs. Stewart
5.  Converging and Diverging Views
6.  Visual Perception—Evolution into Bayesianity?
7.  The Discovery of Neptune
8.  Digression on Alternative Hypotheses
9.  Back to Newton
10.  Horse racing and Weather Forecasting
11.  Discussion
12.  Paradoxes of Intuition
13.  Bayesian Jurisprudence
14.  Comments

 Chapter 6 Elementary Parameter Estimation

1.  Inversion of the Urn Distributions
2.  Both N and R Unknown
3.  Uniform Prior
4.  Predictive Distributions
5.  Truncated Uniform Priors
6. Concave Prior
7.  The Binomial Monkey Prior
8.  Metamorphosis into Continuous Parameter Estimation
9.  Estimation with a Binomial Sampling Distribution
10.  Digression on Optional Stopping
11.  Compound Estimation Problems
12. Simple Bayesian Estimate: Quantitative Prior Information
13.  From Posterior Distribution Function to Estimate
14.  Back to the Problem
15.  Effects of Qualitative Prior Information
16.  Choice of a Prior
17.  On With the Calculation!
18.  The Jeffreys Prior
19.  The Point of It All
20.  Interval Estimation
21.  Calculation of Variance
22.  Generalization and Asymptotic Forms
23.  Rectangular Sampling Distribution
24.  Small Samples
25.  Mathematical Trickery
26.  Comments

 Chapter 7 The Central, Gaussian Or Normal Distribution

1.  The Gravitating Phenomenon
2.  The Herschel-Maxwell Derivation
3.  The Gauss Derivation
4.  Historical Importance of Gauss’ Result
5.  The Landon Derivation
6.  Why the Ubiquitous Use of Gaussian Distributions?
7.  Why the Ubiquitous Success?
8.  What Estimator Should We Use?
9.  Error Cancellation
10.  The Near-Irrelevance of Sampling Frequency Distributions
11.  The Remarkable Efficiency of Information Transfer
12.  Other Sampling Distributions
13.  Nuisance Parameters as Safety Devices
14.  More General Properties
15.  Convolution of Gaussians
16.  The Central Limit Theorem
17.  Accuracy of Computations
18.  Galton’s Discovery
19.  Population Dynamics and Darwinian Evolution
20.  Evolution of Humming-Birds and Flowers
21.  Application to Economics
22.  The Great Inequality of Jupiter and Saturn
23.  Resolution of Distributions into Gaussians
24.  Hermite Polynomial Solutions
25.  Fourier Transform Relations
26.  There is Hope After All
27.  Comments

 Chapter 8 Sufficiency, Ancillarity, And All That

1.  Sufficiency
2.  Fisher Sufficiency
3.  Generalized Sufficiency
4.  Sufficiency Plus Nuisance Parameters
5.  The Likelihood Principle
6.  Ancillarity
7.  Generalized Ancillary Information
8.  Asymptotic Likelihood: Fisher Information
9.  Combining Evidence from Different Sources
10.  Pooling the Data
11.  Sam’s Broken Thermometer
12.  Comments

 Chapter 9 Repetitive Experiments Probability And Frequency

1.  Physical Experiments
2.  The Poorly Informed Robot
3.  Induction
4.  Are There General Inductive Rules?
5.  Multiplicity Factors
6.  Partition Function Algorithms
7.  Entropy Algorithms
8.  Another Way of Looking at it
9.  Entropy Maximization
10.  Probability and Frequency
11.  Significance Tests
12.  Comparison of Psi and Chi-Squared
13.  The Chi-Squared Test
14.  Generalization
15.  Halley’s Mortality Table
16.  Comments
17.  Superstitions

 Chapter 10 Physics Of “random Experiments”

1.  An Interesting Correlation
2.  Historical Background
3.  How to Cheat at Coin and Die Tossing
4.  Bridge Hands
5.  General Random Experiments
6.  Induction Revisited
7.  But What About Quantum Theory?
8.  Mechanics Under the Clouds
9.  More On Coins and Symmetry
10.  Independence of Tosses
11.  The Arrogance of the Uninformed

VOLUME II – ADVANCED APPLICATIONS

Chapter 11 Discrete Prior Probabilities The Entropy Principle

1.  New Kind of Prior Information
2.  Minimum ∑p i
3.  Entropy: Shannon’s Theorem
4.  The Wallis Derivation
5.  An Example
6.  Generalization: A More Rigorous Proof
7.  Formal Properties of Maximum-Entropy Distributions
8.  Conceptual Problems—Frequency Correspondence
9.  Comments

 Chapter 12 Ignorance Priors And Transformation Groups

1.  What Are We Trying to Do?
2.  IGNORANCE PRIORS
3.  Continuous Distributions
4.  TRANSFORMATION GROUPS
5.  Location and Scale Parameters
6. Poisson Rate
7.  Unknown Probability for Success
8.  Bertrand’s Problem
9.  Comments

 Chapter 13 Decision Theory Historical Background

1.  Inference vs. Decision
2.  Daniel Bernoulli’s Suggestion
3.  The Rationale of Insurance
4.  Entropy and Utility
5.  The Honest Weatherman
6.  Reactions to Daniel Bernoulli and Laplace
7.  Wald’s Decision Theory
8.  Parameter Estimation for Minimum Loss
9.  Reformulation of the Problem
10.  Effect of Varying Loss Functions
11.  General Decision Theory
12.  Comments
13.  Decision Theory is not Fundamental
14.  Another Dimension?

 Chapter 14 Simple Applications Of Decision Theory

1.  Definitions and Preliminaries
2.  Sufficiency and Information
3.  Loss Functions and Criteria of Optimum Performance
4. Discrete Example
5.  How Would Our Robot Do It?
6.  Historical Remarks
7.  The Widget Problem
8.  Comments

 Chapter 15 Paradoxes Of Probability Theory

1.  How do Paradoxes Survive and Grow?
2.  Summing a Series the Easy Way
3.  Nonconglomerability
4.  The Tumbling Tetrahedrons
5.  Solution for a Finite Number of Tosses
6.  Finite vs. Countable Additivity
7.  The Borel-Kolmogorov Paradox
8.  The Marginalization Paradox
9.  Discussion
10. a. Useful Result After All?
11.  How to Mass-Produce Paradoxes
12.  Comments
13.  Chapter Orthodox Methods: Historical Background
14.  The Early Problems
15.  Sociology of Orthodox Statistics
16.  Ronald Fisher, Harold Jeffreys, and Jerzy Neyman
17.  Pre-data and Post-data Considerations
18.  The Sampling Distribution for an Estimator
19.  Pro-Causal and Anti-Causal Bias
20.  What is Real; the Probability or the Phenomenon?
21.  Comments

 Chapter 16 Principles And Pathology Of Orthodox Statistics

1.  Information Loss
2.  Unbiased Estimators
3.  Pathology of an Unbiased Estimate
4.  The Fundamental Inequality of the Sampling Variance
5.  Periodicity: The Weather in Central Park
6. Bayesian Analysis:
7.  The Folly of Randomization
8.  Fisher: Common Sense at Rothamsted
9.  Missing Data
10.  Trend and Seasonality in Time Series
11.  The General Case
12.  Comments

 Chapter 17 The Ap Distribution And Rule Of Succession

1.  Memory Storage for Old Robots
2.  Relevance
3. Surprising Consequence
4.  Outer and Inner Robots
5.  An Application
6.  Laplace’s Rule of Succession
7.  Jeffreys’ Objection
8.  Bass or Carp?
9.  So where does this leave the rule?
10.  Generalization
11.  Confirmation and Weight of Evidence
12.  Carnap’s Inductive Methods
13.  Probability and Frequency in Exchangable Sequences
14.  Prediction of Frequencies
15.  One-Dimensional Neutron Multiplication
16.  The de Finette Theorem
17.  Comments

 Chapter 18 Physical Measurements

1.  Reduction of Equations of Condition
2.  Reformulation as a Decision Problem
3.  The Underdetermined Case: K is Singular
4.  The Overdetermined Case: K Can be Made Nonsingular
5.  Numerical Evaluation of the Result
6.  Accuracy of the Estimates
7.  Comments
8.  Chapter 19 Model Comparison
9.  Formulation of the Problem
10.  The Fair Judge and the Cruel Realist
11.  But Where is the Idea of Simplicity?
12.  An Example: Linear Response Models
13.  Comments
14.  Final Causes

 Chapter 20 Outliers And Robustness

1.  The Experimenter’s Dilemma
2.  Robustness
3.  The Two-Model Model
4.  Exchangeable Selection
5.  The General Bayesian Solution
6.  Pure Outliers
7.  One Receding Datum

 Chapter 21 Introduction To Communication Theory

1.  Origins of the Theory
2.  The Noiseless Channel
3.  The Information Source
4.  Does the English Language have Statistical Properties?
5.  Optimum Encoding: Letter Frequencies Known
6.  Better Encoding From Knowledge of Digram Frequencies
7.  Relation to a Stochastic Model
8.  The Noisy Channel
9.  Fixing a Noisy Channel
10.  References

 Appendix A Other Approaches To Probability Theory

1.  The Kolmogorov System of Probability
2.  The de Finetti System of Probability
3.  Comparative Probability
4.  Holdouts Against Universal Comparability
5.  Speculations About Lattice Theories

 Appendix B Mathematical Formalities And Style

1.  Notation and Logical Hierarchy
2.  Our “Cautious Approach” Policy
3.  Willy Feller on Measure Theory
4.  Kronecker vs. Weierstrasz
5.  What is a Legitimate Mathematical Function?
6.  Counting Infinite Sets?
7.  The Hausdorff Sphere Paradox and Mathematical Diseases
8.  What Am I Supposed to Publish?
9.  Mathematical Courtesy

 Appendix C Convolutions And Cumulants

1.  Relation of Cumulants and Moments
2.  Examples