sách giáo trình công nghệ thông tin pdf doc - sách tham khảo

Matrix Algebra

Theory, Computations, and Applications in Statistics

 James E. Gentle

Preface

I began this book as an update of Numerical Linear Algebra for Applications in Statistics, published by Springer in 1998. There was a modest amount of new material to add, but I also wanted to supply more of the reasoning behind the facts about vectors and matrices. I had used material from that text in some courses, and I had spent a considerable amount of class time proving assertions made but not proved in that book. As I embarked on this project, the character of the book began to change markedly. In the previous book, I apologized for spending 30 pages on the theory and basic facts of linear algebra before getting on to the main interest: numerical linear algebra. In the present book, discussion of those basic facts takes up over half of the book.

The orientation and perspective of this book remains numerical linear al - gebra for applications in statistics. Computational considerations inform the narrative. There is an emphasis on the areas of matrix analysis that are im - portant for statisticians, and the kinds of matrices encountered in statistical applications receive special attention.

This book is divided into three parts plus a set of appendices. The three parts correspond generally to the three areas of the book’s subtitle—theory, computations, and applications—although the parts are in a different order, and there is no firm separation of the topics.

Part I, consisting of Chapters 1 through 7, covers most of the material in linear algebra needed by statisticians. (The word “matrix” in the title of the present book may suggest a somewhat more limited domain than “linear algebra”; but I use the former term only because it seems to be more commonly used by statisticians and is used more or less synonymously with the latter term.) The first four chapters cover the basics of vectors and matrices, concen - trating on topics that are particularly relevant for statistical applications. In Chapter 4, it is assumed that the reader is generally familiar with the basics of partial differentiation of scalar functions. Chapters 5 through 7 begin to take on more of an applications flavor, as well as beginning to give more consid - eration to computational methods. Although the details of the computations are not covered in those chapters, the topics addressed are oriented more to - ward computational algorithms. Chapter 5 covers methods for decomposing matrices into useful factors.

Chapter 6 addresses applications of matrices in setting up and solving linear systems, including overdetermined systems. We should not confuse sta - tistical inference with fitting equations to data, although the latter task is a component of the former activity. In Chapter 6, we address the more me - chanical aspects of the problem of fitting equations to data. Applications in statistical data analysis are discussed in Chapter 9. In those applications, we need to make statements (that is, assumptions) about relevant probability distributions.

Chapter 7 discusses methods for extracting eigenvalues and eigenvectors.

There are many important details of algorithms for eigenanalysis, but they are beyond the scope of this book. As with other chapters in Part I, Chap - ter 7 makes some reference to statistical applications, but it focuses on the mathematical and mechanical aspects of the problem.

Although the first part is on “theory”, the presentation is informal; neither definitions nor facts are highlighted by such words as “Definition”, “Theorem”, “Lemma”, and so forth. It is assumed that the reader follows the natural development. Most of the facts have simple proofs, and most proofs are given naturally in the text. No “Proof” and “Q. E. D.” or appear to indicate beginning and end; again, it is assumed that the reader is engaged in the development. For example, on page 270:

If A is nonsingular and symmetric, then A−1 is also symmetric because (A−1) T = (AT) −1 = A−1.

The first part of that sentence could have been stated as a theorem and given a number, and the last part of the sentence could have been introduced as the proof, with reference to some previous theorem that the inverse and transposition operations can be interchanged. (This had already been shown before page 270—in an unnumbered theorem of course!) None of the proofs are original (at least, I don’t think they are), but in most cases I do not know the original source, or even the source where I first saw them. I would guess that many go back to C. F. Gauss. Most, whether they are as old as Gauss or not, have appeared somewhere in the work of C. R. Rao.

Some lengthier proofs are only given in outline, but references are given for the details. Very useful sources of details of the proofs are Harville (1997), especially for facts relating to applications in linear models, and Horn and Johnson (1991) for more general topics, especially those relating to stochastic matrices. The older books by Gantmacher (1959) provide extensive coverage and often rather novel proofs. These two volumes have been brought back into print by the American Mathematical Society.

I also sometimes make simple assumptions without stating them explicitly.

For example, I may write “for all i” when i is used as an index to a vector.

I hope it is clear that “for all i” means only “for i that correspond to indices of the vector”. Also, my use of an expression generally implies existence. For example, if “AB” is used to represent a matrix product, it implies that “A and B are conformable for the multiplication AB”. Occasionally I remind the reader that I am taking such shortcuts.

The material in Part I, as in the entire book, was built up recursively. In the first pass, I began with some definitions and followed those with some facts that are useful in applications. In the second pass, I went back and added definitions and additional facts that lead to the results stated in the first pass. The supporting material was added as close to the point where it was needed as practical and as necessary to form a logical flow. Facts motivated by additional applications were also included in the second pass. In subsequent passes, I continued to add supporting material as necessary and to address the linear algebra for additional areas of application. I sought a bare - bones presentation that gets across what I considered to be the theory necessary for most applications in the data sciences. The material chosen for inclusion is motivated by applications.

Throughout the book, some attention is given to numerical methods for computing the various quantities discussed. This is in keeping with my be - lief that statistical computing should be dispersed throughout the statistics curriculum and statistical literature generally. Thus, unlike in other books on matrix “theory”, I describe the “modified” Gram - Schmidt method, rather than just the “classical” GS. (I put “modified” and “classical” in quotes be - cause, to me, GS is MGS. History is interesting, but in computational matters, I do not care to dwell on the methods of the past.) Also, condition numbers of matrices are introduced in the “theory” part of the book, rather than just in the “computational” part. Condition numbers also relate to fundamental properties of the model and the data.

The difference between an expression and a computing method is em - phasized. For example, often we may write the solution to the linear system Ax = b as A−1b. Although this is the solution (so long as A is square and of full rank), solving the linear system does not involve computing A−1. Wemay write A−1b, but we know we can compute the solution without inverting the matrix.

“This is an instance of a principle that we will encounter repeatedly:

the form of a mathematical expression and the way the expression should be evaluated in actual practice may be quite different.” (The statement in quotes appears word for word in several places in the book.) Standard textbooks on “matrices for statistical applications” emphasize their uses in the analysis of traditional linear models. This is a large and im - portant field in which real matrices are of interest, and the important kinds of real matrices include symmetric, positive definite, projection, and generalized inverse matrices. This area of application also motivates much of the discussion in this book. In other areas of statistics, however, there are different matrices of interest, including similarity and dissimilarity matrices, stochastic matrices, rotation matrices, and matrices arising from graph - theoretic approaches to data analysis. These matrices have applications in clustering, data mining, stochastic processes, and graphics; therefore, I describe these matrices and their special properties. I also discuss the geometry of matrix algebra. This provides a better intuition of the operations. Homogeneous coordinates and special operations in IR3 are covered because of their geometrical applications in statistical graphics.

Part II addresses selected applications in data analysis. Applications are referred to frequently in Part I, and of course, the choice of topics for coverage was motivated by applications. The difference in Part II is in its orientation.

Only “selected” applications in data analysis are addressed; there are ap - plications of matrix algebra in almost all areas of statistics, including the theory of estimation, which is touched upon in Chapter 4 of Part I. Certain types of matrices are more common in statistics, and Chapter 8 discusses in more detail some of the important types of matrices that arise in data analy - sis and statistical modeling. Chapter 9 addresses selected applications in data analysis. The material of Chapter 9 has no obvious definition that could be covered in a single chapter (or a single part, or even a single book), so I have chosen to discuss briefly a wide range of areas. Most of the sections and even subsections of Chapter 9 are on topics to which entire books are devoted;

however, I do not believe that any single book addresses all of them.

Part III covers some of the important details of numerical computations, with an emphasis on those for linear algebra. I believe these topics constitute the most important material for an introductory course in numerical analysis for statisticians and should be covered in every such course.

Except for specific computational techniques for optimization, random number generation, and perhaps symbolic computation, Part III provides the basic material for a course in statistical computing. All statisticians should have a passing familiarity with the principles.

Chapter 10 provides some basic information on how data are stored and manipulated in a computer. Some of this material is rather tedious, but it is important to have a general understanding of computer arithmetic before considering computations for linear algebra. Some readers may skip or just skim Chapter 10, but the reader should be aware that the way the computer stores numbers and performs computations has far - reaching consequences.

Computer arithmetic differs from ordinary arithmetic in many ways; for ex - ample, computer arithmetic lacks associativity of addition and multiplication, and series often converge even when they are not supposed to. (On the com - puter, a straightforward evaluation of ∞ x=1 x converges!) I emphasize the differences between the abstract number system IR, called the reals, and the computer number system IF, the floating - point numbers unfortunately also often called “real”. Table 10.3 on page 400 summarizes some of these differences. All statisticians should be aware of the effects of these differences. I also discuss the differences between Z Z, the abstract number system called the integers, and the computer number system II, the fixed - point numbers. (Appendix A provides definitions for this and other notation that I use.) Chapter 10 also covers some of the fundamentals of algorithms, such as iterations, recursion, and convergence. It also discusses software development.

Software issues are revisited in Chapter 12.

While Chapter 10 deals with general issues in numerical analysis, Chap - ter 11 addresses specific issues in numerical methods for computations in linear algebra.

Chapter 12 provides a brief introduction to software available for com - putations with linear systems. Some specific systems mentioned include the IMSLTM libraries for Fortran and C, Octave or MATLABR [1] (or MatlabR ), and R or S - PLUSR (or S - Plus R ). All of these systems are easy to use, and the best way to learn them is to begin using them for simple problems. I do not use any particular software system in the book, but in some exercises, and particularly in Part III, I do assume the ability to program in either Fortran or C and the availability of either R or S - Plus, Octave or Matlab, and Maple R or MathematicaR. My own preferences for software systems are Fortran and R, and occasionally these preferences manifest themselves in the text.

Appendix A collects the notation used in this book. It is generally “stan - dard” notation, but one thing the reader must become accustomed to is the lack of notational distinction between a vector and a scalar. All vectors are “column” vectors, although I usually write them as horizontal lists of their elements. (Whether vectors are “row” vectors or “column” vectors is generally only relevant for how we write expressions involving vector/matrix multipli - cation or partitions of matrices.) I write algorithms in various ways, sometimes in a form that looks similar to Fortran or C and sometimes as a list of numbered steps. I believe all of the descriptions used are straightforward and unambiguous.

This book could serve as a basic reference either for courses in statistical computing or for courses in linear models or multivariate analysis. When the book is used as a reference, rather than looking for “Definition” or “Theorem”, the user should look for items set off with bullets or look for numbered equa - tions, or else should use the Index, beginning on page 519, or Appendix A, beginning on page 479.

The prerequisites for this text are minimal. Obviously some background in mathematics is necessary. Some background in statistics or data analysis and some level of scientific computer literacy are also required. References to rather advanced mathematical topics are made in a number of places in the text. To some extent this is because many sections evolved from class notes that I developed for various courses that I have taught. All of these courses were at the graduate level in the computational and statistical sciences, but they have had wide ranges in mathematical level. I have carefully reread the sections that refer to groups, fields, measure theory, and so on, and am convinced that if the reader does not know much about these topics, the material is still understandable, but if the reader is familiar with these topics, the references add to that reader’s appreciation of the material. In many places, I refer to computer programming, and some of the exercises require some programming.

a. careful coverage of Part III requires background in numerical programming.

In regard to the use of the book as a text, most of the book evolved in one way or another for my own use in the classroom. I must quickly admit, how - ever, that I have never used this whole book as a text for any single course. I have used Part III in the form of printed notes as the primary text for a course in the “foundations of computational science” taken by graduate students in the natural sciences (including a few statistics students, but dominated by physics students). I have provided several sections from Parts I and II in online PDF files as supplementary material for a two - semester course in mathemati - cal statistics at the “baby measure theory” level (using Shao,2003). Likewise, for my courses in computational statistics and statistical visualization, I have provided many sections, either as supplementary material or as the primary text, in online PDF files or printed notes. I have not taught a regular “applied statistics” course in almost 30 years, but if I did, I am sure that I would draw heavily from Parts I and II for courses in regression or multivariate analysis.

If I ever taught a course in “matrices for statistics” (I don’t even know if such courses exist), this book would be my primary text because I think it covers most of the things statisticians need to know about matrix theory and computations.

Some exercises are Monte Carlo studies. I do not discuss Monte Carlo methods in this text, so the reader lacking background in that area may need to consult another reference in order to work those exercises. The exercises should be considered an integral part of the book. For some exercises, the required software can be obtained from either statlib or netlib (see the bibliography). Exercises in any of the chapters, not just in Part III, may require computations or computer programming.

Penultimately, I must make some statement about the relationship of this book to some other books on similar topics. Much important statisti - cal theory and many methods make use of matrix theory, and many sta - tisticians have contributed to the advancement of matrix theory from its very early days. Widely used books with derivatives of the words “statistics” and “matrices/linear - algebra” in their titles include Basilevsky (1983), Graybill (1983), Harville (1997), Schott (2004), and Searle (1982). All of these are useful books. The computational orientation of this book is probably the main difference between it and these other books. Also, some of these other books only address topics of use in linear models, whereas this book also dis - cusses matrices useful in graph theory, stochastic processes, and other areas of application. (If the applications are only in linear models, most matrices of interest are symmetric, and all eigenvalues can be considered to be real.) Other differences among all of these books, of course, involve the authors’ choices of secondary topics and the ordering of the presentation.

I thank John Kimmel of Springer for his encouragement and advice on this book and other books on which he has worked with me. I especially thank Ken Berk for his extensive and insightful comments on a draft of this book.

I thank my student Li Li for reading through various drafts of some of the chapters and pointing out typos or making helpful suggestions. I thank the anonymous reviewers of this edition for their comments and suggestions. I also thank the many readers of my previous book on numerical linear algebra who informed me of errors and who otherwise provided comments or suggestions for improving the exposition. Whatever strengths this book may have can be attributed in large part to these people, named or otherwise. The weaknesses can only be attributed to my own ignorance or hardheadedness.

I thank my wife, Marıa, to whom this book is dedicated, for everything.

IusedTEXviaL A TEX2ε to write the book. I did all of the typing, program - ming, etc. myself, so all misteaks are mine. I would appreciate receiving suggestions for improvement and notification of errors. Notes on this book, including errata, are available at.

 Conetent:

  Part I: Linear Algebra

1 Basic Vector/Matrix Structure and Notation - Page: 3

1.1 Vectors –Page:  4

1.2 Arrays - Page: 5

1.3 Matrices - Page: 5

1.4 Representation of Data - Page: 7

2 Vectors and Vector Spaces - Page: 9

2.1 Operations on Vectors - Page: 9

2.1.1 Linear Combinations and Linear Independence - Page: 10

2.1.2 Vector Spaces and Spaces of Vectors - Page: 11

2.1.3 Basis Sets - Page: 14

2.1.4 Inner Products - Page: 15

2.1.5 Norms - Page: 16

2.1.6 Normalized Vectors - Page: 21

2.1.7 Metrics and Distances - Page: 22

2.1.8 Orthogonal Vectors and Orthogonal Vector Spaces - Page: 22

2.1.9 The “One Vector” - Page: 23

2.2 Cartesian Coordinates and Geometrical Properties of Vectors - Page: 24

2.2.1 Cartesian Geometry - Page: 25

2.2.2 Projections - Page: 25

2.2.3 Angles between Vectors - Page: 26

2.2.4 Orthogonalization Transformations - Page: 27

2.2.5 Orthonormal Basis Sets - Page: 29

2.2.6 Approximation of Vectors - Page: 30

2.2.7 Flats, Affine Spaces, and Hyperplanes - Page: 31

2.2.8 Cones - Page: 32

2.2.9 Cross Products in IR3 - Page: 33

2.3 Centered Vectors and Variances and Covariances of Vectors - Page: 33

2.3.1 The Mean and Centered Vectors - Page: 34

2.3.2 The Standard Deviation, the Variance, andScaledVectors - Page: 35

2.3.3 Covariances and Correlations between Vectors - Page: 36

Exercises - Page: 37

3 Basic Properties of Matrices - Page: 41

3.1 Basic Definitions and Notation - Page: 41

3.1.1 Matrix Shaping Operators - Page: 44

3.1.2 Partitioned Matrices - Page: 46

3.1.3 Matrix Addition – Page:  47

3.1.4 Scalar-Valued Operators on Square Matrices:  TheTrace - Page: 49

3.1.5 Scalar-Valued Operators on Square Matrices:  TheDeterminant – Page:  50

3.2 Multiplication of Matrices and Multiplication of VectorsandMatrices - Page: 59

3.2.1 Matrix Multiplication (Cayley) - Page: 59

3.2.2 Multiplication of Partitioned Matrices - Page: 61

3.2.3 Elementary Operations on Matrices - Page: 61

3.2.4 Traces and Determinants of Square Cayley Products - Page: 67

3.2.5 Multiplication of Matrices and Vectors - Page: 68

3.2.6 Outer Products - Page: 69

3.2.7 Bilinear and Quadratic Forms; Definiteness - Page: 69

3.2.8 Anisometric Spaces - Page: 71

3.2.9 Other Kinds of Matrix Multiplication - Page: 72

3.3 Matrix Rank and the Inverse of a Full Rank Matrix - Page: 76

3.3.1 The Rank of Partitioned Matrices, Products

ofMatrices,andSumsofMatrices - Page: 78

3.3.2 Full Rank Partitioning - Page: 80

3.3.3 Full Rank Matrices and Matrix Inverses - Page: 81

3.3.4 Full Rank Factorization - Page: 85

3.3.5 Equivalent Matrices - Page: 86

3.3.6 Multiplication by Full Rank Matrices - Page: 88

3.3.7 Products of the Form ATA - Page: 90

3.3.8 A Lower Bound on the Rank of a Matrix Product - Page: 92

3.3.9 Determinants of Inverses - Page: 92

3.3.10 Inverses of Products and Sums of Matrices - Page: 93

3.3.11 Inverses of Matrices with Special Forms - Page: 94

3.3.12 Determining the Rank of a Matrix - Page: 94

3.4 More on Partitioned Square Matrices:  The Schur Complement 95

3.4.1 Inverses of Partitioned Matrices - Page: 95

3.4.2 Determinants of Partitioned Matrices - Page: 96

3.5 Linear Systems of Equations - Page: 96

3.5.1 Solutions of Linear Systems - Page: 97

3.5.2 Null Space:  The Orthogonal Complement - Page: 99

3.6 Generalized Inverses - Page: 100

3.6.1 Generalized Inverses of Sums of Matrices - Page: 101

3.6.2 Generalized Inverses of Partitioned Matrices - Page: 101

3.6.3 Pseudoinverse or Moore-Penrose Inverse - Page: 101

3.7 Orthogonality - Page: 103

3.8 Eigenanalysis; Canonical Factorizations - Page: 105

3.8.1 Basic Properties of Eigenvalues and Eigenvectors - Page: 107

3.8.2 The Characteristic Polynomial - Page: 108

3.8.3 The Spectrum - Page: 110

3.8.4 Similarity Transformations - Page: 114

3.8.5 Similar Canonical Factorization;

Diagonalizable Matrices - Page: 116

3.8.6 Properties of Diagonalizable Matrices - Page: 118

3.8.7 Eigenanalysis of Symmetric Matrices - Page: 119

3.8.8 Positive Definite and Nonnegative Definite Matrices - Page: 124

3.8.9 The Generalized Eigenvalue Problem - Page: 126

3.8.10 Singular Values and the Singular Value Decomposition - Page: 127

3.9 MatrixNorms.128

3.9.1 Matrix Norms Induced from Vector Norms - Page: 129

3.9.2 The Frobenius Norm — The “Usual” Norm - Page: 131

3.9.3 Matrix Norm Inequalities - Page: 133

3.9.4 The Spectral Radius - Page: 134

3.9.5 Convergence of a Matrix Power Series - Page: 134

3.10 Approximation of Matrices - Page: 137

Exercises - Page: 140

 Vector/Matrix Derivatives and Integrals - Page: 145

4.1 Basics of Differentiation - Page: 145

4.2 Types of Differentiation - Page: 149

4.2.1 Differentiation with Respect to a Scalar - Page: 149

4.2.2 Differentiation with Respect to a Vector - Page: 150

4.2.3 Differentiation with Respect to a Matrix - Page: 154

4.3 Optimization of Functions - Page: 156

4.3.1 Stationary Points of Functions - Page: 156

4.3.2 Newton’s Method - Page: 156

4.3.3 Optimization of Functions with Restrictions - Page: 159

4.4 Multiparameter Likelihood Functions - Page: 163

4.5 Integration and Expectation - Page: 164

4.5.1 Multidimensional Integrals and Integrals Involving

VectorsandMatrices 165

4.5.2 Integration Combined with Other Operations - Page: 166

4.5.3 Random Variables - Page: 167

Exercises - Page: 169

5 Matrix Transformations and Factorizations 173

5.1 Transformations by Orthogonal Matrices - Page: 174

5.2 Geometric Transformations - Page: 175

5.2.1 Rotations - Page: 176

5.2.2 Reflections - Page: 178

5.2.3 Translations; Homogeneous Coordinates - Page: 178

5.3 Householder Transformations (Reflections) - Page: 180

5.4 Givens Transformations (Rotations) - Page: 182

5.5 Factorization of Matrices - Page: 185

5.6 LU and LDU Factorizations - Page: 186

5.7 QR Factorization - Page: 188

5.7.1 Householder Reflections to Form the QR Factorization - Page: 190

5.7.2 Givens Rotations to Form the QR Factorization - Page: 192

5.7.3 Gram-Schmidt Transformations to Form the QR Factorization - Page: 192

5.8 Singular Value Factorization - Page: 192

5.9 Factorizations of Nonnegative Definite Matrices - Page: 193

5.9.1 Square Roots - Page: 193

5.9.2 Cholesky Factorization - Page: 194

5.9.3 Factorizations of a Gramian Matrix - Page: 196

5.10 Incomplete Factorizations - Page: 197

Exercises - Page: 198

6 Solution of Linear Systems 201

6.1 Condition of Matrices - Page: 201

6.2 DirectMethods forConsistentSystems.206

6.2.1 Gaussian Elimination and Matrix Factorizations - Page: 207

6.2.2 Choice of Direct Method - Page: 211

6.3 Iterative Methods for Consistent Systems - Page: 211

6.3.1 The Gauss-Seidel Method with Successive Overrelaxation - Page: 212

6.3.2 Conjugate Gradient Methods for Symmetric Positive Definite Systems - Page: 213

6.3.3 Multigrid Methods - Page: 217

6.4 Numerical Accuracy - Page: 218

6.5 Iterative Refinement - Page: 219

6.6 Updating a Solution to a Consistent System - Page: 220

6.7 OverdeterminedSystems;LeastSquares222

6.7.1 Least Squares Solution of an Overdetermined System - Page: 224

6.7.2 Least Squares with a Full Rank Coefficient Matrix - Page: 226

6.7.3 Least Squares with a Coefficient Matrix NotofFullRank 227

6.7.4 Updating a Least Squares Solution ofanOverdeterminedSystem.228

6.8 Other Solutions of Overdetermined Systems - Page: 229

6.8.1 Solutions that Minimize Other Norms of the Residuals - Page: 230

6.8.2 Regularized Solutions - Page: 233

6.8.3 Minimizing Orthogonal Distances - Page: 234

Exercises - Page: 238

7 Evaluation of Eigenvalues and Eigenvectors 241

7.1 General Computational Methods - Page: 242

7.1.1 Eigenvalues from Eigenvectors and Vice Versa - Page: 242

7.1.2 Deflation - Page: 243

7.1.3 Preconditioning - Page: 244

7.2 PowerMethod 245

7.3 JacobiMethod247

7.4 QR Method 250

7.5 KrylovMethods.252

7.6 Generalized Eigenvalues - Page: 252

7.7 Singular Value Decomposition - Page: 253

Exercises - Page: 256

 Part II Applications in Data Analysis

8 Special Matrices and Operations Useful in Modeling and Data Analysis - Page: 261

8.1 Data Matrices and Association Matrices - Page: 261

8.1.1 Flat Files - Page: 262

8.1.2 Graphs and Other Data Structures - Page: 262

8.1.3 Probability Distribution Models - Page: 269

8.1.4 Association Matrices - Page: 269

8.2 SymmetricMatrices - Page: 270

8.3 Nonnegative Definite Matrices; Cholesky Factorization - Page: 275

8.4 Positive Definite Matrices - Page: 277

8.5 Idempotent and Projection Matrices - Page: 280

8.5.1 Idempotent Matrices - Page: 281

8.5.2 Projection Matrices:  Symmetric Idempotent Matrices - Page: 286

8.6 Special Matrices Occurring in Data Analysis - Page: 287

8.6.1 Gramian Matrices - Page: 288

8.6.2 Projection and Smoothing Matrices - Page: 290

8.6.3 Centered Matrices and Variance-Covariance Matrices - Page: 293

8.6.4 The Generalized Variance - Page: 296

8.6.5 Similarity Matrices - Page: 298

8.6.6 Dissimilarity Matrices - Page: 299

8.7 Nonnegative and Positive Matrices - Page: 299

8.7.1 Properties of Square Positive Matrices - Page: 301

8.7.2 Irreducible Square Nonnegative Matrices - Page: 302

8.7.3 Stochastic Matrices - Page: 306

8.7.4 Leslie Matrices - Page: 307

8.8 Other Matrices with Special Structures - Page: 307

8.8.1 Helmert Matrices - Page: 308

8.8.2 Vandermonde Matrices - Page: 309

8.8.3 Hadamard Matrices and Orthogonal Arrays - Page: 310

8.8.4 Toeplitz Matrices - Page: 311

8.8.5 Hankel Matrices - Page: 312

8.8.6 Cauchy Matrices - Page: 313

8.8.7 Matrices Useful in Graph Theory - Page: 313

8.8.8 M-Matrices.317

Exercises - Page: 317

9 Selected Applications in Statistics - Page: 321

9.1 Multivariate Probability Distributions - Page: 322

9.1.1 Basic Definitions and Properties - Page: 322

9.1.2 The Multivariate Normal Distribution 323

9.1.3 Derived Distributions and Cochran’s Theorem - Page: 323

9.2 LinearModels.325

9.2.1 Fitting the Model - Page: 327

9.2.2 Linear Models and Least Squares - Page: 330

9.2.3 Statistical Inference - Page: 332

9.2.4 The Normal Equations and the Sweep Operator - Page: 335

9.2.5 Linear Least Squares Subject to Linear Equality Constraints - Page: 337

9.2.6 Weighted Least Squares - Page: 337

9.2.7 Updating Linear Regression Statistics - Page: 338

9.2.8 Linear Smoothing - Page: 341

9.3 Principal Components - Page: 341

9.3.1 Principal Components of a Random Vector - Page: 342

9.3.2 Principal Components of Data - Page: 343

9.4 Condition of Models and Data - Page: 346

9.4.1 Ill-Conditioning in Statistical Applications - Page: 346

9.4.2 Variable Selection - Page: 347

9.4.3 Principal Components Regression - Page: 348

9.4.4 Shrinkage Estimation - Page: 348

9.4.5 Testing the Rank of a Matrix - Page: 350

9.4.6 Incomplete Data - Page: 352

9.5 Optimal Design - Page: 355

9.6 Multivariate Random Number Generation - Page: 358

9.7 StochasticProcesses - Page: 360

9.7.1 Markov Chains - Page: 360

9.7.2 Markovian Population Models - Page: 362

9.7.3 Autoregressive Processes - Page: 364

Exercises - Page: 365

 Part III Numerical Methods and Software

10 Numerical Methods 375

10.1 Digital Representation of Numeric Data - Page: 377

10.1.1 The Fixed-Point Number System - Page: 378

10.1.2 The Floating-Point Model for Real Numbers - Page: 379

10.1.3 Language Constructs for Representing Numeric Data - Page: 386

10.1.4 Other Variations in the Representation of Data; Portability of Data - Page: 391

10.2 Computer Operations on Numeric Data - Page: 393

10.2.1 Fixed-Point Operations - Page: 394

10.2.2 Floating-Point Operations - Page: 395

10.2.3 Exact Computations; Rational Fractions - Page: 399

10.2.4 Language Constructs for Operations onNumericData- Page: 401

10.3 Numerical Algorithms and Analysis - Page: 403

10.3.1 Error in Numerical Computations - Page: 404

10.3.2 Efficiency - Page: 412

10.3.3 Iterations and Convergence - Page: 417

10.3.4 Other Computational Techniques - Page: 419

Exercises - Page: 422

11 Numerical Linear Algebra - Page: 429

11.1 Computer Representation of Vectors and Matrices - Page: 429

11.2 General Computational Considerations forVectorsandMatrices - Page: 431

11.2.1 Relative Magnitudes of Operands - Page: 431

11.2.2 Iterative Methods - Page: 433

11.2.3 Assessing Computational Errors - Page: 434

11.3 Multiplication of Vectors and Matrices - Page: 435

11.4 Other Matrix Computations - Page: 439

Exercises - Page: 441

12 Software for Numerical Linear Algebra - Page: 445

12.1 Fortran and C - Page: 447

12.1.1 Programming Considerations - Page: 448

12.1.2 Fortran 95 - Page: 452

12.1.3 Matrix and Vector Classes in C++ - Page: 453

12.1.4 Libraries - Page: 454

12.1.5 The IMSLTM Libraries.457

12.1.6 Libraries for Parallel Processing - Page: 460

12.2 Interactive Systems for Array Manipulation - Page: 461

12.2.1 MATLABR - Page:  andOctave463

12.2.2 R and S-PLUSR - Page:  466

12.3 High-Performance Software - Page: 470

12.4 Software for Statistical Applications - Page: 472

12.5 Test Data - Page: 472

Exercises - Page: 475

A Notation and Definitions - Page: 479

A.1 General Notation - Page: 479

A.2 ComputerNumberSystems- Page: 481

A.3 General Mathematical Functions and Operators - Page: 482

A.4 LinearSpacesandMatrices- Page: 484

A.5 ModelsandData- Page: 490

B Solutions and Hints for Selected Exercises - Page: 493

Bibliography - Page: 505

Index  - Page: 519