# Probability, Random Variables, Statistics, and Random Processes

## Fundamentals & Applications

1. Edition June 2019

416 Pages, Hardcover*Textbook*

**978-1-119-30081-6**

Probability, Random Variables, Statistics, and Random Processes: Fundamentals & Applications is a comprehensive undergraduate-level textbook. With its excellent topical coverage, the focus of this book is on the basic principles and practical applications of the fundamental concepts that are extensively used in various Engineering disciplines as well as in a variety of programs in Life and Social Sciences. The text provides students with the requisite building blocks of knowledge they require to understand and progress in their areas of interest. With a simple, clear-cut style of writing, the intuitive explanations, insightful examples, and practical applications are the hallmarks of this book.

The text consists of twelve chapters divided into four parts. Part-I, Probability (Chapters 1 - 3), lays a solid groundwork for probability theory, and introduces applications in counting, gambling, reliability, and security. Part-II, Random Variables (Chapters 4 - 7), discusses in detail multiple random variables, along with a multitude of frequently-encountered probability distributions. Part-III, Statistics (Chapters 8 - 10), highlights estimation and hypothesis testing. Part-IV, Random Processes (Chapters 11 - 12), delves into the characterization and processing of random processes. Other notable features include:

* Most of the text assumes no knowledge of subject matter past first year calculus and linear algebra

* With its independent chapter structure and rich choice of topics, a variety of syllabi for different courses at the junior, senior, and graduate levels can be supported

* A supplemental website includes solutions to about 250 practice problems, lecture slides, and figures and tables from the text

Given its engaging tone, grounded approach, methodically-paced flow, thorough coverage, and flexible structure, Probability, Random Variables, Statistics, and Random Processes: Fundamentals & Applications clearly serves as a must textbook for courses not only in Electrical Engineering, but also in Computer Engineering, Software Engineering, and Computer Science.

Acknowledgments xv

About the Companion Website xvii

Part I Probability 1

1 Basic Concepts of Probability Theory 3

1.1 Statistical Regularity and Relative Frequency 3

1.2 Set Theory and Its Applications to Probability 5

1.3 The Axioms and Corollaries of Probability 12

1.4 Joint Probability and Conditional Probability 18

1.5 Statistically Independent Events and Mutually Exclusive Events 21

1.6 Law of Total Probability and Bayes' Theorem 28

1.7 Summary 32

Problems 32

2 Applications in Probability 37

2.1 Odds and Risk 37

2.2 Gambler's Ruin Problem 41

2.3 Systems Reliability 43

2.4 Medical Diagnostic Testing 47

2.5 Bayesian Spam Filtering 50

2.6 Monty Hall Problem 51

2.7 Digital Transmission Error 54

2.8 How to Make the Best Choice Problem 56

2.9 The Viterbi Algorithm 59

2.10 All Eggs in One Basket 61

2.11 Summary 63

Problems 63

3 Counting Methods and Applications 67

3.1 Basic Rules of Counting 67

3.2 Permutations and Combinations 72

3.2.1 Permutations without Replacement 73

3.2.2 Combinations without Replacement 73

3.2.3 Permutations with Replacement 74

3.2.4 Combinations with Replacement 74

3.3 Multinomial Counting 77

3.4 Special Arrangements and Selections 79

3.5 Applications 81

3.5.1 Game of Poker 81

3.5.2 Birthday Paradox 83

3.5.3 Quality Control 86

3.5.4 Best-of-Seven Championship Series 86

3.5.5 Lottery 89

3.6 Summary 90

Problems 90

Part II Random Variables 95

4 One Random Variable: Fundamentals 97

4.1 Types of Random Variables 97

4.2 The Cumulative Distribution Function 99

4.3 The Probability Mass Function 102

4.4 The Probability Density Function 104

4.5 Expected Values 107

4.5.1 Mean of a Random Variable 107

4.5.2 Variance of a Random Variable 110

4.5.3 Moments of a Random Variable 113

4.5.4 Mode and Median of a Random Variable 114

4.6 Conditional Distributions 116

4.7 Functions of a Random Variable 120

4.7.1 pdf of a Function of a Continuous Random Variable 121

4.7.2 pmf of a Function of a Discrete Random Variable 123

4.7.3 Computer Generation of Random Variables 124

4.8 Transform Methods 125

4.8.1 Moment Generating Function of a Random Variable 125

4.8.2 Characteristic Function of a Random Variable 126

4.9 Upper Bounds on Probability 127

4.9.1 Markov Bound 127

4.9.2 Chebyshev Bound 128

4.9.3 Chernoff Bound 128

4.10 Summary 131

Problems 131

5 Special Probability Distributions and Applications 137

5.1 Special Discrete Random Variables 137

5.1.1 The Bernoulli Distribution 137

5.1.2 The Binomial Distribution 138

5.1.3 The Geometric Distribution 140

5.1.4 The Pascal Distribution 142

5.1.5 The Hypergeometric Distribution 143

5.1.6 The Poisson Distribution 144

5.1.7 The Discrete Uniform Distribution 146

5.1.8 The Zipf (Zeta) Distribution 147

5.2 Special Continuous Random Variables 148

5.2.1 The Continuous Uniform Distribution 148

5.2.2 The Exponential Distribution 149

5.2.3 The Gamma Distribution 151

5.2.4 The Erlang Distribution 152

5.2.5 The Weibull Distribution 152

5.2.6 The Beta Distribution 153

5.2.7 The Laplace Distribution 154

5.2.8 The Pareto Distribution 155

5.3 Applications 156

5.3.1 Digital Transmission: Regenerative Repeaters 156

5.3.2 System Reliability: Failure Rate 157

5.3.3 Queuing Theory: Servicing Customers 158

5.3.4 Random Access: Slotted ALOHA 159

5.3.5 Analog-to-Digital Conversion: Quantization 160

5.4 Summary 161

Problems 162

6 Multiple Random Variables 165

6.1 Pairs of Random Variables 165

6.2 The Joint Cumulative Distribution Function of Two Random Variables 167

6.2.1 Marginal Cumulative Distribution Function 169

6.3 The Joint Probability Mass Function of Two Random Variables 170

6.3.1 Marginal Probability Mass Function 170

6.4 The Joint Probability Density Function of Two Random Variables 171

6.4.1 Marginal Probability Density Function 172

6.5 Expected Values of Functions of Two Random Variables 173

6.5.1 Joint Moments 174

6.6 Independence of Two Random Variables 175

6.7 Correlation between Two Random Variables 178

6.8 Conditional Distributions 185

6.8.1 Conditional Expectations 186

6.9 Distributions of Functions of Two Random Variables 188

6.9.1 Joint Distribution of Two Functions of Two Random Variables 191

6.10 Random Vectors 192

6.11 Summary 197

Problems 198

7 The Gaussian Distribution 201

7.1 The Gaussian Random Variable 201

7.2 The Standard Gaussian Distribution 204

7.3 Bivariate Gaussian Random Variables 210

7.3.1 Linear Transformations of Bivariate Gaussian Random Variables 213

7.4 Jointly Gaussian Random Vectors 215

7.5 Sums of Random Variables 217

7.5.1 Mean and Variance of Sum of Random Variables 217

7.5.2 Mean and Variance of Sum of Independent, Identically Distributed Random Variables 218

7.5.3 Distribution of Sum of Independent Random Variables 218

7.5.4 Sum of a Random Number of Independent, Identically Distributed Random Variables 219

7.6 The Sample Mean 220

7.6.1 Laws of Large Numbers 222

7.7 Approximating Distributions with the Gaussian Distribution 223

7.7.1 Relation between the Gaussian and Binomial Distributions 223

7.7.2 Relation between the Gaussian and Poisson Distributions 225

7.7.3 The Central Limit Theorem 226

7.8 Probability Distributions Related to the Gaussian Distribution 230

7.8.1 The Rayleigh Distribution 230

7.8.2 The Ricean Distribution 231

7.8.3 The Log-Normal Distribution 231

7.8.4 The Chi-Square Distribution 232

7.8.5 The Maxwell-Boltzmann Distribution 232

7.8.6 The Student's t-Distribution 233

7.8.7 The F Distribution 234

7.8.8 The Cauchy Distribution 234

7.9 Summary 234

Problems 235

Part III Statistics 239

8 Descriptive Statistics 241

8.1 Overview of Statistics 241

8.2 Data Displays 244

8.3 Measures of Location 249

8.4 Measures of Dispersion 250

8.5 Measures of Shape 255

8.6 Summary 257

Problems 257

9 Estimation 259

9.1 Parameter Estimation 259

9.2 Properties of Point Estimators 260

9.3 Maximum Likelihood Estimators 265

9.4 Bayesian Estimators 270

9.5 Confidence Intervals 272

9.6 Estimation of a Random Variable 274

9.7 Maximum a Posteriori Probability Estimation 275

9.8 Minimum Mean Square Error Estimation 277

9.9 Linear Minimum Mean Square Error Estimation 279

9.10 Linear MMSE Estimation Using a Vector of Observations 282

9.11 Summary 285

Problems 285

10 Hypothesis Testing 287

10.1 Significance Testing 287

10.2 Hypothesis Testing for Mean 291

10.2.1 p-Value 294

10.3 Decision Tests 300

10.4 Bayesian Test 303

10.4.1 Minimum Cost Test 304

10.4.2 Maximum a Posteriori Probability (MAP) Test 305

10.4.3 Maximum-Likelihood (ML) Test 305

10.4.4 Minimax Test 307

10.5 Neyman-Pearson Test 307

10.6 Summary 309

Problems 309

Part IV Random Processes 311

11 Introduction to Random Processes 313

11.1 Classification of Random Processes 313

11.1.1 State Space 314

11.1.2 Index (Time) Parameter 314

11.2 Characterization of Random Processes 318

11.2.1 Joint Distributions of Time Samples 318

11.2.2 Independent Identically Distributed Random Process 319

11.2.3 Multiple Random Processes 320

11.2.4 Independent Random Processes 320

11.3 Moments of Random Processes 320

11.3.1 Mean and Variance Functions of a Random Process 321

11.3.2 Autocorrelation and Autocovariance Functions of a Random Process 321

11.3.3 Cross-correlation and Cross-covariance Functions 324

11.4 Stationary Random Processes 326

11.4.1 Strict-Sense Stationary Processes 326

11.4.2 Wide-Sense Stationary Processes 327

11.4.3 Jointly Wide-Sense Stationary Processes 329

11.4.4 Cyclostationary Processes 331

11.4.5 Independent and Stationary Increments 331

11.5 Ergodic Random Processes 333

11.5.1 Strict-Sense Ergodic Processes 333

11.5.2 Wide-Sense Ergodic Processes 333

11.6 Gaussian Processes 336

11.7 Poisson Processes 339

11.8 Summary 341

Problems 341

12 Analysis and Processing of Random Processes 345

12.1 Stochastic Continuity, Differentiation, and Integration 345

12.1.1 Mean-Square Continuity 345

12.1.2 Mean-Square Derivatives 346

12.1.3 Mean-Square Integrals 347

12.2 Power Spectral Density 347

12.3 Noise 353

12.3.1 White Noise 353

12.4 Sampling of Random Signals 355

12.5 Optimum Linear Systems 357

12.5.1 Systems Maximizing Signal-to-Noise Ratio 357

12.5.2 Systems Minimizing Mean-Square Error 359

12.6 Summary 362

Problems 362

Bibliography 365

Books 365

Internet Websites 368

Answers 369

Index 387