Module VI: Probability

1. Random Variables

Definition

A random variable is a function that assigns a numerical value to each outcome of a random experiment.

Type	Definition	Example
Discrete	Takes countable values (finite or countably infinite)	Number of heads in 3 coin tosses: 0, 1, 2, 3
Continuous	Takes any value in an interval (uncountably infinite)	Height, weight, time, temperature

2. Discrete Random Variable & Probability Distribution

Probability Mass Function (PMF)

For a discrete random variable \( X \), the PMF is \( P(X = x) = p(x) \) where:

\( 0 \leq p(x) \leq 1 \) for all \( x \)
\( \sum_{\text{all } x} p(x) = 1 \)

A die is thrown. Let X = number on top face. Find probability distribution

X can take values: 1, 2, 3, 4, 5, 6
Each outcome equally likely: \( P(X = x) = \frac{1}{6} \) for each

X	1	2	3	4	5	6
P(X)	1/6	1/6	1/6	1/6	1/6	1/6

Sum = \( 6 \times \frac{1}{6} = 1 \) ✓ (Valid PMF)

Find k if \( P(X = x) = kx \) for \( x = 1, 2, 3, 4 \)

For valid PMF: \( \sum P(X = x) = 1 \)
\( k(1) + k(2) + k(3) + k(4) = 1 \)
\( 10k = 1 \)

\( k = \frac{1}{10} = 0.1 \)

3. Continuous Random Variable & PDF

Probability Density Function (PDF)

For a continuous random variable \( X \), the PDF \( f(x) \) satisfies:

\( f(x) \geq 0 \) for all \( x \)
\( \int_{-\infty}^{\infty} f(x) \, dx = 1 \)
\( P(a \leq X \leq b) = \int_a^b f(x) \, dx \)

For continuous random variable: \( P(X = a) = 0 \) (probability at a single point is zero)

Find k if \( f(x) = kx^2 \) for \( 0 \leq x \leq 3 \), and 0 otherwise

For valid PDF: \( \int_{-\infty}^{\infty} f(x) \, dx = 1 \)
\( \int_0^3 kx^2 \, dx = 1 \)
\( k \left[ \frac{x^3}{3} \right]_0^3 = 1 \)
\( k \cdot \frac{27}{3} = 1 \) → \( 9k = 1 \)

\( k = \frac{1}{9} \)

For the above PDF, find \( P(1 \leq X \leq 2) \)

\( P(1 \leq X \leq 2) = \int_1^2 \frac{1}{9}x^2 \, dx \)
\( = \frac{1}{9} \left[ \frac{x^3}{3} \right]_1^2 = \frac{1}{27}[8 - 1] \)

\( P(1 \leq X \leq 2) = \frac{7}{27} \)

4. Expectation (Mean)

Discrete: \( E(X) = \mu = \sum x \cdot P(X = x) \)

Continuous: \( E(X) = \mu = \int_{-\infty}^{\infty} x \cdot f(x) \, dx \)

Properties of Expectation

\( E(c) = c \) (constant)
\( E(cX) = cE(X) \)
\( E(X + Y) = E(X) + E(Y) \)
\( E(aX + b) = aE(X) + b \)

Find E(X) for the distribution

X	0	1	2	3
P(X)	0.1	0.3	0.4	0.2

\( E(X) = \sum x \cdot P(X = x) \)
\( = 0(0.1) + 1(0.3) + 2(0.4) + 3(0.2) \)
\( = 0 + 0.3 + 0.8 + 0.6 \)

\( E(X) = 1.7 \)

For \( f(x) = \frac{1}{9}x^2 \), \( 0 \leq x \leq 3 \), find E(X)

\( E(X) = \int_0^3 x \cdot \frac{1}{9}x^2 \, dx = \frac{1}{9}\int_0^3 x^3 \, dx \)
\( = \frac{1}{9} \left[ \frac{x^4}{4} \right]_0^3 = \frac{1}{9} \cdot \frac{81}{4} \)

\( E(X) = \frac{9}{4} = 2.25 \)

5. Variance and Standard Deviation

Variance: \[ \text{Var}(X) = \sigma^2 = E(X^2) - [E(X)]^2 \]

Also written as: \( \sigma^2 = E[(X - \mu)^2] \)

Standard Deviation: \[ \sigma = \sqrt{\text{Var}(X)} = \sqrt{E(X^2) - [E(X)]^2} \]

For calculation:
Discrete: \( E(X^2) = \sum x^2 \cdot P(X = x) \)
Continuous: \( E(X^2) = \int_{-\infty}^{\infty} x^2 \cdot f(x) \, dx \)

Properties of Variance

\( \text{Var}(c) = 0 \)
\( \text{Var}(cX) = c^2 \text{Var}(X) \)
\( \text{Var}(X + c) = \text{Var}(X) \)
\( \text{Var}(aX + b) = a^2 \text{Var}(X) \)

Find Variance and S.D. for the distribution

X	0	1	2	3
P(X)	0.1	0.3	0.4	0.2

From before: \( E(X) = 1.7 \)
\( E(X^2) = 0^2(0.1) + 1^2(0.3) + 2^2(0.4) + 3^2(0.2) \)
\( = 0 + 0.3 + 1.6 + 1.8 = 3.7 \)
\( \text{Var}(X) = E(X^2) - [E(X)]^2 = 3.7 - (1.7)^2 = 3.7 - 2.89 = 0.81 \)
\( \sigma = \sqrt{0.81} = 0.9 \)

Variance = 0.81, Standard Deviation = 0.9

For \( f(x) = \frac{1}{9}x^2 \), \( 0 \leq x \leq 3 \), find Var(X)

From before: \( E(X) = \frac{9}{4} \)
\( E(X^2) = \int_0^3 x^2 \cdot \frac{1}{9}x^2 \, dx = \frac{1}{9}\int_0^3 x^4 \, dx \)
\( = \frac{1}{9} \left[ \frac{x^5}{5} \right]_0^3 = \frac{1}{9} \cdot \frac{243}{5} = \frac{27}{5} = 5.4 \)
\( \text{Var}(X) = 5.4 - \left(\frac{9}{4}\right)^2 = 5.4 - 5.0625 = 0.3375 \)

Var(X) = 0.3375, \( \sigma = \sqrt{0.3375} \approx 0.581 \)

6. Moment Generating Function (MGF)

Definition

The moment generating function of a random variable \( X \) is:

Discrete: \( M_X(t) = E(e^{tX}) = \sum e^{tx} \cdot P(X = x) \)

Continuous: \( M_X(t) = E(e^{tX}) = \int_{-\infty}^{\infty} e^{tx} \cdot f(x) \, dx \)

Key Property: Finding Moments from MGF

\[ E(X^n) = \mu_n' = \left. \frac{d^n}{dt^n} M_X(t) \right|_{t=0} \]

First moment: \( E(X) = M_X'(0) \)

Second moment: \( E(X^2) = M_X''(0) \)

\( M_X(t) \) is called "moment generating function" because its derivatives at \( t = 0 \) give moments

Find MGF and first two moments for \( P(X=x) = \frac{1}{2^x} \) for \( x = 1, 2, 3, ... \)

\( M_X(t) = E(e^{tX}) = \sum_{x=1}^{\infty} e^{tx} \cdot \frac{1}{2^x} = \sum_{x=1}^{\infty} \left(\frac{e^t}{2}\right)^x \)
This is a geometric series with first term \( a = \frac{e^t}{2} \) and ratio \( r = \frac{e^t}{2} \)
Sum = \( \frac{a}{1-r} = \frac{e^t/2}{1 - e^t/2} = \frac{e^t}{2 - e^t} \) (valid for \( e^t < 2 \))
First moment: \[ M_X'(t) = \frac{d}{dt}\left(\frac{e^t}{2-e^t}\right) = \frac{e^t(2-e^t) + e^t \cdot e^t}{(2-e^t)^2} = \frac{2e^t}{(2-e^t)^2} \] \( E(X) = M_X'(0) = \frac{2 \cdot 1}{(2-1)^2} = 2 \)
Second moment: After differentiating again: \( E(X^2) = M_X''(0) = 6 \)

\( M_X(t) = \frac{e^t}{2-e^t} \), \( E(X) = 2 \), \( E(X^2) = 6 \), Var(X) = 6 - 4 = 2

Find MGF of \( f(x) = e^{-x} \) for \( x \geq 0 \)

\( M_X(t) = \int_0^{\infty} e^{tx} \cdot e^{-x} \, dx = \int_0^{\infty} e^{(t-1)x} \, dx \)
\( = \left[ \frac{e^{(t-1)x}}{t-1} \right]_0^{\infty} \) (for \( t < 1 \))
\( = 0 - \frac{1}{t-1} = \frac{1}{1-t} \)
First moment: \( M_X'(t) = \frac{1}{(1-t)^2} \), \( E(X) = M_X'(0) = 1 \)
Second moment: \( M_X''(t) = \frac{2}{(1-t)^3} \), \( E(X^2) = M_X''(0) = 2 \)

\( M_X(t) = \frac{1}{1-t} \), \( E(X) = 1 \), \( E(X^2) = 2 \), Var(X) = 1

For a random variable with MGF \( M_X(t) = e^{3t + 2t^2} \), find mean and variance

\( M_X(t) = e^{3t + 2t^2} \)
\( M_X'(t) = (3 + 4t)e^{3t + 2t^2} \)
\( E(X) = M_X'(0) = 3 \cdot e^0 = 3 \)
\( M_X''(t) = 4e^{3t+2t^2} + (3+4t)^2 e^{3t+2t^2} \)
\( E(X^2) = M_X''(0) = 4 + 9 = 13 \)
\( \text{Var}(X) = E(X^2) - [E(X)]^2 = 13 - 9 = 4 \)

Mean = 3, Variance = 4

Quick Reference: All Formulas

Concept	Discrete	Continuous
Valid distribution	\( \sum P(X=x) = 1 \)	\( \int f(x)dx = 1 \)
Mean \( E(X) \)	\( \sum x \cdot P(X=x) \)	\( \int x \cdot f(x)dx \)
\( E(X^2) \)	\( \sum x^2 \cdot P(X=x) \)	\( \int x^2 \cdot f(x)dx \)
Variance	\( \text{Var}(X) = E(X^2) - [E(X)]^2 \)
S.D.	\( \sigma = \sqrt{\text{Var}(X)} \)
MGF	\( \sum e^{tx} \cdot P(X=x) \)	\( \int e^{tx} \cdot f(x)dx \)
Moments from MGF	\( E(X^n) = M_X^{(n)}(0) \)