Statistics II — Parameter Estimation

Statistics II — Parameter Estimation

We almost never measure an entire population, so we estimate its parameters from a sample. But not all estimators are equally good: we want an estimator that is unbiased and has as small a variance as possible. On this page we will learn what an unbiased estimator is, what efficiency means, how to compute the mean squared error (MSE) and the standard error of the mean, and how sample size affects precision.

Background and Basic Definitions

A parameter is a numerical value describing the population (such as the mean \(\mu\) or the variance \(\sigma^2\)), while an estimator is a formula computed from the sample used to guess the parameter (such as the sample mean \(\bar{x}\)). The resulting numerical value is called a point estimate.

Properties of a good estimator:

  • Unbiasedness: on average across many samples the estimator hits the parameter, i.e. \( E(\hat{\theta}) = \theta \). The bias is \( \text{Bias} = E(\hat{\theta}) - \theta \).
  • Efficiency: among two unbiased estimators, the more efficient one has the smaller variance — it spreads less around the parameter.

Mean squared error (MSE) measures the overall quality of an estimator:

\[ \text{MSE}(\hat{\theta}) = \text{Var}(\hat{\theta}) + \big(\text{Bias}(\hat{\theta})\big)^2 \]

For an unbiased estimator the bias is zero, so \( \text{MSE}(\hat{\theta}) = \text{Var}(\hat{\theta}) \).

The sample mean is the best point estimator for \(\mu\) and is unbiased: \( E(\bar{x}) = \mu \). Its spread is measured by the standard error of the mean:

\[ \text{SE}(\bar{x}) = \frac{\sigma}{\sqrt{n}} \]

The unbiased estimator of the population variance uses division by \((n-1)\):

\[ s^2 = \frac{\sum (x_i - \bar{x})^2}{n-1} \]

Solution Steps

  1. Step 1 — Identify the parameter being estimated (\(\mu\), \(\sigma^2\), \(p\)) and the appropriate estimator.
  2. Step 2 — To check unbiasedness, compute \( E(\hat{\theta}) \) and compare to the parameter; equal \(\Rightarrow\) unbiased.
  3. Step 3 — To compare efficiency among unbiased estimators, choose the one with the smaller variance.
  4. Step 4 — To compute MSE, use \( \text{Var} + \text{Bias}^2 \); if unbiased, \( \text{MSE} = \text{Var} \).
  5. Step 5 — For the standard error of the mean, divide \(\sigma\) by \(\sqrt{n}\): \( \text{SE} = \frac{\sigma}{\sqrt{n}} \).
  6. Step 6 — For required sample size, isolate \(n\) from the SE equation: \( n = \left(\frac{\sigma}{\text{SE}}\right)^2 \).

Worked Examples

Example 1: Standard Error of the Mean

Problem: The population standard deviation is known to be \( \sigma = 20 \). A sample of size \( n = 25 \) is drawn. What is the standard error of the sample mean?

Solution:

  1. Use the formula \( \text{SE}(\bar{x}) = \frac{\sigma}{\sqrt{n}} \).
  2. Substitute: \( \text{SE} = \frac{20}{\sqrt{25}} = \frac{20}{5} \).
  3. Compute: \( \frac{20}{5} = 4 \).
  4. Note: the larger the sample, the smaller the standard error — because \(\sqrt{n}\) in the denominator grows.

Answer: The standard error is \( 4 \).

Example 2: Required Sample Size

Problem: We want the standard error of the mean to be no greater than \( 2 \). The population standard deviation is \( \sigma = 16 \). What sample size is required?

Solution:

  1. Start from \( \text{SE} = \frac{\sigma}{\sqrt{n}} \) and isolate \(n\).
  2. Isolate the square root: \( \sqrt{n} = \frac{\sigma}{\text{SE}} = \frac{16}{2} = 8 \).
  3. Square both sides: \( n = 8^2 = 64 \).
  4. Check: \( \frac{16}{\sqrt{64}} = \frac{16}{8} = 2 \) — exactly as required.

Answer: A sample of size \( n = 64 \) is required.

Example 3: Sample Variance from Observations

Problem: A sample of four observations: \( 4, 7, 9, 12 \). Compute the unbiased sample variance \( s^2 \).

Solution:

  1. First, the mean: \( \bar{x} = \frac{4+7+9+12}{4} = \frac{32}{4} = 8 \).
  2. Deviations from the mean: \( -4, -1, 1, 4 \); their squares: \( 16, 1, 1, 16 \).
  3. Sum of squared deviations: \( 16 + 1 + 1 + 16 = 34 \).
  4. Divide by \( (n-1) = 3 \): \( s^2 = \frac{34}{3} \approx 11.33 \).

Answer: \( s^2 = \frac{34}{3} \approx 11.33 \).

Example 4: MSE of an Unbiased Estimator

Problem: The sample mean \( \bar{x} \) is an unbiased estimator of \(\mu\). Given \( \sigma = 12 \) and \( n = 9 \), what is \( \text{MSE}(\bar{x}) \)?

Solution:

  1. Since \( \bar{x} \) is unbiased, its bias is zero, so \( \text{MSE}(\bar{x}) = \text{Var}(\bar{x}) \).
  2. The variance of the sample mean is \( \text{Var}(\bar{x}) = \frac{\sigma^2}{n} \).
  3. Substitute: \( \frac{12^2}{9} = \frac{144}{9} \).
  4. Compute: \( \frac{144}{9} = 16 \) (note this equals \( \text{SE}^2 = 4^2 \)).

Answer: \( \text{MSE}(\bar{x}) = 16 \).

Example 5: Choosing the More Efficient Estimator

Problem: Two unbiased estimators of \(\theta\): estimator \(A\) has \( \text{Var}(A) = 9 \) and estimator \(B\) has \( \text{Var}(B) = 4 \). Which is preferred?

Solution:

  1. Both estimators are unbiased, i.e. \( E(A) = E(B) = \theta \) — no difference in bias.
  2. In this case the criterion is efficiency: prefer the estimator with the smaller variance.
  3. Since \( \text{Var}(B) = 4 \lt 9 = \text{Var}(A) \), estimator \(B\) is more efficient.
  4. For unbiased estimators MSE equals variance, so \(B\) also has a smaller MSE.

Answer: Estimator \(B\) is preferred (more efficient, smaller variance).

Common Mistakes

✗ Common mistake: Computing sample variance by dividing by \(n\) instead of \((n-1)\).

✓ The correct way: Dividing by \(n\) yields a downward-biased estimator (too small). The unbiased estimator of the population variance uses \((n-1)\): \( s^2 = \frac{\sum (x_i - \bar{x})^2}{n-1} \).

✗ Common mistake: Confusing the sample standard deviation \(s\) with the standard error of the mean \(\text{SE}\).

✓ The correct way: \(s\) describes the spread of individual observations, while \( \text{SE} = \frac{\sigma}{\sqrt{n}} \) describes the spread of the mean across samples. SE is always smaller and decreases further as \(n\) grows.

✗ Common mistake: Forgetting to take the square root of \(n\) and computing \( \text{SE} = \frac{\sigma}{n} \).

✓ The correct way: The denominator is \(\sqrt{n}\), not \(n\). Therefore quadrupling the sample size only halves the standard error, because \( \sqrt{4} = 2 \).

Practice Tips

  • Tip — To halve the standard error (reduce it by a factor of 2) you must quadruple the sample size, because \(n\) is under a square root.
  • Tip — A good estimator = unbiased + efficient. First verify unbiasedness, then compare variances to select the efficient one.
  • Tip — Remember MSE: \( \text{MSE} = \text{Var} + \text{Bias}^2 \). If the estimator is unbiased, MSE reduces to variance alone.
  • Tip — To find required sample size, use \( n = \left(\frac{\sigma}{\text{SE}}\right)^2 \) directly and round up to the nearest whole number.

Summary and Key Formulas

  • Unbiased estimator: \( E(\hat{\theta}) = \theta \).
  • Efficiency: among unbiased estimators, the efficient one = smallest variance.
  • MSE \( = \text{Var}(\hat{\theta}) + \text{Bias}^2 \); for unbiased \( \text{MSE} = \text{Var} \).
  • Standard error of the mean: \( \text{SE} = \frac{\sigma}{\sqrt{n}} \).
  • Required sample size: \( n = \left(\frac{\sigma}{\text{SE}}\right)^2 \).
  • Sample variance: \( s^2 = \frac{\sum (x_i-\bar{x})^2}{n-1} \).