Solutions to Homework Assignment 10

Problem 9.72 (8 points)

Suppose that \(Y_1,\dotsc,Y_n\) denote a random sample from a Poisson distribution with mean \(\lambda > 0\). Then the probability mass function is \[ p(y \mid \lambda) = e^{-\lambda}\frac{\lambda^y}{y!}, \qquad y = 0,1,2,3,\dotsc \]

1. We find the likelihood function and the derivative of its natural log. \[\begin{align*} L(y_1,\dotsc,y_n \mid \lambda) &= \prod_{i=1}^{n} p(y_i \mid \lambda) = e^{-n\lambda}\cdot \lambda^{\sum_{i=1}^{n}y_i} / (y_1 ! \dotsb y_n !) \\ \implies \ln L(y_1,\dotsc,y_n \mid \lambda) &= -n \lambda + \left(\sum_{i=1}^{n} y_i\right) \ln \lambda - \ln\left(y_1 ! \dotsb y_n !\right) \\ \implies \frac{\partial}{\partial \lambda} \ln L(y_1,\dotsc,y_n \mid \lambda) &= -n + \frac{1}{\lambda}\sum_{i=1}^{n} y_i. \end{align*}\] We see that \(\hat{\lambda}\) is a critical point for the log likelihood if and only if \(\hat{\lambda} = \frac{1}{n}\sum_{i=1}^{n} y_i\). The second derivative test confirms this critical point is a maximum. Therefore, the Maximum Likelihood Estimator for \(\lambda\) is \[ \hat{\lambda} = \frac{1}{n}\sum_{i=1}^{n} Y_i = \overline{Y}. \]
2. The expected value and variance of the MLE are \[\begin{align*} E[\hat{\lambda}] &= E[\overline{Y}] = E[Y] = \lambda \\ V[\hat{\lambda}] &= V[\overline{Y}] = \frac{1}{n}V[Y] = \frac{1}{n}\lambda. \end{align*}\]
3. Since \[ \lim_{n\rightarrow \infty}V[\hat{\lambda}] = \lim_{n\rightarrow \infty}\frac{\lambda}{n} = 0, \] Theorem \(9.1\) implies that \(\hat{\lambda}\) is a consistent estimator for \(E[\hat{\lambda}] = \lambda\).
4. Let \(t(\lambda) = e^{-\lambda} = P(Y = 0)\). The invariance property for Maximum Likelihood Estimators implies that the MLE for \(e^{-\lambda}\) is \[ t(\hat{\lambda}) = t(\overline{Y}) = e^{-\overline{Y}}. \]

Problem 9.74 (6 points)

Let \(Y_1,\dotsc,Y_n\) denote a random sample from the probability distribution whose density function is \[ f_Y(y \mid \theta) = \begin{cases} \frac{1}{\theta} r y^{r - 1} e^{-y^r/\theta}, & \theta > 0, y > 0 \\ 0, & \textrm{elsewhere} \end{cases}, \] where \(r\) is a known positive constant.

1. We write the likelihood function \[ L(y_1,\dotsc,y_n \mid \theta) = \prod_{i=1}^{n} f(y_i \mid \theta) = \left[ \frac{1}{\theta^n} e^{-\sum_{i=1}^n y_i^r / \theta} \right]\cdot \left[ r^n \left(\prod_{i=1}^{n} y_i\right)^{r-1} \right] = g(u,\theta)\cdot h(y_1,\dots,y_n), \] where \(u = \sum_{i=1}^{n} y_i^r\), \(g(u,\theta) = \frac{1}{\theta^n} e^{-u/\theta}\) and \(h(y_1,\dotsc,y_n) = r^n \left(\prod_{i=1}^{n} y_i\right)^{r-1}\). The factorization criterion implies that a minimal sufficient statistic for \(\theta\) is \[ U = \sum_{i=1}^n Y_i^r. \]
2. Let \(\ell(\theta)\) denote the log likelihood function. Then \[ \ell(\theta) = \ln L(y_1,\dotsc,y_n \mid \theta) = -n \ln \theta - \frac{1}{\theta}\sum_{i=1}^{n} y_i^r + \ln\left[r^n \left(\prod_{i=1}^{n} y_i\right)^{r-1}\right]. \] Then \(\ell'(\theta) = -n \theta^{-1} + \theta^{-2} \sum_{i=1}^{n} y_i^r\), and \(\ell'(\hat{\theta}) = 0\) implies that \(\hat{\theta} = \frac{1}{n}\sum_{i=1}^{n} y_i^r\). A calculation shows that \(\ell''(\hat{\theta}) < 0\), so \(\ell(\theta)\) has a maximum at \(\hat{\theta}\). Therefore, the MLE for \(\theta\) is \[ \hat{\theta} = \frac{1}{n}\sum_{i=1}^{n} Y_i^r. \]

3. Note that \(\hat{\theta} = \frac{1}{n} U\). From part (a), we know that \(U\) is a minimal sufficient statistic for \(\theta\), so if \(\hat{\theta}\) is unbiased then the Rao-Blackwell Theorem implies that \(\hat{\theta}\) is a MVUE.

   Let \(w = y^r\), then \(dw = r y^{r-1} dy\) and \[ f_Y(y \mid \theta)\,dy = \frac{1}{\theta} r y^{r - 1} e^{-y^r/\theta}\,dy = \frac{1}{\theta} e^{-w/\theta}\,dw = f_W(w \mid \theta)\,dw, \] which shows that the random variable \(W = Y^r\) has an Exponential distribution with mean \(\theta\). Thus \(E[Y^r] = E[W] = \theta\), and therefore \(E[\hat{\theta}] = \frac{1}{n}\sum_{i=1}^{n} E[Y_i^r] = \theta\), which shows that \(\hat{\theta}\) is an unbiased estimator for \(\theta\) and hence a MVUE for \(\theta\).

Problem 9.77 (10 points)

Let \(Y_1,\dotsc,Y_n\) denote a random sample from a Gamma distribution whose probability density function is given by \[ f_Y(y \mid \theta) = \begin{cases} \frac{1}{\Gamma(\alpha)\theta^{\alpha}} y^{\alpha - 1} e^{-y/\theta}, & y > 0 \\ 0, & \textrm{elsewhere} \end{cases}, \] where \(\alpha\) is known.

1. We write the likelihood function \[ L(y_1,\dotsc,y_n \mid \theta) = \prod_{i=1}^{n} f_Y(y_i \mid \theta) = \left[\Gamma(\alpha) \theta^{\alpha}\right]^{-n}\cdot \left(\prod_{i=1}^{n} y_i\right)^{\alpha - 1}\cdot \exp\left(-\frac{1}{\theta}\sum_{i=1}^{n}y_i\right). \] Let \(\ell(\theta)\) denote the log likelihood function. Then \[ \ell(\theta) = \ln L(y_1,\dotsc,y_n \mid \theta) = -n \alpha \ln \theta - \frac{1}{\theta}\sum_{i=1}^{n} y_i + \ln \left[ \left(\Gamma(\alpha)\right)^{-n} \left(\prod_{i=1}^{n} y_i \right)^{\alpha-1} \right]. \] The derivative of the log likelihood is \(\ell'(\theta) = -n \alpha \theta^{-1} + \theta^{-2}\sum_{i=1}^{n} y_i\), thus \(\ell'(\hat{\theta}) = 0\) implies \(\hat{\theta} = \frac{1}{n \alpha}\sum_{i=1}^{n}y_i\). A calculation shows that \(\ell''(\hat{\theta}) < 0\), which shows that \(\hat{\theta}\) is a maximum. Therefore, the Maximum Likelihood Estimator for \(\theta\) is \[ \hat{\theta} = \frac{1}{n \alpha}\sum_{i=1}^{n} Y_i = \frac{1}{\alpha} \overline{Y}. \]
2. \(Y\) is a Gamma distributed random variable, so \(E[Y] = \alpha \theta\) and \(V[Y] = \alpha \theta^2\). Hence \[\begin{align*} E[\hat{\theta}] &= \frac{1}{\alpha}E[\overline{Y}] = \frac{\alpha \theta}{\alpha} = \theta. \\ V[\hat{\theta}] &= \frac{1}{\alpha^2}V[\overline{Y}] = \frac{\alpha \theta^2}{n\alpha^2} = \frac{\theta^2}{n\alpha}. \end{align*}\]
3. Since \[ \lim_{n\rightarrow \infty}V[\hat{\theta}] = \lim_{n\rightarrow \infty} \frac{\theta^2}{n\alpha} = 0, \] Theorem \(9.1\) shows that \(\hat{\theta}\) is a consistent estimator for \(E[\hat{\theta}] = \theta\).
4. We rewrite the likelihood function as \[\begin{align*} L(y_1,\dotsc,y_n \mid \theta) &= \left[ \theta^{-n\alpha}e^{-\frac{1}{\theta}\sum_{i=1}^{n} y_i} \right]\cdot \left[ \left(\Gamma(\alpha)\right)^{-n} \left(\prod_{i=1}^{n}y_i\right)^{\alpha - 1} \right] \\ &= g(u,\theta)\cdot h(y_1,\dotsc,y_n), \end{align*}\] where \(u = \sum_{i=1}^{n} y_i\). By the factorization criterion, \[ U = \sum_{i=1}^{n} Y_i \] is a minimal sufficient statistic for \(\theta\).
5. Let \(n = 5\) and \(\alpha = 2\). Since \(Y\) has a Gamma distribution with \(\alpha = 2\) and \(\beta = \theta\), Problem \(6.40\) shows that \(W = \frac{2}{\theta} Y\) has a \(\chi^2\)-distribution with \(\nu = 2\alpha = 4\) degrees of freedom. Thus \[ \frac{2}{\theta}U = \frac{2}{\theta}\sum_{i=1}^{5} Y_i \] has a \(\chi^2\)-distribution with \(\nu=20\) degrees of freedom. Using this as a pivotal quantity we find \[\begin{align*} P\left(\chi^{2}_{0.95} \le \frac{2}{\theta}U \le \chi^{2}_{0.05}\right) &= 0.90 \\ P\left(\frac{2U}{\chi^{2}_{0.05}} \le \theta \le \frac{2U}{\chi^{2}_{0.95}}\right) &= 0.90. \end{align*}\] With \(\nu = 20\), \(\chi^{2}_{0.05} = 31.4104\), \(\chi^{2}_{0.95} = 10.8508\). Therefore, a \(90\%\) confidence interval for \(\theta\) is \[ \left(0.0637\sum_{i=1}^{5} Y_i , 0.1843\sum_{i=1}^{5} Y_i\right) \]

Problem 9.80 (2 points)

Let \(Y_1,\dotsc,Y_n\) denote a random sample from a probability distribution whose density function is given by \[ f_Y(y \mid \theta) = \begin{cases} (\theta + 1) y^{\theta}, & 0 < y < 1, \theta > -1 \\ 0, & \textrm{elsewhere} \end{cases}. \] We write the likelihood function \[ L(y_1,\dotsc,y_n \mid \theta) = \prod_{i=1}^{n} f_Y(y_i \mid \theta) = (\theta + 1)^{n} \left(\prod_{i=1}^{n} y_i\right)^{\theta}. \] Let \(\ell(\theta)\) denote the log likelihood function. Then \[\begin{align*} \ell(\theta) &= \ln L(y_1,\dotsc,y_n \mid \theta) = n \ln (\theta + 1) + \theta \ln\left(\prod_{i=1}^{n} y_i\right) \\ \implies \ell'(\theta) &= \frac{n}{\theta + 1} + \ln\left(\prod_{i=1}^{n} y_i\right). \end{align*}\]

A calculation shows that \(\ell'(\hat{\theta}) = 0\) if \[ \hat{\theta} = -\frac{n + \ln \prod_{i=1}^{n} y_i} {\ln \prod_{i=1}^{n} y_i}. \] Therefore, the Maximum Likelihood Estimator for \(\theta\) is \[ \hat{\theta} = -\frac{n + \ln \prod_{i=1}^{n} Y_i} {\ln \prod_{i=1}^{n} Y_i}, \] which is quite different from the Method Of Moments estimator \((2\overline{Y} - 1)/(1 - \overline{Y})\).

Problem 9.82 (2 points)

\[\begin{align*} L(p) &= \left[p^{25}(1-p)^{75}\right]\cdot \left[p^{30}(1-p)^{70}\right] = p^{55}(1-p)^{145} \\ \ell(p) &= \ln L(p) = 30 \ln p + 145 \ln (1 - p) \\ \ell'(p) &= \frac{30}{p} - \frac{145}{1-p} = \frac{55(1 - p) - 145p}{p(1 - p)} \\ \ell'(p) &= \frac{55 - 200 p}{p(1 - p)}. \end{align*}\]

We see that \(\ell'(\hat{p}) = 0\) if and only if \(\hat{p} = 55/200 = 0.275\). Since \(\ell''(\hat{p}) < 0\), we see that the MLE is \(\hat{p} = 0.275\).

Problem 9.84 (6 points)

Let \(Y_1,\dotsc,Y_n\) be a random sample from a population whose density function is \[ f(y \mid \theta) = \begin{cases} \frac{3y^2}{\theta^3}, & 0 \le y \le \theta \\ 0, & \textrm{elsewhere} \end{cases}. \] Exercise \(9.44\) showed that \(Y_{(n)} = \max(Y_1,\dotsc,Y_n)\) is sufficient for \(\theta\).

1. We write the likelihood function \[ L(y_1,\dotsc,y_n \mid \theta) = \prod_{i=1}^{n} f(y_i \mid \theta) = 3^{n}\left(\prod_{i=1}^{n} y_i\right)^{2} \theta^{-3n}\prod_{i=1}^{n} I_{[0,\theta]}(y_i). \] As a function of \(\theta\), \(\prod_{i=1}^{n} I_{[0,\theta]}(y_i) = I_{[0,\theta]}(y_{(n)})\cdot I_{[0,\infty)}(y_{(1)}) = 1\) for \(\theta \ge y_{(n)}\), and equals \(0\) for \(\theta < y_{(n)}\). Since \(\theta^{-3n}\) is a decreasing function of \(\theta\), we conclude that \(L(y_1,\dotsc,y_n \mid \theta)\) achieves its maximum value at \(\hat{\theta} = y_{(n)}\). Therefore \(Y_{(n)}\) is a Maximum Likelihood Estimator for \(\theta\).
2. The Cumulative Distribution Function (CDF) for \(Y\) is \[ F_Y(y) = P(Y \le y) = \begin{cases} 1, & y > \theta \\ (y / \theta)^{3}, & 0 \le y \le \theta \\ 0, & y < 0 \end{cases}. \] From this we deduce that the CDF for \(Y_{(n)}\) is \(G_{(n)}(y) = P(Y_{(n)} \le y) = [F_Y(y)]^{n}\). The corresponding Probability Density Function (PDF) for \(Y_{(n)}\) is \(g_{(n)}(y) = G'_{(n)}(y)\), \[ g_{(n)}(y) = \begin{cases} \frac{3n}{\theta^{3n}} y^{3n - 1}, & 0 \le y \le \theta \\ 0, & \textrm{elsewhere} \end{cases}. \] A quick calculation shows that if we let \(w = y/\theta\), then \(g_{(n)}(y)\) transforms into the density function \[ f_W(w) = \begin{cases} 3n w^{3n - 1}, & 0 \le w \le 1 \\ 0, & \textrm{elsewhere} \end{cases}, \] which shows that \(W = \frac{1}{\theta}Y_{(n)}\) has a Beta distribution with parameters \(\alpha = 3n\) and \(\beta = 1\). Moreover, since \(f_W(w)\) is independent of \(\theta\), \(W\) is a pivotal quantity.
3. Another quick calculation shows that the CDF for \(W\) is \[ F_W(w) = \begin{cases} 1, & w > 1 \\ w^{3n}, & 0 \le w \le 1 \\ 0, & w < 0 \end{cases}. \] Let \(0 < \alpha < 1\), then \[\begin{align*} P\left((\alpha/2)^{\frac{1}{3n}} \le W \le (1 - \alpha/2)^{\frac{1}{3n}}\right) &= F_W\left((1 - \alpha/2)^{\frac{1}{3n}}\right) - F_W\left((\alpha/2)^{\frac{1}{3n}}\right) \\ &= 1 - \alpha/2 - \alpha/2 \\ &= 1 - \alpha. \end{align*}\] Since \(W = Y_{(n)} / \theta\) and since \[ P\left((\alpha/2)^{\frac{1}{3n}} \le Y_{(n)}/\theta \le (1 - \alpha/2)^{\frac{1}{3n}}\right) = P\left( (1 - \alpha/2)^{-\frac{1}{3n}} Y_{(n)} \le \theta \le (\alpha/2)^{-\frac{1}{3n}} Y_{(n)}\right), \] we deduce that a \((1 - \alpha)\cdot 100\%\) confidence interval for \(\theta\) is \[ \left( (1 - \alpha/2)^{-\frac{1}{3n}} Y_{(n)} , (\alpha/2)^{-\frac{1}{3n}} Y_{(n)}\right). \]

Problem 9.86 (2 Points)

Let \(\beta = t(\theta)\) be an invertible function. Let \(\hat{\theta}\) be a maximum likelihood estimator for the likelihood function \(L(y_1,\dotsc,y_n \mid \theta)\), then \[\begin{equation} L(y_1,\dotsc,y_n \mid \theta) \le L(y_1,\dotsc,y_n \mid \hat{\theta}) \end{equation}\]

for all \(\theta\).

Let \(\mathscr{L}(y_1,\dotsc,y_n \mid \beta)\) be the corresponding likelihood function involving \(\beta\). The two likelihood functions are related as follows: \[\begin{equation} L(y_1,\dotsc,y_n \mid \theta) = \mathscr{L}(y_1,\dotsc,y_n \mid t(\theta)). \end{equation}\] For example, let \(Y\) be a normal random variable with \(E[Y] = 0\) and \(V[Y] = \sigma^2\), \(\sigma > 0\). Let \(t(x) = x^2\), let \(\theta = \sigma\), then \(\beta = t(\theta) = \theta^2 = \sigma^2\), and \[\begin{align*} \mathscr{L}(y_1,\dotsc,y_n \mid \beta) &= \prod_{i=1}^{n} \left(2 \pi \beta \right)^{-1/2} e^{-\frac{1}{2}y_i^2 / \beta}. \\ L(y_1,\dotsc,y_n \mid \theta) &= \prod_{i=1}^{n} \left(2 \pi \theta^2 \right)^{-1/2} e^{-\frac{1}{2}y_i^2 / \theta^2} \\ &= \mathscr{L}(y_1,\dotsc,y_n \mid \theta^2) \\ &= \mathscr{L}(y_1,\dotsc,y_n \mid t(\theta)). \end{align*}\]

Returning to the general case, let \(\beta\) be arbitrary, then since \(t\) is invertible we define \(\theta = t^{-1}(\beta)\). Thus, \[ \mathscr{L}(y_1,\dotsc,y_n \mid \beta) = L(y_1,\dotsc,y_n \mid \theta) \le L(y_1,\dotsc,y_n \mid \hat{\theta}) = \mathscr{L}(y_1,\dotsc,y_n \mid t(\hat{\theta})). \] Since \(\mathscr{L}(y_1,\dotsc,y_n \mid \beta) \le \mathscr{L}(y_1,\dotsc,y_n \mid t(\hat{\theta}))\) for all \(\beta\), \(t(\hat{\theta})\) is a Maximum Likelihood Estimator for \(\beta\).

Problem 9.88 (2 points)

Let \(Y_1,\dotsc,Y_n\) be a random sample from a normal distribution with mean \(\mu\) and variance \(\sigma^2\), both unknown. Example \(9.15\) (pages \(450 - 451\)) shows that \[ \hat{\mu} = \overline{Y},\qquad \widehat{\sigma^2} = \frac{1}{n}\sum_{i=1}^{n}\left(Y_i - \overline{Y}\right)^2 \] are Maximum Likelihood Estimators for \(\mu\) and \(\sigma^2\). Since \(\sigma = \sqrt{\sigma^2}\), the invariance property for MLEs implies that \[ \hat{\sigma} = \sqrt{ \frac{1}{n}\sum_{i=1}^{n}\left(Y_i - \overline{Y}\right)^2 } \] is an MLE for \(\sigma\).