Minimum mean square error
Let be a point estimator of the parameter :
Suppose there are two estimators and of the parameter Set and equal to the mean square errors of those two estimators. Then the relative efficiency of and can be defined as:
Unfortunately, the correct distribution from which to estimate the mean-squared error of the estimator is a point of contention between Bayesian and frequentist schools of probability theory. Orthodox statistics employs a transformation of variables to get the probability distribution of the estimator from the sampling distribution, giving the estimator's probability independent of the actual data set obtained. This distribution correctly describes the variation of the estimator over all possible data sets.
Bayesian statistics instead holds that the correct distribution to use is that which represents the probability an observer would give to the variable after observing the actual data set.
Where I represents some information the observer has about the nature of the variable . This distribution correctly describes the observer's state of knowledge about the parameter to be estimated after taking the observed data set into consideration.
It is interesting to note that these alternate viewpoints can sometimes (but not always) produce the same mean +/- standard deviation answer, as for example, estimation of the mean of a Normally-distributed data set. (Jaynes)
- Johnson, D. (2004, November 22). Minimum Mean Squared Error Estimators. Connexions
- Prediction and Improved Estimation in Linear Models, by J. Bibby, H. Toutenburg (Wiley, 1977). This book looks almost exclusively at minimum mean-square error estimation and inference.
- Jaynes, E. T. Probability Theory: The Logic of Science. Cambridge University Press, 2003.