Rao-Blackwellisation of sampling schemes
Casella and Robert (1996, Biometrika) presented a general Rao-Blackwellisation principle for accept-reject and Metropolis-Hastings schemes that leads to significant decreases in the variance of the resulting estimators, but at a potential high cost in computing and storage. Adopting a completely different perspective, we introduce instead a universal scheme that guarantees variance reductions in all Metropolis-Hastings based estimators while keeping the computing cost under control. The principle relates to the availability of an unbiased estimator of the acceptance probability. In a second if related part, we consider the implications of the fact that parallel raw-power can be exploited by a generic Metropolis--Hastings algorithm if the proposed values are independent. In particular, we present improvements to the independent Metropolis--Hastings algorithm that significantly decrease the variance of any estimator derived from the MCMC output, for a null computing cost since those improvements are based on a fixed number of target density evaluations. Furthermore, those techniques do not jeopardize the Markovian convergence properties of the algorithm, since they are based on the Rao--Blackwell principles of Gelfand and Smith (1990), already exploited in Casella and Robert (1996). We illustrate those improvements both on a toy normal example and on a classical probit regression model, but stress the fact that they are applicable in any case where the independent Metropolis-Hastings is applicable. Extensions to the random walk Metropolis--Hastings algorithm will also be discussed.