Summary: Estimation of genewise variance arises from two important applications in microarray data analysis: selecting significantly differentially expressed genes and validation tests for normalization of microarray data. We approach the problem by introducing a two-way nonparametric model, which is an extension of the famous J. Neyman and E. L. Scott model [Econometrica, Chicago 16, 1–32 (1948; Zbl 0034.07602)] and is applicable beyond microarray data. The problem itself poses interesting challenges because the number of nuisance parameters is proportional to the sample size and it is not obvious how the variance function can be estimated when the measurements are correlated.
In such a high-dimensional nonparametric problem, we proposed two novel nonparametric estimators for genewise variance functions and semiparametric estimators for measurement correlations, via solving a system of nonlinear equations. Their asymptotic normality is established. The finite sample property is demonstrated by simulation studies. The estimators also improve the power of the tests for detecting statistically differentially expressed genes. The methodology is illustrated by the data from the microarray quality control (MAQC) project.