Statistical inference for high-dimension, low-sample-size data

Abstract:

In this paper, we consider statistical inference for high-dimension, low-sample-size (HDLSS) data. We first show that HDLSS data have distinct geometric representations depending on whether or not the data meets a certain boundary condition. We clarify the limit of the conventional principal component analysis (PCA) for HDLSS data. In order to overcome the curse of dimensionality, we introduce two effective PCAs called the noise-reduction methodology and the cross-data-matrix (CDM) methodology. We further introduce the extended CDM methodology, which offers an unbiased estimator having small asymptotic variance and low computational cost, for feature parameters appearing in high-dimensional data analysis. We give correlation tests and several inferences on multiclass mean vectors for HDLSS data, and discuss sample size determination to ensure prespecified high accuracy for inference. Finally, we introduce two effective discriminant procedures: the geometric classifier and the distance-based classifier, that can hold misclassification rates less than a threshold.