Structuring the Population

Abstract

Many recipes for how to do an NIR calibration begin with a step that combines rejecting outliers with splitting the available samples into calibration and validation sets. This is sometimes referred to as structuring the population, though it has other names as well. Though the aims of having clean data and a sensible split cannot be argued with, my personal view is that there is a general tendency to take things a bit too far, with the result being an over-optimistic view of how good is the resulting calibration.