For an X–Y stage’s six-degrees-of-freedom (6-DOF), displacement measurement system using multi-interferometers, this paper builds the geometric error’s transfer model first, and then analyses the sensitivity of the measurement error to the geometric error. On this basis, a method of a geometric error tolerance design for the accuracy of a measurement system is proposed using an improved genetic algorithm (GA) with minimal weighted and multi-source error bounds. For the high correlation of transfer coefficients in the propagation process of various errors, using GA’s global optimal searching capability in the entire motion range of each stage, this method overcomes the shortcoming that conservative results are usually produced when an equivalent effect method is used. Therefore, this method acquires a more objective result of the tolerance design of a geometric error for the X–Y stage measurement system accuracy.