A spectral-domain interferometer with dual reference paths and orthogonal polarization states to avoid measurement errors when interference signals overlap is proposed and realized. In our previous study, by using dual reference mirrors, two inherent problems of the spectral-domain interferometer, the non-measurable range and the directional ambiguity problem, were successfully solved. However, because of the overlap of interference signals, the absolute distance values were distorted. In this study, the polarization states of beams from two reference paths were made orthogonal to eliminate the interference signal between them, so that the overlap can be essentially avoided. First, we performed a numerical simulation on the measurement error with respect to the degree of superposition of the interference signals. Simulation results show that with the previous method the measurement error can be up to approximately 1 μm within the overlap region, but the proposed method drastically reduced this error to below 100 nm. Then, the proposed method was experimentally realized and verified. In conclusion, the proposed method can measure the absolute distances without the inherent problems as well as the measure errors caused by the overlap of the interference signals.