Spectrally resolved interferometry (SRI) is an attractive technique to measure absolute distances without any moving components. In the spectral interferogram obtained by a spectrometer, the optical path difference (OPD) can simply be extracted from the linear slope of the spectral phase. However, SRI has a fundamental measuring range limitation due to maximum and minimum measurable distances. In addition, SRI cannot distinguish the OPD direction because the spectral interferogram is in the form of a natural sinusoidal function. In this investigation, we describe a direction determining SRI and propose the optimal conditions for determining OPD direction. Spectral phase nonlinearity, caused by a dispersive material, effects OPD direction but deteriorates spectral interferogram visibility. In the experiment, various phase nonlinearities were measured by adjusting the dispersive material (BK7) thickness. We observed the interferogram visibility and the possibility of direction determination. Based on the experimental results, the optimal dispersion conditions are provided to distinguish OPD directions of SRI.