This paper presents a line-of-sight (LOS) stabilization control method for portable optical systems by analyzing fast steering mirror, image sensor, and gyro sensor system. To compensate for LOS errors caused by hand tremors in portable optical systems, we present the configuration of an image sensor-based LOS stabilization control system and a control strategy considering the phase delay effect caused by low sampling frequency of the image sensor. The phase delay effect of the image sensor caused restricted bandwidth, which limited the stabilization performance. To overcome such limitations, we present disturbance feedforward control using the gyro sensor and controller design method considering characteristics of the gyro sensor. Through overall system modeling, we constructed a control simulation model. The LOS stabilization performance against hand tremor disturbances was analyzed based on the proposed controller design. Simulation results demonstrated that integrating a gyro sensor-based disturbance feedforward control with the image sensor-based LOS stabilization control significantly enhanced the stabilization performance.