The brake squeal noise is a high-frequency noise over 1 kHz range generated by the contact between the brake pad and the disk. The purpose of this paper was to investigate the behavior of the squeal noise characteristics of the brake system from an instability point of view, according to the variation of major parameters such as friction coefficient between the flexible pad and the disk, brake pressure, and Young’s modulus of disk. Full nonlinear perturbed modal analysis using commercial finite element analysis program was performed to derive complex eigenvalue results of the model. And the sensitivity behavior was observed. Increasing the coefficient of friction or Young’s modulus of disk tended to make the squeal mode of the model more unstable. However, the change in brake pressure has a complicated nonlinear relationship with the squeal mode of the model. The judgment technique conducted in this study should be considered to be used in the design of the vibration point of the disk and pad of railway vehicles in the future.