This paper demonstrates how ambient vibration measurements at a limited number of locations can be effectively utilized to estimate parameters of a finite element model of a large-scale structural system involving a large number of elements. System identification using ambient vibration measurements presents a challenge requiring the use of special identification techniques, which ran deal with very small magnitudes of ambient vibration contaminated by noise without the knowledge of input farces. In the present study, the modal parameters such as natural frequencies, damping ratios, and mode shapes of the structural system were estimated by means of appropriate system identification techniques including the random decrement method. Moreover, estimation of parameters such as the stiffness matrix of the finite element model from the system response measured by a limited number of sensors is another challenge. In this study, the system stiffness matrix was estimated by using the quadratic optimization involving the computed and measured modal strain energy of the system, with the aid of a sensitivity relationship between each element stiffness and the modal parameters established by the second order inverse modal perturbation theory. The finite element models thus identified represent the actual structural system very well, as their calculated dynamic characteristics satisfactorily matched the observed ones from the ambient vibration test performed on a large-scale structural system subjected primarily to ambient wind excitations. The dynamic models identified by this study will be used for design of an active mass damper system to be installed on this structure fer suppressing its wind vibration.