Operational modal analysis is based on data collected with a network of sensors installed on the monitored structure to measure its response to the external stimuli. As the instrumentation can be costly, sensors get located at a limited number of spots where hopefully damage-sensitive features can be sensed. Hence, the actual capability to detect in real time a drift from the undamaged structural state might get detrimentally affected. Non-contact measurement methods resting on e.g. digital video cameras, which have gained interest in recent years, can instead provide high-resolution and diffused measurements/information. In this study, moving from videos of a vibrating structure and by combining them with optical flow estimation methods, the vibration response of the said structure is assessed. By means of a phase-based optical flow methodology, a linear correlation between the phase and the structural motion is assumed by e.g. the Gabor filter. Since such a phase extracted from the frames does not result to be linear in every case study, a primary objective is the linearization of the extracted phase, and then the derivation of the motion matrix for all the frames. By applying a dimensional reduction technique, the dimensions of the aforementioned matrix can be reduced to a value close to the number of the excited vibration modes of the structure. By using the blind source separation method, mode shapes and vibration frequencies are finally obtained. The performance of the proposed method is investigated using numerical examples, to testify the accuracy in extracting the dynamic features of the considered structure.