The utilization of tower cranes at construction sites entails inherent risks, notably the potential for loads to fall on individuals. To address these risks, laws in many countries explicitly prohibit individuals from occupying the vicinity directly beneath suspended loads, known as the fall zone. This study proposes a novel method for identifying the tower crane load fall zone and determining workers' locations relative to this zone. The dynamic nature of crane load fall zones has not been adequately addressed in previous studies, mainly due to the difficulties in detecting various types of crane loads. Past studies have heavily relied on detecting a load based on its color and shape, which inevitably limits the range of possible identifications. Thus, this study presents a method that recognizes crane loads based on their movement patterns and elevation, using stereo cameras and computer vision algorithms. In addition to the previously mentioned limitation, earlier studies were constrained by the assumptions made to measure workers and load fall zone locations in 2D image space, such as assuming that all construction site entities were at the same height. To address this issue, the YOLOv7 deep learning algorithm was employed to accurately detect workers, while stereo camera depth data were utilized to measure their positions in the 3D world coordinate system. The effectiveness of the proposed method was validated through tests in a simulated small-scale project. The results indicate that this method can recognize a diverse range of loads with a high level of accuracy, exceeding 90%, which is a substantial improvement over earlier studies that identified only a limited set of load types. Additionally, the proposed method outperforms prior approaches in terms of analysis speed, achieving 8 frames per second speed compared to a maximum of 1 frame per second in earlier research.