The bolted joint has been widely used to connect load-bearing elements in aerospace, civil, and mechanical engineering systems. During its service life, particularly under external dynamical loads, a bolted joint may undergo self-loosening. Bolt-looseness causes a reduction in the load-bearing capacity and eventually leads to the failure of the bolted joint. This paper presents an automated image-based algorithm combining the Faster RCNN model with image processing for quick detection of loosened bolts in a structural connection. The algorithm is validated using a lab-scale bolted joint model for which various bolt-loosening events are simulated. The imagery data of the joint is captured and passed through the algorithm for bolt-looseness detection. The obtained results show that the loosened bolts in the joint were well detected and their loosening degrees were precisely quantified. Therefore, the image-based algorithm is promising for real-time structural health monitoring of realistic bolted joints.