Data collection plays an important role in pavement health monitoring, which is usually performed using costly devices, including point-based lasers and laser scanners. The main aim of this study is measuring pavement characteristics using an RGB-D sensor. By recording the depth and color images simultaneously, the sensor benefits the data fusion. By mounting the sensor on a moving cart, and fixing the vertical distance from the ground, data was collected along 100 m of the asphalt pavement using MATLAB. At each stop point, multiple frames were captured and the central region of interests was stored followed by applying low pass filters. To create a 3D dimensional surface of the pavement, sensor calibration was performed to map the RGB and depth infrared images. The SURF (Speeded-up Robust Features) and MSAC (M-estimator Sample Consensus) algorithm were used to match the stitched images along the longitudinal profile. A case study of measuring roughness and rutting is applied to test the validity of the method. The result confirms that the proposed system is capable of measuring such indices with acceptable accuracy.