As a common surface defect in pipeline steel, surface scratches significantly affect hydrogen embrittlement sensitivity due to their roughness characteristics. It is generally recognized that rougher surface scratches tend to result in higher susceptibility to hydrogen embrittlement. However, in this study, the influence of surface scratch roughness on hydrogen embrittlement sensitivity was systematically investigated in X80 pipeline steel using slow strain rate tensile testing and hydrogen microprint techniques, leading to the discovery of an unexpected phenomenon. Specifically, as the scratch features transitioned from coarse to fine, the hydrogen embrittlement sensitivity first decreased and then increased—a non-monotonic trend that has not been previously reported in the literature. This behavior originates from the synergistic effect between the surface scratch morphology and the residual compressive stress induced by the scratching process, with the two factors exhibiting a competitive relationship. As the grit size of the abrasive paper decreases (i.e., the finer the grit, the smaller the scratch roughness), the scratch dimensions decrease, which helps reduce hydrogen embrittlement sensitivity. At the same time, however, the surface residual compressive stress also diminishes, thereby increasing hydrogen embrittlement sensitivity. The competition between these two effects results in the observed non-monotonic variation in hydrogen embrittlement sensitivity with increasing abrasive paper grit size. Hydrogen-induced cracks almost invariably initiate at the scratch grooves, likely due to localized hydrogen enrichment driven by stress concentration during tensile loading—a mechanism strongly supported by finite element simulation results.