Diabetes mellitus (DM) is a metabolic disorder that may lead to chronic hyperglycemia; the most important type of DM is type 2 diabetes mellitus (T2DM). In recent years, the treatment strategies for DM have been progressed; however, anti-diabetic drugs have negative impacts such as kidney and liver disorders and hypoglycemic problems. The most important anti-diabetic activities of medicinal herbs and plant are hypoglycemic, anti-lipidemic, anti-hyperglycemic, and insulin mimicking. Some of the most important medicinal plants with anti-diabetic activities are garlic, bitter melon, ginger, and roselle plant. This literature review was conducted using different databases such as Google Scholar, MEDLINE, PubMed Central, and Science Direct. Many countries rely on plant-based medicine and medicinal plants because they are cost-effective and affordable. Medicinal plants can manage blood sugar levels and promote the immune system. Some chemical components such as fructans, kaempferol, quercetin, sulfur compounds, and allicin have shown anti-diabetic activities. The plants have no side effects, and most of them are effective in the treatment of DM. The goal of this manuscript is to highlight and study the biomedical significance and pharmaceutical benefits of some of the most important medicinal herbs and plants with anti-diabetic activities that have shown high potential to treat and prevent DM.