Fractional calculus (FC), with its roots dating back to 1965 and pioneered by Leibniz and L'Hospital, is increasingly attracting attention in the applications of machine learning, a rising star in artificial intelligence (AI). Machine learning (ML), particularly in artificial neural network (ANN) applications, captures an internal memory dynamic by utilizing patterns in datasets. Fractional differential calculus, fractional derivatives, and integrals are used in models developed to internalize this dynamic and make learning algorithms more effective. According to studies in the literature, the Caputo definition of the fractional derivative is used in artificial neural networks called Fractional-order Artificial Neural Networks (FAANs). Here, we observe that activation functions based on fractional derivatives stand out. These activation functions are more flexible than their classical counterparts due to the need to carry adjustable parameters, and they increase the learning capacity and accuracy of the network. Based on this, this study will introduce the activation functions found in the literature at the intersection of FC and ANNs. This study shall discuss the convergence results in Banach spaces by considering the binary and multiple components of the relevant activation functions actively used in ANN operators. New activation functions obtained from the combination of these very general activation functions with more specific ones will be examined. Convergence analysis will be discussed using Banach space-valued Caputo fractional derivative and Caputo fractional Taylor formulas. Finally, a mathematical analysis of these activation functions—recently added to the literature—will be performed. The work closes with a nuanced examination of existing challenges and identifies several fruitful directions for future exploration in uniting fractional calculus with neural network methodologies.