Statistical mechanics has a universal appeal that extends beyond molecular systems, and yet, as its tools are being transplanted to fields outside physics, the fundamental questions, what is thermodynamics and how it may be applied outside the realm of physical particles, have remained unanswered. We answer these questions here: Statistical mechanics in its most general form is variational calculus applied to probability distributions and by extension to stochastic processes in general; as a mathematical theory, it is independent of physical hypotheses but provides the means to incorporate our knowledge and model assumptions about the particular problem. The fundamental ensemble is a microcanonical space of probability distributions sampled via a bias functional that establishes a probability measure on this space. The maximization of this measure expresses the most probable distribution via a set of parameters (microcanonical partition function, canonical partition function and generalized temperature) that are connected through a set of mathematical relationships that we recognize as the familiar equations of thermodynamic. Any distribution in in this space maybe endowed with the status of the most probable distribution under an appropriately constructed bias functional. Entropy, Kullback-Leibler divergence and the second law have simple interpretations in this theory. We obtain statistical mechanics as a special application to molecular systems and make contact with Information Theory and Bayesian inference. We use numerical examples to demonstrate the thermodynamic treatment of generic probability distributions, present a thermodynamic algorithm (the cluster ensemble) to sample arbitrary distributions with positive argument by analogy to reacting particles and discuss the extension of statistical mechanics to stochastic processes in general.