Fast tuning of topic models: an application of Rényi entropy and renormalization theory

In practice, the critical step in build machine learning models of big data (BD) often involves costly in terms of time and computing resources procedure of parameter tuning with grid search. Due to the size BD are comparable to mesoscopic physical systems. Hence, methods of statistical physics could be applied to BD. The paper shows that topic modeling (a clustering method for large document collections) demonstrates self-similar behavior under the condition of a varying number of clusters. Such behavior allows using a renormalization technique. A combination of renormalization procedure with Rényi entropy approach allows for fast searching of the optimal number of clusters. In this paper, the renormalization procedure is developed for the Latent Dirichlet Allocation (LDA) model with variational Expectation-Maximization algorithm. The experiments were conducted on two document collections with a known number of clusters in Russian and English languages, respectively. The paper presents results for three versions of the renormalization procedure: (1) a renormalization with the random merging of clusters, (2) a renormalization based on minimal values of Kullback-Leibler divergence and (3) a renormalization with merging clusters with minimal values of Rényi entropy where entropy is computed for each topic separately. The paper shows that the renormalization procedure allows finding the optimal number of topics ten times faster than grid search without significant loss of quality.