Maximum Entropy Approach for Reconstructing Bivariate Probability Distributions

The main purpose for this study is to provide a useful algorithm that combines the Maximum Entropy Method (MEM) and a computational method to predict the unique form of a two-dimensional maximum entropy distributions. In this paper, we present the application of MEM to determine the important bivariate distributions which are very effective in industrial and engineering fields especially in Cybernetics and internet systems. The combination of MEM and numerical method as a proposed algorithm is a useful method with the ability to model datasets which includes both missing and presence datas, The new algorithm provides reasonable estimations for reconstructing the target bivariate distributions which has maximum entropy. We examined the effectiveness of our algorithm to make predictions for two famus distributions. The resulting distributions have minimum error with respect to missing information. In fact, maximum entropy distribution is able to fit the known density given the prior knowledge of the target distribution. So, the MEM modeling procedure can be applied in present form for a variety of applications with precsence-only datasets. Possessing the simple and accurate mathematical formulation and using presence-only data, MEM has become a well-suited method for different kinds of distribution modeling.


Introduction
The method of maximum entropy is a very effective procedure in the determination of the general form of a density via solving optimization problems which is introduced by Rubinstein (1999).The concept of maximum entropy method was first proposed by Jaynes (1957).It is a useful method of reconstructing a density given finite numbers of moment constraints from incomplete datasets.There are many different standard statistical methods for modeling missing/presence available data such as generalized linear and additive models (GLM and GAM).The essence of MEM is to detect the probability distribution of maximum entropy, subject to a set of constraints that represents our presence-only information about the target distribution.In other words, the MEM makes it possible to find a unique distribution that gives rise to these moments.
Shannon et al. (1948), Introduced different methods to calculate the maximum entropy distributions.In this paper, we apply the MEM to estimate two-dimensional distributions and different classes of distributions.Note that not all classes of distributions contain a maximum entropy distribution.
The case of maximum entropy bivariate distribution has been studied by Djafari (2011).He has noted that an important problem in statistics is the construction of a joint probability distribution from its marginals.He linked this problem to an important problem in Computed tomography (CT) as a reconstruction of an image from its projections.To determine proposed to use copula by maximum entropy method as the solution to the problem of reconstructing a bivariate density from its marginals.
In this study, we consider the determination of a two-dimensional distribution in ME problem as an optimization problem by introducing a combined algorithm.The new algorithm is able to resolve a nonlinear system and calculate Lagrange multipliers to determine the class of distribution.The solution of this non-linear system requires calculation of two-dimensional integrals.Since the integrals are complex, the classical and existing numerical methods such as Simpson, etc., are not efficient and the results are not sufficiently accurate, the first step of the algorithm is a implementation of Newton method to transform this system to a system of linear equations.Results show that the proposed algorithm both advantages, of high accuracy and good agreement with exact solutions.
In the present study, various distributions such as Normal, beta etc., are considered.The rest of this paper is organized as follows: In Section 2, we present a summary of the ME method.In Section 3, we will show Some preliminary numerical experiments in various classes of distributions.Sections 4 is devoted to conclusions.

The Principle of Maximum Entropy
The MEM is determining the distribution that maximizes the information entropy.Jaynes' (1957) stated that maximum entropy principle provides a method for solving this problem when the available information is in the form of moment constraints.Suppose a probabilistic system in which X is a continuous random-variate with probability ) (x p , then the information entropy S , of a distribution where ¦ â is the support of the distribution and the information available is in the form of the normal constraint of probability Our purpose is to find ) (x p that maximizes the information entropy S given in Eq.( 1) subject to , 0,..., where k g and k a are known constants for N k 0,..., = .
ME method solves this problem by maximizing the Shannon entropy subject to the given known constraints.The problem of Shannon entropy maximization can be solved by Lagrange multiplier method (Fletcher, 1991).Where 1) is the number of known moments.Then , we define the entropy functional by introducing Lagrangian multipliers k This functional is a maximum when Eq. ( 4) gives us the constraints defined in Eq. ( 2) and Eq. ( 5) leads to Lagrange multiplier from a set of nonlinear equations system, N data constraints and normalization constraint.To obtain the 1  N Lagrange multipliers, the following set of 1  N nonlinear equations system should be solved: and a globally convergent Newton solver may be used to calculate k

Implementation and Results
As MEM is not available in standard statistical packages, we applied the MEM by writting code in MATLAB.In this code, we have shown the performance of Newton's method for solving twodimensional nonlinear equations and their quadrature numerical integration.We implemented the combination of two methods for the solver, one that uses the probabilistic expression for the constraints, and one that uses numerical technique to estimate Lagrange multipliers.To test the code, a number of moments were generated by the use of known distributions.We used a finite support to carry out the integrations for solving nonlinear system and the results have a reasonable approximation to the target distributions.The following examples include the new ME characterizations of many bivariate distributions.

Bivariate Normal Distribution
For normal distribution, we consider these constraints: Hence, the density that satisfies the constraints and also maximizes the entropy is We apply numerical method for calculating Lagrange multipliers.The exact form of this normal distribution in terms of these conditions should be given as

Bivariate Pareto Distribution
Consider the known constraints of pareto distribution : 2

Conclusions
In the present paper, an application of maximum entropy method is discussed to reconstruct bivariate distribution.A new algorithm is presented based on the standard Newton's method and probabilistic procedure.In addition, we define the critical boundary key points, which a new measure on monitoring wear condition and identifying probable wear faults are stated.The new algorithm is applied to identify distributions for cybernetics data as illustrated by four examples.By using this algorithm, the effectiveness of MEM were improved, and all kinds of bivariate distributions can be detected successfully.So the accurancy of this method is verified.
) (x p , is named Shannon entropy introduced by Shanoon et al. in 1948:

,
to determine the class of maximum entropy distributions.Our goal is determination of 1  N support of X and Y .Starting with a bivariate normal distribution, the PDF is given by


Figure1 shows the plot of error between exact distribution and estimated bivariate normal distribution.