Entropy On the Daróczy-Tsallis capacities of discrete channels

In the past there has been an extensive work on generalized entropies and generalized channel capacities. One of the first was Daróczy, who introduced new parameterized generalization of Shannon entropy, which reduces to the Shannon case if the parameter is set to one. A variant of this entropy, with a different normalization constant, was later proposed by Tsallis, who set it up as a basis for non-extensive statistical mechanics. Based on the generalized entropy, Daróczy introduced generalized mutual information which shares several important properties with the Shannon case, such as symmetry with respect to input/output channel distributions, non-negativity (if the parameter is greater than one) and obeying the chain rule. Daróczy also introduced a generalized channel capacity as the maximum of the generalized mutual information and derived expressions for the capacities of symmetric channel and binary symmetric channel as a special case. In this paper we provide new expressions for Daróczy capacities of weakly symmetric channel, binary erasure channel and z-channel, extending the previous work by Daróczy. Similarly to the Shannon case, the capacity of weakly symmetric channel is expressed as the source entropy reduced by the entropy of the transition matrix row (scaled by appropriate constant), capacity of binary erasure channel is expressed as the q-average number of bits which can be recovered after transmission, while the capacity of z-channel is expressed in terms of q-logarithm and q-exponential of generalized binary entropy function. All the expressions are general and can be directly applied to Tsallis entropy, reducing to the Shannon capacity results in a limit case, when the parameter tends to one.


Introduction
Shannon entropy follows the composition rule by which the entropy of a joint system can be represented as the sum of the entropy of one system and the conditional entropy of another, with respect to the first one.In the past, there was extensive work on paremeterized generalization of Shannon entropy, Daróczy-Tsallis entropies [1], [2], which follow more general q-addition rule [3] and reduce to the Shannon entropy if the parameter q is set to 1.
Daróczy-Tsallis entropies has successfully been used in a number of different fields, such as physics, chemistry, biology, economics, linguistics, medicine, cognitive sciences, computer sciences and social sciences [4], and have also been widely studied in information theory.Fundamental results such as Shannon-Khinchin axioms [5], [6], source coding theorems [7], rate-distortion theory [8], Fano's and data processing inequalities [9] has been established for the generalzied case.A generalized channel capacity have also been introduced and anylized by Daróczy [1], and revisited later by Landsberg and Vedral [10] (Tsallis entropy case), but only for binary symmetric channel.
In this paper we provide new expressions for Daróczy-Tsallis capacities of weakly symmetric channel, binary erasure channel and z-channel, extending the previous work from [1] and [10].Similarly to the Shannon case, the capacity of weakly symmetric channel is expressed as the source entropy reduced by the entropy of the transition matrix row (scaled by appropriate constant), capacity of binary erasure channel is expressed as the q-average number of bits which can be recovered after transmission, while the capacity of z-channel is expressed in terms of q-logarithm and q-exponential [9] of generalized binary entropy function.
The paper is organized as follows.In Section 2 we introduce the basic definitions and equalities for Daróczy-Tsallis entropy.The Daróczy-Tsallis capacity is introduced in section 3 and the channel capacity formula for weakly symmetric channel is derived in section 4. The Daróczy-Tsallis capacities of binary erasure channel and z-channel are derived in sections 5 and 6, respectively.
j=1 r ij and q i = n i=1 r ij .We define the joint entropy of P and Q as For a conditional distribution Q |k = (q 1|k , . . ., q m|k ) ∈ ∆ m , where q i|k = r ik /p k , p k > 0(k = 1, . . ., n), we define the conditional entropy of Q for a given P as Daróczy-Tsallis entropy has the following properties (for all n ∈ N+): [SA1] H q is continuous in ∆ n and with respect to q for all q ∈ R + ; [SA2] H q takes its largest value for the uniform distribution, U n = (1/n, . . ., 1/n) ∈ ∆ n , i.e.H q (P ) ≤ H q (U n ), for any P ∈ ∆ n ; [SA3] H q is expandable: [SA4] H q follows the chain rule, or all [SA5] H q is symmetric: , for any π is a permutation π of {1, . . ., n}, and all (p 1 , . . ., p n ) ∈ ∆ n Note that form the symmetry and the chain rule we have directly: The properties [SA1] to [SA4] are refereed as generalized Shannon-Khinchin axioms.In [5], [6], it is shown that uniquely determines Daróczy-Tsallis entropy.In the following text we will fix the constant k to 1.

Daróczy-Tsallis channel capacity
A discrete constant channel with the space X = {x 1 , . . ., x n } of input symbols and with the space Y = {y 1 , . . ., y n } of output symbols is characterized by the (m×n) transition matrix, Q, with conditional probabilities Q ˙|k = (q 1|k , . . ., q m|k ), m j=1 q j|k = 1, for k = 1, . . ., n, as rows: where If X is distributed according to P = (p 1 , . . ., p n ), than the output distribution Q = (q 1 , . . ., q m ) is given by scalar products q k = Q T ˙|k • P , k = 1, . . ., m, where Q ˙|k and P are taken as vectors, and The joint distribution P Q can be obtained as If q k > 0 (k = 1, . . ., m), we can define the inverse transition matrix is given with where P Q :,k stands for k-th column of P Q: The generalized mutual information between X and Y is denoted as I q (P, Q) and defined as Two following theorems represents the basic properties of the q mutual information.
Proof.The mutual information can be represented as On the other side, for q ≥ 1, is convex function of p i and −H(Q|P ) is concave, so that I(P, Q) concave since it is represented as a sum of concave functions.
In the case q < 1 the mutual information can take negative values for a choice of P and Q |k which can be treated as negative information transfer between P and Q [10].Moreover, for q < 1 the mutual information is not necessary concave which makes problem with the definition of the generalized channel capacity .For these reasons, we restrict the definition of the mutual information to the case q ≥ 1.
Daróczy-Tsallis channel capacity is defined as a maximum mutual information which can be conveyed between X and Y , where the maximum is taken with respect to the input distribution P : In this paper, we derive the expressions for the channel capacities of some common discrete channels.

Daróczy-Tsallis capacity of weakly symmetric channel
Definition 4.1.A channel is said to be symmetric if the rows of the channel transition matrix t (i) are permutations of each other, Q |k = (q π(1) , . . ., q π(n) ), and the columns are permutations of each other.A channel is said to be weakly symmetric if every row of the transition matrix is a permutation of every other row and all the column sums n i=1 a ij are equal.
Since all rows are permutations of each other, due to the symmetry property H q (Q |k ) = H q (Q |1 ) for all k = 1, . . ., n so that Now, due to the maximality property, mutual information satisfies the following: By the convexity of t q (q > 1), we obtain The equality is achieved for uniform input distribution P = (1/n, . . ., 1/n) since column sums are equal, m j=1 a ij = c, so that Q = A T • P = (c/n, . . ., c/n) is uniform, and we have the for q = 1, we have: and the channel capacity (28) reads Let us chose the function φ(q) so that C q scales to 1 for p = 0 and binary symmetric channel (case n = 2).Then, φ(q) = 2 1−q − 1 and the result reduces to the theorem proven by Daróczy [1]

Daróczy-Tsallis capacity of binary symmetric channel
The binary symmetric channel (BSC) is a weakly symmetric channel represented by two dimensional transition matrix: If we define binary Daróczy-Tsallis entropy function as: Daróczy-Tsallis entropy capacity can be represented as: In the limit case which agrees with [11].
If chose the function φ(q) = 2 1−q − 1 so that C q scales to 1 for α = 0 , the BCS capacity reduces to In Fig. 1 we show the C q dependence on p on BSC for different values of q.

Daróczy-Tsallis capacity of binary erasure channel
Binary erasure channel is two input-tree output channel described by the transition matrix: so that H q (P |1 ) = H q (P |3 ) = 0 and H q (P |2 ) = h q (p), so that we have On the other side H q (P ) = h q (p), so that the mutual information is given by and is maximized for uniform distribution p = 1/2, so that In the limit case which agrees with [11].In the case of Havrda-Charvat entropy φ(q) = 2 1−q −1, channel capacity reduces to which can be interpreted in a similar way like for the Shannon case [11].Let X be a random variable taking 1 if a bit is lost during the transmission and 0 if a correct bit is transmitted.The probability that the bit will be lost is then P (X = 1) = α and the q average number of lost bits is given with P (X = 0) q • 0 + P (X = 1) q • 1 = α q .In accordance C q stands for the q-average number of bits which can be recovered.The C q dependence on α and q for BEC, is given in Fig. 2.

Daróczy-Tsallis capacity of z-channel
The z-channel two input-two output channel described by the transition matrix: If P = (1 − p, p), then Q = (1 − αp, αp), so that H q (Q) = h q (αp).On the other side, H q (Q |1 ) = 0 and H q (Q |2 ) = h q (α) and we have The optimal p can be found by taking the derivative of I q (P, Q) and equating it to zero.If we introduce q-exponential, the optimal distribution can expressed as If we introduce q-logarithm, which is the inverse function of e q , Log q On the other side: Then, the C q can be expressed as e q − h q (α) α q 1−q 1 + e q − h q (α) α q −1 1−q − 1 φ(q) (42) and finally C q = Log q 1 + e q − h q (α) α q (43) In the limit case lim q→1 C q = log 2 1 + 2 − h(α) α (44) which agrees with [11].The C q dependence on α and q for z-channel (with φ(q) = 2 1−q − 1), is given in Fig. 3.

Conclusions
New expressions for Daróczy-Tsallis capacities of weakly symmetric channel, binary erasure channel and z-channel, were derived, which is extends the previous work by Daróczy.We shown that, similarly to the Shannon case, the capacity of weakly symmetric channel can be expressed as the source entropy reduced by the entropy of the transition matrix row (scaled by appropriate constant), capacity of binary erasure channel can be expressed as the q-average number of bits which can be recovered after transmission, while the capacity of z-channel can be expressed in terms of q-logarithm and q-exponential of generalized binary entropy function.All the expressions were derived in general way so that they can be directly applied to Tsallis entropy, reducing to the Shannon capacity results in a limit case.

Example 4 . 1 .
If the transition matrix of a discrete constant channel has the form[1]

Figure 1 .
Figure 1.C q as a function of transition probability p and parameter q for BSC

Figure 2 .
Figure 2. C q as a function of transition probability α and parameter q for BEC

Figure 3 .
Figure 3. C q as a function of transition probability α and parameter q for z-channel