Draft:Information theory/Permutations and combinations

'''THIS DRAFT IS BEING USED TO HELP ME EDIT Information theory/Permutations and combinations. DO NOT DELETE'''

The rate at which $n!$ increases for large $n$ boggles the mind: $3!=6$, and $5!=120.$ But $50!$$$\approx 3\times 10^{80}$$, which likely outnumbers the number of atoms in the visible universe.

Suppose you have with the following collection of animal crackers: Two are Elephants, three are Crows, and two are Rats. How many seven-letter words can you make?

Combinations


The traditional view of combinations focuses on selecting k items from a collection (or set) of n items. In this discussion, the $n$ items are positive integers, i.e., the set: $$\{1,2,3,...,n\}$$. Rules of set theory permit us to change the order of these $N$ objects. However for our purposes, these $n$ objects (integers) are generally listed in the proper order because the integers denote the order in which the "letters" of a message are presented. In keeping with a parallel discussion of Shannon entropy, we associate the bins with Crow and Rat, using the notation $C$ and $R$. These refer to "letters" in a binary alphabet. - Permute "Bin labels", not the order of the letters in the word
 * Instead of placing the members of the set $$\{1,2\ldots N\}$$ into "bins" $$\{C,R\ldots\}$$, we shall instead permute $$N$$ "bin-labels" and attach them to the set ${#1,#2,#3.}$

-

Simple example: 3-choose-2
This is equivalent to finding 3-choose-1. Our goal is to well to counting all the three-letter words that use $C$ twice and $R$ once. For reasons that will be understood later, we solve this problem attaching labels to letter that is used twice: $$(C_1,C_2).$$ As shown in the left-most column in $$, we begin by counting all $3!=6$ permutations of $$(C_1C_2R).$$

The stationary columns represent ${#1,#2,#3.}$, which are the order in which the letters appear. Equivalently ${#1,#2,#3.}$ represent the 3 distinguishable objects to be placed in either the C or R bins. The other two columns in the figure illustrate that there there is a two-fold redundancy in that each of the three combinations appears twice.

Multinomial coefficients for non-binary alphabets
Application of the binomial theorem for, with $n=3$ and $k=2:$ leads to the familiar result:

$$   $$\binom{n}{k}=\frac{n!}{k!(n-k)!}=\frac{3!}{2!1!}$$

If more than two "bins" are involved, we define, $$n_1=k$$ and $$n_2=n-k.$$ This results in a formula that is easier to remember and can be generalized to include counting the words that can be written with three or more letters:

$$   $$\frac{(n_1+n_2)!}{n_1!n_2!} \Rarr\frac{(n_1+n_2+n_3)!}{n_1!n_2!n_3!}\Rarr\frac{\left(\sum_j n_j\right)!}{\prod_j(n_j!)}$$

Larger numbers: 5-choose-3


$$illustrates how one can visualize the counting of 5-choose-3. As before, we begin with the 120 permutations of the 5 subscripted letters $$\{C_1C_2C_3R_1R_2\}.$$ The goal is to organize these permutations in such a way that the duplicates associated with dropping the subscripts are accounted for. With 3-choose-1 the only duplicates involved the order of Crows ($$C_1C_2$$ versus $$C_2C_1$$).

With 3 Crows and 2 Rats, we have $$3!=6$$ ways to permute the Crows and $$2!=2$$ permutations for the Rat. It is instructive to first consider how many of the 120 permutations are associated with a single word. Consider, for example all the ways subscripts can be added to the word that starts with two $$R$$s, namely the word: $$RRCCC$$:

well



\begin{matrix} \text{positive} \\ \text{permutations} \\ \text{of }C_1C_2C_3 \end{matrix} \overbrace{ \begin{cases} R_1R_2C_1C_2C_3 & R_2R_1C_1C_2C_3 \\ R_1R_2C_2C_3C_1 & R_2R_1C_2C_3C_1 \\ R_1R_2C_3C_1C_2 & R_2R_1C_3C_1C_2 \\ \end{cases}}^{R_1R_2\quad\quad\quad\quad R_2R_1}$$   $$



\begin{matrix} \text{negative} \\ \text{permutations} \\ \text{of }C_1C_2C_3 \end{matrix} \underbrace{ \begin{cases} R_1R_2C_3C_2C_1 & R_2R_1C_3C_2C_1 \\ R_1R_2C_2C_1C_3 & R_2R_1C_2C_1C_3  \\ R_1R_2C_1C_3C_2 & R_2R_1C_1C_3C_2 \\ \end{cases}}_{R_1R_2\quad\quad\quad\quad R_2R_1}$$

Counting the words in $$, we see that there are 12 ways to attach our subscripts to $RRCCC.$ It can be shown that all words involving 3 $C$s and 2 $R$s can be subscripted in 12 different ways.

The product rule
It is instructive to closely examine how $$enumerates the $12$ ways that subscripts can be assigned to repeated letters in any given word. The $12$ words are listed in two columns of six rows each. The first column places $$R_1$$ before $$R_2$$, while the second column presents the $R$s in reverse order. Each of the six rows represents one of the $6$ permutations of the subscripts $123$ that are assigned to the Crows (these permutations were presented in $$.)

The fact that all (six) ways to assign to subscripts to the three Crow letters is multiplied by all (two) ways to assign subscripts to the two Rat letters is known as the product rule.

This same $$(2\times 6=12)$$ element array of possible subscripts for Crow and Rat is also shown in $$. The boxed element at tells us the order in which the subscripted letters appear. The top three digits $213$ informs us that the subscripted Crows follow the order $$C_2..C_1..C_3$$. Meanwhile the lower pair $12$ informs us that $$R_1$$ precedes $$R_2$$.

Recap on efforts to count 5-choose-3

 * 1) We know from counting the permutations of $5$ symbols that there are $120$ ways to arrange the five subscripted letters $$C_1C_2C_3$$.
 * 2) For any given five letter word (involving $3C$s and $2R$s) there are $12$ ways to to separately attach subscripts to the letters. Since these $12$ versions of the word are identical if the subscripts are  removed, we shall refer to these as duplicates of the word.
 * This is all we need to calculate $5-choose-3$.

Geometric "proof"
$$illustrates how one can arrange the $(5!=120)$ permutation this collection. Taking the height to be an unknown, we create the box by stacking rectangles of $12$ duplicates for each word. Since $$12\times 10 = 120$$, we conclude that $10$ is our desired result:

$$   $$\binom 5 3 =\frac{5!}{3!2!}=10$$

Algebraic proof and multinomial theorem
It is not difficult to extend the geometric argument associated with $$to arbitrary values on $n$ and $k$ as as stated at $$. On the other hand, an important restriction results from using a three-dimensional box to account for duplicate words. However, counting the words one can create with more than

On the other hand,

stophere

$${n \choose k_1, k_2, \ldots, k_m}$$


 * 1) There are $5!=120$ ways to permute the subscripted version of RRCCC.
 * 2) Each combination (of 5-take-3) is duplicated 12 time if the subscripts are removed.
 * &there4; The value of 5-take-3 is $120/12=10$

For reasons soon to be discussed, we express these ways as the product, $$6\times 2 = 12$$. Since there there are $5!=120$ ways to permute the subscripted version of

Multinomials
Although the value of Shannon entropy is independent of how many letters are contained in the "alphabet", some users might wish to count how many words can be created by a non-binary alphabet. There are a number ways to do it

THIS SECTION NEEDS TO BE MOVED TO ANOTHER PAGE (MyOpenMath?)

File:Multinomial theorem mississippi.svg