Quick and Easy Way to Make a Joint Probability Matrix

Linear Algebra and Related Introductory Topics

Barry Kurt Moser , in Linear Models, 1996

1.3 RANDOM VECTORS

Let the n × 1 random vector Y = (Y ₁, Y ₂,…, Y_n )′ where Y_i is a random variable for i = 1,…, n. The vector Y is a random entity. Therefore, Y has an expectation; each element of Y has a variance; and any two elements of Y have a covariance (assuming the expectations, variances, and covariances exist). The following definitions and theorems describe the structure of random vectors.

Definition 1.3.1

Joint Probability Distribution : The probability distribution of the n × 1 random vector Y = (Y ₁,…, Y_n )′ equals the joint probability distribution of Y ₁,…, Y_n . Denote the distribution of Y by f _Y (y) − f _Y (y ₁,…, y_n ).

Definition 1.3.2

Expectation of a Random Vector: The expected value of the n × 1 random vector Y = (Y ₁,…, Y_n )′ is given by E(Y) = [E(Y ₁),…, E(Y_n )]′.

Definition 1.3.3

Covariance Matrix of a Random Vector Y: The n × 1 random vector Y = (Y ₁,…, Y_n )′ has n × n covariance matrix given by

$cov\left(\text{Y}\right)=\text{E}\left\{\left[\text{Y}-\text{E}\left(\text{Y}\right)\right]{\left[\text{Y}-\text{E}\left(\text{Y}\right)\right]}^{\prime }\right\}\cdot$

The ij ^th element of cov(Y) equals E{[Y_i − E(Y_i )][Y_j − E(Y_j )]} for i, j = 1,…, n.

Definition 1.3.4

Linear Transformations of a Random Vector Y: If B is an m × n matrix of constants and Y is an n × 1 random vector, then the m × 1 random vector BY represents m linear transformations of Y.

The following theorem provides the covariance matrix of linear transformations of a random vector.

Theorem 1.3.1

If B is an m × n matrix of constants, Y is an n × 1 random vector, and cov(Y) is the n × n covariance matrix of Y, then the m × 1 random vector BY has an m × m covariance matrix given by B [cov(Y)]B′.

Proof:

$\begin{array}{ll}cov\left(B\text{Y}\right)& =\text{E}\left\{\left[\text{B}\text{Y}-\text{E}\left(\text{B}\text{Y}\right)\right]{\left[\text{B}\text{Y}-\text{E}\left(\text{B}\text{Y}\right)\right]}^{\prime }\right\}\\ & =\text{E}\left\{\text{B}\left[\text{Y}-\text{E}\left(\text{Y}\right)\right]{\left[\text{Y}-\text{E}\left(\text{Y}\right)\right]}^{\prime }{\text{B}}^{\prime }\right\}\\ & =\text{B}\left\{\text{E}\left\{\left[\text{Y}-\text{E}\left(\text{Y}\right)\right]{\left[\text{Y}-\text{E}\left(\text{Y}\right)\right]}^{\prime }\right\}\right\}{\text{B}}^{\prime }\\ & =\text{B}\left[cov\left(\text{Y}\right)\right]{\text{B}}^{\prime }\cdot \end{array}$

The next theorem provides the expected value of a quadratic form.

Theorem 1.3.2

Let Y be an n × 1 random vector with mean vector μ = E(Y) and n × n covariance matrix Σ = cov(Y) then E(Y′AY) = tr(AΣ) + μ′Aμ where A is any n × n symmetric matrix of constants.

Proof:

Since (Y − μ)′A (Y − μ) is a scalar and using Result 1.5,

${\left(\text{Y}-\mu \right)}^{\prime }\text{A}\left(\text{Y}-\mu \right)=\text{tr}\left[{\left(\text{Y}-\mu \right)}^{\prime }\text{A}\left(\text{Y}-\mu \right)\right]=\text{tr}\left[\text{A}\left(\text{Y}-\mu \right){\left(\text{Y}-\mu \right)}^{\prime }\right]\cdot$

Therefore,

$\begin{array}{ll}\text{E}\left[{\text{Y}}^{\prime }\text{A}\text{Y}\right]& =\text{E}\left[{\left(\text{Y}-\mu \right)}^{\prime }\text{A}\left(\text{Y}-\mu \right)2{\text{Y}}^{\prime }\text{A}\mu -{\mu }^{\prime }\text{A}\mu \right]\\ & =\text{E}\left\{\text{tr}\left[\text{A}\left(\text{Y}-\mu \right){\left(\text{Y}-\mu \right)}^{\prime }\right]\right\}+2\text{E}\left({\text{Y}}^{\prime }\text{A}\mu \right)-{\mu }^{\prime }\text{A}\mu \\ & =\text{tr}\left[\text{AE}\left\{\left(\text{Y}-\mu \right){\left(\text{Y}-\mu \right)}^{\prime }\right\}\right]+{\mu }^{\prime }\text{A}\mu \\ & =\text{tr}\left[\text{A}\Sigma \right]+{\mu }^{\prime }\text{A}\mu \cdot \end{array}$

The moment generating function of a random vector is used extensively in the next chapter. The following definitions and theorems provide some general moment generating function results.

Definition 1.3.5

Moment Generating Function (MGF) of a Random Vector Y: The MGF of an n × 1 random vector Y is given by

$m_{\text{Y}}\left(t\right)=\text{E}\left(e^{t^{\prime }\text{Y}}\right)$

where the n × 1 vector of constants t = (t ₁,…, t_n )′ if the expectation exists for −h < t_i < h where h > 0 and i = 1,…, n.

There is a one-to-one correspondence between the probability distribution of Y and the MGF of Y, if the MGF exists. Therefore, the probability distribution of Y can be identified if the MGF of Y can be found. The following two theorems and corollary are used to derive the MGF of a random vector Y.

Theorem 1.3.3

Let the n × 1 random vector Y = (Y′ ₁, Y′ ₂,…, Y′ _m )′ where Y _i is an n_i × 1 random vector for i = 1,…, m and $n={\Sigma }_{i=1}^m{n}_i$ . Let $m_{\text{Y}}(.),m_{{\text{Y}}_1}(.),\dots ,m_{{\text{Y}}_{\text{m}}}(.)$ represent the MGFs of Y, Y ₁,…, Y _m respectively. The vectors Y ₁,…, Y _m are mutually independent if and only if

$m_{\text{Y}}\left(\text{t}\right)=m_{{\text{Y}}_1}\left(t_1\right)m_{{\text{Y}}_2}\left(t_2\right)\dots m_{{\text{Y}}_m}\left(t_m\right)$

for all t = (t′ ₁,…, t′ _m )′ on the open rectangle around 0.

Theorem 1.3.4

If Y is an n × 1 random vector, g is an n × 1 vector of constants, and c is a scalar constant, then

$m_{{\text{g}}^{\prime }\text{Y}}\left(c\right)=m_{\text{Y}}\left(cg\right)\cdot$

Proof:

$m_{\text{g}\prime \text{Y}}(c)=\text{E}[e^{\text{cg ′ Y}}]=\text{E}[e^{\text{(cg)}\prime \text{Y}}]=m_{\text{Y}}(\text{cg})$ .

Corollary 1.3.4

Let m _Y1, (.),…, m _{Y m} (.) represent the MGFs of the independent random variables Y ₁,…, Y_m, respectively. If Z = ${\Sigma }_{i=1}^m{\text{Y}}_i$ then the MGF of Z is given by

$m_Z\left(s\right)=\underset{i=1}{\overset{m}{\Pi }}{m}_{{\text{Y}}_i}\left(s\right)\cdot$

Proof:

$\begin{array}{l}\begin{array}{cc}{m}_Z\left(s\right)=m_{{1^{\prime }}_m\text{Y}}\left(s\right)=m_{\text{Y}}\left(s{1}_m\right)& \text{by\hspace{0.17em}Theorem\hspace{0.17em}1}\cdot \text{3}\cdot \text{4}\end{array}\\ \begin{array}{cc}=\underset{i=1}{\overset{m}{\Pi }}{m}_{{\text{Y}}_i}\left(s\right)& \text{by\hspace{0.17em}Theorem\hspace{0.17em}1}\cdot \text{3}\cdot \text{3}\cdot \end{array}\end{array}$

Moment generating functions are used in the next example to derive the distribution of the sum of independent chi-square random variables.

Example 1.3.1

Let Y ₁,…, Y_m be m independent central chi-square random variables where Y_i and n_i degrees of freedom for i = 1,…, m. For any i

$\begin{array}{ll}{m}_{{\text{Y}}_i}\left(t\right)& =\text{E}\left(e^{t{\text{Y}}_i}\right)\\ & ={{\int }_0^{\infty }{e}^{t{y}_i}\left[\Gamma \left(n_i/2\right)2^{n_i/2}\right]}^{-1}{y}_i^{n_i/2-1}{e}^{-y_i/2}d{y}_i\\ & ={\left[\Gamma \left(n_i/2\right)2^{n_i/2}\right]}^{-1}{\int }_0^{\infty }{y}_i^{n_i/2-1}{e}^{-y_i/{\left(1/2-t\right)}^{-1}}d{y}_i\\ & ={\left[\Gamma \left(n_i/2\right)2^{n_i/2}\right]}^{-1}\left[\Gamma \left(n_i/2\right){\left(1/2-t\right)}^{-n_i/2}\right]\\ & ={\left(1-2t\right)}^{-n_i/2}\cdot \end{array}$

Let Z = ${\Sigma }_{i=1}^m{\text{Y}}_i$ . By Corollary 1.3.4,

$\begin{array}{ll}{m}_Z\left(t\right)& =\underset{i=1}{\overset{m}{\Pi }}{m}_{{\text{Y}}_i}\left(t\right)\\ & =\left(1-2t\right)-{\Sigma }_{i=1^{n_i/2}}^m\cdot \end{array}$

Therefore, ${\Sigma }_{i=1}^m{\text{Y}}_i$ is distributed as a central chi-square random variable with ${\Sigma }_{i=1}^m{n}_i$ degrees of freedom.

The next theorem is useful when dealing with functions of independent random vectors.

Theorem 1.3.5

Let g ₁(Y ₁),.…g_m (Y _m ) be m functions of the random vectors Y ₁,…, Y _m, respectively. If Y ₁,…, Y _m are mutually independent, then g ₁,…, g_m are mutually independent.

The next example demonstrates that the sum of squares of n independent N₁(0, 1) random variables has a central chi-square distribution with n degrees of freedom.

Example 1.3.2

Let Z ₁,…, Z_n be a random sample of normally distributed random variables with mean 0 and variance 1. Let Y_i = Z_i ² for i = 1,…, n. The moment generating function of Y_i is

$\begin{array}{ll}{m}_{{\text{Y}}_i}\left(t\right)=m_{Z_i^2}\left(t\right)& =\text{E}\left(e^{t{Z}_i^2}\right)\\ & ={\int }_{-\infty }^{\infty }{\left(2\pi \right)}^{-1/2}{e}^{t{z}_i^2-z_i^2/2}d{z}_i\\ & ={\int }_{-\infty }^{\infty }{\left(2\pi \right)}^{-1/2}{e}^{-\left(1-2t\right)z_i^2/2}d{z}_i\\ & ={\left(1-2t\right)}^{-1/2}\cdot \end{array}$

That is, each Y_i , has a central chi-square distribution with one degree of freedom. Furthermore, by Theorem 1.3.5, the Y_i 's are independent random variables. Therefore, by Example 1.3.1, ${\Sigma }_{i=1}^n{\text{Y}}_i={\Sigma }_{i=1}^n{Z}_i^2$ is a central chi-square random variable with n degrees of freedom.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780125084659500016

Random Variables

Sheldon Ross , in Introduction to Probability Models (Eleventh Edition), 2014

2.5.1 Joint Distribution Functions

Thus far, we have concerned ourselves with the probability distribution of a single random variable. However, we are often interested in probability statements concerning two or more random variables. To deal with such probabilities, we define, for any two random variables $X$ and $Y$ , the joint cumulative probability distribution function of $X$ and $Y$ by

$F(a\text{,}b)=P\{X\le a\text{,}Y\le b\}\text{,}-\infty <a\text{,}b<\infty$

The distribution of $X$ can be obtained from the joint distribution of $X$ and $Y$ as follows:

$\begin{array}{cc}\hfill F_X(a)=& P\{X\le a\}\hfill \\ \hfill =& P\{X\le a\text{,}Y<\infty \}\hfill \\ \hfill =& F(a\text{,}\infty )\hfill \end{array}$

Similarly, the cumulative distribution function of $Y$ is given by

$F_Y(b)=P\{Y\le b\}=F(\infty \text{,}b)$

In the case where $X$ and $Y$ are both discrete random variables, it is convenient to define the joint probability mass function of $X$ and $Y$ by

$p(x\text{,}y)=P\{X=x\text{,}Y=y\}$

The probability mass function of $X$ may be obtained from $p(x\text{,}y)$ by

$p_X(x)={\displaystyle \sum _{y:p(x\text{,}y)>0}}p(x\text{,}y)$

Similarly,

$p_Y(y)={\displaystyle \sum _{x:p(x\text{,}y)>0}}p(x\text{,}y)$

We say that $X$ and $Y$ are jointly continuous if there exists a function $f(x\text{,}y)$ , defined for all real $x$ and $y$ , having the property that for all sets $A$ and $B$ of real numbers

$P\{X\in A\text{,}Y\in B\}={\int }_B{\int }_{A}f(x\text{,}y)\mathit{dx}\mathit{dy}$

The function $f(x\text{,}y)$ is called the joint probability density function of $X$ and $Y$ . The probability density of $X$ can be obtained from a knowledge of $f(x\text{,}y)$ by the following reasoning:

$\begin{array}{cc}\hfill P\{X\in A\}=& P\{X\in A\text{,}Y\in (-\infty \text{,}\infty )\}\hfill \\ \hfill =& {\int }_{-\infty }^{\infty }{\int }_{A}f(x\text{,}y)\mathit{dx}\mathit{dy}\hfill \\ \hfill =& {\int }_A{f}_X(x)\mathit{dx}\hfill \end{array}$

where

$f_X(x)={\int }_{-\infty }^{\infty }f(x\text{,}y)\mathit{dy}$

is thus the probability density function of $X$ . Similarly, the probability density function of $Y$ is given by

$f_Y(y)={\int }_{-\infty }^{\infty }f(x\text{,}y)\mathit{dx}$

Because

$F(a\text{,}b)=P(X\le a\text{,}Y\le b)={\int }_{-\infty }^a{\int }_{-\infty }^{b}f(x\text{,}y)\mathit{dy}\mathit{dx}$

differentiation yields

$\frac{d^2}{\mathit{da}\mathit{db}}F(a\text{,}b)=f(a\text{,}b)$

Thus, as in the single variable case, differentiating the probability distribution function gives the probability density function.

A variation of Proposition 2.1 states that if $X$ and $Y$ are random variables and $g$ is a function of two variables, then

$\begin{array}{cc}\hfill E[g(X\text{,}Y)]=& {\displaystyle \sum _y}{\displaystyle \sum _x}g(x\text{,}y)p(x\text{,}y)\mathrm{in}\mathrm{the}\mathrm{discrete}\mathrm{case}\hfill \\ \hfill =& {\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }g(x\text{,}y)f(x\text{,}y)\mathit{dx}\mathit{dy}\mathrm{in}\mathrm{the}\mathrm{continuous}\mathrm{case}\hfill \end{array}$

For example, if $g(X\text{,}Y)=X+Y$ , then, in the continuous case,

$\begin{array}{cc}\hfill E[X+Y]=& {\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }(x+y)f(x\text{,}y)\mathit{dx}\mathit{dy}\hfill \\ \hfill =& {\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }\mathit{xf}(x\text{,}y)\mathit{dx}\mathit{dy}+{\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }\mathit{yf}(x\text{,}y)\mathit{dx}\mathit{dy}\hfill \\ \hfill =& E[X]+E[Y]\hfill \end{array}$

where the first integral is evaluated by using the variation of Proposition 2.1 with $g(x\text{,}y)=x$ , and the second with $g(x\text{,}y)=y$ .

The same result holds in the discrete case and, combined with the corollary in Section 2.4.3, yields that for any constants $a\text{,}b$

(2.10) $E[\mathit{aX}+\mathit{bY}]=\mathit{aE}[X]+\mathit{bE}[Y]$

Joint probability distributions may also be defined for $n$ random variables. The details are exactly the same as when $n=2$ and are left as an exercise. The corresponding result to Equation (2.10) states that if $X_1\text{,}{X}_2\text{,}\dots \text{,}{X}_n$ are $n$ random variables, then for any $n$ constants $a_1\text{,}{a}_2\text{,}\dots \text{,}{a}_n$ ,

(2.11) $E[a_1{X}_1+a_2{X}_2+\cdots +a_n{X}_n]=a_{1}E[X_1]+a_{2}E[X_2]+\cdots +a_{n}E[X_n]$

Example 2.29

Calculate the expected sum obtained when three fair dice are rolled.

Solution:  Let $X$ denote the sum obtained. Then $X=X_1+X_2+X_3$ where $X_i$ represents the value of the $i$ th die. Thus,

$E[X]=E[X_1]+E[X_2]+E[X_3]=3\left(\frac{7}{2}\right)=\frac{21}{2}■$

Example 2.30

As another example of the usefulness of Equation (2.11), let us use it to obtain the expectation of a binomial random variable having parameters $n$ and $p$ . Recalling that such a random variable $X$ represents the number of successes in $n$ trials when each trial has probability $p$ of being a success, we have

$X=X_1+X_2+\cdots +X_n$

where

$X_i=\left\{\begin{array}{cc}1\text{,}\hfill & \mathrm{if}\mathrm{the}i\mathrm{th}\mathrm{trial}\mathrm{is}\mathrm{a}\mathrm{success}\hfill \\ 0\text{,}\hfill & \mathrm{if}\mathrm{the}i\mathrm{th}\mathrm{trial}\mathrm{is}\mathrm{a}\mathrm{failure}\hfill \end{array}\right.$

Hence, $X_i$ is a Bernoulli random variable having expectation $E[X_i]=1(p)+0(1-p)=p$ . Thus,

$E[X]=E[X_1]+E[X_2]+\cdots +E[X_n]=\mathit{np}$

This derivation should be compared with the one presented in Example 2.17.   ■

Example 2.31

At a party $N$ men throw their hats into the center of a room. The hats are mixed up and each man randomly selects one. Find the expected number of men who select their own hats.

Solution:  Letting $X$ denote the number of men that select their own hats, we can best compute $E[X]$ by noting that

$X=X_1+X_2+\cdots +X_N$

where

$X_i=\left\{\begin{array}{cc}1\text{,}\hfill & \mathrm{if}\mathrm{the}i\mathrm{th}\mathrm{man}\mathrm{selects}\mathrm{his}\mathrm{own}\mathrm{hat}\hfill \\ 0\text{,}\hfill & \mathrm{otherwise}\hfill \end{array}\right.$

Now, because the $i$ th man is equally likely to select any of the $N$ hats, it follows that

$P\{X_i=1\}=P\{i\mathrm{th}\mathrm{man}\mathrm{selects}\mathrm{his}\mathrm{own}\mathrm{hat}\}=\frac{1}{N}$

and so

$E[X_i]=1P\{X_i=1\}+0P\{X_i=0\}=\frac{1}{N}$

Hence, from Equation (2.11) we obtain

$E[X]=E[X_1]+\cdots +E[X_N]=\left(\frac{1}{N}\right)N=1$

Hence, no matter how many people are at the party, on the average exactly one of the men will select his own hat.   ■

Example 2.32

Suppose there are 25 different types of coupons and suppose that each time one obtains a coupon, it is equally likely to be any one of the 25 types. Compute the expected number of different types that are contained in a set of 10 coupons.

Solution:  Let $X$ denote the number of different types in the set of 10 coupons. We compute $E[X]$ by using the representation

$X=X_1+\cdots +X_{25}$

where

$X_i=\left\{\begin{array}{cc}1\text{,}\hfill & \mathrm{if}\mathrm{at}\mathrm{least}\mathrm{one}\mathrm{type}i\mathrm{coupon}\mathrm{is}\mathrm{in}\mathrm{the}\mathrm{set}\mathrm{of}10\hfill \\ 0\text{,}\hfill & \mathrm{otherwise}\hfill \end{array}\right.$

Now,

$\begin{array}{cc}\hfill E[X_i]=& P\{X_i=1\}\hfill \\ \hfill =& P\{\mathrm{at}\mathrm{least}\mathrm{one}\mathrm{type}i\mathrm{coupon}\mathrm{is}\mathrm{in}\mathrm{the}\mathrm{set}\mathrm{of}10\}\hfill \\ \hfill =& 1-P\{\mathrm{no}\mathrm{type}i\mathrm{coupons}\mathrm{are}\mathrm{in}\mathrm{the}\mathrm{set}\mathrm{of}10\}\hfill \\ \hfill =& 1-{\left(\frac{24}{25}\right)}^{10}\hfill \end{array}$

when the last equality follows since each of the 10 coupons will (independently) not be a type $i$ with probability $\frac{24}{25}$ . Hence,

$E[X]=E[X_1]+\cdots +E[X_{25}]=25\left[1-{\left(\frac{24}{25}\right)}^{10}\right]■$

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780124079489000025

RANDOM VARIABLES AND EXPECTATION

Sheldon M. Ross , in Introduction to Probability and Statistics for Engineers and Scientists (Fourth Edition), 2009

SOLUTION

We start by determining the distribution function of X/Y. For a > 0

$\begin{array}{l}{F}_{X/Y}(a)=P\{X/Y\le a\}\\ =\underset{x/y\le a}{\iint }f(x,y)dxdy\\ =\underset{x/y\le a}{\iint }{e}^{-x}{e}^{-y}dxdy\\ ={\int }_0^{\infty }{\int }_0^{ay}{e}^{-x}{e}^{-y}dxdy\\ ={\int }_0^{\infty }(1-e^{-ay})e^{-y}dy\\ =\left[-e^{-y}+\frac{e^{-(a+1)y}}{a+1}\right]{|}_0^{\infty }\\ =1-\frac{1}{a+1}\end{array}$

Differentiation yields that the density function of X/Y is given by

We can also define joint probability distributions for n random variables in exactly the same manner as we did for n = 2. For instance, the joint cumulative probability distribution function F (a ₁, a ₂,…, a_n ) of the n random variables X ₁, X ₂,…, X_n is defined by

$F(a_1,a_2,\dots ,a_n)=P\{X_1\le a_1,X_2\le a_2,\dots ,X_n\le a_n\}$

If these random variables are discrete, we define their joint probability mass function p (x ₁, X ₂,…, X ⁿ) by

$p(x_1,x_2,\dots ,x_n)=P\{X_1=x_1,X_2=x_2,\dots ,X_n=x_n\}$

Further, the n random variables are said to be jointly continuous if there exists a function f (x ₁, x ₂,…, x_n ), called the joint probability density function, such that for any set C in n -space

$P\{(X_1,X_2,\dots ,X_n)\in C\}=\int {\int }_{(x_1,\dots ,x_n)\in C}\dots \int f(x_1,\dots ,x_n)d{x}_{1}d{x}_2\dots d{x}_n$

In particular, for any n sets of real numbers A ₁, A ₂,…, A _n

$P\{X_1\in A_1,X_2\in A_2,\dots X_n\in A_n\}={\int }_{A_n}{\int }_{A_{n-1}}\dots {\int }_{A_1}f(x_1,\dots ,x_n)d{x}_{1}d{x}_2\dots d{x}_n$

The concept of independence may, of course, also be defined for more than two random variables. In general, the n random variables X ₁, X ₂,…, X_n are said to be independent if, for all sets of real numbers A ₁, A ₂,…, A_n ,

$P\{X_1\in A_1,X_2\in A_2,\dots ,X_n\in A_n\}=\prod _{i=1}^{n}P\{X_i\in A_i\}$

As before, it can be shown that this condition is equivalent to

$\begin{array}{l}P\{X_1\le a_1,X_2\le a_2,\dots ,X_n\le a_n\}\\ =\prod _{i=1}^{n}P\{X_1\le a_j\}\text{for}\text{all}{a}_1,a_2,\dots a_n\end{array}$

Finally, we say that an infinite collection of random variables is independent if every finite subcollection of them is independent.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780123704832000096

Graphical Models

R.G. Almond , in International Encyclopedia of Education (Third Edition), 2010

Factorization and Conditional Independence

A graphical model is a joint probability distribution over a collection of variables that can be factored according to the cliques of an undirected graph. Let $\mathcal{G}=〈v,\varepsilon 〉$ be a graph whose nodes correspond to the variables in the model, and let C be the set of cliques in the graph. Let v be an instantiation of the values in ν and let v _C be the corresponding set of values for the variables in C   ⊂   ν. Then we can write the joint probability distribution as:

[1] $p(\text{v})={\displaystyle \prod _{\text{C}\in C}}{\varphi }_{\text{C}}({\text{v}}_{\text{C}})$

In this equation, p(v) should be interpreted as a probability mass function if all of the variables in ν are discrete, and a density function if any of the variables are continuous. The factors, ϕ _C(·) are called potentials (sometimes Gibbs potentials; the term comes from statistical physics). The potentials can be probability distributions, conditional probability distributions, or products of any of the above. Generally, they are not interpreted directly and often they are left unnormalized.

The original use of the term graphical model was to distinguish a subset of the number of possible log-linear models for contingency tables that could be represented as a graph. Consider the set of possible models over three discrete variables A, B, and C. The models [A][B][C], [A][BC], [AB][BC], and [ABC] (and ones that are equivalent after relabeling the variables) are all graphical – the factors correspond to the cliques. The no-three-way interaction model [AB][BC][CA] is not graphical because the clique should be [ABC] (the three factors do not correspond to cliques). Difference in the substantive interpretation of the no-three-way interaction model from the three-way interaction model is quite subtle; thus, restricting consideration to graphical models does not greatly limit the interpretability. Although the no-three-way interaction model has fewer parameters, it also is more difficult to work with, requiring iterative solutions to fit the model to data.

Restricting the variables to have a multivariate normal distribution creates a set of models corresponding to covariance selection models. (Note that for normally distributed data, the pairwise correlations define the joint distribution, so all models can be expressed with a graph, although it might be the saturated graph in which all nodes are connected.) In fact, there are three classes of graphical models for which most computations can be done exactly: (1) models in which all variables are discrete (graphical log-linear models), (2) multivariate normal distributions, and (3) finite mixtures of multivariate normal distributions. The graphical notation can be used to express other kinds of models, but operations on those kinds of models frequently involves integrals that must be solved numerically. (For example, the standard unidimensional item response theory model can be drawn as a graphical model with the observable outcome variables conditionally independent given the latent ability variable. However, the model is not a conditional Gaussian distribution; therefore estimating examinee ability using this model requires numeric integration.)

The Markov property of a graphical model states that if X and Y are variables in the graph, and S is a set that separates X from Y in the graph, then X is conditionally independent of Y given S, sometimes written X ∐ Y|S. This Markov property drives a number of efficient algorithms for computation, as well as aiding in the interpretation of the model. Probability distributions with this property are sometimes called Markov random fields.

Consider the well-known example of Simpson's paradox in which a university has several departments, each of which admits a different proportion of their applicants, and each of which has a different proportion of women in their applicant pool. Further assume that each department is completely gender neutral in their admissions decisions. The graph shown in Figure 2 illustrates these assumptions. The nodes gender and acceptance are connected in the graph, so they are marginally dependent. Depending on whether women prefer more- or less-selective departments, the university will appear biased against or toward women. The graph tells the story visually: it is the preference of women for more (or less) selective departments that causes the marginal dependency.

Figure 2. A Graphical model illustrating simpson's paradox.

A fundamental result in the theory of graphical models is that factorization (according to a graph) implies the Markov property (with respect to that graph) and vice versa. This is sometimes known as the Gibbs–Markov equivalence theorem. Going from the factorization to the Markov property requires no special assumptions, but going from the conditional independence statements to the factorization requires that the probability for all configurations of the variables be positive. This can present a problem if there is a deterministic constraint between two or more variables; however, in practical circumstances, it can usually be made to work.

The Gibbs–Markov equivalence property has particular salience when building graphical models from expert opinion. The usual practice is to first elicit the conditional independence properties, then specify the probabilities for the factors defined by the cliques of the graph.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780080448947013348

Crystal Structure Determination

R. Spagna , in Encyclopedia of Condensed Matter Physics, 2005

Probability Methods

Probability methods were introduced by Hauptman and Karle, and led to the joint probability distributions of a set of normalized structure factors on the basis that the atomic coordinates were the primitive random variables uniformly and independently distributed, and the reciprocal vectors were assumed to be fixed. It may also be assumed that the reciprocal vectors are the primitive random variables while the crystal structure is fixed. The probabilistic approach yielded formulas to calculate phases or combination of phases which are s.i.'s by the | E _h | alone. Considering the most important class of s.i's, the three-phase s.i. (triplets), the distribution associated derived by Cochran for a non-centrosymmetric structure is given by:

$P({\text{Φ}}_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">hk</mi> </mi>}})=\frac{1}{L}\text{exp}(G_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">hk</mi> </mi>}}\text{cos}{\text{Φ}}_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">hk</mi> </mi>}})$

where L is a normalization factor and for equal atoms

$G_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">hk</mi> </mi>}}=\frac{2}{\sqrt{N}}|E_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">h</mi> </mi>}}{E}_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}}{E}_{-\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">h</mi> </mi>}-\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}}|$

where N is the number of atoms in the unit cell.

In Figure 13, the probability distributions for different values of the parameter G _hk are shown to have a maximum at Φ _hk equal to zero, and the variance decreases as G _hk increases.

Figure 13. Trends of the probability distributions for different values of the parameter G _hk .

In centrosymmetric crystals, the relation becomes:

$S(-\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">h</mi> </mi>})S\left(\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}\right)S(\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">h</mi> </mi>}-\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>})\cong +$

where S( h ) stands for the sign of the reflection h and the symbol ≅ stands for "probably equal." In this case, the basic conditional formula for sign determination is given by Cochran and Woolfson:

$P_{+}=\frac{1}{2}+\frac{1}{2}\text{tanh}(\frac{1}{\sqrt{N}}|E_{-\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">h</mi> </mi>}}{E}_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}}{E}_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">h</mi> </mi>}-\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}}|)$

The larger the absolute value of the argument of tanh, the more reliable is the sign indication.

For fixed h , let the vector k range over the set of known values E _k E _{− h − k} , then the total probability distribution of φ _h is given by the product of the single distributions, increasing the reliability of the estimate, in mathematical terms.

Calculation of the conditional joint probability distributions for φ _h , under the condition that several phases φ _k and φ _{h − k} are known, led to the tangent formula:

$\text{tan}{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">ϕ</mi> </mi>}}_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">h</mi> </mi>}}=\frac{{\sum }_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}}|E_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}}{E}_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">h</mi> </mi>}-\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}}|\text{sin}({\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">ϕ</mi> </mi>}}_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}}+{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">ϕ</mi> </mi>}}_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">h</mi> </mi>}-\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}})}{{\sum }_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}}|E_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}}{E}_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">h</mi> </mi>}-\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}}|\text{cos}({\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">ϕ</mi> </mi>}}_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}}+{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">ϕ</mi> </mi>}}_{\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">h</mi> </mi>}-\text{<mi mathvariant="bold" is="true"> <mi mathvariant="italic" is="true">k</mi> </mi>}})}$

in which the summations are taken over the same sample of reciprocal vectors k .

Formulations of the "nested neighborhood principle" and of the "representation theory" gave the basis to obtain joint probability distributions which improve the estimates for s.i.'s Furthermore, the latter formulated precise general rules for identifying the phasing magnitudes using the space-group symmetry. More complex formulas derived were also able to give indication of Φ _hk away from 0°.

Modified tangent techniques are used in almost all computer programs for the phase determination process.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B0123694019004253

Stochastic Dynamics

Don Kulasiri , Wynand Verwoerd , in North-Holland Series in Applied Mathematics and Mechanics, 2002

2.8 Relationship between White Noise and Brownian Motion

consider a stochastic process X(t,ω ) having a stationary joint probability distribution and E(X(t,ω))=0, i.e. the mean value of the process is zero. The Fourier transform of Var(X (t,ω)) can be written as,

(2.23) $S\left(\lambda ,\omega \right)=\frac{1}{2\pi }{\int }_{-\infty }^{\infty }var(X\left(\tau ,\omega \right)e^{-\mathit{i\lambda \tau }}\mathit{d\tau }$

S(λ,ω) is called the spectral density of the process X(t,ω) and is also a function of angular frequency λ. The inverse of the Fourier transform is given by

(2.24) $\mathit{Var}\left(X\left(\tau ,\omega \right)\right)={\int }_{-\infty }^{\infty }S\left(\lambda ,\omega \right)e^{\mathit{i\lambda \tau }}\mathit{d\lambda }$

and when τ = 0,

(2.25) $\mathit{Var}\left(X\left(0,\omega \right)\right)={\int }_{-\infty }^{\infty }S\left(\lambda ,\omega \right)\mathit{d\lambda }\text{.}$

Therefore, variance of X(0,ω)is the area under a graph of spectral density S(λ,ω) against λ:

(2.26) $\mathit{Var}\left(X\left(0,\omega \right)\right)=E\left(X^2\left(0,\omega \right)\right)\text{,}$

because E(X(t,ω))= 0

Spectral density S(λ,ω) is considered as the "average power" per unit frequency at λ which gives rise to the variance of X(t, ω) at τ = o. If the average power is a constant which means that the power is distributed uniformly across the frequency spectrum, such as the case for white light, then X(t,ω) is called white noise. White noise is often used to model independent random disturbances in engineering systems, and the increments of Brownian motion have the same characteristics as white noise. Therefore white noise (ζ(t)) is defined as

$\zeta \left(t\right)=\frac{\mathit{dB}\left(t\right)}{\mathit{dt}}\text{,}$

(2.27) $\mathit{dB}\left(t\right)=\zeta \left(t\right)\mathit{dt}\text{.}$

We will use this relationship to formulate stochastic differential equations.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/S0167593102800031

Financial Markets with Continuous Time

Yuliya Mishura , in Finance Mathematics, 2016

2.5.3 Pricing look-back options in the Black-Scholes model

In order to evaluate the look-back option in some market model, we need to know the joint probability distribution of the final value S _T of underlying asset's price and minimal value S _T ^min and/or maximal value S _T ^max of underlying asset's price on the interval [0, T]. Unfortunately, the joint distribution of these three random variables has a simple form only in Bachelier model: if W is a Wiener process, W _T ^min and W _T ^max are its minimal and maximal values on the interval [0, T], then the joint distribution for $x\in \left[a,b\right]$ is given by the following formula:

$\begin{array}{l}\mathbb{P}\left(a\le W_T^{\text{min}}\le W_T^{\text{max}}\le b,W_T\in dx\right)\\ =\frac{1}{\sqrt[]{2\pi t}}{\displaystyle \sum _{k=-\infty }^{\infty }}\left(exp\left\{-\frac{y_k^2\left(x\right)}{2t}\right\}-exp\left\{-\frac{{\left(yk\left(x\right)-2b\right)}^2}{2\mathrm{t}}\right\}\right)dx\text{,}\end{array}$

where $yk\left(x\right)=x+2k\left(b-a\right)\text{.}$ For the Black-Scholes model, there is no such explicit formula. However, there is a fairly simple formula for the joint distribution of the final value and maximal value (for the joint distribution of the final value and minimal value), which is sufficient for look-back options and barrier options. Let $X_t=W_t+\mu t+x$ be a Wiener process with drift coefficient μ and initial value $x,X_T^{\text{max}}=\underset{t\in \left[0,T\right]}{max}{X}_t$ is its maximal value on the interval [0, T], and t _max is a random point where this maximal value is achieved. Then, for $zVx\le y\text{,}$ the joint distribution is given by the formula

$\begin{array}{l}\mathbb{P}\left(X_T\in dz,X_T^{\text{max}}\in \mathit{dy},t_{\text{max}}\in dt\right)=\frac{\left(y-x\right)\left(y-z\right)}{\pi \sqrt[]{t^3{\left(T-t\right)}^3}}\\ \times exp\left\{-\frac{{\left(y-x\right)}^2}{2t}-\frac{{\left(y-z\right)}^2}{2\left(T-t\right)}-\mu \left(x-z\right)-\frac{{\mu }^{2}T}{2}\right\}dz\mathit{dy}dt\end{array}$

=: pT,x,μ(z, y, t)dz dy dt.

Therefore, the joint probability density of X _T and X _T ^max equals ${\displaystyle \underset{0}{\overset{T}{\int }}}PT,x,\mu \left(z,y,t\right)dt\text{.}$ The joint probability density of X _T and X _t ^min is readily determined from this formula, if we have to consider the process $-X_t\text{,}$ that is also a Wiener process with a constant drift coefficient. Consider the Black-Scholes model of the financial market. Recall that w.r.t. the martingale measure this model has the form $B_t=e^{rt},S_t=S_0{e}^{\left(r-{\sigma }^2/2\right)t+\sigma W_t}\text{.}$ Let the look-back derivative have the payoff function $C=\left(S_T,S_T^{\text{max}}\right)$ . Taking into account that log S _t is a Wiener process with the drift coefficient $\mu =r-{\sigma }^2/2$ and initial value $x=log{S}_0$ , we can determine the price of the corresponding discounted option by the formula

$\begin{array}{c}\hfill \pi \left(D\right)=\mathbb{E}\left({\mathrm{e}}^{-\mathrm{r}\mathrm{T}}\mathrm{C}\right)=e^{-rT}{\displaystyle \underset{x}{\overset{\infty }{\int }}}{{\displaystyle \int }}_{\mathbb{R}}g\left(e^z,e^y\right){\displaystyle \underset{0}{\overset{T}{\int }}}PT,x,\mu \left(z,y,t\right)dtdz\mathit{dy}\hfill \\ \hfill =e^{-rT}{\displaystyle \underset{S_0}{\overset{\infty }{\int }}}{\displaystyle \underset{0}{\overset{\infty }{\int }}}\frac{g\left(z,y\right)}{zy}{\displaystyle \underset{0}{\overset{T}{\int }}}PT,x,\mu \left(logz,logy,t\right)dtdz\mathit{dy}\text{.}\hfill \end{array}$

For specific options, this formula can be converted and we can obtain rather simple expressions. In particular, the look-back call option price with floating strike equals

$S_0\mathrm{\Phi }\left(a_1\left(S_0,S_0\right)\right)-S_0{e}^{-rT}\mathrm{\Phi }\left({\mathrm{a}}_2\left({\mathrm{S}}_0,{\mathrm{S}}_0\right)\right)$

$=\frac{S_0{\sigma }^2}{2r}\left(\mathrm{\Phi }\left(-a_1\left(S_0,S_0\right)\right)-e^{-rT}\mathrm{\Phi }\left(-a_3\left(S_0,S_0\right)\right)\right)\text{,}$

Where for $x>0$ and $y>0$

$\begin{array}{c}\hfill a_1\left(x,y\right)=\frac{\log \left(\frac{x}{y}\right)+\left(\raisebox{1ex}{$r+{\sigma }^2$}\!\left/ \!\raisebox{-1ex}{$2$}\right.\right)T}{\sigma \sqrt[]{T},}\hfill \\ \hfill a_2\left(x,y\right)=\frac{\log \left(\frac{x}{y}\right)-\left(r-\raisebox{1ex}{${\sigma }^2$}\!\left/ \!\raisebox{-1ex}{$2$}\right.\right)T}{\sigma \sqrt[]{T}}=a_1\left(x,y\right)-\sigma \sqrt[]{T},\hfill \\ \hfill a_3\left(x,y\right)=\frac{\log \left(\frac{x}{y}\right)-\left(r-\raisebox{1ex}{${\sigma }^2$}\!\left/ \!\raisebox{-1ex}{$2$}\right.\right)T}{\sigma \sqrt[]{T}}=a_1\left(x,y\right)-2r\sqrt[]{T}\text{,}\hfill \end{array}$

and Φ is, as usual, a standard Gaussian distribution function. The look-back call option price with fixed strike equals

$S_0\mathrm{\Phi }\left({\mathrm{a}}_1\left(S_0,K\right)\right)-K{e}^{-rT}\mathrm{\Phi }\left(a_2\left(S_0,K\right)\right)+$

$+\frac{S_0{\sigma }^2}{2r}\left(\mathrm{\Phi }\left(a_1,\left(S_0,K\right)\right)-e^{-rT}{\left(\raisebox{1ex}{$K$}\!\left/ \!\raisebox{-1ex}{$S$}\right.\right)}^{\frac{2r}{{\sigma }^2}}\mathrm{\Phi }\left({\mathrm{a}}_3\left({\mathrm{S}}_0,\mathrm{K}\right)\right)\right)$

for $S_0\le K$ and

$\left(S_0-K\right)e^{-rT}+S_0\mathrm{\Phi }\left({\mathrm{a}}_1\left({\mathrm{S}}_0,{\mathrm{S}}_0\right)\right)-{\mathrm{S}}_0{\mathrm{e}}^{-\mathrm{r}\mathrm{T}}\mathrm{\Phi }\left({\mathrm{a}}_2\left({\mathrm{S}}_0,{\mathrm{S}}_0\right)\right)+$

$+\frac{S_0{\sigma }^2}{2r}\left(\mathrm{\Phi }\left(a_1\left(S_0,S_0\right)\right)-e^{-rT}\mathrm{\Phi }\left(a_3\left(S_0,S_0\right)\right)\right)$

for $S_0>K$ . It is easy to understand the dependence of the option value on K and S ₀: for $S_0>K$ , option is "in money". Other formulas for look-back options, including pricing formula for any time, are discussed in [MUS 02].

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9781785480461500028

OCKHAM'S RAZOR, TRUTH, AND INFORMATION

Kevin T. Kelly , in Philosophy of Information, 2008

5 SOME EXAMPLES

For some guidance in the general developments that follow, consider some familiar examples.

Polynomial structures. Let S be a finite set of natural numbers and suppose that the truth is some unknown polynomial law:

$y=f\left(x\right)=\underset{i\in S}{\Sigma }{a}_i{x}^i,$

where for each i ∈ S, a_i ≠ 0. Say that S is the structure of the law, as it determines the form of the law as it would be written in a textbook. Suppose that the problem is to infer the true structure S of the law. It is implausible to suppose that for a given value of the independent variable x one could observe the exact value of the dependent variable y, so suppose that for each queried value of x at stage k of inquiry, the scientist receives an arbitrarily small, open interval around the corresponding value of y and that repeated queries of x result in an infinite sequence of open intervals converging to {y}.

It is impossible to be sure that one has selected S correctly by any finite time, since there may be some i ∈ S such that a ₁ is set to a very small value in f, making it appear that the monomial a_i xⁱ is missing from f. Ockham's razor urges the conclusion that i ∉ S until the corresponding monomial is noticed in the data.

There is a connection between the complexity of the true polynomial structure and what scientists and engineers call effects. Suppose that S ₀ = {0}, so for some a_i > 0, f ₀(x) = a_i . Let experience e _o present a finite sequence of interval observations of the sort just described for f₀ . Then there is a bit of wiggle room in each such interval, so that for some suitably small a ₁ > 0, the curve f ₁(x) = a ₁ x + a ₀ of form S ₁ = {0, 1} is compatible with e ₀. Eventually, some open interval around y = a ₀ is presented that excludes f ₀. Call such information a first-order effect. If e₁ extends that information and presents an arbitrary, finite number of shrinking, open intervals around f ₁ then, again, there exists suitably small a ₂ > 0 such that f ₂(x) = a₂x² + a ₁ x + a ₀ of form S₂ = {0, 1, 2} passes through each of the intervals presented in e ₁. Eventually, the intervals tighten so that no linear curve passes between them. Call such information a second-order effect, and so forth. The number of effects presented by a world corresponds to the cardinality of S, so there is a correspondence between empirical effects and empirical complexity. A general account of empirical effects is provided in Section 16 below.

Linear dependence. Suppose that the truth is a multivariate linear law

$y=f\left(x\right)=\underset{i\in S}{\Sigma }{a}_i{x}^i,$

where for each i ∈ S, a_i ≠ 0. Again, the problem is to infer the structure S of f. Let the data be presented as in the preceding example. As before, it seems that complexity corresponds with the cardinality of S which is connected, in turn, to the number of effects presented by nature if f is true.

Conservation laws. Consider an idealized version of explaining reactions with conservation laws, as in the theory of elementary particles [Schulte, 2001; Valdez-Perez, 1996]. Suppose that there are n observable types of particles, and it is assumed that they interact so as to conserve n distinct quantities. In other words, each particle of type P_i carries a specific amount of each of the conserved quantities and for each of the conserved quantities, the total amount of that quantity going into an arbitrary reaction must be the total amount that emerges. Usually, one thinks of a reaction in terms of inputs and outputs; e.g.,

$r=\left(p_1,p_1,p_1,p_2,p_2,p_3,\to p_1,p_1,p_2,p_3,p_3,\right).$

One can represent the inputs by a vector in which entry i is the number of input particles of type p_i in r, and similarly for the output:

$\begin{array}{ccc}a& =& \left(\mathrm{3,2,1}\right);\\ b& =& \left(\mathrm{2,1,2}\right);\\ r& =& \left(a\to b\right);\end{array}$

A quantity q (e.g., mass or spin) is an assignment of real numbers to particle types, as in q = (1, 0, 1), which says that particles a ₁, a ₃ both carry a unit of q and a ₂ carries none. Quantity q is conserved in r just in case the total q in is the total q out. That condition is just:

$\underset{i=1}{\overset{3}{\Sigma }}{q}_i{a}_i=\underset{i=1}{\overset{3}{\Sigma }}{q}_i{b}_i,$

or, in vector notation,

$q\cdot a=q\cdot b,$

which is equivalent to:

$q\cdot \left(a-b\right)=0.$

Since reaction r enters the condition for conservation solely as the vector difference a – b, there is no harm, so far as conservation is concerned, in identifying reaction r with the difference vector:

$r=a-b=\left(\mathrm{1,1},-1\right).$

Then the condition for r conserving q can be rewritten succinctly as:

$q\cdot r=0$

which is the familiar condition for geometrical orthogonality of q with r. Thus, the reactions that preserve quantity q are precisely the integer-valued vectors orthogonal to q. In this example, r does conserve q, for:

$\left(\mathrm{1,0,1}\right)\cdot \left(\mathrm{1,1},-1\right)=1+0-1=0.$

But so do reactions u = (1, 0, −1) and v = (0, 1, 0), which are linearly independent. Since the subspace of vectors orthogonal to q is two-dimensional, every reaction that conserves q is a linear combination of u and v (e.g., r = u + v). If the only conserved quantity were q, then it would be strange to observe only scalar multiples of r. In that case, one would expect that the possible reactions are constrained by some other conserved quantity linearly independent of q, say q' = (0, 1, 1). Now the possible reactions lie along the intersection of the planes respectively orthogonal to q and q', which are precisely the scalar multiples of r. Notice that any two linearly independent quantities orthogonal to r would suffice — the quantities, themselves, are not uniquely determined.

Now suppose that the problem is to determine how many quantities are conserved, assuming that some conservation theory is true and that every possible reaction is observed, eventually. Let an "effect" be the observation of a reaction linearly independent of the reactions seen so far. As in the preceding applications, effects may appear at any time but cannot be taken back after they occur and the correct answer is uniquely determined by the (finite) number of effects that occur.

In this example, favoring the answer that corresponds to the fewest effects corresponds to positing the greatest possible number of conserved quantities, which corresponds to physical practice (cf. [Ford, 1963]). In this case, simplicity intuitions are consonant with testability and explanation, but run counter to minimization of free parameters (posited conserved quantities).

Discovering causal structure. If one does not have access to experimental data, due to cost, feasibility, or ethical considerations, one must base one's policy recommendations on purely observational data. In spite of the usual advice that correlation does not imply causation, sometimes it does. The following setup is based upon [Spirtes et al., 2000]. Let V be a finite set of empirical variables. A causal structure associates with each unordered pair of variables {X, Y} one of the following statements:

$X\to Y;X\leftarrow Y;X\left|\right|Y;$

interpreted, respectively, as X is a direct cause of Y, Y is a direct cause of X, and X, Y have no direct causal connection. The first two cases are direct causal connections and the fourth case denies such a connection. A causal structure can, therefore, be presented as a directed, acyclic graph (DAG) in which variables are vertices and arrows are direct causal connections. The notation X — Y means that there is a direct connection in either direction between X and Y without specifying which. A partially oriented graph with such ambiguous edges is understood, for present purposes, to represent the disjunction of the structures that result from specifying them in each possible way.

At the core of the approach is a rule for associating causal structures with probability distributions. Let p be a joint probability distribution on variables V. If S is a subset of V, let (X ∐ Y)|S abbreviate that X is statistically independent of Y conditional on S in p. A sequence of variables is a path if each successive pair is immediately causally connected. A collision on a path is a variable with arrows coming in from adjacent variables on the path (e.g., variable Y in path X → Y ← Z). A path is activated by variable set S just in case the only variables in S that occur on the path are collisions and every collision on the path has a descendent in S. Then the key assumption relating probabilities to causal structures is simply:

(X ∐ Y)|S if and only if no path between X and Y is activated by S.

Let T_p denote the set of all causal structures satisfying this relation to probability measure p.

To see why it is intuitive to associate T_p with p, suppose that X → Y → Z and that none of these variables are in conditioning set S. Then knowing something about Z tells one something about X and knowing something about the value of X tells one something about Z. But the ultimate cause X yields no further information about Z when the intermediate cause Y is known (unless there is some other activated path between X and Y). On the other hand, suppose that the path is X → Y ← Z with collision Y. If there is no further path connecting X with Z, knowing about X says nothing about Z (X and Z are independent causes of Y), but since X and Z may cooperate or compete in a systematic way to produce Y, knowing the value of Y together with the value of X yields some information about the corresponding setting of Z. The dependency among causes given the state of the common effect turns out to be an important clue to causal orientation.

It follows from the preceding assumption that there is a direct connection X — Y just in case X and Y are dependent conditional on each set of variables not including X, Y. There is a collision (X → Y ← Z) if (X – Y – Z) holds (by the preceding rule) and (X — Z) does not hold (by the preceding rule) and, furthermore, X, Z are dependent given every set of variables including Y but not X, Z [Spirtes et al., 2000, theorem 3.4]. Further causal orientations may be entailed in light of background assumptions. The preceding rules (actually, more computationally efficient heuristic versions thereof) have been implemented in "data-mining" software packages that search for causal structures governing large sets of observational variables. The key points to remember are that (1) a direct causal connection is implied by the appearance of some set of statistical dependencies and (2) edge orientations depend both on the appearance of some statistical dependencies and on the non-appearance in the future of further statistical dependencies.

The above considerations are taken to be general. However, much of the literature on causal discovery focuses on two special cases. In the discrete multinomial case, say that G ∈ D_g if and only if G ∈ T_p and p is a discrete, joint distribution over a finite range of possible values for each variable in G. In the linear Gaussian case, say that G ∈ L_p if and only if G ∈ T_p and p is generated from G as follows: each variable in G is assumed to be a linear function of its parents, together with an extra, normally distributed, unobserved variable called an error term and the error terms are assumed to be uncorrelated. For brevity, say that G is standard for p if and only if G ∈ D_p or G ∈ L_p . The following discussion is restricted to the standard cases because that is where matters are best understood at present.

In practice, not all variables are measured, but assume, optimistically, that all causally relevant variables are measured. Even then, in the standard cases, the DAGs in T_p cannot possibly be distinguished from one another from samples drawn from p, so one may as well require only convergence to T_p in each p compatible with background assumptions. ⁴

Statistical dependencies among variables must be inferred from finite samples, which can result in spurious causal conclusions because finite samples cannot reliably distinguish statistical independence from weak statistical dependence. Idealizing, as in the preceding examples, suppose that one receives the outputs of a data-processing laboratory that merely informs one of the dependencies ⁵ that have been verified so far (at the current, growing sample size) by a standard statistical dependency test, where the null hypothesis is independence. ⁶ Think of an effect as data verifying that a partial correlation is non-zero. Absence of an effect is compatible with noticing it later (the correlation could be arbitrarily small). If it is required only that one infer the true indistinguishability class T(p) for arbitrary p representable by a DAG, then effects determine the right answer.

What does Ockham say? In the light of the preceding examples, something like: assume no more dependencies than one has seen so far, unless background knowledge and other dependencies entail them. It follows, straightforwardly, that direct causal connections add complexity, and that seems intuitively right. Causal orientation of causal connections is more interesting. It may seem that causal orientation does affect complexity, because, with binary variables, a common effect depends in some manner that must be specified upon four states of the joint causes whereas a common cause affects each effect with just two states. Usually, free parameters contribute to complexity, as in the curve-fitting example above. But given the overall assumptions of causal discovery, a result due to Chickering [2003] implies that these extra parameters do not correspond to potential empirical effects and, hence, do not really contribute to empirical complexity. In other words, given that no further edges are coming, one can afford to wait for data that decide all the discernable facts about orientation [Schulte, 2007]. Standard MDL procedures that tax free parameters can favor non-collisions over collisions before the data resolve the issue, risking extra surprises. ⁷

For example, when there are three variables X, Y, Z and (X − Y −Z) is known, then, excluding unobserved causes, there are two equivalence classes of graphs, the collision orientation (X → Y ← Z) in one class C and all the other orientations in the complementary class ¬C. Looking at the total set of implied dependencies for C, C′, it turns out that the only differences are that C entails ¬((X ∐ Z)|Y) but not ¬(X ∐ Z), whereas ¬C entails ¬(X ∐ Z) but not ¬((X ∐ Z)|Y), so there is no inclusion relationship between the dependencies characterizing C and the dependencies characterizing ¬C. Therefore, both hypotheses are among the simplest compatible with the data, so Ockham's razor does not choose among them. Moreover, given that the truth is (X — Y — Z), nature must present either ¬(X ∐ Z) or ¬(X ∐ Z)|Y) eventually (given that the causal truth can be represented by some graph over the observable variables) so it seems that science can and should wait for nature to resolve the matter instead of racing ahead — and that is just how Ockham's razor is interpreted in the following discussion. Regardless of which effect nature elects to present, it remains possible, thereafter, to present the other effect as well, in which case each variable is connected immediately to every other and one can infer nothing about causal directionality. This situation involves more effects than either of the two preceding cases, but another direct causal connection is also added, reflecting the increase in complexity.

The preceding evolution can result in spectacular reversals of causal conclusions as experience increases, not just in terms of truth, but in terms of practical consequences as well. Suppose that it is known that (X → Y —Z) and none of these variables has yet exhibited any dependence with W. Then discovery of ¬((X ∐ Z)|Y), background knowledge, and Ockham's razor unambiguously imply (X → Y ← Z), a golden invitation to exploit Z to control Y. Indeed, the connections may be obvious and strong, inviting one to invest serious resources to exploit Z. But the conclusion rests entirely on Ockham's razor, for the further discovery of ¬(X ∐ Z) is incompatible with (X → Y ← Z) and the new Ockham answer is (X → Y – Z) with edge (X – Z) added. Further discovery that ¬((Z ∐ W)|X, Y) and that ¬((Y ∐ W)|Z) results in the conclusion Y → Z ← W, reversing the original

conclusion that Y can be controlled by Z. ⁸ The orientation of the direct causal connection Y — Z can be flipped n times in sequence by assuming causes X ₀, … X_n of Y in the role of X and potential collisions W ₀, …, W_n in the role of W. There is no way that a convergent strategy can avoid such discrete flips of Y — Z; they are an ineluctable feature of the problem of determining the efficacy of Z on Y from non-experimental data, no matter how strong the estimate of the strength of the cause Y → Z is prior to the reversal. Indeed, standard causal discovery algorithms exhibit the diachronic retractions just discussed in computer simulations. The practical consequences of getting the edge orientation wrong are momentous, for if Z does not cause Y, the policy of manipulating Z to achieve results for Y will have no benefits at all to justify its cost. Indeed, in the case just described, sample size imposes no non-trivial bound on arbitrarily large mis-estimates of the effectiveness of Y in controlling Z (cf. [Robins et al., 2003; Zhang and Spirtes, 2003]). Therefore a skeptical stance toward causal inference is tempting:

We could try to learn the correct causal graph from data but this is dangerous. In fact it is impossible with two variables. With more than two variables there are methods that can find the causal graph under certain assumptions but they are large sample methods and, furthermore, there is no way to ever know if the sample size you have is large enough to make the methods reliable [Wasserman, 2003, p. 275].

This skepticism is one more symptom of the unrealizable demand that simplicity should reliably point toward or inform one of the true theoretical structure, a popular — if infeasible — view both in statistics and philosophy [Goldman, 1986; Mayo, 1996; Dretske, 1981]. The approach developed below is quite different: insofar as finding the truth makes reversals of opinion unavoidable, they are not only justified but laudable — whereas, insofar as they are avoidable, they should be avoided. So the best possible strategies are those that converge to the truth with as few course-reversals as possible. That is what standard causal inference algorithms tend to do, and it is the best they could possibly do in the standard cases.

To summarize, an adequate explanation of Ockham's razor should isolate what is common to the simplicity intuitions in examples like the preceding ones and should also explain how favoring the simplest theory compatible with experience helps one find the truth more directly or efficiently than competing strategies when infallibility or even probable infallibility is hopeless. Such an explanation, along the lines of the freeway metaphor, will now be presented. First, simplicity and efficient convergence to the truth must be defined with mathematical rigor and then a proper proof must be provided that Ockham's razor is the most efficient possible strategy for converging to the truth.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780444517265500145

Clinical Decisions Based on Models

R.H. Riffenburgh , in Statistics in Medicine (Third Edition), 2012

The Gibbs Sampler

We know that each of the probability estimates in the transition matrix has a probability distribution and so together they have a joint probability distribution. The Gibbs sampler, named for physicist J.W. Gibbs but developed by the Geman brothers³⁸, is an algorithm for sampling randomly for each parameter conditional upon (often said "given") all the others to find the conditional distribution for each and use the collection of all the conditionals to find the joint distribution of the set of parameters.

The method will be easier to follow couched in terms of a simple clinical example. Let us return to the flu example. We have a vector (a list) of estimates on the number of patients falling into each state (well, slightly ill, severely ill, recovered, and dead) and a probability distribution for each state. The vector will also have a joint probability distribution. A random frequency is drawn from each distribution in turn. However, each of these frequencies will depend on all the other influences. The number slightly ill will depend on (be conditional on) the number who have become severely ill, the number who have died, and so forth. We can develop a joint distribution from the conditionals, but each time we find a new probability, it changes all the others. The Gibbs sampler adjusts the joint distribution for each conditional in turn, generating a second vector of frequencies. But this second vector is again subject to dependencies on subsequent conditionals, so must "loop" through the same path to a third, and so forth. This is a form of simulation. In most cases, the sequence of computations exhibits smaller and smaller changes with successive simulations so that at some point, perhaps after thousands of "loops", the probabilities become stable, that is, fixed or stationary. Then we have dependable estimates of what the rates for the various flu states really are.

Computer-based sampling and processing is necessary because the simulation is very computationally intensive. The Gibbs sampler provides a mechanism to pass quickly through thousands of loops.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780123848642000202

Computational Analysis and Understanding of Natural Languages: Principles, Methods and Applications

Indranil Ghosh , in Handbook of Statistics, 2018

3 Conceptual Exercises

1.

Let us have an arbitrary set of (conditional) independence relationships among N variables that is associated with a joint probability distribution.

(a)

Can we always find a DAG that perfectly maps this set (perfectly maps = preserves all the (conditional) independence relationships, it neither removes nor adds any)?

(b)

Can we always find an undirected graph that perfectly maps this set?

(c)

Can directed acyclic models represent the conditional independence relationships of all possible undirected models?

(d)

Can undirected models represent the conditional independence relationships of all possible directed acyclic models?

(e)

Can we always find a directed acyclic model or an undirected model? (Scutari and Denis, 2015, collected by Jiri Klema)

2.

Suppose our goal is inference for a parameter δ based on data that would ideally consist of n independent pairs (X, Y), but that some values of Y are missing, as shown by an indicator variable I. Thus the data on an individual have form (x, y, 1) or (x, ?, 0). We suppose that although the missing mechanism Pr(I = 0|x, y) may depend on x and y, it does not involve δ. There are now three possibilities (Davison, 2003, Exercise: 6.2.10):

•

data are missing completely at random, that is Pr(I = 0|x, y) = Pr(I = 0) is independent of both x and y;

•

data are missing at random, that is Pr(I = 0|x, y) = Pr(I = 0|x) depends on x but not on y;

•

there is nonignorable nonresponse, meaning that Pr(I = 0|x, y) depends on y and possibly also on x.

Write down the DAG and its moral graph for X, Y, and I under the missing data models described above. Use them to give an equation-free explanation of the differences among the models and of their consequences.

3.

Show that for a DAG, the local directed Markov property (DL) implies the ordered directed Markov property (DO) (course by Yu, 2010).

4.

A language L has a four-character vocabulary W = {δ, A, B, C}, where δ is the empty symbol. The probability that character w _i will be followed by w _j is given by the following matrix:

In transmitting messages from L, some characters may be corrupted by noise and be confused with others. The probability that the transmitted character w _j will be interpreted as w _k is given by the following confusion matrix:

The string, δ, δ, B, C, A, δ, δ is received, and it is known that the transmitted string begins and ends with δ. Then,

(a)

Find the probability that the i-th transmitted symbol is B, for i = 1, 2, …, 7.

(b)

Find the probability that the string transmitted is the one received.

(c)

Find the probability that no message (a string of δs) was transmitted.

(d)

Find the message most likely to have been transmitted.

(e)

Find the most likely seven-symbol string in L that starts and ends with δ (course by Yu, 2010).

5.

Let us have three tram lines —–7, 17, and 21—regularly coming to the stop in front of the faculty building. Line 17 operates more frequently than line 21, 21 goes more often than line 7 (the ratio is 5:3:2, it is kept during all the hours of operation). Line 7 uses a single car setting in 9 out of 10 cases during the daytime, in the evening it always has the only car. Line 17 has one car rarely and only in evenings (1 out of 10 tram cars). Line 21 can be short whenever, however, it takes a long setting with 2 cars in 8 out of 10 cases. Albertov is available by line 21, lines 7 and 17 are headed in the direction of IP Pavlova. The line changes appear only when a tram goes to depot (let 21 have its depot in the direction of IP Pavlova, 7 and 17 have their depots in the direction of Albertov). Every 10th tram goes to the depot evenly throughout the operation. The evening regime is from 5   pm to 22   pm, the daytime regime is from 5.30   am to 6.30   pm. Then,

(a)

Draw a correct, efficient, and causal BN.

(b)

is evening. A short tram is approaching the stop. What is the probability it will go to Pavlova?

(c)

There is a tram 17 standing in the stop. How many cars does it have?

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/S0169716118300166

parsonsmoul1992.blogspot.com

Source: https://www.sciencedirect.com/topics/mathematics/joint-probability-distribution

Share this post