Marginalisation

Marginalization in conditional probability involves summing over the possible values of some variables to obtain the probability distribution of a subset of the variables. This is useful when we are interested in the probability distribution of a subset of variables, without considering the others.

Definition and Formula

Given a joint probability distribution $P(X, Y)$, marginalization can be used to find the marginal probability distribution of $X$ by summing over all possible values of $Y$:

In the context of conditional probability, marginalization can help us find the marginal conditional probability. For instance, given the joint distribution $P(X, Y, Z)$, the conditional probability $P(X \mid Z)$ can be obtained by marginalizing over $Y$:

Alternatively, if you have the conditional probability $P(X, Y \mid Z)$, you can marginalize over $Y$ to find $P(X \mid Z)$:

Example

Consider three variables: $A$, $B$, and $C$. Suppose we have the joint probability distribution $P(A, B, C)$, and we want to find the marginal conditional probability $P(A \mid C)$.

1. Joint Distribution: $P(A, B, C)$

2. Marginalize over $B$ to find $P(A, C)$:

3. Normalize to get $P(A \mid C)$:

However, using conditional probabilities directly, we can marginalize over $B$:

Chain Rule for Joint Distributions

The chain rule of probability, also known as the chain rule for joint distributions, is a method to express the joint probability of a set of random variables in terms of conditional probabilities.

For a set of $n$ random variables $X_1, X_2, \ldots, X_n$, the chain rule of probability states that the joint probability $P(X_1, X_2, \ldots, X_n)$ can be factored as:

In general form, this can be written as:

This formula allows you to break down the joint probability into a product of conditional probabilities, which can be easier to handle, especially when dealing with complex distributions or large numbers of variables.

Example

For three random variables $X, Y, Z$, the chain rule can be written as:

Each term in this product represents a conditional probability, showing the dependency of each variable on the preceding ones.

Use in Bayesian Networks

In Bayesian networks, the chain rule is often used to represent the joint probability distribution over the network's variables by considering the network's structure (i.e., the dependencies among the variables).

Factor Conditioning

Factor conditioning refers to the process of conditioning a probability distribution on a subset of its variables, resulting in a new distribution over the remaining variables.

Example with Joint Probability Distribution

Let's consider a simple example with three random variables: $A$, $B$, and $C$. Suppose we have the joint probability distribution $P(A, B, C)$.

Step-by-Step Factor Conditioning

1. Original Joint Distribution:

   The joint distribution $P(A, B, C)$ can be factored into conditional probabilities:

2. Conditioning on a Variable: Suppose we want to condition on $A = a$. This means we are interested in the distribution of $B$ and $C$ given $A = a$.

3. Conditional Distribution: The new distribution after conditioning on $A = a$ is:

   $$P(B, C \mid A = a)$$

   This can be computed using the original factors:

Using the factorization of the joint distribution, we get:

Simplifying, we get: