5. Expected Values of Functions of Random Vectors

Let $(X,Y)$ be a two-dimensional random variable and $D_{(X,Y)}$ the set of points of discontinuity of the joint cumulative distribution function $F_{X,Y}(x,y).$

Definition: Let be a function of the two-dimensional random variable $(X,Y)$. Then, the expected value of is given by

• $(X,Y)$ is a two-dimensional discrete random variable: $$E\left[g\right(X,Y\left)\right]=\sum _{(x,y)\in D_{(X,Y)}}g(x,y)f_{X,Y}(x,y)$$ provided that ${\sum }_{(x,y)\in D_{(X,Y)}}\left|g\right(x,y\left)\right|f_{X,Y}(x,y)<+\infty .$

• $(X,Y)$ is a two-dimensional continuous random variable: $$E\left[g\right(X,Y\left)\right]={\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }g(x,y)f_{X,Y}(x,y)dxdy$$ provided that ${\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }\left|g\right(x,y\left)\right|f(x,y)dxdy<+\infty .$

Theorem: Let $(X,Y)$ be a discrete two-dimensional random variable with joint probability function $f_{X,Y}(x,y)$:

1. If $g(X,Y)=h\left(X\right)$ that is $g(X,Y)$ only depends on $X$ , then $$\begin{array}{cc}E\left(g\right(X,Y\left)\right)& =E\left[h\right(X\left)\right]=\sum _{(x,y)\in D_{(X,Y)}}h\left(x\right)f_{X,Y}(x,y)\\ & =\sum _{x\in D_X}h\left(x\right)\sum _{y\in D_Y}{f}_{X,Y}(x,y)=\sum _{x\in D_X}h\left(x\right)f_X\left(x\right)\end{array}$$ provided that ${\sum }_{(x,y)\in D_{(X,Y)}}\left|h\right(x\left)\right|f_{X,Y}(x,y)<+\infty .$

2. If $g(X,Y)=v\left(Y\right)$ that is $g(X,Y)$ only depends on $Y$ , then $$\begin{array}{cc}E\left[v\right(Y\left)\right]& =\sum _{(x,y)\in D_{(X,Y)}}v\left(y\right)f_{X,Y}(x,y)\\ & =\sum _{y\in D_Y}v\left(y\right)\sum _{x\in D_X}{f}_{X,Y}(x,y)=\sum _{y\in D_Y}v\left(y\right)f_Y\left(y\right)\end{array}$$ provided that ${\sum }_{(x,y)\in D_{(X,Y)}}\left|v\right(y\left)\right|f_{X,Y}(x,y)<+\infty .$

Theorem: Let $(X,Y)$ be a continuous two-dimensional random variable with joint probability function $f_{X,Y}(x,y):$

• If $g(X,Y)=h\left(X\right)$ that is $g(X,Y)$ only depends on $X$ , then

$$\begin{array}{cc}E\left[h\right(X\left)\right]& ={\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }h\left(x\right)f_{X,Y}(x,y)dxdy\\ & ={\int }_{-\infty }^{+\infty }h\left(x\right)\left({\int }_{-\infty }^{+\infty }{f}_{X,Y}\right(x,y\left)dy\right)dx={\int }_{-\infty }^{+\infty }h\left(x\right)f_X\left(x\right)dx\end{array}$$ provided that ${\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }\left|h\right(x\left)\right|f_{X,Y}(x,y)dxdy<+\infty .$

• If $g(X,Y)=v\left(Y\right)$ that is $g(X,Y)$ only depends on $Y$ , then

$$\begin{array}{cc}E\left[v\right(Y\left)\right]& ={\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }v\left(y\right)f_{X,Y}(x,y)dxdy\\ & ={\int }_{-\infty }^{+\infty }v\left(y\right)\left({\int }_{-\infty }^{+\infty }{f}_{X,Y}\right(x,y\left)dx\right)dy={\int }_{-\infty }^{+\infty }v\left(Y\right)f_Y\left(y\right)dy\end{array}$$ provided that ${\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }\left|v\right(y\left)\right|f_{X,Y}(x,y)dxdy<+\infty .$

Properties:

1. $E\left[h\right(X)+v(Y\left)\right]=E\left[h\right(X\left)\right]+E\left[v\right(Y\left)\right]$ provided that $E\left[\left|h\right(X\left)\right|\right]<+\infty ,$ $E$

2. $E\left[{\sum }_{i=1}^N{X}_i\right]={\sum }_{i=1}^{N}E\left[X_i\right],$ where $N$ is a finite integer, provided that $E\left[\left|X_i\right|\right]<+\infty$ for $i=1,2,...,N.$

Definition: The $r$ th and $s$ th moment of products about the origin of the random variables $X$ and $Y$, denoted by is the expected value of $X^r{Y}^s,$ for $r=1,2,...;$ $s=1,2,...$ which is given by

• if $X$ and $Y$ are discrete random variables: $${\mu }_{r,s}^{\prime }=E\left[X^r{Y}^s\right]=\sum _{(x,y)\in D_{(X,Y)}}{x}^r{y}^s{f}_{X,Y}(x,y)$$

• if $X$ and $Y$ are continuous random variables: $${\mu }_{r,s}^{\prime }=E\left[X^r{Y}^s\right]={\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }{x}^r{y}^{s}f(x,y)dxdy$$

Remarks:

• If $r=s=1,$ we have ${\mu }_{1,1}^{\prime }=E\left[XY\right]$

• Cauchy-Schwarz Inequality: For any two random variables $X$ and $Y$, we have $\left|E\left[XY\right]\right|\le E{\left[X^2\right]}^{1/2}E{\left[Y^2\right]}^{1/2}$ provided that $E\left[\left|XY\right|\right]$ is finite.

• If $X$ and $Y$ are independent random variables, $E\left[h\left(X\right)v\left(Y\right)\right]=E\left(h\left(X\right)\right)E\left(v\left(Y\right)\right)$ for any two functions $h\left(X\right)$ and $v\left(Y\right).$

  [Warning: The reverse is not true.]

• If $X_1,X_2,...,X_n$ are independent random variables independent, $E[X_1{X}_2...X_n]=E\left(X_1\right)E\left(X_2\right)...E\left(X_n\right).$

  [Warning: The reverse is not true.]

Definition: The $r$ th and $s$ th moment of products about the mean of the discrete random variables $X$ and $Y$, denoted by ${\mu }_{r,s}$ is the expected value of ${(X-{\mu }_X)}^r{(Y-{\mu }_Y)}^s,$ for $r=1,2,...;$ $s=1,2,...$ which is given by $$\begin{array}{cc}{\mu }_{r,s}& =E\left[{(X-{\mu }_X)}^r{(Y-{\mu }_Y)}^s\right]\\ & =\sum _{(x,y)\in D_{(X,Y)}}{(x-{\mu }_X)}^r{(y-{\mu }_Y)}^s{f}_{X,Y}(x,y)\end{array}$$

Definition: The $r$ th and $s$ th moment of products about the mean of the continuous random variables $X$ and $Y$, denoted by for $r=1,2,...;$ $s=1,2,...$ is given by $$\begin{array}{cc}{\mu }_{r,s}& =E\left[{(X-{\mu }_X)}^r{(Y-{\mu }_Y)}^s\right]\\ & ={\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }{(x-{\mu }_X)}^r{(y-{\mu }_Y)}^{s}f(x,y)dxdy\end{array}$$

The covariance is a measure of the joint variability of two random variables. Formally it is defined as $$Cov(X,Y)={\sigma }_{XY}={\mu }_{1,1}=E\left[(X-{\mu }_X)(Y-{\mu }_Y)\right]$$

How can we interpret the covariance?

• When the variables tend to show similar behavior, the covariance is positive:

  • If high (small) values of one variable mainly correspond to high (small) values of the other variable;

• When the variables tend to show opposite behavior, the covariance is negative:

  • When high (small) values of one variable mainly correspond to low (high) values of the other;

• If there is no linear association, then the covariance will be zero.

Properties:

• $Cov(X,Y)=E\left(XY\right)-E\left(X\right)E\left(Y\right).$

• If $X$ and $Y$ are independent $Cov(X,Y)=0.$

• If $Y=bZ$, where $b$ is constant, $$Cov(X,Y)=bCov(X,Z).$$

• If $Y=V+W$, $$Cov(X,Y)=Cov(X,V)+Cov(X,W).$$

• If $Y=b$, where $b$ is constant, $$Cov(X,Y)=0.$$

• If follows from the Cauchy-Schwarz Inequality that $\left|Cov(X,Y)\right|\le \sqrt{Var\left(X\right)Var\left(Y\right)}.$

The covariance has the inconvenient of depending on the scale of both random variables. For what values of the covariance can we say that there is a strong association between the two random variables?

The correlation coefficient is a measure of the joint variability of two random variables that do not depend on the scale: $${\rho }_{X,Y}=\frac{Cov(X,Y)}{\sqrt{Var\left(X\right)Var\left(Y\right)}}.$$

Properties:

• If follows from the Cauchy-Schwarz Inequality that $-1\le {\rho }_{X,Y}\le 1.$

If $Y=bX+a,$ where $b$ and $a$ are constants

• ${\rho }_{X,Y}=1$ if $b>0.$

• ${\rho }_{X,Y}=-1$ if $b<0.$

• If $b=0,$ it is not defined.

Summary of important results:

• If $Y=V\pm W,$ $$Var\left(Y\right)=Var\left(V\right)+Var\left(W\right)\pm 2Cov(V,W).$$

• If $X_1,....,X_n$ are random variables and $a_1,...,a_n$ are constants and $Y={\sum }_{i=1}^n{a}_i{X}_i,$ then $$Var\left(Y\right)=\sum _{i=1}^n{a}_i^{2}Var\left(X_i\right)+2\underset{=0,\text{if}{X}_i,X_j\text{are independent}}{\underset{}{\sum _{i=1}^n\sum _{j=1,j<i}^n{a}_i{a}_{j}Cov(X_i,X_j)}}.$$

• If $X_1,....,X_n$ are random variables, $a_1,...,a_n$ are constants and $b_1,...,b_n$ are constants, $Y_1=$ and $Y_2={\sum }_{i=1}^n{b}_i{X}_i$ then

$$Cov(Y_1,Y_2)=\sum _{i=1}^n{a}_i{b}_{i}Var\left(X_i\right)+\underset{=0,\text{if}{X}_i,X_j\text{are independent}}{\underset{}{\sum _{i=1}^n\sum _{j=1,j<i}^n(a_i{b}_j+a_j{b}_i)Cov(X_i,X_j)}}.$$

Definition: Let $(X,Y)$ be a two dimensional random variable and $u(Y,X)$ a function of $Y$ and $X.$ Then, the conditional expectation of $u(Y,X)$ given $X=x$, is given by

• if $X$ and $Y$ are discrete random variables $$E\left[u\right(Y,X)|X=x]=\sum _{y\in D_Y}u(y,x)f_{Y|X=x}\left(y\right)$$ where $D_Y$ is the set of discontinuity points of $F_Y\left(y\right)$ and $f_{Y|X=x}\left(y\right)$ is the value of the conditional probability function of $Y$ given $X=x$ at $y$

• if $X$ and $Y$ are continuous random variables $$E\left[u\right(Y,X)|X=x]=\underset{-\infty }{\overset{+\infty }{\int }}u(y,x)f_{Y|X=x}\left(y\right)dy$$ where $f_{Y|X=x}\left(y\right)$ is the value of the conditional probability density function of $Y$ given $X=x$ at $y$.

provided that the expected values exist and are finite.

Remarks:

1. If $u(Y,X)=Y$, then we have the conditional mean of $Y,$ $E\left[u\right(Y,X\left)\right|X=x]=E\left[Y\right|X=x]={\mu }_{Y|x}$ (notice that this is a function of $x$)$.$

2. If $u(Y,X)={(Y-{\mu }_{Y|x})}^2$, then we have the conditional variance of $Y$ $$\begin{array}{ccc}E\left[u\right(Y,X\left)\right|X=x]& =& E\left[{(Y-{\mu }_{Y|x})}^2\right|X=x]\\ & =& E\left[{(Y-E\left[u\right(Y\left)\right|X=x])}^2\right|X=x]\\ & =& Var\left[Y\right|X=x]\end{array}$$

3. As usual, $Var\left[Y\right|X=x]=E\left[Y^2\right|X=x]-E{\left[Y\right|X=x]}^2.$

4. If $Y$ and $X$ are [independent], $E\left(Y\right|X=x)=E(Y).$

5. Of course we can reverse the roles of $Y$ and $X,$ that is we can compute $E(u(X,Y)|Y=y),$ using definitions similar to those above.

Example: Let $(X,Y)$ be two-dimensional random variable such that $$f_{X,Y}(x,y)=\{\begin{array}{cc}1/2,& 0<x<2,0<y<x\\ 0,& \text{c.c.}\end{array}.$$ Then the conditional density function of $Y|X=1$ is given by $$\begin{array}{cc}{f}_{Y|X=1}\left(y\right)& =\{\begin{array}{cc}\frac{f_{X,Y}(1,y)}{f_X\left(1\right)},& 0<y<1\\ 0,& \text{c.c.}\end{array}=\{\begin{array}{cc}\frac{1/2}{1/2},& 0<y<1\\ 0,& \text{c.c.}\end{array}\\ & =\{\begin{array}{cc}1,& 0<y<1\\ 0,& \text{c.c.}\end{array}\end{array}$$ where $$\begin{array}{c}{f}_X\left(x\right)=\{\begin{array}{cc}{\int }_0^x{f}_{X,Y}(x,y)dy,& 0<x<2\\ 0,& \text{c.c.}\end{array}=\{\begin{array}{cc}\frac{x}{2},& 0<x<2\\ 0,& \text{c.c.}\end{array}.\end{array}$$

Example: The conditional expected value can be computed as follows:

$$\begin{array}{c}E\left(Y\right|X=1)={\int }_0^{1}y{f}_{Y|X=1}(y)dy={\int }_0^{1}ydy=\frac{1}{2}.\end{array}$$ To compute the conditional variance, one may start by computing the following conditional expected value $$\begin{array}{c}E\left(Y^2\right|X=1)={\int }_0^1{y}^2{f}_{Y|X=1}(y)dy={\int }_0^1{y}^{2}dy=\frac{1}{3}.\end{array}$$ Therefore $$\begin{array}{cc}Var\left(Y\right|X=1)& =E\left(Y^2\right|X=1)-{\left(E\right(Y|X=1))}^2\\ & =\frac{1}{3}-\frac{1}{4}=\frac{1}{12}\end{array}$$

Example: Let $X$ and $Y$ be two random variables such that $$f_{X,Y}(x,y)=\frac{1}{9},\text{for}x=1,2,3,y=0,1,2,3,y\le x$$

To compute the conditional expected value one has to compute the condition probability function:

$$f_{Y|X=1}\left(y\right)=\{\begin{array}{cc}\frac{f_{X,Y}(1,Y)}{f_X\left(1\right)},& y=0,1\\ 0,& \text{otherwise}\end{array}=\{\begin{array}{cc}\frac{1}{2},& y=0,1\\ 0,& \text{otherwise}\end{array}$$ where $$f_X\left(1\right)=\sum _{y=0}^1{f}_{X,Y}(1,y)=\sum _{y=0}^1\frac{1}{9}=\frac{2}{9}$$ Therefore, $$E(Y|X=1)=\sum _{y\in D_Y}y{f}_{Y|X=1}\left(y\right)=0\times \frac{1}{2}+1\times \frac{1}{2}=\frac{1}{2}.$$

Notice that $g\left(y\right)=E\left(X\right|Y=y)$ is indeed a function of $y$. Therefore, $g\left(Y\right)$ is a random variable because $Y$ can take different values according its distribution, i.e, if $Y$ can take the value $y$, then $g\left(Y\right)$ can take $g\left(y\right)$ with probability $P(Y=y)>0$.

• Discrete random variables

The random variable $Z=g\left(Y\right)=E\left(X\right|Y)$ takes the values $g\left(y\right)=E\left(X\right|Y=y)$. Assume that all values of $g\left(y\right)$ are different. Then, $$Z\text{takes the value}g\left(y\right)\text{with probability}P(Y=y)$$

In general, the probability function of $Z=g\left(Y\right)=E\left(X\right|Y)$ can be computed in the following way $$\begin{array}{c}P(Z=z)=P\left(g\right(Y)=z)=P(Y\in \{y:g\left(y\right)=z\left\}\right)\end{array}$$

Example: Let $(X,Y)$ be a discrete random variable such that $f_{X,Y}(x,y)$ is represented in the following table

X/Y	1	2	3
0	0.2	0.1	0.15
1	0.05	0.35	0.15

One may compute the following conditional probability functions: $$\begin{array}{c}{f}_{Y|X=0}=\{\begin{array}{cc}4/9,& y=1\\ 2/9,& y=2\\ 3/9,& y=3\\ 0,& \text{otherwise}\end{array}\text{and}{f}_{Y|X=1}=\{\begin{array}{cc}1/11,& y=1\\ 7/11,& y=2\\ 3/11,& y=3\\ 0,& \text{otherwise}\end{array}.\end{array}$$ Consequently, $E\left(Y\right|X=0)=17/9$ and $E\left(Y\right|X=1)=24/11$. Therefore, the random variable $Z=E\left(Y\right|X)$ has the following probability function $$P(Z=z)=\{\begin{array}{cc}P(X=0),& z=17/9\\ P(X=1),& z=24/11\\ 0,& \text{otherwise}\end{array}=\{\begin{array}{cc}0.45,& z=17/9\\ 0.55,& z=24/11\\ 0,& \text{otherwise}\end{array}.$$

• Continuous random variables

The cumulative distribution function of $Z=g\left(Y\right)=E\left(X\right|Y)$ is, indeed $$\begin{array}{c}{F}_Z\left(z\right)=P(Z\le z)=P\left(g\right(Y)\le z)=P(Y\in \{y:g\left(y\right)\le z\left\}\right)\end{array}$$ When $g$ is an injective function, we get that $$F_Z\left(z\right)=F_Y\left(g^{-1}\right(z\left)\right)\text{or}{F}_Z\left(g\right(y\left)\right)=F_Y\left(y\right).$$

Therefore, we can calculate all the quantities that we know (the expected value, variance, ...) for $E\left(X\right|Y)$ or $E\left(Y\right|X)$

Theorem (Law of iterated Expectations) Let $(X,Y)$ be a two dimensional random variable. Then, $E\left(Y\right)=E\left(E\left[Y\right|X]\right)$ provided that $E\left(\left|Y\right|\right)$ is finite and $E\left(X\right)=E\left(E\right[X\left|Y\right])$ provided that $E\left(X\right)$ is finite.

Remark: This theorem shows that there are two ways to compute $E\left(Y\right)$ (resp., $E\left(X\right)$). The first is the direct way. The second way is to consider the following steps:

1. compute $E\left[Y\right|X=x]$ and notice that this is a function solely of $x$ that is we can write $g\left(x\right)=E\left[Y\right|X=x],$

2. according to the theorem replacing $g\left(x\right)$ by $g\left(X\right)$ and taking the mean we obtain $E\left[g\right(X\left)\right]=E\left[Y\right]$ for this specific form of $g\left(X\right).$

3. This theorem is useful in practice in the calculation of $E\left(Y\right)$ if we know $f_{Y|X=x}\left(y\right)$ or $E\left[X\right|X=x]$ and $f_X\left(x\right)$ (or some moments of $X$), but not $f_{X,Y}(x,y).$

Remarks: The results presented can be generalized for functions of $X$ and $Y$, i.e., $E\left(u\right(X,Y\left)\right)=E\left(E\right(u(X,Y)\left|X\right))$, if $E\left(u\right(X,Y\left)\right)$ exists.

Example: Let $(X,Y)$ be a bi-dimensional continuous random variable such that $$E\left(X\right|Y=y)=\frac{3y-1}{3}\text{and}{f}_Y(y)=\{\begin{array}{cc}1/2,& 0<y<2\\ 0,& \text{otherwise}\end{array}$$ Taking into account the previous theorem, $$E\left(X\right)=E\left(E\right(X\left|Y\right))=E\left(\frac{3Y-1}{3}\right)={\int }_0^2\frac{3y-1}{6}dy=2/3.$$

Theorem: Assuming that $E\left(Y^2\right)$ exists then $$Var\left(Y\right)=Var\left[E\right(Y\left|X\right)]+E[Var\left[Y\right|X\left]\right].$$

Theorem: Let $X$ and $Y$ be two random variables then $$Cov(X,Y)=Cov(X,E(Y\left|X\right))$$

Example: Let $(X,Y)$ be a bidimensional random variable such that $$\begin{array}{cc}& f_{X|Y=y}\left(x\right)=\frac{1}{y},0<x<y(\text{for a fixed}y>1)\\ & f_Y\left(y\right)=3y^{-4},y>1\end{array}$$ Compute $Var\left(X\right)$ using the previous theorem.

Exam question: Let $X$ and $Y$ be two random variables such that $$E\left(X\right|Y=y)=y,$$ for all $y$ such that $f_Y\left(y\right)>0$. Prove that $Cov(X,Y)=Var\left(Y\right)$. Are the random variables independent? Justify your answer.