5. Expected Values of Functions of Random Vectors

Let \((X,Y)\) be a two-dimensional random variable and \(D_{\left( X,Y\right) }\) the set of points of discontinuity of the joint cumulative distribution function \(F_{X,Y}\left( x,y\right) .\)

Definition: Let \(% g\left( X,Y\right)\) be a function of the two-dimensional random variable \((X,Y)\). Then, the expected value of \(% g\left( X,Y\right)\) is given by

• \((X,Y)\) is a two-dimensional discrete random variable: \[ E[g(X,Y)]=\sum_{\left( x,y\right) \in D_{\left( X,Y\right) }}g(x,y)f_{X,Y}(x,y) \] provided that \(\sum_{\left( x,y\right) \in D_{\left( X,Y\right) }}\left \vert g(x,y)\right \vert f_{X,Y}(x,y)<+\infty .\)

• \((X,Y)\) is a two-dimensional continuous random variable: \[ E[g(X,Y)]=\int \nolimits_{-\infty }^{+\infty }\int \nolimits_{-\infty }^{+\infty }g(x,y)f_{X,Y}(x,y)dxdy \] provided that \(\int \nolimits_{-\infty }^{+\infty }\int \nolimits_{-\infty }^{+\infty }\left \vert g(x,y)\right \vert f(x,y)dxdy<+\infty .\)

Theorem: Let \((X,Y)\) be a discrete two-dimensional random variable with joint probability function \(f_{X,Y}(x,y)\):

1. If \(g(X,Y)=h\left( X\right)\) that is \(g(X,Y)\) only depends on \(X\) , then \[\begin{aligned} E(g(X,Y)) &=E[h(X)] =\sum_{\left( x,y\right) \in D_{\left( X,Y\right) }}h(x)f_{X,Y}(x,y)\\ &=\sum_{x\in D_{X}}h(x)\sum_{y\in D_{Y}}f_{X,Y}(x,y)=\sum_{x\in D_{X}}h(x)f_{X}\left( x\right) \end{aligned} \] provided that \(\sum_{\left( x,y\right) \in D_{\left( X,Y\right) }}\left \vert h(x)\right \vert f_{X,Y}(x,y)<+\infty .\)

2. If \(g(X,Y)=v(Y)\) that is \(g(X,Y)\) only depends on \(Y\) , then \[ \begin{aligned} E[v(Y)]&=\sum_{\left( x,y\right) \in D_{\left( X,Y\right) }}v(y)f_{X,Y}(x,y)\\ &=\sum_{y\in D_{Y}}v(y)\sum_{x\in D_{X}}f_{X,Y}(x,y)=\sum_{y\in D_{Y}}v(y)f_{Y}\left( y\right) \end{aligned} \] provided that \(\sum_{\left( x,y\right) \in D_{\left( X,Y\right) }}\left \vert v(y)\right \vert f_{X,Y}(x,y)<+\infty .\)

Theorem: Let \((X,Y)\) be a continuous two-dimensional random variable with joint probability function \(f_{X,Y}(x,y):\)

• If \(g(X,Y)=h\left( X\right)\) that is \(g(X,Y)\) only depends on \(X\) , then

\[\begin{aligned} E[h(X)]&=\int_{-\infty }^{+\infty }\int_{-\infty }^{+\infty }h(x)f_{X,Y}(x,y)dxdy\\ &=\int_{-\infty }^{+\infty }h(x)\left( \int_{-\infty }^{+\infty }f_{X,Y}(x,y)dy\right) dx=\int_{-\infty }^{+\infty }h(x)f_{X}\left( x\right) dx \end{aligned}\] provided that \(\int_{-\infty }^{+\infty }\int_{-\infty }^{+\infty }\left \vert h(x)\right \vert f_{X,Y}(x,y)dxdy<+\infty .\)

• If \(g(X,Y)=v(Y)\) that is \(g(X,Y)\) only depends on \(Y\) , then

\[ \begin{aligned} E[v(Y)]&=\int_{-\infty }^{+\infty }\int_{-\infty }^{+\infty }v(y)f_{X,Y}(x,y)dxdy\\ &=\int_{-\infty }^{+\infty }v(y)\left( \int_{-\infty }^{+\infty }f_{X,Y}(x,y)dx\right) dy=\int_{-\infty }^{+\infty }v(Y)f_{Y}\left( y\right) dy \end{aligned} \] provided that \(\int_{-\infty }^{+\infty }\int_{-\infty }^{+\infty }\left \vert v(y)\right \vert f_{X,Y}(x,y)dxdy<+\infty .\)

Properties:

1. \(E\left[ h(X)+v(Y)\right] =E\left[ h(X)\right] +E\left[ v(Y)\right]\) provided that \(E\left[ \left \vert h(X)\right \vert \right] <+\infty ,\) \(E% \left[ \left \vert v(Y)\right \vert \right] <+\infty\)

2. \(E\left[ \sum_{i=1}^{N}X_{i}\right] =\sum_{i=1}^{N}E\left[ X_{i}\right] ,\) where \(N\) is a finite integer, provided that \(E\left[ \left \vert X_{i}\right \vert \right] <+\infty\) for \(i=1,2,...,N.\)

Definition: The \(r\) th and \(s\) th moment of products about the origin of the random variables \(X\) and \(Y\), denoted by \(% \mu _{r,s}^{\prime }\) is the expected value of \(X^{r}Y^{s},\) for \(r=1,2,...;\) \(s=1,2,...\) which is given by

• if \(X\) and \(Y\) are discrete random variables: \[ \mu _{r,s}^{\prime }=E[X^{r}Y^{s}]=\sum_{\left( x,y\right) \in D_{\left( X,Y\right) }}x^{r}y^{s}f_{X,Y}(x,y) \]

• if \(X\) and \(Y\) are continuous random variables: \[ \mu _{r,s}^{\prime }=E[X^{r}Y^{s}]=\int \nolimits_{-\infty }^{+\infty }\int \nolimits_{-\infty }^{+\infty }x^{r}y^{s}f(x,y)dxdy \]

Remarks:

• If \(r=s=1,\) we have \(\mu _{1,1}^{\prime }=E[XY]\)

• Cauchy-Schwarz Inequality: For any two random variables \(X\) and \(Y\), we have \(\left \vert E\left[ XY\right] \right \vert \leq E\left[ X^{2} \right] ^{1/2}E\left[ Y^{2}\right] ^{1/2}\) provided that \(E\left[ \left \vert XY\right \vert \right]\) is finite.

• If \(X\) and \(Y\) are independent random variables, \(E\left[ h\left( X\right) v\left( Y\right) \right] =E\left( h\left( X\right) \right) E\left( v\left( Y\right) \right)\) for any two functions \(h\left( X\right)\) and \(v\left( Y\right) .\)

  [Warning: The reverse is not true.]

• If \(X_{1},X_{2},...,X_{n}\) are independent random variables independent, \(E\left[ X_{1}X_{2}...X_{n}\right] =E\left( X_{1}\right) E\left( X_{2}\right) ...E\left( X_{n}\right) .\)

  [Warning: The reverse is not true.]

Definition: The \(r\) th and \(s\) th moment of products about the mean of the discrete random variables \(X\) and \(Y\), denoted by \(\mu _{r,s}\) is the expected value of \(\left( X-\mu _{X}\right) ^{r}\left( Y-\mu _{Y}\right) ^{s},\) for \(r=1,2,...;\) \(s=1,2,...\) which is given by \[ \begin{aligned} \mu _{r,s}&=E[\left( X-\mu _{X}\right) ^{r}\left( Y-\mu _{Y}\right) ^{s}]\\&=\sum_{\left( x,y\right) \in D_{\left( X,Y\right) }}\left( x-\mu _{X}\right) ^{r}\left( y-\mu _{Y}\right) ^{s}f_{X,Y}(x,y) \end{aligned} \]

Definition: The \(r\) th and \(s\) th moment of products about the mean of the continuous random variables \(X\) and \(Y\), denoted by \(% \mu _{r,s},\) for \(r=1,2,...;\) \(s=1,2,...\) is given by \[ \begin{aligned} \mu _{r,s}&=E[\left( X-\mu _{X}\right) ^{r}\left( Y-\mu _{Y}\right) ^{s}]\\ &=\int \nolimits_{-\infty }^{+\infty }\int \nolimits_{-\infty }^{+\infty }\left( x-\mu _{X}\right) ^{r}\left( y-\mu _{Y}\right) ^{s}f(x,y)dxdy \end{aligned} \]

The covariance is a measure of the joint variability of two random variables. Formally it is defined as \[ Cov\left( X,Y\right) =\sigma _{XY}=\mu _{1,1}=E\left[ \left( X-\mu _{X}\right) \left( Y-\mu _{Y}\right) \right] \]

How can we interpret the covariance?

• When the variables tend to show similar behavior, the covariance is positive:

  • If high (small) values of one variable mainly correspond to high (small) values of the other variable;

• When the variables tend to show opposite behavior, the covariance is negative:

  • When high (small) values of one variable mainly correspond to low (high) values of the other;

• If there is no linear association, then the covariance will be zero.

Properties:

• \(Cov\left( X,Y\right) =E(XY)-E(X)E(Y).\)

• If \(X\) and \(Y\) are independent \(Cov\left( X,Y\right) =0.\)

• If \(Y=bZ\), where \(b\) is constant, \[ Cov\left( X,Y\right) =bCov\left( X,Z\right) . \]

• If \(Y=V+W\), \[ Cov\left( X,Y\right) =Cov\left( X,V\right) +Cov\left( X,W\right) . \]

• If \(Y=b\), where \(b\) is constant, \[ Cov\left( X,Y\right) =0. \]

• If follows from the Cauchy-Schwarz Inequality that \(\left \vert Cov\left( X,Y\right) \right \vert \leq \sqrt{Var\left( X\right) Var\left( Y\right) }.\)

The covariance has the inconvenient of depending on the scale of both random variables. For what values of the covariance can we say that there is a strong association between the two random variables?

The correlation coefficient is a measure of the joint variability of two random variables that do not depend on the scale: \[ \rho _{X,Y}=\frac{Cov\left( X,Y\right) }{\sqrt{Var\left( X\right) Var\left( Y\right) }}. \]

Properties:

• If follows from the Cauchy-Schwarz Inequality that \(-1\leq \rho _{X,Y}\leq 1.\)

If \(Y=bX+a,\) where \(b\) and \(a\) are constants

• \(\rho _{X,Y}=1\) if \(b>0.\)

• \(\rho _{X,Y}=-1\) if \(b<0.\)

• If \(b=0,\) it is not defined.

Summary of important results:

• If \(Y=V\pm W,\) \[ Var\left( Y\right) =Var\left( V\right) +Var\left( W\right) \pm 2Cov\left( V,W\right) . \]

• If \(X_{1},....,X_{n}\) are random variables and \(a_{1},...,a_{n}\) are constants and \(Y=\sum_{i=1}^{n}a_{i}X_{i},\) then \[ Var\left( Y\right) =\sum_{i=1}^{n}a_{i}^{2}Var\left( X_{i}\right) +2\underbrace{\sum_{i=1}^{n}\sum_{j=1,j<i}^{n}a_{i}a_{j}Cov(X_{i},X_{j})}_{=0,\text{ if $X_i,X_j$ are independent}}. \]

• If \(X_{1},....,X_{n}\) are random variables, \(a_{1},...,a_{n}\) are constants and \(b_{1},...,b_{n}\) are constants, \(Y_{1}=% \sum_{i=1}^{n}a_{i}X_{i},\) and \(Y_{2}=\sum_{i=1}^{n}b_{i}X_{i}\) then

\[ Cov\left( Y_{1},Y_{2}\right) =\sum_{i=1}^{n}a_{i}b_{i}Var\left( X_{i}\right) +\underbrace{\sum_{i=1}^{n}\sum_{j=1,j<i}^{n}\left( a_{i}b_{j}+a_{j}b_{i}\right) Cov(X_{i},X_{j})}_{=0,\text{ if $X_i,X_j$ are independent}}. \]

Definition: Let \((X,Y)\) be a two dimensional random variable and \(u(Y,X)\) a function of \(Y\) and \(X.\) Then, the conditional expectation of \(u\left( Y,X\right)\) given \(X=x\), is given by

• if \(X\) and \(Y\) are discrete random variables \[ E\left[ u(Y,X)% %TCIMACRO{\U{a6}}% %BeginExpansion {\vert}% %EndExpansion X=x\right] =\sum \limits_{y\in D_{Y}}u(y,x)f_{Y|X=x}(y) \] where \(D_{Y}\) is the set of discontinuity points of \(F_{Y}\left( y\right)\) and \(f_{Y|X=x}(y)\) is the value of the conditional probability function of \(Y\) given \(X=x\) at \(y\)

• if \(X\) and \(Y\) are continuous random variables \[ E\left[ u(Y,X)% %TCIMACRO{\U{a6}}% %BeginExpansion {\vert}% %EndExpansion X=x\right] =\int \limits_{-\infty }^{+\infty }u(y,x)f_{Y|X=x}(y)dy \] where \(f_{Y|X=x}(y)\) is the value of the conditional probability density function of \(Y\) given \(X=x\) at \(y\).

provided that the expected values exist and are finite.

Remarks:

1. If \(u(Y,X)=Y\), then we have the conditional mean of \(Y,\) \(E\left[ u(Y,X) \vert X=x\right] =E\left[ Y \vert X=x\right] =\mu_{Y|x}\) (notice that this is a function of \(x\))\(.\)

2. If \(u(Y,X)=\left( Y-\mu _{Y|x}\right) ^{2}\), then we have the conditional variance of \(Y\) \[ \begin{aligned} E\left[ u(Y,X) \vert X=x\right] &=& E\left[ \left( Y-\mu _{Y|x}\right)^{2} \vert X=x\right] \\ &=& E\left[ \left( Y-E\left[ u(Y) \vert X=x \right] \right)^{2} \vert X=x\right] \\ &=& Var\left[ Y \vert X=x \right] \end{aligned} \]

3. As usual, \(Var\left[ Y \vert X=x\right] =E\left[ Y^{2} \vert X=x\right] - E\left[ Y \vert X=x\right] ^{2}.\)

4. If \(Y\) and \(X\) are [independent], \(E(Y \vert X=x)=E(Y).\)

5. Of course we can reverse the roles of \(Y\) and \(X,\) that is we can compute \(E\left( u\left( X,Y\right) |Y=y\right) ,\) using definitions similar to those above.

Example: Let \((X,Y)\) be two-dimensional random variable such that \[ f_{X,Y}(x,y)= \begin{cases} 1/2,& 0<x<2,0<y<x\\ 0,&\text{c.c.} \end{cases}. \] Then the conditional density function of \(Y\vert X=1\) is given by \[ \begin{aligned} f_{Y|X=1}(y)&= \begin{cases} \frac{f_{X,Y}(1,y)}{f_{X}(1)},&0<y<1\\ 0,& \text{c.c.} \end{cases}= \begin{cases} \frac{1/2}{1/2},& 0<y<1\\ 0,& \text{c.c.} \end{cases}\\ &= \begin{cases} 1,& 0<y<1\\ 0,& \text{c.c.} \end{cases} \end{aligned} \] where \[ \begin{aligned} f_{X}(x)= \begin{cases} \int_{0}^{x}f_{X,Y}(x,y)dy,&0<x<2\\ 0,&\text{c.c.} \end{cases}= \begin{cases} \frac{x}{2},&0<x<2\\ 0,&\text{c.c.} \end{cases}. \end{aligned} \]

Example: The conditional expected value can be computed as follows:

\[ \begin{aligned} E(Y\vert X=1)=\int_{0}^{1}yf_{Y\vert X=1}(y)dy=\int_{0}^{1}ydy=\frac{1}{2}. \end{aligned} \] To compute the conditional variance, one may start by computing the following conditional expected value \[ \begin{aligned} E(Y^2\vert X=1)=\int_{0}^{1}y^2f_{Y\vert X=1}(y)dy=\int_{0}^{1}y^2dy=\frac{1}{3}. \end{aligned} \] Therefore \[ \begin{aligned} Var(Y\vert X=1)&=E(Y^2\vert X=1)-\left(E(Y\vert X=1)\right)^2\\ &=\frac{1}{3}-\frac{1}{4}=\frac{1}{12} \end{aligned} \]

Example: Let \(X\) and \(Y\) be two random variables such that \[ f_{X,Y}(x,y)=\frac{1}{9}, \,\text{ for }x=1,2,3,\,y=0,1,2,3,\,y\leq x \]

To compute the conditional expected value one has to compute the condition probability function:

\[ f_{Y|X=1}(y)=\begin{cases} \frac{f_{X,Y}(1,Y)}{f_{X}(1)},&y=0,1\\ 0,&\text{otherwise} \end{cases}=\begin{cases} \frac{1}{2},&y=0,1\\ 0,&\text{otherwise} \end{cases} \] where \[ f_{X}(1)=\sum_{y=0}^1f_{X,Y}(1,y)=\sum_{y=0}^1\frac{1}{9}=\frac{2}{9} \] Therefore, \[ E(Y|X=1) = \sum_{y \in D_Y}yf_{Y|X=1}(y)= 0\times\frac{1}{2}+1\times\frac{1}{2}=\frac{1}{2}. \]

Notice that \(g(y)=E(X\vert Y=y)\) is indeed a function of \(y\). Therefore, \(g(Y)\) is a random variable because \(Y\) can take different values according its distribution, i.e, if \(Y\) can take the value \(y\), then \(g(Y)\) can take \(g(y)\) with probability \(P(Y=y)>0\).

• Discrete random variables

The random variable \(Z=g(Y)=E(X\vert Y)\) takes the values \(g(y)=E(X\vert Y=y)\). Assume that all values of \(g(y)\) are different. Then, \[ Z\text{ takes the value }g(y) \text{ with probability } P(Y=y) \]

In general, the probability function of \(Z=g(Y)=E(X\vert Y)\) can be computed in the following way \[ \begin{aligned} P(Z=z)=P(g(Y)=z)=P(Y\in\{y\,:g(y)= z\}) \end{aligned} \]

Example: Let \((X,Y)\) be a discrete random variable such that \(f_{X,Y}(x,y)\) is represented in the following table

X/Y	1	2	3
0	0.2	0.1	0.15
1	0.05	0.35	0.15

One may compute the following conditional probability functions: \[ \begin{aligned} f_{Y\vert X=0}=\begin{cases} 4/9,&y=1\\ 2/9,&y=2\\ 3/9,&y=3\\ 0,&\text{otherwise} \end{cases}\quad\text{and}\quad f_{Y\vert X=1}=\begin{cases} 1/11,&y=1\\ 7/11,&y=2\\ 3/11,&y=3\\ 0,&\text{otherwise} \end{cases}.\end{aligned} \] Consequently, \(E(Y\vert X=0)=17/9\) and \(E(Y\vert X=1)=24/11\). Therefore, the random variable \(Z=E(Y\vert X)\) has the following probability function \[ P(Z=z)=\begin{cases} P(X=0),& z=17/9\\ P(X=1),&z=24/11\\ 0,& \text{otherwise} \end{cases}=\begin{cases} 0.45,& z=17/9\\ 0.55,&z=24/11\\ 0,& \text{otherwise} \end{cases}. \]

• Continuous random variables

The cumulative distribution function of \(Z=g(Y)=E(X\vert Y)\) is, indeed \[ \begin{aligned} F_Z(z)=P(Z\leq z)=P(g(Y)\leq z)=P(Y\in\{y\,:g(y)\leq z\})\end{aligned} \] When \(g\) is an injective function, we get that \[ F_Z(z)=F_Y(g^{-1}(z)) \text{ or } F_Z(g(y))=F_Y(y). \]

Therefore, we can calculate all the quantities that we know (the expected value, variance, ...) for \(E(X\vert Y)\) or \(E(Y\vert X)\)

Theorem (Law of iterated Expectations) Let \((X,Y)\) be a two dimensional random variable. Then, \(E(Y)=E(E\left[ Y \vert X\right] )\) provided that \(E(\left \vert Y \right \vert )\) is finite and \(E(X)=E(E[X\vert Y])\) provided that \(E(X)\) is finite.

Remark: This theorem shows that there are two ways to compute \(E(Y )\) (resp., \(E(X)\)). The first is the direct way. The second way is to consider the following steps:

1. compute \(E\left[Y \vert X=x\right]\) and notice that this is a function solely of \(x\) that is we can write \(g(x)=E\left[ Y \vert X=x\right] ,\)

2. according to the theorem replacing \(g(x)\) by \(g(X)\) and taking the mean we obtain \(E\left[ g(X)\right] =E\left[ Y \right]\) for this specific form of \(g(X).\)

3. This theorem is useful in practice in the calculation of \(E\left( Y \right)\) if we know $ f_{Y|X=x}(y)$ or \(E\left[ X \vert X=x\right]\) and \(f_{X}(x)\) (or some moments of \(X\)), but not \(f_{X,Y}(x,y).\)

Remarks: The results presented can be generalized for functions of \(X\) and \(Y\), i.e., \(E(u(X,Y))=E(E(u(X,Y)\vert X))\), if \(E(u(X,Y))\) exists.

Example: Let \((X,Y)\) be a bi-dimensional continuous random variable such that \[ E(X\vert Y=y)=\frac{3y-1}{3}\quad\text{and}\quad f_Y(y)= \begin{cases} 1/2,&0<y<2\\ 0,& \text{otherwise} \end{cases} \] Taking into account the previous theorem, \[ E(X)=E(E(X\vert Y))=E\left(\frac{3Y-1}{3}\right)=\int_{0}^2\frac{3y-1}{6}dy=2/3. \]

Theorem: Assuming that \(E(Y^2)\) exists then \[ Var(Y)=Var[E(Y\vert X)]+E[Var[Y\vert X]]. \]

Theorem: Let \(X\) and \(Y\) be two random variables then \[ Cov(X,Y)=Cov(X,E(Y\vert X)) \]

Example: Let \((X,Y)\) be a bidimensional random variable such that \[ \begin{aligned} &f_{X\vert Y=y}(x)=\frac{1}{y},~~~0<x<y~~ (\text{for a fixed} y>1)\\ &f_Y(y)=3y^{-4},~~~y>1\end{aligned} \] Compute \(Var(X)\) using the previous theorem.

Exam question: Let \(X\) and \(Y\) be two random variables such that \[ E(X\vert Y=y)=y, \] for all \(y\) such that \(f_Y(y)>0\). Prove that \(Cov(X,Y)=Var(Y)\). Are the random variables independent? Justify your answer.