2.1 Random Experiment

Random experiment: an experiment whose outcome cannot be determined in advance, but is nevertheless still subject to analysis.

• The outcome cannot be predicted with certainty.

• The collection of every possible outcomes can be described and perhaps listed.

Probability Theory \(\approx\) Uncertainty Measurement

3.1 Sample Spaces

Sample Space: The set of all possible outcomes of an experiment is called the sample space \((S)\). Each outcome in a sample space is called an element of the sample space \((s)\). \[S=\{s_1,s_2, \cdots, s_n\}\]

3.2 Events

Event: is a subset of the sample space, usually designated with capital letters \(A,B,C, \cdots\). \[A\subset S\] Remarks:

• outcomes are different from events;

• \(A\subset B\) if all the elements of the sample space contained in \(A\) are also contained in \(B\);

• The sample space \(S\) is an event;

• The empty set is an event.

An event (\({ A}\)) occurs when the outcome \((s)\) of the experiment belongs to \(A\). \[s\in A\]

3.3 Cardinality of an Event

Cardinality of \(A\):

• \(\#A\) is finite; \[\begin{aligned} A=\{1,2,3,4\},\quad\text{and}\quad \#A=4\\ B=\{H,T\},\quad\text{and}\quad \#B=2 \end{aligned}\]

• \(\#A\) is countable and infinite \[\begin{aligned} A=\mathbb{N},\quad\text{and}\quad \#A=+\infty\\ B=\{2n\,: n\in\mathbb{N}\},\quad\text{and}\quad \#B=+\infty \end{aligned}\]

• \(\#A\) is uncountable \[\begin{aligned} A=[3,5],\quad\text{and}\quad \#A=+\infty\\ B=\mathbb{R}^+,\quad\text{and}\quad \#B=+\infty \end{aligned}\]

The sample space \(S\) is said to be

• Discrete, whenever \(\#S\) is finite or countably infinite;

• Continuous, whenever \(\#S\) uncountable.

3.4 Operations with Events

Often, we are interested in events that are actually combinations of two or more events.

Let \(A\) and \(B\) be two events of \(S\), i.e. \(A,B\subset S\). Then we can define the following operations with events:

• \(A\cap B\): the intersection of \(A\) and \(B\) is the subset of \(S\) that contains all the elements that are in both \(A\) and \(B\);

• \(A\cup B\): the union of \(A\) and \(B\) is the subset of \(S\) that contains all the elements that are either in \(A\), in \(B\), or in both;

• \(A-B\): the difference of \(A\) and \(B\) is the subset of \(A\) that contains all the elements of \(A\) that are not in \(B\);

• \(\overline{A}\) (or \(A'\)): the complement of \(A\) is the subset of \(S\) that contains all the elements of \(S\) that are not in \(A\), i.e. \(\overline{A}=S-A\).

3.5 Venn Diagrams

3.6 Properties of Operations with Events

Let \(A, B\) and \(C\) be events of \(S\). Then, the following properties hold true:

• Associativity: \[(A\cup B)\cup C=A\cup(B\cup C),\quad\text{and}\quad (A\cap B)\cap C=A\cap(B\cap C)\]

• Comutatitvity: \[A\cup B=B\cup A\quad\text{and}\quad A\cap B=B\cap A\]

• Distributivity: \[A\cup(B\cap C)=(A\cup B)\cap (A\cup C)\quad\text{and}\quad A\cap(B\cup C)=(A\cap B)\cup (A\cap C)\]

• De Morgan’s Laws: \[\overline{A\cup B}=\overline{A}\cap\overline{B}\quad\text{and}\quad\quad \overline{A\cap B}=\overline{A}\cup\overline{B}\]

Mutually Exclusive Events: Two events having no elements in common are said to be mutually exclusive (or Disjoint or Incompatible). In other words, the events \(A\) and \(B\), with \(A,B\subset S\), are disjoint if \[A\cap B=\emptyset.\]

Prove the following properties:

Let \(A\) and \(B\) be two events of \(S\), the sample space.

• \(A\cup A=A\cap A=A\);

• \(A\subset B\Rightarrow A\cap B=A\) and \(A\cup B=B\);

• \(A\cap S=A\cup\emptyset=A;\)

• \(\overline{\overline{A}}=A;\)

• \(A\cap\overline{A}=\emptyset\) (\(A\) and \(\overline{A}\) are mutually exclusive events);

• \(A\cup\overline{A}=S.\)

• \(A-B=A\setminus B=A\cap\overline{B}\)

4.1 Probability Space

A probability space is a triplet \((S,{\cal F}, P)\), where \(S\) is the sample space, \({\cal F}\) is a \(\sigma-\)algebra and \(P\) is a probability measure.

Roughly speaking, \({\cal F}\) is a collection of all events contained in \(S\), i.e., \({\cal F}=\{A_1,A_2,A_3,\cdots\}\).

Definition: A \(\sigma-\)algebra \(\cal F\) is a set of events that satisfies the following properties:

• \(\cal F\) contain the sample space \((S\in {\cal F})\);

• If \(A\in{\cal F}\), then \(\overline{A}\in{\cal F}\);

• If \(A_1, A_2, A_3, \cdots\), then \(\bigcup_{i=1}^{+\infty}A_i\in{\cal F}\)

With the probability measure, we assigna probability to each event in \(\cal F\)

4.2 Properties of \(P\)

A probability measure is a function with domain \({\cal F}\) that satisfies the Kolmogorov Axioms:

• \(P(S)=1\)

• \(P(A)\geq 0\), for all \(A\in{\cal F}\)

• If \(A_1, A_2, A_3, \cdots\) are mutually exclusive events, then \[P(\bigcup_{i=1}^{+\infty}A_i)=\sum_{i=1}^{+\infty}P(A_i).\]

Theorem: Let \(A\) and \(B\) be two events of \(S\). The following properties follow from the Kolmogorov Axioms.

• \(P(\overline{A})=1-P(A)\) and \(P(\emptyset)=0\);

• \(A\subset B\Rightarrow P(A)\leq P(B)\);

• \(P(A)\leq 1\);

• \(P(B- A)=P(B)-P(A\cap B)\);

• \(P(A\cup B)=P(A)+P(B)-P(A\cap B)\).

Prove the previous Theorem!

• \(P(\overline{A})=1-P(A)\);

Proof: We know that \(A\cup \overline{A}=S\). Then, \[\begin{aligned} P(S)&=P(A\cup \overline{A})=\underbrace{P(A)+P(\overline{A})}_{A\text{ and }\overline{A}\text{ are disjoint}}\\ \Leftrightarrow 1&= P(A)+P(\overline{A}) \end{aligned}\]

• \(P(\emptyset)=0\);

Proof: Notice that \(\overline{S}=\emptyset\). Then, from the previous property, we get that \[P(\emptyset)=P(\overline{S})=1-P(S)=1-1=0.\]

• \(A\subset B\Rightarrow P(A)\leq P(B)\);

Proof: If \(A\subset B\), then \(B=A\cup(B-A)\). Additionally \(A\cap (B-A)=\emptyset\) (Check these two statements!)

Therefore, \[\begin{aligned} P(B)=P(A\cup (B-A))=\underbrace{P(A)+ P((B-A))}_{A\text{ and }(B-A)\text{ are disjoint}}. \end{aligned}\]

Since \(P((B-A))\geq 0\), then \(P(B)\geq P(A)\).

• \(P(A)\leq 1\);

Proof: Since \(A\subset S\), then \(P(A)\leq P(S)=1\).

• \(P(B- A)=P(B)-P(A\cap B)\);

Proof: We stat by noticing that \(B=(B-A)\cup(A\cap B)\). Additionally, \((B-A)\cap(A\cap B)=\emptyset\). (Check these two statements!). \[\begin{aligned} P(B)=P((B-A)\cup(A\cap B))=\underbrace{P(B-A)+P(A\cap B)}_{A\cap B\text{ and }(B-A)\text{ are disjoint}} \end{aligned}\]

• \(P(A\cup B)=P(A)+P(B)-P(A\cap B)\).

Proof: We start by noticing that \(A\cup B=(A-B)\cup(A\cap B)\cup (B-A)\). Then, \[\begin{aligned} P(A\cup B)&=P(A-B)+P(A\cap B)+P(B-A)\\ &=P(A)+P(B)-P(A\cap B), \end{aligned}\] the second equality following from the previous property.

5.1 Laplace’s definition of probability

Let \(S\) be a sample space such that

• \(S\) is composed of \(n\) elements \((\# S=n)\)

• all outcomes are distinct and equally likely.

Let \(A\) be an event of \(S\): \[P(A)=\frac{\# A}{\# S}.\]

5.2 Relative frequency definition of probability

Relative frequency of an event \(A\): Repeat an experiment \(N\) times and assume that the event \(A\) occurs \(n_A\) times throughout these \(N\) repetitions. Then, we say that the relative frequency of \(A\) is \[f_N(A)=\frac{n_A}{N}.\]

Remark: \(f_N\) satisfies the following properties:

• \(0\leq f_N(A)\leq 1\), for all \(A\in S\)

• \(f_N(S)=1\)

• \(f_N(A\cup B)=f_N(A)+f_N(B)\), if \(A\cap B=\emptyset\)

Frequencist interpretation of probability: The frequncist probability of the event \(A\) is given by: \[\lim_{N\to +\infty}f_N(A)=P(A).\] This is also known as Law of Large Numbers.

Relative frequency of occurrence of a tail.

It is clear that: \[P("\text{A tail is obtained}")=\lim_{N\to+\infty}=\frac{1}{2}\]

\(N\)

5.3 Subjective interpretation of probability

• The subjective definition of probability deals with the problem of calculating probabilities when the experiment is not symmetric and cannot be successively repeated.

• Let \(A\) be an event of \(S\). The subjective probability of \(A\) is a number in \([0,1]\) that represents the degree of confidence that a person assigns to the occurrence of \(A\).

6.1 Multiplication rule

Let \(A\) and \(B\) be two events in a sample space \(S\) such that \(P(B)\neq 0\). Then, the multiplication rule \[P(A\cap B)=P(A\vert B)\times P(B).\] In the same way, if \(P(A)\neq 0\), then \[P(A\cap B)=P(B\vert A)\times P(A).\]

The multiplication rule can be generalized for \(3,4,5,\cdots\) events.

Theorem: Let \(A\), \(B\) and \(C\) be three events in a sample space \(S\) such that \(P(B\cap C)\neq 0\). Then, the multiplication rule is \[P(A\cap B\cap C)=P(C)\times P(B\vert C)\times P(A\vert B\cap C).\] Proof: By definition \[P(A\vert B\cap C)=\frac{P(A\cap B\cap C)}{P(B\cap C)}\text{ and }P(C)\times P(B\vert C)=P(B\cap C).\] Therefore, \[P(C)\times P(B\vert C)\times P(A\vert B\cap C)= P(A\cap B\cap C).\]

\[ \begin{aligned} P(``\text{draw 1 Heart (H), 1 Heart and 1 Diamond (D)}'')=\\ =P(H)\times P(H|H)\times P(D\vert H\cap H)=\frac{13}{52}\times\frac{12}{51}\times\frac{13}{50}. \end{aligned} \] Suppose know that we draw 4 cards. What is the probability that we draw in order 1 Heart (H), 1 Heart (H), 1 Diamond (D) and 1 Club (C).

\[ \begin{aligned} &P(``\text{draw 1 Heart (H), 1 Heart, 1 Diamond (D) \,and \,1 Club (C)}'')=\\ &P(H)P(H|H)P(D\vert H\cap H)P(C\vert H\cap H\cap D)=\frac{13}{52}\times\frac{12}{51}\times\frac{13}{50}\times \frac{13}{49}. \end{aligned} \] ## Partition

Partition: Let \(A_1, A_2, \cdots, A_n\) be \(n\) events of \(S\). We say that \(A_1, A_2, \cdots, A_n\) is a partition to \(S\) whenever the following conditions are satisfied:

• the events \(A_1, A_2, \cdots, A_n\) are pairwise disjoint: \[A_i\cap A_j=\emptyset,\quad\forall i\neq j\in\{1,2,\cdots n\};\]

• the of all events is the sample space: \[\bigcup_{i=1}^n A_i= S.\]

Remark:

• If \(B\subset S\), then \[B=\bigcup_{i=1}^{n}(A_i\cap B);\]

• If \(A\subset S\), then \(\{A,\overline{A}\}\) is a partition of \(S\).

6.2 Total probability theorem

Total probability theorem: Let \(B\) be an event of \(S\) and \(A_1, A_2, \cdots, A_n\) a partition of \(S\) such that \(P(A_j)>0\), for all \(j=1,2,\cdots,n\). Then,

\[ \begin{aligned} P(B)&=\sum_{i=1}^n P(B\vert A_i)\times P(A_i)\\ &=\sum_{i=1}^n P(B\cap A_i) \end{aligned} \] Proof: The result comes from the previous remark and the multiplication rule.

where the events \(B_i\), with \(i=1,2,3\), are such that \[B_i=B\cap A_i.\]

6.3 Bayes’ theorem

Bayes’ theorem: Let \(B\) be an event of \(S\) and \(A_1, A_2, \cdots, A_n\) a partition of \(S\). Additionally, assume that \(P(B)>0\) and \(P(A_j)>0\), for all \(j=1,2,\cdots,n\). Then, \[\begin{aligned} P(A_j\vert B)&=\frac{P(A_j)\times P(B\vert A_j)}{P(B)}\\ &=\frac{P(A_j)\times P(B\vert A_j)}{\sum_{i=1}^n P(B\cap A_i)}\\ &=\frac{P(A_j)\times P(B\vert A_j)}{\sum_{i=1}^n P(A_i)\times P(B\vert A_i)} \end{aligned} \]

Proof: By definition, \[P(A_j\vert B)=\frac{P(A_j\cap B)}{P(B)}.\]

Use the multiplication rule to rewrite the numerator and the total probability law to rewrite the denominator.

6.4 Examples

image

\[ P(\mbox{Choose box 1})=\frac{1}{2} \]

Let’s start from the beginning:

\(A=``\)Select a blue ball\("\) \(B_i=\)“Choose box" \(i\), \(i=1,2\).

\(P(A\vert B_1)=\frac{1}{3},\quad P(A\vert B_2)=\frac{2}{3}\)

\(P(A)=?\)

image

\[ \begin{aligned} P(B_i)=\frac{1}{2},\quad P(A\vert B_1)=\frac{1}{3}\\ P(A\vert B_2)=\frac{2}{3} \end{aligned} \]

From the total probability theorem: \[ \begin{aligned} P(A)&=P(A\cap B_1)+P(A\cap B_2)\\ &=P(A\vert B_1)P(B_1)\\ &+P(A\vert B_2)P(B_2)=\frac{1}{2} \end{aligned} \] Then,

\[ \begin{aligned} P(B_1\vert A)=&\frac{P(A\vert B_1)P(B_1)}{P(A)}\\ =&\frac{1/6}{1/2}=\frac{1}{3} \end{aligned} \]

Compute the probability that the price increases (PI) and probability that the demand increases (PI). \[P(PI)=0.55\quad\text{and}\quad P(DI)=0.25\]

\[ P(PI~\vert~ DI)=\frac{P(PI~\cap~ DI)}{P(DI)}=\frac{0.1}{0.25}=\frac{2}{5}. \]

Compute the probability that the demand increase (DI) given that the price increases (PI). \[ \begin{aligned} P(DI~\vert~ PI)=&\frac{P(PI~\vert~ DI)P(DI)}{P(PI)}\\ =&\frac{P(PI~\cap~ DI)}{P(PI)}=\frac{2}{11} \end{aligned} \]