Autocovariance of Continuous Joint Probability Distribution

Time-Frequency Signal and System Analysis

In Time Frequency Analysis, 2003

Joint distributions generalize single-variable distributions that measure the energy content of some physical quantity in a signal. Given a quantity a represented by the Hermitian (self-adjoint) operator A, we obtain the density |(IF _A s)(a)|² measuring the "a content" of the signal s simply by squaring the projection of s onto the formal eigenfunctions ¹ $u_a^{\text{A}}$ of A [5]

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780080443355500255

Load and Global Response of Ships

J.J. Jensen , in Elsevier Ocean Engineering Series, 2001

3.1.6 Probability Distributions of Several Variables

The probability distribution F(x ₁, x ₂,…,x_n ) of n random variables X_i , i =   1,2,…, n, is defined by

(3.55) $F\left(x_1,x_2,x_3,\dots ,x_n\right)=P\left(X_1\le x_1,X_2\le x_2,\dots ,X_n\le x_n\right)$

as a generalisation of Eq.(3.1). For continuous variables, the joint probability density function p(x ₁, x ₂ ,…,x_n ) is given by

(3.56) $F\left(x_1,x_2,\dots ,x_n\right)={\displaystyle \underset{a_1}{\overset{x_1}{\int }}{\displaystyle \underset{a_2}{\overset{x_2}{\int }}\cdots {\displaystyle \underset{a_n}{\overset{x_n}{\int }}p\left(u_1,u_2,\dots ,u_n\right)d{u}_{1}d{u}_2\dots d{u}_n}}}$

where a_i is the lower boundary on X_i. Provided that F(x ₁ , x ₂ ,…,x_n ) is differentiable for all values of x_i, then

(3.57) $p\left(x_1,x_2,\dots ,x_n\right)=\frac{{\partial }^{n}F\left(x_1,x_2,\dots ,x_n\right)}{\partial x_1\partial x_2\cdots \partial x_n}$

If the individual random variables X_i are statistically independent, then Eqs. (3.55) - (3.57) yield

(3.58) $F\left(x_1,x_2,\dots ,x_n\right)=F_{x_1}\left(x_1\right)F_{x_2}\left(x_2\right)\dots F_{x_n}\left(x_n\right)$

and

(3.59) $p\left(x_1,x_2,\dots ,x_n\right)=p_{x_1}\left(x_1\right)p_{x_2}\left(x_2\right)\dots p_{x_n}\left(x_n\right)$

where $F_{x_i}\left(x_i\right)$ and $p_{x_i}\left(x_i\right)$ are denoted the marginal distribution and marginal density functions, respectively. For statistically dependent variables, the marginal density function for X_i becomes

(3.60) $p_{x_i}\left(x_i\right)\equiv {\displaystyle \underset{a_1}{\overset{b_1}{\int }}\cdots {\displaystyle \underset{a_{i-1}}{\overset{b_{i-1}}{\int }}{\displaystyle \underset{a_{i+1}}{\overset{b_{i+1}}{\int }}{\displaystyle \underset{a_n}{\overset{b_n}{\int }}p\left(x_1,x_2,\dots ,x_n\right)d{x}_1\dots d{x}_{i-1}d{x}_{i+1}\dots d{x}_n}}}}$

where b_j is the upper boundary on X_j. The marginal distribution is obtained by integration of Eq.(3.60):

(3.61) $F_{x_i}\left(x_i\right)={\displaystyle \underset{a_i}{\overset{x_i}{\int }}{p}_{x_i}\left(u_i\right)d{u}_i}$

The marginal distribution function $F_{x_i}\left(x_i\right)$ expresses the probability that a variable X_i is less than or equal to x_i, irrespective of the values of all the other variables X_j, j =  1, .., n ; j  ≠ i. If, on the contrary, the values of X_j are known, i.e. X_j   = x_j , j =   1,2, .., n ; j   ≠   i, then the conditional probability distribution for X_i is defined as

(3.62) $\begin{array}{l}P\left(X_i|X_j=x_j;j=1,2,..,n;j\ne i\right)=\\ F\left(x_i|x_1,x_2,..,x_{i-1},x_{i+1},..,x_n\right)=\frac{{\displaystyle \underset{a_i}{\overset{x_i}{\int }}p\left(x_1,x_2,..,x_{i-1},u_i,x_{i+1},..,x_n\right)d{u}_i}}{{\displaystyle \underset{a_i}{\overset{b_i}{\int }}p\left(x_1,x_2,..,x_{i-1},u_i,x_{i+1},..,x_n\right)d{u}_i}}\end{array}$

where the denominator is the marginal distribution of X ₁ , X ₂ , .., X _{i – 1}, X _{i +  1} , ..,X_n. The conditional probability density function is obtained by replacing the numerator on the right-hand side with p(x ₁ , x ₂ , ..,x_n ), i.e. by differentiation of Eq.(3.62) with respect to x_i .

For two random variables (X, Y) Eqs.(3.60) - (3.62) yield the following relation between joint, marginal and conditional probability densities:

(3.63) $p\left(x,y\right)=p\left(x|y\right)p_y\left(y\right)$

The moments E[G(X ₁ , X ₂ , .., X_n )] of any combination G(Χ ₁ , X ₂ , .., X_n ) of the random variables are defined by

(3.64) $E\left[G\left(X_1,X_2,..,X_n\right)\right]={\displaystyle \underset{a_1}{\overset{b_1}{\int }}{\displaystyle \underset{a_2}{\overset{b_2}{\int }}..}}{\displaystyle \underset{a_n}{\overset{b_n}{\int }}G\left(x_1,x_2,..,x_n\right)p\left(x_1,x_2,..,x_n\right)d{x}_{1}d{x}_2..d{x}_n}$

The most useful moments are the central moments:

(3.65) ${\zeta }_{m_1,m_2,\dots ,m_n}=E\left[{\left(X_1-{\mu }_1\right)}^{m_1}{\left(X_2-{\mu }_2\right)}^{m_2}\cdots {\left(X_n-{\mu }_n\right)}^{m_n}\right]$

where the mean values are given as

(3.66) $\begin{array}{l}{\mu }_i=E\left[X_i\right]={\displaystyle \underset{a_1}{\overset{b_1}{\int }}{\displaystyle \underset{a_2}{\overset{b_2}{\int }}..{\displaystyle \underset{a_n}{\overset{b_n}{\int }}{x}_{i}p\left(x_1,x_2,..,x_n\right)d{x}_{1}d{x}_2..d{x}_n}}}\\ ={\displaystyle \underset{a_i}{\overset{b_i}{\int }}{x}_i{p}_{\mathit{xi}}\left(x_i\right)d{x}_i}\end{array}$

Of special importance is the covariance matrix $\underset{¯}{\underset{¯}{\Sigma }}$ :

(3.67) $\underset{¯}{\underset{¯}{\Sigma }}=\left[\begin{array}{lll}{\sigma }_{11}\hfill & {\sigma }_{12}\cdots \hfill & {\sigma }_{1n}\hfill \\ .\hfill & \hfill & .\hfill \\ .\hfill & \hfill & .\hfill \\ {\sigma }_{n1}\hfill & \hfill & {\sigma }_{nn}\hfill \end{array}\right]$

with the components

(3.68) ${\sigma }_{ij}\equiv cov\left(x_i,x_j\right)=E\left[\left(X_i-{\mu }_i\right)\left(X_j-{\mu }_j\right)\right]$

The diagonal term σ_ii is seen to be the variance of the variable X_i.

A non-dimensional measure of covariance is the correlation matrix $\underset{¯}{\underset{¯}{\rho }}$ :

(3.69) $\underset{¯}{\underset{¯}{\rho }}=\left[\begin{array}{lll}{\rho }_{11}\hfill & {\rho }_{12}\cdots \hfill & {\rho }_{1n}\hfill \\ .\hfill & \hfill & .\hfill \\ .\hfill & \hfill & .\hfill \\ {\rho }_{n1}\hfill & \hfill & {\rho }_{nn}\hfill \end{array}\right]$

where each of the correlation coefficients is defined as

(3.70) ${\rho }_{ij}=\frac{{\sigma }_{ij}}{s_i{s}_j}$

by use of the standard deviations

(3.71) $s_i=\sqrt{E\left[{\left(X_i-{\mu }_i\right)}^2\right]}=\sqrt{{\sigma }_{ii}}$

as normalisation factors. If the variables X_i and X_j are independent, then

(3.72) ${\sigma }_{ij}=E\left[\left(X_i-{\mu }_i\right)\right]E\left[\left(X_j-{\mu }_j\right)\right]=0$

due to Eq.(3.66). Thus, statistically independent pairs of random variables have zero off-diagonal covariance and correlation coefficients. The reverse is not always true but holds for instance for the multivariate normal distribution defined by the joint probability density function:

(3.73) $p\left(x_1,x_2,\dots ,x_n\right)=\frac{1}{{\left(\sqrt{2\pi }\right)}^n\sqrt{\left|\underset{¯}{\underset{¯}{\Sigma }}\right|}}\exp \left\{-\frac{1}{2}{\left(\underset{¯}{x}-\underset{¯}{\mu }\right)}^T{\underset{¯}{\underset{¯}{\Sigma }}}^{-1}\left(\underset{¯}{x}-\underset{¯}{\mu }\right)\right\}$

where $\underset{¯}{\underset{¯}{\Sigma }}$ is given by Eq.(3.67). | | denotes the matrix determinant and the subscript bar denotes a vector.

This joint distribution clearly becomes the product of the density functions of each of the variables x_i if σ_ij =  0 for i   ≠   j. Thus, in this case, zero correlation also implies statistical independence.

Example 3.1.3

Consider two variables (X ₁ , X ₂) with the joint probability density function

$p\left(x_1,x_2\right)=\frac{6}{7}\left(x_1+x_2\right)+\frac{1}{7};\left(x_1,x_2\right)\in \left[0,1\right]$

It is seen that p(x ₁ , x ₂) satisfies

${\displaystyle \underset{0}{\overset{1}{\int }}{\displaystyle \underset{0}{\overset{1}{\int }}p\left(x_1,x_2\right)d{x}_{1}d{x}_2=1}}$

as well as

$p\left(x_1,x_2\right)\ge 0\mathrm{f}\mathrm{o}\mathrm{r}\left(x_1,x_2\right)\in \left[0,1\right]$

as required for a probability density function. Furthermore, the mean values are found to be

${\mu }_i={\displaystyle \underset{0}{\overset{1}{\int }}{\displaystyle \underset{0}{\overset{1}{\int }}{x}_{i}p\left(x_1,x_2\right)d{x}_{1}d{x}_2=\frac{9}{14};i=1,2}}$

and the covariance to be

$\begin{array}{l}cov\left(x_1,x_2\right)=E\left[\left(X_1-{\mu }_1\right)\left(X_2-{\mu }_2\right)\right]=\\ {\displaystyle \underset{0}{\overset{1}{\int }}{\displaystyle \underset{0}{\overset{1}{\int }}\left(x_1-{\mu }_1\right)\left(x_2-{\mu }_2\right)p\left(x_1,x_2\right)}d{x}_{1}d{x}_2=0}\end{array}$

The variables X ₁ and X ₂ are thus uncorrelated. By application of Eq.(3.60) the marginal probability densities become

$\begin{array}{l}{p}_{x_1}\left(x_1\right)={\displaystyle \underset{0}{\overset{1}{\int }}p\left(x_1,x_2\right)d{x}_2=\frac{6}{7}{x}_1+\frac{4}{7}}\\ p_{x_2}\left(x_2\right)={\displaystyle \underset{0}{\overset{1}{\int }}p\left(x_1,x_2\right)d{x}_1=\frac{6}{7}{x}_2+\frac{4}{7}}\end{array}$

As

$p\left(x_1,x_2\right)\ne p_{x_1}\left(x_1\right)p_{x_2}\left(x_2\right)$

the two variables are, however, not statistically independent.

The correlation coefficients ρ_ij , Eq.(3.70), are bounded:

(3.74) $-1\le {\rho }_{ij}\le 1$

as

$E{\left[\left(X_i-{\mu }_i\right)\left(X_j-{\mu }_j\right)\right]}^2\le E\left[{\left(X_i-{\mu }_i\right)}^2\right]E\left[{\left(X_j-{\mu }_j\right)}^2\right]$

due to Schwarz' inequality*.

A correlation coefficient equal to zero signifies uncorrelated variables. For ρ_ij  =  ±   1, it follows from Eq.(3.70) that

$E\left[\left(X_i-{\mu }_i\right)\left(X_j-{\mu }_j\right)\right]=\pm s_i{s}_j$

This can only be satisfied for

$\frac{X_i-{\mu }_i}{s_i}=\pm \frac{X_j-{\mu }_j}{s_j}$

which is a complete correlation between X_i and X_j. The value of ρ_ij therefore serves as a convenient measure of the correlation between two random variables.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/S1571995201800052

Biomethane for transport applications

Mattias Svensson , in The Biogas Handbook, 2013

18.2.3 Synergies of joint distribution and utilization of biomethane and natural gas

The synergies of joint distribution of biomethane and natural gas are quite obvious. It is the same molecule, so allowing biomethane to utilize the natural gas transport infrastructure decreases the total costs and makes it possible to reach full utilization of the energy of the biogas potential ( Thamsiriroj et al. 2011). At the same time, the renewability of the energy gas infrastructure is increased.

When the natural gas grid distribution system is not an option, several synergies are at hand when allowing for joint utilization of natural gas and biomethane for automotive fuel purposes. The irrefutable environmental benefits of biomethane make it the preferred choice at all times, but in an emerging market situation its production is too insecure and small to adapt smoothly. Here, natural gas can not only initiate and accelerate market penetration during the build-up of the biomethane production capacity and hence facilitate the unavoidable hen-and-egg situation, but can also serve as a backup and secure supply source in the event of production failures or sudden growth in demand.

The Swedish NGV market is a showcase for this type of synergy. Gas grid coverage is limited to the west coast of Sweden, making it necessary to utilize biogas as the main source of gas in the rest of the country. Natural gas in compressed and liquefied form is used as backup to sustain the biomethane market development. At times of accelerated market expansion, the use of natural gas may increase for a time, but customer preferences motivate the gas suppliers to strive for a growing share of renewable methane, even in the parts of Sweden with natural gas grid access. Over time, the volumes of biomethane on the Swedish NGV market have continually increased, in 2011 reaching 60% on an energy basis in a total market of more than 1200 GWh, supplying 38 600 vehicles, of which a significant portion are buses.

Road transport of biomethane should be avoided for larger volumes because grid transport is so much better in terms of both costs and energy expenditure. This is addressed by investing in local gas grids. The expansion and connection of local grids to the national grid is a natural progression in an expanding biomethane market, once again showing the importance of natural gas and biomethane working together in the same market.

It can be envisaged that LBG will change the market conditions in a very positive manner for countries with conditions similar to Sweden. The three types of distribution – by grid, LNG by road and CNG by road – will coexist, fulfilling different needs of the market.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B978085709498850018X

Modeling Financial Data with Stable Distributions

John P. Nolan , in Handbook of Heavy Tailed Distributions in Finance, 2003

7 Multivariate application

Here we will examine the joint distribution of the German Mark and the Japanese Yen. The data set is the one described above in the univariate example. We are interested in both assessing whether joint distribution is bivariate stable and in estimating the fit.

Figure 9 shows a sequence of smoothed density, q-q plot and variance stabilized p-p plot for projections in 8 different directions: $\pi /2$ , $\pi /3$ , $\pi /4$ , $\pi /6,0,$ $-\pi /6,$ $-\pi /4,$ $-\pi /3$ . (We restrict to the right half plane because projections in the left half plane are reflections of those in the right half plane.) These projections are similar to Figure 2, in fact the fifth row of Figure 9 is exactly the same as Figure 2. Except on the extreme tails, the stable fit does a goodjob of describing the data.

Fig. 9. Projection diagnostics for the German Mark and Japanese Yen exchange rates.

The projection functions $\alpha \left(t\right)$ , $\beta \left(t\right)$ , $\gamma \left(t\right)$ , and $\delta \left(t\right)$ were estimated and used to compute an estimate of the spectral measure using the projection method. The results are shown in Figure 10. It shows a discrete estimate of the spectral measure (with $m=100$ evenly spaced point masses) in polar form, a cumulative plot of the spectral measure in rectangular form, and then four plots for the parameter estimates $\left(\alpha \right(t),\beta (t),\gamma (t),\delta (t\left)\right)$ . Also on the $\alpha \left(t\right)$ plot is a horizontal line showing the average value of all the estimated indices which is taken as the estimate of the common α that should come from jointly stable distribution. The plots of $\beta \left(t\right)$ and $\gamma \left(t\right)$ also show the skewness and scale functions computed from the estimated spectral measure substituted into (9). These curves, which are based on ajoint estimate of the spectral measure, are indistinguishable from the direct, separate estimates of the directional parameters.

Fig. 10. Estimation results for the German Mark and Japanese Yen exchange rates.

The fitted spectral measure was used to plot the fitted bivariate density shown in Figure 11. The spread of the spectral measure is spiky, and masks a pattem that is more obvious in the density surface: the approximate elliptical contours of the fitted density. This suggests modeling the data by a sub-Gaussian stable distribution, a topic discussed in the next section.

Fig. 11. Estimated density surface and level curves for a bivariate stable fit to the German Mark and Japanese Yen exchange rates.

Some comments on these plots. The polar plots of the spectral measure show a unit circle and lines connecting the points $({\theta }_j,r_j)$ , where ${\theta }_j=2\pi (j-1)/m$ and $r_j=1+({\lambda }_j/{\lambda }_{\max })$ , where ${\lambda }_{\max }=\max {\lambda }_j$ . The polar plots are spiky, because we are estimating a discrete object. What should be looked at is the overall spread of mass, not specific spikes in the plot. In cases where the spectral measure is really smooth, it may be appropriate to smooth these plots out to better show it's true nature. In cases where the measure is discrete, i.e., the independent case, then one wants to emphasize the spikes. So there is no satisfactory general solution and wejust plot the raw data.

Finally, most graphing programs will set vertical scale so that the graph fills the graph. This emphasizes minor Iiuctuations in the data that are not of practical significance. In the graphs below, the vertical scales for the parameter functions $\alpha \left(t\right)$ , $\beta \left(t\right)$ , $\gamma \left(t\right)$ are respectively [0, 2], [-1, 1], and $[0,1.2\times \max \gamma (t\left)\right]$ . These bounds show how the functions vary over their possible range. For $\delta \left(t\right)$ , we used the bounds $[-1.2\times \max |\delta \left(t\right)|,1.2\times \max |\delta \left(t\right)\left|\right]$ , which visually exaggerates the changes in $\delta \left(t\right)$ . A scale that depends on $\max \gamma \left(t\right)$ may be more appropriate.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780444508966500054

THEORY OF MULTIPLICITY IN NUCLEAR SAFEGUARDS

IMRE PÁZSIT , LÉNÁRD PÁL , in Neutron Fluctuations, 2008

11.2 BASIC EQUATIONS

Since the number distribution of neutrons and gamma quanta generated in the sample will be derived by starting with one initial event, a backward equation formalism will be used. Then, as usual, one has to proceed in two steps, first by deriving an equation for the distribution of neutrons induced by one initial particle, and then another equation connecting the source-induced and the single-particle-induced distributions.

In order to investigate the joint distribution of the random variables v and μ , define the probability

(11.29) $\mathcal{P}\{v=n_{1,}\mu =n_2|{ℐ}_r\}=w(n_1,n_2|1)$

that the numbers of neutrons and gamma quanta emitted from a sample are exactly n ₁ and n ₂, respectively, provided that the cascade was started by one neutron. ⁴ One can write that

(11.30) $w(n_1,n_2|1)=(1-\text{p}){\delta }_{n_{1,}1}{\delta }_{n_{2,}0}+\text{p}{\displaystyle \sum _{k=0}^{\infty }{p}_r(k){\displaystyle \sum _{\ell =0}^{\infty }{f}_r}}(\ell ){\displaystyle \sum _{\begin{array}{l}{n}_{11}+\dots +n_{1k}=n_1\\ n_{21}+\dots +n_{2k}=n_2-\ell \end{array}}{\displaystyle \prod _{i=1}^{k}w(n_{1i,}{n}_{2i}|1).}}$

Introducing the generating function

(11.31) $u(z_1,z_2|1)\equiv u(z_1,z_2)={\displaystyle \sum _{n_1=0}^{\infty }{\displaystyle \sum _{n_2=0}^{\infty }w(n_1,n_2|1)}}{z}_1^{n_1}{z}_2^{n_2},$

one obtains

(11.32) $u(z_1,z_2)=(1-\text{p})z_1+\text{p}{r}_r(z_2)q_r[u(z_1,z_2)].$

Let

(11.33) $\mathcal{P}\{v=n_1,\mu =n_2|{ℐ}_s\}=W(n_1,n_2|{ℐ}_s)$

be the probability that the numbers of neutrons and gamma quanta emitted from a sample are exactly n ₁ and n ₂, respectively, provided that the cascade was started by one source event ${ℐ}_s$ . Since

(11.34) $w(n_1,n_2|{ℐ}_s)={\displaystyle \sum _{\ell =0}^{\infty }{f}_s(\ell ){\displaystyle \sum _{k=0}^{\infty }{p}_s}}(k){\displaystyle \sum _{\begin{array}{l}{n}_{11}+\dots +n_{1k}=n_1\\ n_{21}+\dots +n_{2k}=n_2-\ell \end{array}}{\displaystyle \prod _{i=1}^{k}w(n_{1i,}{n}_{2i}|1),}}$

it can be immediately shown that the generating function

(11.35) $U(z_1,z_2|{ℐ}_s)={\displaystyle \sum _{n_1=0}^{\infty }{\displaystyle \sum _{n_2=0}^{\infty }W(n_1,n_2|{ℐ}_s)}}{z}_1^{n_1}{z}_2^{n_2}$

satisfies the equation

(11.36) $U(z_1,z_2|{ℐ}_s)\equiv U(z_1,z_2)=r_s(z_2)q_s[u(z_1,z_2)].$

From equations (11.32) and (11.36) all the joint and individual moments and probability distributions of the numbers of the generated neutrons and gamma photons can be derived. The individual distributions for neutrons and gamma quanta are contained as special cases that can be obtained by taking z ₂ = 1 and z ₁ = 1, respectively. They will be first described, before turning to the joint moments and distributions.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780080450643500144

11th International Symposium on Process Systems Engineering

Zhonggai Zhao , ... Fei Liu , in Computer Aided Chemical Engineering, 2012

3.2 Smoothing for Joint distribution

If t   <   T , the joint distribution is [R. B. Gopaluni, 2008]

(8) $p\left(x_{i,t},x_{i,t+1}|\text{Y}\right)=p\left(x_{i,t+1}|\text{Y}\right)\frac{p\left(x_{i,t+1}|x_{i,t}\right)p\left(x_{i,t}|y_{i,1:t},Y_{1:i}\right)}{\int p\left(x_{i,t+1}|x_{i,t}\right)p\left(x_{i,t}|y_{i,1:t},Y_{1:i}\right)d{x}_{i,t}}$

Then the smoother weights are ${\omega }_{i,t,t+1}^{\left(n\right)}=\frac{{\eta }_{i,t}^n}{{\sum }_{n=1}^N{\eta }_{i,t}^n}$ , where ${\eta }_{i,t}^n={\omega }_{i,t}^{\left(n\right)}{\omega }_{i,t+1|\mathit{IT}}^{\left(n\right)}\frac{p\left(x_{i,t+1}^n|x_{i,t}^n\right)}{{\sum }_{k=1}^N{\omega }_{i,t}^{\left(k\right)}p\left(x_{i,t+1}^n|x_{i,t}^k\right)}$ .

If t = T, the joint distribution involves the initial states as

(9) $\begin{array}{l}p\left(x_{i,1},x_{i,T},x_{i+1,1}|\text{Y}\right)=p\left(x_{i+1,1}|\text{Y}\right)\frac{p\left(x_{i+1,1}|x_{i,1},x_{i,T}\right)p\left(x_{i,T}|Y_{1:i}\right)p\left(x_{i,1}|Y_{1:i}\right)}{\int p\left(x_{i+1,1}|x_{i,1},x_{i,T}\right)p\left(x_{i,T}|Y_{1:i}\right)p\left(x_{i,1}|Y_{1:i}\right)d{x}_{i,1}d{x}_{i,T}}\\ =\sum _{n=1}^N{\omega }_{i,T,1}^n\delta \left(x_{i+1,1}-x_{i+1,1}^n\right)\delta \left(x_{i,1}-x_{i,1}^n\right)\delta \left(x_{i,T}-x_{i,T}^n\right)\end{array}$

The smoother weights are ${\omega }_{i,T,1}^{\left(n\right)}=\frac{{\eta }_{i,T}^n}{{\sum }_{n=1}^N{\eta }_{i,T}^n}$ where ${\eta }_{i,t}^n={\omega }_{i,T}^{\left(n\right)}{\omega }_{i+1|\mathit{iT}}^{\left(n\right)}{\omega }_{i+1,1|\mathit{IT}}^{\left(n\right)}\frac{p\left(x_{i+1,1}^n|x_{i,T}^n,x_{i,1}^n\right)}{{\sum }_{k=1}^N{\omega }_{i,1|\mathit{iT}}^{\left(k\right)}{\omega }_{i,T}^{\left(k\right)}p\left(x_{i+1,1}^n|x_{i,T}^k,x_{i,1}^k\right)}$ .

To summarize, the Q function can be expressed as

10 $\begin{array}{l}Q\left({\theta }_i,\theta \right)={\displaystyle \sum _{k=1}^N{\omega }_{1,\left.1\right|\mathit{IT}}^k\mathit{log}{p}_{\theta }\left(x_{1,1}^k\right)+{\displaystyle \sum _{i=2}^I{\displaystyle \sum _{j=2}^T{\displaystyle \sum _{k=1}^N{\omega }_{i,j-1,j}^k}\mathit{log}}}}{p}_{\theta }\left(\left.x_{i,j}^k\right|x_{i,j-1}^k\right)+\\ {\displaystyle \sum _{i=2}^I{\displaystyle \sum _{k=1}^N{\omega }_{i-1,T,1}^k\mathit{log}{\mathit{p}}_{\theta }\left(\left.x_{i,1}^k\right|x_{i-1,T}^k,x_{i-1,1}^k\right)}}+\\ {\displaystyle \sum _{j=2}^T{\displaystyle \sum _{k=1}^N{\omega }_{1,j-1,j}^k\mathit{log}{\mathit{p}}_{\theta }}}\left(\left.x_{1,j}^k\right|x_{1,j-1}^k\right)+{\displaystyle \sum _{i=1}^I{\displaystyle \sum _{j=1}^T{\displaystyle \sum _{k=1}^N{\omega }_{i,j\left|\mathit{IT}\right.}^k}\mathit{log}}}{p}_{\theta }\left(\left.y_{i,j}\right|x_{i,j}^k\right)\end{array}$

If an analytical solution of θ to maximize (10) is intractable, numerical methods can be employed to obtain the parameter estimation. Computations of (10) and θ are iterated until the convergence of θ .

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780444595065500183

Time varying signals

Grant E Hearn , Andrew V Metcalfe , in Spectral Analysis in Engineering, 1995

3.5.1 Stationarity

A stochastic process is strictly stationary if the joint distribution of

$\left(X_{t_l},\mathrm{\dots },X_{t_n}\right)$

is the same as the joint distribution of

$\left(X_{t_1+k,\mathrm{\dots },}{X}_{t_n+k}\right)$

for any set of times {t ₁,…, t_n } and any value of k. As we are limiting our attention to the first two moments we will adopt a rather weaker notion of second-order stationarity. If we are prepared to assume a multivariate normal distribution, then the distributions are equivalent. A stochastic process is second-order stationary if

(3.4) $E\left(X_t\right)=\mu ,\mathrm{a}\text{constant}\text{for}\text{all}t$

and γ(t ₁, t ₂) depends only on the difference between t ₂ and t ₁, known as the lag and usually denoted by k. That is

(3.5) $\begin{array}{cc}\text{cov}\left(X_t,X_{t+k}\right)-\gamma \left(k\right),& \text{where}k=t_2-t_1\end{array}$

and the acf becomes

(3.6) $\rho \left(k\right)=\gamma \left(k\right)/\gamma \left(0\right)$

It follows from the definition of ρ(k) that ρ(0) is 1 and ρ(-k) is equal to ρ(k). Also, as ρ(k) is a special case of a correlation coefficient it is less than or equal to 1 in absolute value. A plot of ρ(k) against k is called a correlogram. In summary, a random process is second-order stationary if its mean and variance do not change with time and if the covariance depends only on lag and not on absolute time. If a random process includes a trend or seasonal effects, or a change in variance, it is non-stationary. A plot of the data is an important first step in looking for non-stationarity in time series. Figures 3.5(a) and (b) show realizations from non-stationary random processes. The first includes an obvious trend, the second an obvious increase in variance with time.

Fig. 3.5. (a) An obvious trend with time; (b) increase in variance with time

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780340631713500051

Application of Bayes Network analysis to RWGD siting

Thea Hincks , ... Steve Sparks , in Geological Repository Systems for Safe Disposal of Spent Nuclear Fuels and Radioactive Waste (Second Edition), 2017

19.2 Bayes Networks: introduction and methods

The basic concept behind the treatment of uncertainty in Bayes Networks is conditional probability (Jensen, 2001). A BN is a directed acyclic graph, comprising of a set of variables (nodes) representing states of the system, together with a set of directed links (arcs) representing conditional dependencies between the nodes (Jensen, 2001). Bayes networks form the basis of many expert systems, from medical diagnosis and decision making to bioinformatics, and more recently, widespread application in hazard and risk assessment (Spiegelhalter et al., 1993; Ale et al., 2008; Aspinall and Woo, 2014).

For the simple case where nodes can be assigned discrete states, node relationships are described by conditional probability tables (CPTs, e.g., Table 19.1). Continuous variables are represented by joint probability distributions. Static networks describe the state of a system with no history (Fig. 19.1). Dynamic BNs (e.g., the special case Hidden Markov Model, Fig. 19.2) describe systems where the state of the system at any time is dependent on any number of past states.

Table 19.1. Example conditional probability table

	P(Y|X)
	Y   =   0	Y   =   1
X   =   0	0.8	0.2
X   =   1	0.96	0.04

Figure 19.1. A simple Bayes Network. For a Bayes Network B   =   {X,Y,U}, let X represent the set of unobserved or hidden states (nodes shaded gray), Y the set of observable states (the evidence) and U the set of directed links between nodes. Arrows indicate direction of causality or influence.

Figure 19.2. The Hidden Markov Model. A simple dynamic model used to infer the state of a hidden variable X (shaded gray) given an observation on Y (Murphy, 2002). This is a first-order dynamic model—the state of X at time t is influenced by its state at time t − 1. Higher order processes may be tied over several time slices. The prior distribution P(X₀) and transition model P(X_t|X_t−1) can be learned from data or determined by expert elicitation or physical models.

For data-rich applications, network parameters and structure can be estimated entirely from data, using learning algorithms (e.g., Murphy, 2002; Hanea et al., 2010; Nickel et al., 2015). Where data are limited, or it is necessary to model states or scenarios outside the set of observations, expert judgment can be applied to develop a more comprehensive model (Morales et al., 2008; Aspinall and Cooke, 2013). When the model is fully parameterized, inference can be performed in any direction, e.g., to estimate the probability distribution of an unobservable node, or forecast future states.

Methods of analyzing information in the BN, such as entropy (a measure of unpredictability) and "mutual information" (MI) (see below, and results in Section 19.4), have considerable value in communicating model uncertainty, sensitivity to different assumptions, and the strength of relationships between nodes. High entropy can result from high natural variability (which cannot be reduced); however it could also be an indication of an inadequate model, resulting from a poor understanding of the processes involved, or an inaccurate representation of the key states and interactions. MI is a measure of conditional dependency between two variables and is an elegant way of expressing dependencies between elements in a system that may not be immediately obvious, especially in complex models with numerous links.

19.2.1 The marginal distribution

For discrete random variables X,Y with joint distribution function P(x,y), the marginal distribution of X, P(x), is the sum of the joint distribution of X and Y, P(x,y), over all values of Y:

(19.1) $P\left(x\right)=\sum _{y}P\left(x,y\right)=\sum _{y}P\left(x|y\right)P\left(y\right)$

19.2.2 Entropy

Entropy is a measure of unpredictability in a system (Bedford and Cooke, 2001). Zero entropy means the state is known exactly, with a probability of 1. Maximum entropy corresponds to all states having equal probability (maximum unpredictability). For a node (variable) X, with n discrete states, the entropy H(X) is defined as follows:

(19.2) $H\left(X\right)=\sum _{i=1}^{n}P\left(x_i\right){\log }_2\left(\frac{1}{P\left(x_i\right)}\right)$

where P(x _i ) is the probability of X being in state x _i .

The maximum entropy for any given node will therefore depend on the number of possible states:

(19.3) $H_{\max }\left(X\right)={\log }_{2}n$

To compare the entropy of nodes with different numbers of states, it can be useful to compute the "normalized entropy" H ₀(X):

(19.4) $H_0\left(X\right)=\frac{H\left(X\right)}{H_{\max }\left(X\right)}=\frac{H\left(X\right)}{{\log }_{2}n}$

19.2.3 Mutual information

The MI of nodes X and Y, MI(X,Y), is the reduction of uncertainty in Y by knowing X. MI can be computed for any pair of nodes, regardless of whether they are directly connected. MI is used in algorithms for estimating network structure from data, as high MI between a pair of nodes indicates a strong conditional dependency (in some direction). Low MI, typically defined by some cut-off point, implies conditional independence (Kane et al., 2003).

As the BN presented here has been developed using expert opinion (rather than observational data), computing the MI gives a measure of the perceived strength of connection between nodes. This can be used to assess the extent to which individual states and processes are expected to impact on the various risk factors.

For two variables X and Y with discrete states, MI(X,Y) is defined as:

(19.5) $\mathrm{MI}\left(X,Y\right)=H\left(Y\right)-H\left(Y|X\right)=H\left(X\right)-H\left(X|Y\right)$

where H(X) denotes the entropy of node X (Ebert-Uphoff, 2006). The function is symmetric in X and Y and does not indicate the direction of influence (does not distinguish X as a parent of Y or vice versa).

MI percentage is the percentage reduction of uncertainty in target node Y due to knowledge of X and is not symmetric in X and Y:

(19.6) $\mathrm{MI}\%\left(X,Y\right)=\frac{H\left(Y\right)-H\left(Y|X\right)}{H\left(Y\right)}\times 100$

To conclude, BNs provide a clear and computationally efficient way to model complex causal relationships between observable and unobservable states and enable diverse sources of information to be jointly interpreted. By representing the system components and interactions graphically, it can be easier to identify conceptual problems or logical inconsistencies in the model and aid discussion and communication of key scientific ideas between both experts and nontechnical stakeholders. This is particularly important where model development requires input from experts from a range of disciplines. Subcomponents of the system can also be developed separately, for ease of discussion and integration of expert judgment, before being integrated into a full BN model.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780081006429000190

Foundations of Distributed Source Coding

Krishnan Eswaran , Michael Gastpar , in Distributed Source Coding, 2009

Definition A.1

Let X, Y be discrete random variables with joint distribution p(x, y). Then the entropy H(X) of X is a real number given by the expression

(A.53) $H\left(X\right)=-E\left[logp\left(X\right)\right]=-\sum _{x}p\left(x\right)logp\left(x\right),$

the joint entropy H(X, Y) of X and Y is a real number given by the expression

(A.54) $H\left(X,Y\right)=-E\left[logp\left(X,Y\right)\right]=-\sum _{x,y}p\left(x,y\right)logp\left(x,y\right),$

and the conditional entropy H(X|Y) of X given Y is a real number given by the expression

(A.55) $H\left(X|Y\right)=-E\left[logp\left(X|Y\right)\right]=H\left(X,Y\right)-H\left(Y\right).$

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780123744852000056

Validation of internal rating systems and PD estimates*

Dirk Tasche , in The Analytics of Risk Model Validation, 2008

3.5 Equivalence of the both descriptions

So far, we have seen two descriptions of the joint distribution of the score variable and the state variable, which are quite different at first glance. However, thanks to Bayes' formula both descriptions are actually equivalent.

Suppose, first that a description of the joint distribution of score and state by the total probability of default p and the two conditional densities f_D and f_N according to Equations 3.1a and 3.1b is known. Then, the unconditional score density f can be expressed as

(3.3a) $f\left(s\right)=p{f}_D\left(s\right)+\left(1-p\right)f_N\left(s\right)\text{,}$

and the conditional PD given that the score variable takes on the value s can be written as

(3.3b) $\mathrm{P}\left[D|S=s\right]=\frac{p{f}_D\left(s\right)}{f\left(s\right)}\text{.}$

Assume now that the unconditional score density f in the sense of Equation 3.2b and the function representing the conditional PDs given the scores in the sense of Equation 3.2a are known. Then, the total PD p can be calculated as an integral of the unconditional density f and the conditional PD as

(3.4a) $p={\int }_{-\infty }^{\infty }\mathrm{P}\left[D|S=s\right]f\left(s\right)\mathrm{d}s\text{,}$

and both the conditional densities of the score variable can be obtained via

(3.4b) $\begin{array}{l}{f}_D\left(s\right)=\mathrm{P}\left[D|S=s\right]f\left(s\right)/p\mathrm{and}\\ f_N\left(s\right)=\mathrm{P}\left[N|S=s\right]f\left(s\right)/\left(1-p\right)\text{.}\end{array}$

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780750681582500147

reedycorwaseed1985.blogspot.com

Source: https://www.sciencedirect.com/topics/engineering/joint-distribution

Share this post