PMML 4.3 - Bayesian Network Models
 PMML4.3 Menu Home Changes XML Schema Conformance Interoperability General Structure Field Scope Header Data Dictionary Mining Schema Transformations Statistics Taxomony Targets Output Functions Built-in Functions Model Verification Model Explanation Multiple Models Association Rules Baseline Models Bayesian Network Cluster Models Gaussian Process General Regression k-Nearest Neighbors Naive Bayes Neural Network Regression Ruleset Scorecard Sequences Text Models Time Series Trees Vector Machine

## PMML 4.3 - Bayesian Network Models

A Bayesian Network (BN) is a directed acyclic graphical model representing a joint probability distribution over a set of random variables. The graphical model of a BN consists of nodes and arcs, where nodes represent the random variables (continuous and/or categorical) and arcs represent the causal dependence relationships between several random variables. Given new data on any of the random variables, the Bayesian network model can be used to infer all the remaining random variables. Thus, a Bayesian network can be used for both probabilistic regression and classification.

If X={X1,X2...,Xn} are a set of n random variables, then a Bayesian network B, over these random variables can be represented as

PrB(X) =
n
i=1
PrB(Xi | ΠXi)

where ΠXi represents the set of parent nodes of Xi and PrB(Xi | ΠXi) represents the conditional probability distribution of Xi, given its parent nodes. If Xi has no parent nodes, then PrB(Xi | ΠXi) represents the marginal probability distribution of Xi. Variables with no parent nodes are often referred to as root variables. Note that, as mentioned above, the variables in a Bayesian network can be categorical or continuous. For continuous variables, conditional or marginal probability distributions are defined whereas for categorical variables, conditional or marginal probability tables are defined.

Let Nobs represent the subset of variables about which new data (D) is available. This new information can be used along with the available Bayesian network model to estimate the posterior probabilities for the variables for which there is no new data (unobserved variables), represented as Nobs, using the Bayes theorem as

Pr(Nobs | Nobs = D) ∝ Pr(Nobs = D | Nobs ) Pr(Nobs)

where Pr(Nobs) and Pr(Nobs | Nobs = D) represent the prior and posterior distributions of the unobserved variables, respectively. Pr(Nobs = D | Nobs ) represents the likelihood of observing the data (D) conditioned on the unobserved variables. Exact computation of the posterior distributions can become computationally expensive; in such cases, approximate techniques such as Monte Chain Monte Carlo (MCMC) methods, Approximate Bayesian Computation (ABC) can be used to obtain the posterior distributions.

Representation of a Bayesian network in a PMML format requires the storage of the following information
• Marginal probability distributions (distribution type, distribution parameters) for continuous root variables.
• Marginal probability tables for categorical root variables, i.e., the possible values and associated probabilities.
• Conditional probability distributions (distribution type and distribution parameters dependent on the parent nodes) for non-root continuous nodes.
• Conditional probability tables (dependent on the parent nodes) for non-root categorical nodes, i.e., the possible values and associated probabilities.

Bayesian Network Model schema

```<xs:element name="BayesianNetworkModel">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="MiningSchema"/>
<xs:element ref="Output" minOccurs="0"/>
<xs:element ref="ModelStats" minOccurs="0"/>
<xs:element ref="ModelExplanation" minOccurs="0"/>
<xs:element ref="Targets" minOccurs="0"/>
<xs:element ref="LocalTransformations" minOccurs="0"/>
<xs:element ref="BayesianNetworkNodes" minOccurs="1"/>
<xs:element ref="ModelVerification" minOccurs="0"/>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="modelName" type="xs:string"/>
<xs:attribute name="functionName" type="MINING-FUNCTION" use="required"/>
<xs:attribute name="algorithmName" type="xs:string"/>
<xs:attribute name="isScorable" type="xs:boolean" default="true"/>
</xs:complexType>
</xs:element>
```

modelName and algorithmName can have arbitrary strings describing the specific model.

functionName can only be classification or regression.

isScorable: This attribute indicates if the model is valid for scoring. If this attribute is true or if it is missing, then the model should be processed normally. However, if the attribute is false, then the model producer has indicated that this model is intended for information purposes only and should not be used to generate results. In order to be valid PMML, all required elements and attributes must be present, even for non-scoring models. For more details, see General Structure.

Node Types

The Bayesian Network can have two types of nodes - discrete and continuous. Discrete nodes have a limited set of possible values. They have value probabilities if they are not conditioned on other nodes, or have a conditional probability table if they are conditioned on other nodes. The sum of the value probabilities must equal 1. The optional attribute count can be used to track the total number of cases used to generate the probabilities.
Continuous nodes can have a continuous value represented by a continuous distribution (The ContinuousDistribution element is described in greater detail below). If the node is conditioned on other nodes, then a conditional probability table is used to specify the distribution. These nodes are represented in the following way:

```<xs:element name="BayesianNetworkNodes">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:choice maxOccurs="unbounded">
<xs:element ref="DiscreteNode"/>
<xs:element ref="ContinuousNode"/>
</xs:choice>
</xs:sequence>
</xs:complexType>
</xs:element>

<xs:element name="DiscreteNode">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="DerivedField" minOccurs="0" maxOccurs="unbounded"/>
<xs:choice maxOccurs="unbounded">
<xs:element ref="DiscreteConditionalProbability"/>
<xs:element ref="ValueProbability" maxOccurs="unbounded"/>
</xs:choice>
</xs:sequence>
<xs:attribute name="name" type="FIELD-NAME" use="required"/>
<xs:attribute name="count" type="REAL-NUMBER" use="optional"/>
</xs:complexType>
</xs:element>

<xs:element name="ContinuousNode">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="DerivedField" minOccurs="0" maxOccurs="unbounded"/>
<xs:choice maxOccurs="unbounded">
<xs:element ref="ContinuousConditionalProbability"/>
<xs:element ref="ContinuousDistribution" maxOccurs="unbounded"/>
</xs:choice>
</xs:sequence>
<xs:attribute name="name" type="FIELD-NAME" use="required"/>
<xs:attribute name="count" type="REAL-NUMBER" use="optional"/>
</xs:complexType>
</xs:element>
```

Conditional Probabilities

When a node is conditioned upon its parent nodes in the network, its value is given by a conditional probability table that specifies the value probabilities for different combinations of values of the parent nodes. DiscreteConditionalProbability elements are used to specify conditional probabilities for discrete nodes. The DiscreteConditionalProbability element contains a ParentValue element and a ValueProbability element, to specify the value probability for each value of the parent node. The schema for representing discrete conditional probability tables is given below.

```<xs:element name="DiscreteConditionalProbability">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="ParentValue" minOccurs="1" maxOccurs="unbounded"/>
<xs:element ref="ValueProbability" minOccurs="1" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="count" type="REAL-NUMBER" use="optional"/>
</xs:complexType>
</xs:element>

<xs:element name="ParentValue">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="parent" type="FIELD-NAME" use="required"/>
<xs:attribute name="value" type="xs:string" use="required"/>
</xs:complexType>
</xs:element>

<xs:element name="ValueProbability">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="value" type="xs:string" use="required"/>
<xs:attribute name="probability" type="PROB-NUMBER" use="required"/>
</xs:complexType>
</xs:element>
```

For continuous nodes, the value is represented by a continuous distribution. The schema allows the parameters of the distribution to be represented by expressions, allowing them to be functions of other node values. ContinuousConditionalProbability elemnts are is used to represent the conditional probability of continuous nodes. In this case, ParentValue is optional, and only needed to specify conditional probabilities for discrete values of parent nodes. For continues parent nodes, the conditional probabilities are represented by incorporating the parent value in the distribution paramenters. The schema for representing continuous distributions and conditional probability tables is given below.

```<xs:element name="ContinuousConditionalProbability">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="ParentValue" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="ContinuousDistribution" minOccurs="1" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="count" type="REAL-NUMBER" use="optional"/>
</xs:complexType>
</xs:element>

<xs:element name="ContinuousDistribution">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:choice>
<xs:element ref="TriangularDistributionForBN"/>
<xs:element ref="NormalDistributionForBN"/>
<xs:element ref="LognormalDistributionForBN"/>
<xs:element ref="UniformDistributionForBN"/>
</xs:choice>
</xs:sequence>
</xs:complexType>
</xs:element>

<xs:element name="TriangularDistributionForBN">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="Mean" minOccurs="1" maxOccurs="1"/>
<xs:element ref="Lower" minOccurs="1" maxOccurs="1"/>
<xs:element ref="Upper" minOccurs="1" maxOccurs="1"/>
</xs:sequence>
</xs:complexType>
</xs:element>

<xs:element name="NormalDistributionForBN">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="Mean" minOccurs="1" maxOccurs="1"/>
<xs:element ref="Variance" minOccurs="1" maxOccurs="1"/>
</xs:sequence>
</xs:complexType>
</xs:element>

<xs:element name="LognormalDistributionForBN">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="Mean" minOccurs="1" maxOccurs="1"/>
<xs:element ref="Variance" minOccurs="1" maxOccurs="1"/>
</xs:sequence>
</xs:complexType>
</xs:element>

<xs:element name="UniformDistributionForBN">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="Lower" minOccurs="1" maxOccurs="1"/>
<xs:element ref="Upper" minOccurs="1" maxOccurs="1"/>
</xs:sequence>
</xs:complexType>
</xs:element>

<xs:element name="Mean">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:group ref="EXPRESSION"/>
</xs:sequence>
</xs:complexType>
</xs:element>

<xs:element name="Lower">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:group ref="EXPRESSION"/>
</xs:sequence>
</xs:complexType>
</xs:element>

<xs:element name="Upper">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:group ref="EXPRESSION"/>
</xs:sequence>
</xs:complexType>
</xs:element>

<xs:element name="Variance">
<xs:complexType>
<xs:sequence>
<xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
<xs:group ref="EXPRESSION"/>
</xs:sequence>
</xs:complexType>
</xs:element>
```

Scoring Procedure with Example

A scoring example with PMML/BN is provided here for illustration. An example BN is shown below, along with the prior probabilities for the nodes. The BN in the figure below has a combination of continuous and discrete nodes. D1, D2, D3, and D4 are discrete nodes, while C1, C2, C3, and C4 are continuous nodes. The prior probabilities and discrete and continuous conditional probability tables are listed below. In the Bayesian network shown below, let us say that new observations on two variables D4, C4 are available, for example, D4 = 0 and C4 = 7. This new information will be used to estimate unobserved variables, i.e., D1, D2, D3, C1, C2, and C3.

 Value of D1 Probability 0 0.3 1 0.7

 Value of D2 Probability 0 0.6 1 0.3 2 0.1

 D1 = 0 D1 = 1 C1 | D1 N(10, 2) N(14, 2)

 D2 = 0 D2 = 1 D2 = 2 C2 | D2 N(6, 1) N(8, 1) N(14, 1)

 C4 | C2 = N(0.1 C22 + 0.6C2 + 1, 2)

 D3 = 0 D3 = 1 C3 | C2, D3 N(0.15C22, 2) N(0.15C2, 1)

 Value of D3 (D1, D2) = (0, 0) (D1, D2) = (0, 1) (D1, D2) = (0, 2) (D1, D2) = (1, 0) (D1, D2) = (1, 1) (D1, D2) = (1, 2) 0 0.1 0.3 0.4 0.6 0.8 0.9 1 0.9 0.7 0.6 0.4 0.2 0.1

 Value of D4 (D3, C3) = (0, C3 < 9) (D3, C3) = (0, 9 < C3 < 11) (D3, C3) = (0, C3 > 11) (D3, C3) = (1, C3 < 9)) (D3, C3) = (1, C3 < 11)) (D3, C3) = (1, C3 > 11)) 0 0.4 0.3 0.6 0.4 0.1 0.3 1 0.6 0.7 0.4 0.6 0.9 0.7

The PMML file for the BN example above are shown here:

```<PMML xmlns="http://www.dmg.org/PMML-4_3" version="4.3">
<DataField dataType="double" name="C1" optype="continuous"/>
<DataField dataType="double" name="C2" optype="continuous"/>
<DataField dataType="double" name="C3" optype="continuous"/>
<DataField dataType="double" name="C4" optype="continuous"/>
<DataField dataType="string" name="D1" optype="categorical">
<Value value="0"/>
<Value value="1"/>
</DataField>
<DataField dataType="string" name="D2" optype="categorical">
<Value value="0"/>
<Value value="1"/>
<Value value="2"/>
</DataField>
<DataField dataType="string" name="D3" optype="categorical">
<Value value="0"/>
<Value value="1"/>
</DataField>
<DataField dataType="string" name="D4" optype="categorical">
<Value value="0"/>
<Value value="1"/>
</DataField>
<BayesianNetworkModel modelName="BN Example" functionName="regression">
<MiningSchema>
<MiningField name="D4" usageType="active"/>
<MiningField name="C4" usageType="active"/>
<MiningField name="D1" usageType="target"/>
<MiningField name="D2" usageType="target"/>
<MiningField name="D3" usageType="target"/>
<MiningField name="C1" usageType="target"/>
<MiningField name="C2" usageType="target"/>
<MiningField name="C3" usageType="target"/>
</MiningSchema>
<BayesianNetworkNodes>
<!--  -->
<DiscreteNode name="D1">
<ValueProbability value="0" probability="0.3"/>
<ValueProbability value="1" probability="0.7"/>
</DiscreteNode>
<!--  -->
<DiscreteNode name="D2">
<ValueProbability value="0" probability="0.6"/>
<ValueProbability value="1" probability="0.3"/>
<ValueProbability value="2" probability="0.1"/>
</DiscreteNode>
<!--  -->
<DiscreteNode name="D3">
<DiscreteConditionalProbability>
<ParentValue parent="D1" value="0"/>
<ParentValue parent="D2" value="0"/>
<ValueProbability value="0" probability="0.1"/>
<ValueProbability value="1" probability="0.9"/>
</DiscreteConditionalProbability>
<DiscreteConditionalProbability>
<ParentValue parent="D1" value="0"/>
<ParentValue parent="D2" value="1"/>
<ValueProbability value="0" probability="0.3"/>
<ValueProbability value="1" probability="0.7"/>
</DiscreteConditionalProbability>
<DiscreteConditionalProbability>
<ParentValue parent="D1" value="0"/>
<ParentValue parent="D2" value="2"/>
<ValueProbability value="0" probability="0.4"/>
<ValueProbability value="1" probability="0.6"/>
</DiscreteConditionalProbability>
<DiscreteConditionalProbability>
<ParentValue parent="D1" value="1"/>
<ParentValue parent="D2" value="0"/>
<ValueProbability value="0" probability="0.6"/>
<ValueProbability value="1" probability="0.4"/>
</DiscreteConditionalProbability>
<DiscreteConditionalProbability>
<ParentValue parent="D1" value="1"/>
<ParentValue parent="D2" value="1"/>
<ValueProbability value="0" probability="0.8"/>
<ValueProbability value="1" probability="0.2"/>
</DiscreteConditionalProbability>
<DiscreteConditionalProbability>
<ParentValue parent="D1" value="1"/>
<ParentValue parent="D2" value="2"/>
<ValueProbability value="0" probability="0.9"/>
<ValueProbability value="1" probability="0.1"/>
</DiscreteConditionalProbability>
</DiscreteNode>
<!--  -->
<ContinuousNode name="C1">
<ContinuousConditionalProbability>
<ParentValue parent="D1" value="0"/>
<ContinuousDistribution>
<NormalDistributionForBN>
<Mean>
<Constant dataType="double">10</Constant>
</Mean>
<Variance>
<Constant dataType="double">2</Constant>
</Variance>
</NormalDistributionForBN>
</ContinuousDistribution>
</ContinuousConditionalProbability>
<ContinuousConditionalProbability>
<ParentValue parent="D1" value="1"/>
<ContinuousDistribution>
<NormalDistributionForBN>
<Mean>
<Constant dataType="double">14</Constant>
</Mean>
<Variance>
<Constant dataType="double">2</Constant>
</Variance>
</NormalDistributionForBN>
</ContinuousDistribution>
</ContinuousConditionalProbability>
</ContinuousNode>
<!--  -->
<ContinuousNode name="C2">
<ContinuousConditionalProbability>
<ParentValue parent="D2" value="0"/>
<ContinuousDistribution>
<NormalDistributionForBN>
<Mean>
<Constant dataType="double">6</Constant>
</Mean>
<Variance>
<Constant dataType="double">1</Constant>
</Variance>
</NormalDistributionForBN>
</ContinuousDistribution>
</ContinuousConditionalProbability>
<ContinuousConditionalProbability>
<ParentValue parent="D2" value="1"/>
<ContinuousDistribution>
<NormalDistributionForBN>
<Mean>
<Constant dataType="double">8</Constant>
</Mean>
<Variance>
<Constant dataType="double">1</Constant>
</Variance>
</NormalDistributionForBN>
</ContinuousDistribution>
</ContinuousConditionalProbability>
<ContinuousConditionalProbability>
<ParentValue parent="D2" value="2"/>
<ContinuousDistribution>
<NormalDistributionForBN>
<Mean>
<Constant dataType="double">14</Constant>
</Mean>
<Variance>
<Constant dataType="double">1</Constant>
</Variance>
</NormalDistributionForBN>
</ContinuousDistribution>
</ContinuousConditionalProbability>
</ContinuousNode>
<!--  -->
<ContinuousNode name="C4">
<ContinuousConditionalProbability>
<ContinuousDistribution>
<NormalDistributionForBN>
<Mean>
<Apply function="+">
<Apply function="*">
<Constant dataType="double">0.1</Constant>
<Apply function="pow">
<FieldRef field="C2"/>
<Constant dataType="integer">2</Constant>
</Apply>
</Apply>
<Apply function="+">
<Apply function="*">
<Constant dataType="double">0.6</Constant>
<FieldRef field="C2"/>
</Apply>
<Constant dataType="integer">1</Constant>
</Apply>
</Apply>
</Mean>
<Variance>
<Constant dataType="double">2</Constant>
</Variance>
</NormalDistributionForBN>
</ContinuousDistribution>
</ContinuousConditionalProbability>
</ContinuousNode>
<!--  -->
<ContinuousNode name="C3">
<ContinuousConditionalProbability>
<ParentValue parent="D3" value="0"/>
<ContinuousDistribution>
<NormalDistributionForBN>
<Mean>
<Apply function="*">
<Constant dataType="double">0.15</Constant>
<Apply function="pow">
<FieldRef field="C2"/>
<Constant dataType="integer">2</Constant>
</Apply>
</Apply>
</Mean>
<Variance>
<Constant dataType="double">2</Constant>
</Variance>
</NormalDistributionForBN>
</ContinuousDistribution>
</ContinuousConditionalProbability>
<ContinuousConditionalProbability>
<ParentValue parent="D3" value="1"/>
<ContinuousDistribution>
<NormalDistributionForBN>
<Mean>
<Apply function="*">
<Constant dataType="double">1.5</Constant>
<FieldRef field="C2"/>
</Apply>
</Mean>
<Variance>
<Constant dataType="double">1</Constant>
</Variance>
</NormalDistributionForBN>
</ContinuousDistribution>
</ContinuousConditionalProbability>
</ContinuousNode>
<!--  -->
<DiscreteNode name="D4">
<DerivedField name="C3_Discretized" optype="categorical" dataType="string">
<Discretize field="C3">
<DiscretizeBin binValue="0">
<Interval closure="openClosed" rightMargin="9"/>
</DiscretizeBin>
<DiscretizeBin binValue="1">
<Interval closure="openClosed" leftMargin="9" rightMargin="11"/>
</DiscretizeBin>
<DiscretizeBin binValue="2">
<Interval closure="openOpen" leftMargin="11"/>
</DiscretizeBin>
</Discretize>
</DerivedField>
<DiscreteConditionalProbability>
<ParentValue parent="D3" value="0"/>
<ParentValue parent="C3_Discretized" value="0"/>
<ValueProbability value="0" probability="0.4"/>
<ValueProbability value="1" probability="0.6"/>
</DiscreteConditionalProbability>
<DiscreteConditionalProbability>
<ParentValue parent="D3" value="0"/>
<ParentValue parent="C3_Discretized" value="1"/>
<ValueProbability value="0" probability="0.3"/>
<ValueProbability value="1" probability="0.7"/>
</DiscreteConditionalProbability>
<DiscreteConditionalProbability>
<ParentValue parent="D3" value="0"/>
<ParentValue parent="C3_Discretized" value="2"/>
<ValueProbability value="0" probability="0.6"/>
<ValueProbability value="1" probability="0.4"/>
</DiscreteConditionalProbability>
<DiscreteConditionalProbability>
<ParentValue parent="D3" value="1"/>
<ParentValue parent="C3_Discretized" value="0"/>
<ValueProbability value="0" probability="0.4"/>
<ValueProbability value="1" probability="0.6"/>
</DiscreteConditionalProbability>
<DiscreteConditionalProbability>
<ParentValue parent="D3" value="1"/>
<ParentValue parent="C3_Discretized" value="1"/>
<ValueProbability value="0" probability="0.1"/>
<ValueProbability value="1" probability="0.9"/>
</DiscreteConditionalProbability>
<DiscreteConditionalProbability>
<ParentValue parent="D3" value="1"/>
<ParentValue parent="C3_Discretized" value="2"/>
<ValueProbability value="0" probability="0.3"/>
<ValueProbability value="1" probability="0.7"/>
</DiscreteConditionalProbability>
</DiscreteNode>
</BayesianNetworkNodes>
</BayesianNetworkModel>
</PMML>
```

Let Nobs, Nobs and D denote the set of observed, unobserved variables and available data, respectively. Using the Bayes theorem, the updated (posterior) probability distributions of the unobserved variables can be estimated as

Pr(Nobs | Nobs = D) ∝ Pr(Nobs = D | Nobs) Pr(Nobs)

In this scoring example, D4, C4 are observed variables (aka inputs). Given the values of D4 = 0, and C4 = 7, the rest of the variables can be inferred using the Bayes theorem as:

Pr⁡(D1,D2,D3,C1,C2,C3D4=0,C4=7 )∝Pr⁡(D4=0,C4=7 | D1,D2,D3,C1,C2,C3) Pr⁡(D1,D2,D3,C1,C2,C3)

Solving the above expression can be computationally expensive; therefore, approximate methods such as Markov Chain Monte Carlo (MCMC) methods are used to obtain posterior distributions of unobserved variables. Note that MCMC are a class of algorithms, not a single algorithm. For example, within the MCMC class of algorithms, we can use the Metropolis-Hastings algorithm in the scoring example to generate samples from the joint posterior distribution. Since the method is based on generation of samples, it is possible that the results are not exactly the same in different executions. All MCMC class of algorithms are stochastic. However, there would not be a drastic variation in the results from multiple executions.

Using the Metropolis-Hastings algorithm, samples from the joint posterior distribution can be obtained and hence marginal distributions can be constructed. For categorical variables, D1, D2, and D3, the change in the probabilities of each of the states due to the new information is given below.

 Prior Posterior D1 = 0 0.3 0.316 D1 = 1 0.7 0.683

 Prior Posterior D2 = 0 0.6 0.847 D2 = 1 0.3 0.153 D2 = 2 0.3 0

 Prior Posterior D3 = 0 0.535 0.437 D3 = 1 0.465 0.563

Similarly, for continuous variables C1, C2, and C3, their entire prior and posterior probability distributions can be constructed. Along the distributions, their mean’s and variance’s, shown below, are obtained.

 C1 Prior Posterior Mean 12.792 12.717 Variance 7.432 7.344
 C2 Prior Posterior Mean 7.368 5.727 Variance 6.408 0.704
 C3 Prior Posterior Mean 10.257 6.941 Variance 42.626 6.104
 e-mail info at dmg.org