Model Composition
PMML3.0 Menu

Home


PMML Notice and License

Changes


Conformance

General Structure

Header

Data
Dictionary


Mining
Schema


Transformations

Statistics

Taxomony

Targets

Output

Functions

Built-in Functions

Model Composition

Model Verification


Association Rules

Cluster
Models


General
Regression


Naive
Bayes


Neural
Network


Regression

Ruleset

Sequences

Text Models

Trees

Vector Machine

PMML 3.0 - Model Composition: Sequences of Models and Model Selection

Model Composition allows the combination of simple models into a single composite PMML models. There are two main variants:
  1. Model sequencing: two or more models are combined into a sequence where the results of one model are used as input in another model.
  2. Model selection: one of many models can be selected based on decision rules.
PMML supports the combination of decision trees and simple regression models. More general variants would be possible and may be defined in future versions of PMML.

Sample scenarios

Model sequencing and selection in PMML covers a variety of scenarios such as the following examples:
  • A logistic regression model may require non-trivial rules for replacing missing values like
    if Age is missing
        if Occupation is "Student" then Age := 20
        else if Occupation is "Retired" then Age := 70
        else Age := 40
    These preprocessing rules can be defined by a simple decision tree model that is put into a sequence with the regression model.

  • A common method for optimizing prediction models is the combination of segmentation and regression. Data are grouped into segments and for each segment there may be different regression equations. If the segmentation can be expressed by decision rules then this kind of segment based regression can be implemented by a decision tree where any leaf node in the tree can contain an embedded regression model.

  • Prediction results may have to be combined with a cost or profit matrix before a decision can be derived. A mailing campaign model may use tree classification to determine response probabilities per customer and channel. The cost matrix can be appended as a regression model that applies cost weighting factors to different channels, e.g., high cost for phone and low cost for email. The final decision is then based on the outcome of the regression model.

  • A voting scheme that merges results from multiple models can also be implemented by model composition in PMML. For example, there may be an ensemble of three classification models A, B, C, and D for the same target with values "yes" and "no". The final classification result may be defined as the average of the results from A, B, C, and D. The average can be computed by a regression model with equations
    pyes = 0.25*pAyes + 0.25*pByes + 0.25*pCyes + 0.25*pDyes
    pno = 0.25*pAno + 0.25*pBno + 0.25*pCno + 0.25*pDno
    where pXyes stands for the probability of class "yes" and pXno stands for the probability of class "no" in the model X.

XML Schema

Model composition uses three syntactical concepts
  1. The essential elements of a predictive model are captured in elements that can be included in other models.
  2. Embedded models can define new fields, similar to derived fields.
  3. The leaf nodes in a decision tree can contain another predictive model.
For example, using a sequence of models, a field could be defined by a regression equation. This field is then used as an ordinary input field in a decision tree. The basic idea is that we capture the essential elements of a model, in this example from a regression model, and use them to define new fields. That is similar to defining a derived field.

Mining models and their corresponding embedded elements

The first steps in making models reusable in other models is the definition of 'model expression' elements that can be embedded in another model. PMML defines the two elements Regression and DecisionTree.
Standalone model element Embedded model element Main content
RegressionModel Regression RegressionTable(s)
TreeModel DecisionTree Node(s)

EmbeddedModel does not contain a MiningSchema. There is only one MiningSchema at the top-level.


  <xs:group name="EmbeddedModel">
    <xs:sequence>
      <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded" />
      <xs:choice>
        <xs:element ref="Regression" />
        <xs:element ref="DecisionTree" />
      </xs:choice>
    </xs:sequence>
  </xs:group>

Model selection is enabled by allowing an EmbeddedModel within a tree Node.

The element Regression contains the essential elements of a RegressionModel:


  <xs:element name="Regression">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded" />
        <xs:element ref="LocalTransformations" minOccurs="0" />
        <xs:element ref="ResultField" minOccurs="0" maxOccurs="unbounded" />
        <xs:element ref="RegressionTable" maxOccurs="unbounded" />
      </xs:sequence>
      <xs:attribute name="modelName" type="xs:string" />
      <xs:attribute name="functionName" type="MINING-FUNCTION" use="required" />
      <xs:attribute name="algorithmName" type="xs:string" />
      <xs:attribute name="normalizationMethod" type="REGRESSIONNORMALIZATIONMETHOD" default="none" />
    </xs:complexType>
  </xs:element>

ResultFields are elements that define named results, see below.

The element DecisionTree contains the essential elements of a TreeModel:


  <xs:element name="DecisionTree">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded" />
        <xs:element ref="LocalTransformations" minOccurs="0" />
        <xs:element ref="ResultField" minOccurs="0" maxOccurs="unbounded" />
        <xs:element ref="Node" />
      </xs:sequence>
      <xs:attribute name="modelName" type="xs:string" />
      <xs:attribute name="functionName" type="MINING-FUNCTION" use="required" />
      <xs:attribute name="algorithmName" type="xs:string" />
      <xs:attribute name="splitCharacteristic" default="multiSplit">
        <xs:simpleType>
          <xs:restriction base="xs:string">
            <xs:enumeration value="binarySplit" />
            <xs:enumeration value="multiSplit" />
          </xs:restriction>
        </xs:simpleType>
      </xs:attribute>
    </xs:complexType>
  </xs:element>

Regression and DecisionTree can exclusively be used to build a model using the MiningModel model type:


  <xs:element name="MiningModel">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
        <xs:element ref="MiningSchema"/>
        <xs:element ref="Output" minOccurs="0"/>
        <xs:element ref="ModelStats" minOccurs="0"/>
        <xs:element ref="Targets" minOccurs="0"/>
        <xs:element ref="LocalTransformations" minOccurs="0" />
        <xs:choice maxOccurs="unbounded">
          <xs:element ref="Regression"/>
          <xs:element ref="DecisionTree"/>
        </xs:choice>
        <xs:element ref="ModelVerification" minOccurs="0"/>
        <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
      </xs:sequence>
      <xs:attribute name="modelName" type="xs:string" use="optional"/>
      <xs:attribute name="functionName" type="MINING-FUNCTION" use="required"/>
      <xs:attribute name="algorithmName" type="xs:string" use="optional"/>
    </xs:complexType>
  </xs:element>

The element ResultField is very similar to OutputField and DerivedField


  <xs:element name="ResultField">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="Extension" minOccurs="0" maxOccurs="unbounded"/>
      </xs:sequence>
      <xs:attribute name="name" type="FIELD-NAME" use="required" />
      <xs:attribute name="displayName" type="xs:string" />
      <xs:attribute name="optype" type="OPTYPE" />
      <xs:attribute name="dataType" type="DATATYPE"/>
      <xs:attribute name="feature" type="RESULT-FEATURE" />
      <xs:attribute name="value" type="xs:string" />
    </xs:complexType>
  </xs:element>

Model Sequencing for Input Transformations

The following example demonstrates how a regression equation can be used to define an input transformation in another model which happens to be a TreeModel.

  <?xml version="1.0" ?>
  <PMML version="3.0">
    <Header copyright="DMG.org"/>
    <DataDictionary numberOfFields="5">
      <DataField name="age" optype="continuous"/>
      <DataField name="income" optype="continuous"/>
      <DataField name="gender" optype="categorical">
        <Value value="  female"/>
        <Value value="    male"/>
      </DataField>
      <DataField name="weight" optype="continuous"/>
    </DataDictionary>
    <MiningModel functionName="regression">
      <MiningSchema>
        <MiningField name="age"/>
        <MiningField name="income"/>
        <MiningField name="gender"/>
        <MiningField name="weight" usageType="predicted"/>
      </MiningSchema>
      <LocalTransformations>
        <DerivedField name="mc" optype="continuous">
          <MapValues outputColumn="mapped" mapMissingTo="-1">
            <FieldColumnPair field="gender" column="sourceval"/>
            <InlineTable>
              <row><sourceval>  female</sourceval><mapped>1</mapped></row>
              <row><sourceval>    male</sourceval><mapped>0</mapped></row>
            </InlineTable>
          </MapValues>
        </DerivedField>
      </LocalTransformations>
      <Regression>
        <ResultField name="term" feature="predicted"/>
        <RegressionTable intercept="2.34">
          <NumericPredictor name="income" coefficient="0.03"/>
          <PredictorTerm coefficient="1.23">
            <FieldRef field="age"/>
            <FieldRef field="mc"/>
          </PredictorTerm>
        </RegressionTable>
      </Regression>
      <DecisionTree>
        <Node>
          <True/>
          <Node score="32.32">
            <SimplePredicate field="term" operator="lessThan" value="42"/>
          </Node>
          <Node score="78.91">
            <SimplePredicate field="term" operator="greaterOrEqual" value="42"/>
          </Node>
        </Node>
      </DecisionTree>
    </MiningModel>
  </PMML>

Model selection

Model selection in PMML allows for combining multiple 'embedded models', aka model expressions, into the decision logic that selects one of the models depending on the current input values.

The following example shows how regression elements are used within the nodes of a decision tree:


  <?xml version="1.0" ?>
  <PMML version="3.0">
    <Header copyright="DMG.org"/>
    <DataDictionary numberOfFields="5">
      <DataField name="age" optype="continuous"/>
      <DataField name="income" optype="continuous"/>
      <DataField name="gender" optype="categorical">
        <Value value="  female"/>
        <Value value="    male"/>
      </DataField>
      <DataField name="weight" optype="continuous"/>
    </DataDictionary>
    <MiningModel functionName="regression">
      <MiningSchema>
        <MiningField name="age"/>
        <MiningField name="income"/>
        <MiningField name="gender"/>
        <MiningField name="weight" usageType="predicted"/>
      </MiningSchema>
      <LocalTransformations>
        <DerivedField name="mc" optype="continuous">
          <MapValues outputColumn="mapped" mapMissingTo="-1">
            <FieldColumnPair field="gender" column="sourceval"/>
            <InlineTable>
              <row><sourceval>  female</sourceval><mapped>1</mapped></row>
              <row><sourceval>    male</sourceval><mapped>0</mapped></row>
            </InlineTable>
          </MapValues>
        </DerivedField>
      </LocalTransformations>
      <DecisionTree>
        <Node>
          <True/>
          <Node>
            <SimplePredicate field="age" operator="lessOrEqual" value="50"/>
            <Regression>
              <RegressionTable>
                <NumericPredictor name="income" coefficient="0.03"/>
                <PredictorTerm coefficient="1.23">
                  <FieldRef field="age"/>
                  <FieldRef field="mc"/>
                </PredictorTerm>
              </RegressionTable>
            </Regression>
          </Node>
          <Node>
            <SimplePredicate field="age" operator="greaterThan" value="50"/>
            <Regression>
              <RegressionTable intercept="2.22">
                <NumericPredictor name="income" coefficient="0.01"/>
                <PredictorTerm coefficient="-0.11">
                  <FieldRef field="age"/>
                  <FieldRef field="mc"/>
                </PredictorTerm>
              </RegressionTable>
            </Regression>
          </Node>
        </Node>
      </DecisionTree>
    </MiningModel>
  </PMML>


Here is another example how model composition can be applied. It demonstrates how to implement cost-based decisions in PMML. A prediction model such as regression or tree classification is assumed to compute the probability that a customer is interested in a certain product offer. Now it has to be decided whether the probability is high enough so it makes sense to contact a specific customer. There may be two options, contact by phone and contact by mail. Calling someone by phone is more costly than sending mail, 4.5 versus 1.2 in the matrix below. The expected profit is calculated by combining the probabilities that a customer is interested p(I) with weighting factors. E.g., the formula for the expected profit for contact by mail may be
    -1.2 + 3.2*p(I=YES) - 0.8*p(I=NO)
A complete matrix will look like this:
                 phone   mail   noAction
    cost           4.5    1.2     0.0
    p(I=YES)      11.7    3.2    -5
    p(I=NO)       -4     -0.8     0.1
In PMML this can be modeled by a regression expression within a mining model:

  <?xml version="1.0" ?>
  <PMML version="3.0">
    <Header copyright="DMG.org"/>
    <DataDictionary numberOfFields="5">
      <DataField name="x1" optype="continuous"/>
      <DataField name="x2" optype="continuous"/>
      <DataField name="interest" optype="categorical">
        <Value value="YES"/>
        <Value value="NO"/>
      </DataField>
    </DataDictionary>
    <MiningModel functionName="classification">
      <MiningSchema>
        <MiningField name="x1"/>
        <MiningField name="x2"/>
        <MiningField name="interest" usageType="predicted"/>
      </MiningSchema>
      <DecisionTree>
        <ResultField name="p_YES" feature="probability" value="YES" />
        <ResultField name="p_NO" feature="probability" value="NO" />
        <Node>
          <True/>
          <Node score="YES">
            <SimplePredicate field="x1" operator="lessThan" value="50"/>
          </Node>
          <Node>
            <SimplePredicate field="x1" operator="greaterOrEqual" value="50"/>
            <Node score="YES">
              <SimplePredicate field="x2" operator="greaterOrEqual" value="28"/>
            </Node>
            <Node score="NO">
              <SimplePredicate field="x2" operator="greaterOrEqual" value="28"/>
            </Node>
          </Node>
        </Node>
      </DecisionTree>

      <!-- Now use results of decision tree in a profit matrix -->
      <!-- Technically we put the decision tree in a sequence with a Regression element -->
      <!-- The Regression implements the profit computation -->

      <Regression normalizationMethod="none">
        <Extension> <!-- optional -->
          Add any proprietary XML notation that can be used for visualizing 
          a cost/profit matrix together with other parameters 
          that were used for the construction of
          this decision logic.
        </Extension>
        <ResultField name="DEC" feature="predictedValue" />
        <ResultField name="profit" feature="probability" />
        <RegressionTable targetCategory="call" intercept="-4.5">
          <NumericPredictor name="p_YES" coefficient="11.7" />
          <NumericPredictor name="p_NO" coefficient="-4" />
        </RegressionTable>
        <RegressionTable targetCategory="mail" intercept="-1.2">
          <NumericPredictor name="p_YES" coefficient="3.2" />
          <NumericPredictor name="p_NO" coefficient="-0.8" />
        </RegressionTable>
        <RegressionTable targetCategory="noAction" intercept="0.0">
          <NumericPredictor name="p_YES" coefficient="-5" />
          <NumericPredictor name="p_NO" coefficient="0.1" />
        </RegressionTable>
      </Regression>
    </MiningModel>
  </PMML>

Remarks:

  • Decisions are mapped to target categories in regression.
  • The regression equations define the total value of a decision.
  • With normalizationMethod="none" the predicted value is the category where the total value is the maximum.


e-mail info at dmg.org