|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
PMML 4.0 - Transformation Dictionary & Derived FieldsAt various places the mining models use simple functions in order to map user data to values that are easier to use in the specific model. For example, neural networks internally work with numbers, usually in the range from 0 to 1. Numeric input data are mapped to the range [0..1], and categorical fields are mapped to series of 0/1 indicators. Similarly, Naïve Bayes models internally map all input data to categorical values. PMML defines various kinds of simple data transformations:
The corresponding XML elements appear as content of a surrounding markup DerivedField, which provides a common element for the various mappings. They can also appear at several places in the definition of specific models such as neural network or Naïve Bayes models. Transformed fields have a name such that statistics and the model can refer to these fields. The transformations in PMML do not cover the full set of preprocessing functions which may be needed to collect and prepare the data for mining. There are too many variations of preprocessing expressions. Instead, the PMML transformations represent expressions that are created automatically by a mining system. A typical example is the normalization of input values in neural networks. Similarly, a discretization might be constructed by a mining system that computes quantile ranges in order to transform skewed data.
DerivedFields in the TransformationDictionary or LocalTransformations together with DataFields in the DataDictionary must have unique names. If a DerivedField is contained in TransformationDictionary or LocalTransformations, then the name attribute is required. For DerivedFields which are contained inline in models, name is optional. If a DerivedField is contained in TransformationDictionary or LocalTransformations, then the name attribute is required. For DerivedFields which are contained inline in models, name is optional. DerivedFields in the TransformationDictionary or LocalTransformations together with DataFields in the DataDictionary must have unique names within the context of each model. This means that DerivedFields in TransformationDictionary must be unique across all models. DerivedFields in LocalTransformations have to be unique within each model but not across different models. The TransformationDictionary allows for transformations to be defined once and used by different models. Transformations defined in the TransformationDictionary are instantiated by each particular model when applicable. This means that policies for treating outliers and missing values defined in the MiningSchema are applied before transformations in the TransformationDictionary. DerivedFields in the TransformationDictionary may refer either to DataFields or other DerivedFields in the TransformationDictionary. Note that for a transformation to be instantiated by a certain model, its DerivedFields must ultimately refer back to active MiningFields of that model's MiningSchema. Within a particular model, DerivedFields in LocalTransformations may refer to active MiningFields in the MiningSchema, any of the applicable DerivedFields in the TransformationDictionary, or other DerivedFields in the same LocalTransformations. The transformation expression in the content of DerivedField defines how the values of the new field are computed. The attribute optype is needed in order to eliminate cases where the resulting type is not known. If there is a value mapping in a DerivedField, it is not known how to interpret the output. A value mapping might look like this: But it is not known whether 0.1 has to be interpreted as a number or as a string (=categorical value). Hence optype is required in order to make parsing and interpretation of models simpler."cat" -> "0.1" "dog" -> "0.2" "elephant" -> "0.3" etc. A DerivedField may have a list of Value elements that define the ordering of the values for an ordinal field. The attribute property must not be used for Value elements within a DerivedField. That is, the list cannot specify values that are interpreted as missing or invalid. ConstantConstant values can be used in expressions which have multiple arguments. The actual value of a constant is given by the content of the element. For example, <Constant>1.05</Constant> represents the number 1.05. The dataType of Constant can be optionally specified. If the ParameterField definition includes a dataType, the Constant will inherit the dataType specified in the ParameterField. If the dataType is not specified in the ParameterField definition or Constant element, it will be inferred by the content of the element. A Constant that consists solely of numbers will be treated as an integer, if a decimal point is present among the numbers the Constant will be treated as a float. The presence of any non-numeric characters will result in the Constant being treated as a string. Conflicting dataType specifications will be resolved as specified in the Functions document.
FieldRefField references are simply pass-throughs to fields previously defined in the DataDictionary, a DerivedField, or a result field. For example, they are used in clustering models in order to define center coordinates for fields that don't need further normalization.
NormalizationThe elements for normalization provide a basic framework for mapping input values to specific value ranges, usually the numeric range [0 .. 1]. Normalization is used, e.g., in neural networks and clustering models.
NormContinuous: defines how to normalize an input field by piecewise linear interpolation. The mapMissingTo attribute defines the value the output is to take if none of the sections given by the LinearNorms match the input. If the mapMissingTo attribute is not specified, then missing input values are treated as missing (see missing values treatment).
The sequence of LinearNorm elements defines a sequence of points for a stepwise linear interpolation function. The sequence must contain at least two elements. Within NormContinous the elements LinearNorm must be strictly sorted by ascending value of orig. Given two points (a1, b1) and (a2, b2) such that there is no other point (a3, b3) with a1<a3<a2, then the normalized value is b1+ ( x-a1)/(a2-a1)*(b2-b1) for a1 ≤ x ≤ a2. Missing input values are mapped to missing output. asIs: Extrapolates the normalization from the nearest interval. asMissingValues: Maps to a missing value. asExtremeValues: Maps to the next value from the nearest interval so that the function is continuous. The graph below depicts a mapping where outliers are mapped using asExtremeValues: NormContinuous can be used to implement simple normalization functions such as the z-score transformation" (X - m ) / s, where m is the mean value and s is the standard deviation.
Normalize discrete valuesMany mining models encode string values into numeric values in order to perform mathematical computations. For example, regression and neural network models often split categorical and ordinal fields into multiple dummy fields. This kind of normalization is supported in PMML by the element NormDiscrete.
With the indicator method, an element (f, v) defines that the unit has value 1.0 if the value of input field f is v, otherwise it is 0. The set of NormDiscrete instances which refer to a certain input field define a fan-out function which maps a single input field to a set of normalized fields. If the input value is missing and the attribute mapMissingTo is not specified then the result is a missing value as well. If the input value is missing and the attribute mapMissingTo is specified then the result is the value of the attribute mapMissingTo.DiscretizationDiscretization of numerical input fields is a mapping from continuous to discrete values using intervals.
The attribute field defines the name of the input field. The elements DiscretizeBin define a set of mappings from an intervali to a binValuei. The value of the DerivedField is binValuei if the input value is contained in intervali for some i. Two intervals may be mapped to the same categorical value but the mapping for each numerical input value must be unique, i.e., the intervals must be disjoint. The intervals should cover the complete range of input values. Decision table for Discretize('*' stands for any combination)
Example: A definition such as:
In SQL this definition corresponds to a CASE expression CASE When "Profit" < 0 Then 'negative' When "Profit" >= 0 Then 'positive' End Discretize can also be used to create a missing value indicator for a continuous variable. In this case, the DiscretizeBin element is superfluous and can be dropped. Example:
Map ValuesAny discrete value can be mapped to any possibly different discrete value by listing the pairs of values. This list is implemented by a table, so it can be given inline by a sequence of XML markups or by a reference to an external table. The same technique is used for a Taxonomy because the tables can become quite large.
The types InlineTable and TableLocator are defined in the Taxonomy schema. The mapMissingTo attribute defines the value the output column is to take if any of the input columns are missing. Different string values can be mapped to one value but it is an error if the table entries used for matching are not unique. The value mapping may be partial. I.e., if an input value does not match a value in the mapping table, then the result can be a missing value. See the decision table below for the possible combinations. Decision table for MapValues('*' stands for any combination)
Example: A definition such as
In SQL this definition corresponds to a CASE expression CASE "gender" When 'm' Then 'male' When 'f' Then 'female' End Example: Here is an example for the multi-dimensional case, mapping from state and band to salary: Respective PMML would look like this:
Example: The MapValues element can be used to create missing value indicators for categorical variables. In this case, only one FieldColumnPair needs to be specified and the column attribute can be omitted.
AggregationsAssociation rules and sequences refer to sets of items. These sets can be defined by an aggregation over sets of input records. The records are grouped together by one of the fields and the values in this grouping field partition the sets of records for an aggregation. This corresponds to the conventional aggregation in SQL with a GROUP BY clause. Input records with missing value in the groupField are simply ignored. This behavior is similar to the aggregate functions in the presence of NULL values in SQL.
A definition such as:
builds sets of item values; for each transaction, i.e. for each value in the field transaction there is one set of items. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|