Retrieving project results
The DataRobot project created by the command described above fits 13
models to the Ames dataframe. Detailed information about
all of these models can be obtained with the ListModels
function, invoked with the project list returned by the
StartProject function.
listOfAmesModels <- ListModels(project)The ListModels function returns an S3 object of class
‘listOfModels’, with one element for each model in the project. A
summary method has been implemented for this object class, and it
provides the following view of the contents of this list:
summary(listOfAmesModels)## $generalSummary
## [1] "First 6 of 13 models from: listOfAmesModels (S3 object of class listOfModels)"
##
## $detailedSummary
## modelType
## 1 eXtreme Gradient Boosted Trees Regressor (Gamma Loss)
## 2 AVG Blender
## 3 eXtreme Gradient Boosted Trees Regressor (Gamma Loss)
## 4 eXtreme Gradient Boosted Trees Regressor (Gamma Loss)
## 5 Light Gradient Boosting on ElasticNet Predictions (Gamma Loss)
## 6 Light Gradient Boosted Trees Regressor with Early Stopping (Gamma Loss)
## expandedModel
## 1 eXtreme Gradient Boosted Trees Regressor (Gamma Loss)::Geospatial Location Converter::Missing Values Imputed
## 2 AVG Blender::Average Blender
## 3 eXtreme Gradient Boosted Trees Regressor (Gamma Loss)::Geospatial Location Converter::Missing Values Imputed
## 4 eXtreme Gradient Boosted Trees Regressor (Gamma Loss)::Geospatial Location Converter::Missing Values Imputed
## 5 Light Gradient Boosting on ElasticNet Predictions (Gamma Loss)::Numeric Data Cleansing::Smooth Ridit Transform::Elastic-Net Regressor (L2 / Gamma Deviance)
## 6 Light Gradient Boosted Trees Regressor with Early Stopping (Gamma Loss)::Geospatial Location Converter::Missing Values Imputed
## modelId blueprintId
## 1 62dd8b7b6dcff238c40735cd bda168d7725c76940aaf66a5b87dd936
## 2 62dd8b924e47bc6fed6916c2 00a968c06d6324a9630b1357a0a96748
## 3 62dd8a50748806bd8b60c35d 782081a59965e1cb2c171d21b47505d1
## 4 62dd8b5e7665693929b242b2 782081a59965e1cb2c171d21b47505d1
## 5 62dd8a4f748806bd8b60c35a acd16a05344349c0f17a3c99675ddeb2
## 6 62dd8a4f748806bd8b60c359 6ee066b11e7e618b16ab5f9adf2ec9f8
## featurelistName featurelistId samplePct validationMetric
## 1 Informative Features 62dd8a24b4b51b9ba2954729 100.0000 0.01535
## 2 Multiple featurelists Multiple featurelist ids 63.9932 0.01538
## 3 Informative Features 62dd8a24b4b51b9ba2954729 63.9932 0.01539
## 4 Informative Features 62dd8a24b4b51b9ba2954729 80.0000 0.01549
## 5 Informative Features 62dd8a24b4b51b9ba2954729 63.9932 0.01549
## 6 Informative Features 62dd8a24b4b51b9ba2954729 63.9932 0.01572The first element of this list is generalSummary, which
lets us know that the project includes 13 models, and that the second
list element describes the first 6 of these models. This number is
determined by the optional parameter nList for the
summary method, which has the default value 6. The second
list element is detailedSummary, which gives the first
nList rows of the dataframe created when the
as.data.frame method is applied to
listOfAmesModels. Methods for the
as.data.frame generic function are included in the
datarobot package for all four ‘list of’ S3 model object classes:
listOfBlueprints, listOfFeaturelists,
listOfModels, and projectSummaryList. (Use of
this function is illustrated in the following discussion; see the help
files for more complete details.) This dataframe has the following eight
columns:
- modelType: character string; describes the structure of each model
- expandedModel: character string; modelType with short descriptions of any preprocessing steps appended
- modelId: unique alphanumeric model identifier
- blueprintId: unique alphanumeric identifier for the blueprint used to fit the model
- featurelistName: name of the featurelist defining the modeling variables
- featurelistId: unique alphanumeric featurelist identifier
- samplePct: fraction of the training dataset used in fitting the model
- validationMetric: the value of the metric optimized in fitting the model, evaluated for the validation dataset
It is possible to obtain a more complete dataframe from any object of
class ‘listOfModels’ by using the function as.data.frame
with the optional parameter simple = FALSE. Besides the
eight characteristics listed above, this larger dataframe includes, for
every model in the project, additional project information along with
validation, cross-validation, and holdout values for all of the
available metrics for the project. For the project considered in this
note, the result is a dataframe with 13 rows and 51 columns.
In addition to the summary method, a plot method has also been provided for objects of class ‘listOfModels’:
plot(listOfAmesModels, orderDecreasing = TRUE)
Horizontal barplot of modelType and validation set Gamma Deviance values for all project models
This function generates a horizontal barplot that lists the name of
each model (i.e., modelType) in the center of each bar,
with the bar length corresponding to the value of the model fitting
metric, evaluated for the validation dataset (i.e., the
validationMetric value). The only required parameter for
this function is the ‘listOfModels’ class S3 object to be plotted, but
there are a number of optional parameters that allow the plot to be
customized. In the plot shown above, the logical parameter
orderDecreasing has been set to TRUE so that
the plot - generated from the bottom up - shows the models in decreasing
order of validationMetric. For a complete list of optional
parameters for this function, refer to the help files.
Since smaller values of Gamma Deviance.validation are
better, this plot shows the worst model at the bottom and the best model
at the top. The identities of these models are most conveniently
obtained by first converting listOfAmesModels into a
dataframe, using the as.data.frame generic function
mentioned above:
You can also coerce the list of models to a data.frame, which may make it easier to see specific things (such as model metrics):
modelFrame <- as.data.frame(listOfAmesModels)
head(modelFrame[, c("modelType", "validationMetric")])## modelType
## 1 eXtreme Gradient Boosted Trees Regressor (Gamma Loss)
## 2 AVG Blender
## 3 eXtreme Gradient Boosted Trees Regressor (Gamma Loss)
## 4 eXtreme Gradient Boosted Trees Regressor (Gamma Loss)
## 5 Light Gradient Boosting on ElasticNet Predictions (Gamma Loss)
## 6 Light Gradient Boosted Trees Regressor with Early Stopping (Gamma Loss)
## validationMetric
## 1 0.01535
## 2 0.01538
## 3 0.01539
## 4 0.01549
## 5 0.01549
## 6 0.01572It is interesting to note that this single best model, which is fairly complex in structure, actually outperforms the blender model (the second model in the barplot above), formed by averaging the best individual project models. This behavior is unusual, since the blender models usually achieve at least a small performance advantage over the component models on which they are based. In fact, since the individual component models may be regarded as constrained versions of the blender models (e.g., as a weighted average with all of the weight concentrated on one component), the training set performance can never be worse for a blender than it is for its components, but this need not be true of validation set performance, as this example demonstrates.
Or you can see the worst models:
tail(modelFrame[, c("modelType", "validationMetric")])## modelType
## 8 eXtreme Gradient Boosted Trees Regressor (Gamma Loss)
## 9 RandomForest Regressor
## 10 Generalized Additive2 Model (Gamma Loss)
## 11 Elastic-Net Regressor (L2 / Gamma Deviance)
## 12 Elastic-Net Regressor (mixing alpha=0.5 / Gamma Deviance)
## 13 Keras Slim Residual Neural Network Regressor using Training Schedule (1 Layer: 64 Units)
## validationMetric
## 8 0.01669
## 9 0.01772
## 10 0.01885
## 11 0.02282
## 12 0.02338
## 13 0.03272It is also important to note that several of the models appear to be
identical, based on their modelType values, but they
exhibit different performances. This is most obvious from the four
models labelled “eXtreme Gradient Boosted” but it is also true of six
other modelType values, each of which appears two or three
times in the plot, generally with different values for
Gamma Deviance.validation. In fact, these models are not
identical, but differ in the preprocessing applied to them, or in other
details.
In most cases, these differences may be seen by examining the
expandedModel values from modelFrame. For
example, lets get all the different models that use the Elastic-Net
Regressor:
Filter(function(m) grepl("Elastic-Net", m), modelFrame$expandedModel)## [1] "Light Gradient Boosting on ElasticNet Predictions (Gamma Loss)::Numeric Data Cleansing::Smooth Ridit Transform::Elastic-Net Regressor (L2 / Gamma Deviance)"
## [2] "Elastic-Net Regressor (L2 / Gamma Deviance)::Geospatial Location Converter::Smooth Ridit Transform::Numeric Data Cleansing"
## [3] "Elastic-Net Regressor (mixing alpha=0.5 / Gamma Deviance)::Geospatial Location Converter::Smooth Ridit Transform::Missing Values Imputed"In particular, note that the modelType value appears at
the beginning of the expandedModel character string, which
is then followed by any pre-processing applied in fitting the model.
Thus, comparing elements from this list (see below), we can see that
they differ in their preprocessing steps and primary ML model. In the
case of the Light Gradient Boosting, the difference is that Elastic-Net
model is a pre-processing step that is fed into a different model.
