Skater Files

This release includes: * Various bug fixes and performance improvements * new feature importance calculation method * introduction of model scorers * model types can now be determined explicitly

Model Scoring

Now, after you create a Skater model with:

model = InMemoryModel(predict_fn, examples=examples, model_type="classifier")

The model object now provides a .scorers api, which allows you to store predictions against training labels. Based on whether your model is a regressor, classifier that returns labels, or classifier that returns probabilities, scorers will automatically expose various scoring algorithms specific to your model. For instance, in the example above, we could do:

model.scorers.f1(labels, model(X))
model.scorers.cross_entropy(labels, model(X))

if it were a regression, we could do:

model.scorers.mse(labels, model(X))

Calling model.scorers.default(labels, model(X)) or simply model.scorers(labels, model(X)) will execute the default scorer for your model, which are:

regression: mean absolute error classifier (probabilities): cross entropy classifier (labels): f1

Let us know if you'd like more scorers, or even better, feel free to make a PR to add more yourself!

Feature Importance Calculation

The default method of computing feature importance is done by perturbing each feature, and observing how much those perturbations affect predictions.

With the addition of model scoring, we now also provide a method based on observing changes in model scoring functions; the less accurate your model becomes based on perturbing a feature, the more important it is.

To enable scoring based feature importance, you must load training labels into your interpretation object, like:

interpreter = Interpretation(training_data=training_data, training_labels=training_labels)
interpreter.feature_importance.plot_feature_importance(model, method='model-scoring')

Explicit Model Types

Originall Skater tried to infer the type of your model based on the types of predictions it made. Now when you create a model, you can define these explicitely with model_type and probability keyword arguments to skater model types:

model = InMemoryModel(predict_fn, model_type='classifier', probability=True)

or

model = InMemoryModel(predict_fn, model_type='regressor')