Interpretable Machine Learning Models

February 5, 2025

A machine learning model from Amazon selected only males from a pile of resumes¹. Another model fired teachers who were underperforming, according to the model². Such models are discriminatory and can be bad for society. They can make wrong decisions and affect people’s lives in a negative way. To solve this problem, you can start interpreting your models. Interpretability means that a human can understand the cause of the decision.

A lot of research has been done on the interpretability of machine learning models. There are different ways to interpret your machine learning models. The easiest split is between interpretable models and model-agnostic methods. Interpretable models are models who explain themselves, for instance from a decision tree you can easily extract decision rules. Model-agnostic methods are methods you can use for any machine learning model, from support vector machines to neural nets. In this article, the focus will be on interpretable models, like linear regression, logistic regression and decision trees. Here’s a different article about model-agnostic methods.

Dataset

A field where you can use model interpretability is health care. To find out how a model decides whether or not a person has heart disease, we use a dataset from the Cleveland database with the following features:

We will try to predict the target with logistic regression and a decision tree and interpret the models we have built. You can find the Heart Disease UCI dataset on Kaggle.

Code

The code for this article about interpretable models (and for the article about model-agnostic methods) can be found on GitHub.

Interpretable Models

Models you can interpret are, among others, linear regression models, logistic regression models and decision trees. So there is no need for a model-agnostic method in these cases, although you could use them too for more insights. We will build a logistic regression model and decision tree on the dataset and see how we can interpret the results, after a short note about linear regression.

Linear Regression

In linear regression you can use the weights (or coefficients) to find out which features are the most important, these are the features with the highest weight. If, for one record, you add one to a feature value while keeping the others the same the prediction will increase by the weight of the feature.

Logistic Regression

For logistic regression, the way to interpret is a bit different because a coefficient is the natural logarithm of the odds ratio. To interpret the coefficients, you need to do the transformation from the coefficient to the odds ratio. We can use the functions below:

According to the first two equations, we can replace b with ln(a), to get a. So if we raise e to the power of the natural logarithm of the odds ratio (equal to the coefficient we get from a logistic regression model), we get the odds ratio.

Great! After this transformation, it’s a bit the same as linear regression, but instead of increasing it when you change one feature value it will change by a factor. An odds ratio with a value around 1 has the least impact, while a higher or lower one has more influence³.

So let’s build a logistic regression model on the heart disease dataset to understand what happens. We got the following coefficients for each feature, and we can transform them (e raised to the power of the coefficient) to receive the odds ratio. You can see the values in the table below, the values are sorted in descending order:

Now we can start interpreting the model:

Chest pain type has a high impact. When this value increases by one, the probability having a heart disease increases by 120.1 percent.
If the slope variable increases by one, the probability increases by 94.2 percent.
Fasting blood sugar is a binary variable. If this value is one (true), chances are 13.9 percent higher to have a heart disease then when this value is equal to zero (false).
On the bottom we see the sex variable. This odds ratio is below 1, around 1/4. This means that having a heart disease is almost four times more likely when you’re female!

When results like this seems strange, the best thing to do is investigate your data. You might be wondering why age and cholesterol level have a really low impact. This model tells us that a higher cholesterol level is a bit better than a lower one. Is this the truth? The big difference between males and females also raises questions! To investigate this we could use a target plot, you can find one for the gender feature in the notebook on GitHub.

Decision Trees

Let’s see what a Decision Tree shows us! Decision trees are like a rule system, you start in the root node and then you follow the path for a record to a leaf node where you can see the prediction. It’s harder to interpret when your tree has a higher depth, but still doable.

Below you can see an image of a decision tree, build on the heart disease dataset.

On every decision node you see four lines with text. The first line represents the decision rule. If the condition is true for a record, you follow the left branche, if it’s false, you follow the right one. Gini, on the second line, is the value of the Gini Impurity, the lower this value, the better the split⁴. The third and fourth line show how many samples from the training set are in the current node and how they are divided. You might wonder what the colors mean. If a node is orange or blue, it means that for the largest part of the samples in this node the target variable (having a heart disease) is equal to zero or one, respectively. The darker the color, the more homogeneous the node.

So let’s take a record from the test set and follow the nodes of the tree to see what the final prediction will be. This is the new record:

Let’s start in the root node of the decision tree. The decision rule is: feature X₂ (cp, chest pain type) smaller than or equal to 0.5. For our new record, the value is 0, so smaller than 0.5. This statement is true and we follow the left branche. The following feature is X₁₁ (ca, number of major vessels colored by fluoroscopy), this value is greater than 0.5, so the statement is false and we continue to see if X₃ (trestbps, resting blood pressure) is smaller or equal to 109. This is not the case, since it is equal to 150. So false again. The next node checks if X₁ (sex) is smaller than 0.5. No, it’s 1 for this record, he is male. Now we found the leaf node for this record, the prediction is 0. This prediction is correct! So for this record, the model uses only four features, cp, ca, trestbps and sex.

Path for the new record from root to leaf node

Okay, that’s a way to find out why one record has a certain prediction. With just the image of the tree, it’s difficult to see which features are important to the model. We can extract feature importances of trees in different ways. The easiest way is to use the feature importances of scikit-learn. The importance of a feature is computed as the (normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance.

Should you trust these feature importances? You could say that features like cp, ca, chol and age are really important in predicting a heart disease. It’s possible they do, but first test your model! Keep in mind that those feature importances are based only on the training set and the current tree. The way feature importances are calculated in scikit-learn isn’t the best one. It is better to use one of the model-agnostic methods, like permutation feature importance. Curious? Read about it in the next article about model-agnostic methods!

[1] J. Dastin, Amazon scraps secret AI recruiting tool that showed bias against women (2018), Reuters

[2] C. O’Neil, Weapons Of Math Destruction (2016), Crown New York

[3] How do I interpret odds ratios in logistic regression?, UCLA: Statistical Consulting Group

The post Interpretable Machine Learning Models appeared first on Towards Data Science.