How can I handle unknown values for label encoding in sk-learn? The label encoder will only blow up with an exception that new labels were detected.
What I want is the encoding of categorical variables via one-hot-encoder. However, sk-learn does not support strings for that. So I used a label encoder on each column.
My problem is that in my cross-validation step of the pipeline unknown labels show up.
The basic one-hot-encoder would have the option to ignore such cases.
An apriori pandas.getDummies /cat.codes
is not sufficient as the pipeline should work with real-life, fresh incoming data which might contain unknown labels as well.
Would it be possible to use a CountVectorizer
for this purpose?
There are several answers that mention
pandas.get_dummies
as a method for this, but I feel thelabelEncoder
approach is cleaner for implementing a model. Other similar answers mention usingDictVectorizer
for this, but again converting the entireDataFrame
to dict is probably not a great idea.Let's assume the following problematic columns:
Utila is a rarer city, and it isn't present in the training data but in the test set, that we can consider new data at inference time.
The trick is converting this value to "other" and including this in the labelEncoder object. Then we can reuse it in production.
To apply it to new data all we need is to save a
le
object for each column which can be easily done with Pickle.This answer is based on this question which I feel wasn't totally clear to me, therefore added this example.