我训练了scikit learn的TfidVectorizer
的一个实例,并希望将其保存到磁盘。我将IDF矩阵(IDF_
属性)作为numpy数组保存到磁盘,并将词汇表(词汇表_
)作为JSON对象保存到磁盘(出于安全和其他原因,我避免pickle)。我正在尝试这样做:
import json
from idf import idf # numpy array with the pre-computed IDFs
from sklearn.feature_extraction.text import TfidfVectorizer
# dirty trick so I can plug my pre-computed IDFs
# necessary because "vectorizer.idf_ = idf" doesn't work,
# it returns "AttributeError: can't set attribute."
class MyVectorizer(TfidfVectorizer):
TfidfVectorizer.idf_ = idf
# instantiate vectorizer
vectorizer = MyVectorizer(lowercase = False,
min_df = 2,
norm = 'l2',
smooth_idf = True)
# plug vocabulary
vocabulary = json.load(open('vocabulary.json', mode = 'rb'))
vectorizer.vocabulary_ = vocabulary
# test it
vectorizer.transform(['foo bar'])
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 1314, in transform
return self._tfidf.transform(X, copy=False)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 1014, in transform
check_is_fitted(self, '_idf_diag', 'idf vector is not fitted')
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn/utils/validation.py", line 627, in check_is_fitted
raise NotFittedError(msg % {'name': type(estimator).__name__})
sklearn.utils.validation.NotFittedError: idf vector is not fitted
那么,我做错了什么?我无法愚弄矢量器对象:不知何故,它知道我在作弊(即,传递它预先计算的数据,而不是用实际文本训练它)。我检查了矢量器对象的属性,但找不到类似“istrained”、“isfitted”之类的东西。那么,我如何愚弄矢量器呢?
好的,我想我明白了:矢量器实例有一个属性\u tfidf
,而这个属性又必须有一个属性\u idf\u diag
。transform
方法调用一个check\u fitted
函数,该函数检查\u idf\u diag
是否存在。(我错过了它,因为它是属性的属性。)因此,我检查了TfidfVectorizer源代码,以了解\u idf\u diag
是如何创建的。然后我将其添加到\u tfidf
属性中:
import scipy.sparse as sp
# ... code ...
vectorizer._tfidf._idf_diag = sp.spdiags(idf,
diags = 0,
m = len(idf),
n = len(idf))
现在矢量化工作了。