我在一组短文档上训练分类器,并在获得二进制分类任务的合理f1和准确度分数后对其进行酸洗。
在培训期间,我使用sciki learnCountVectorier
cv:
cv = CountVectorizer(min_df=1, ngram_range=(1, 3), max_features = 15000)
然后使用fit_transform()
和transform()
方法获得转换后的列车和测试集:
transformed_feat_train = numpy.zeros((0,0,))
transformed_feat_test = numpy.zeros((0,0,))
transformed_feat_train = cv.fit_transform(trainingTextFeat).toarray()
transformed_feat_test = cv.transform(testingTextFeat).toarray()
这对于训练和测试分类器都很有效。然而,我不确定如何使用fit_transform()
和转换()
与训练分类器的腌制版本一起预测看不见的、未标记的数据的标签。
我正在提取未标记数据上的特征,方法与训练/测试分类器时完全相同:
## load the pickled classifier for labeling
pickledClassifier = joblib.load(pickledClassifierFile)
## transform data
cv = CountVectorizer(min_df=1, ngram_range=(1, 3), max_features = 15000)
cv.fit_transform(NOT_SURE)
transformed_Feat_unlabeled = numpy.zeros((0,0,))
transformed_Feat_unlabeled = cv.transform(unlabeled_text_feat).toarray()
## predict label on unseen, unlabeled data
l_predLabel = pickledClassifier.predict(transformed_feat_unlabeled)
错误消息:
Traceback (most recent call last):
File "../clf.py", line 615, in <module>
if __name__=="__main__": main()
File "../clf.py", line 579, in main
cv.fit_transform(pickledClassifierFile)
File "../sklearn/feature_extraction/text.py", line 780, in fit_transform
vocabulary, X = self._count_vocab(raw_documents, self.fixed_vocabulary)
File "../sklearn/feature_extraction/text.py", line 727, in _count_vocab
raise ValueError("empty vocabulary; perhaps the documents only"
ValueError: empty vocabulary; perhaps the documents only contain stop words
您应该使用相同的矢量器实例来转换训练和测试数据。您可以通过使用矢量器分类器创建管道、在培训集上培训管道、酸洗整个管道来实现这一点。稍后加载酸洗管道并调用predict。
请参阅此相关问题:将分类器投入生产。