提问者:小点点

如何使用CountVectorizer.fit_transform()的酸洗分类器标记数据


我在一组短文档上训练分类器,并在获得二进制分类任务的合理f1和准确度分数后对其进行酸洗。

在培训期间,我使用sciki learnCountVectoriercv:

    cv = CountVectorizer(min_df=1, ngram_range=(1, 3), max_features = 15000) 

然后使用fit_transform()transform()方法获得转换后的列车和测试集:

    transformed_feat_train = numpy.zeros((0,0,))
    transformed_feat_test = numpy.zeros((0,0,))

    transformed_feat_train = cv.fit_transform(trainingTextFeat).toarray()
    transformed_feat_test = cv.transform(testingTextFeat).toarray()

这对于训练和测试分类器都很有效。然而,我不确定如何使用fit_transform()转换()与训练分类器的腌制版本一起预测看不见的、未标记的数据的标签。

我正在提取未标记数据上的特征,方法与训练/测试分类器时完全相同:

## load the pickled classifier for labeling
pickledClassifier = joblib.load(pickledClassifierFile)

## transform data
cv = CountVectorizer(min_df=1, ngram_range=(1, 3), max_features = 15000)
cv.fit_transform(NOT_SURE)

transformed_Feat_unlabeled = numpy.zeros((0,0,))
transformed_Feat_unlabeled = cv.transform(unlabeled_text_feat).toarray()

## predict label on unseen, unlabeled data
l_predLabel = pickledClassifier.predict(transformed_feat_unlabeled)

错误消息:

    Traceback (most recent call last):
      File "../clf.py", line 615, in <module>
        if __name__=="__main__": main()
      File "../clf.py", line 579, in main
        cv.fit_transform(pickledClassifierFile)
      File "../sklearn/feature_extraction/text.py", line 780, in fit_transform
        vocabulary, X = self._count_vocab(raw_documents, self.fixed_vocabulary)
      File "../sklearn/feature_extraction/text.py", line 727, in _count_vocab
        raise ValueError("empty vocabulary; perhaps the documents only"
    ValueError: empty vocabulary; perhaps the documents only contain stop words

共1个答案

匿名用户

您应该使用相同的矢量器实例来转换训练和测试数据。您可以通过使用矢量器分类器创建管道、在培训集上培训管道、酸洗整个管道来实现这一点。稍后加载酸洗管道并调用predict。

请参阅此相关问题:将分类器投入生产。