提问者:小点点

更新 Apache Lucene 索引文件


我正在使用Apache Lucene库来创建我的网站的搜索功能.该网站正在从Sharepoint RSSFeed获取其所有内容,因此每次我都必须通过所有的RSSFeed网址并阅读内容.为了使搜索功能更快,我创建了一个计划任务,每隔一小时进行一次索引:

    <bean id="rssIndexerService" class="com.lloydsbanking.webmi.service.RSSIndexerService" />
<task:scheduled-tasks> <task scheduled ref="rssIndexerService" method="indexUrls" cron="0 0 * * * MON-FRI" /></task:scheduled-tasks>

问题是,如果我创建一个新的内容,那么在服务器运行时,搜索不会显示新的内容,并且在调用计划任务后,如果我删除一个条目,它仍然不会显示从索引文件中删除的条目。以下是索引代码:

@Service
public class RSSIndexerService extends RSSReader {

    @Autowired
    private RSSFeedUrl rssFeedUrl;

    private IndexWriter indexWriter = null;

    private String indexPath = "C:\\MI\\index";

    Logger log = Logger.getLogger(RSSIndexerService.class.getName());

    public void indexUrls() throws IOException {
        Date start = new Date();
        IndexWriter writer = getIndexWriter();
        log.info("Reading all the Urls in the Sharepoint");     
        Iterator<Entry<String, String>> entries = rssFeedUrl.getUrlMap().entrySet().iterator();
        try {
            while (entries.hasNext()) {
                Entry<String, String> mapEntry = entries.next();
                String url = mapEntry.getValue();
                SyndFeed feed = rssReader(url);
                for (Object entry : feed.getEntries()) {
                    SyndEntry syndEntry = (SyndEntry) entry;
                    SyndContent desc = syndEntry.getDescription();
                    if (desc != null) {
                        String text = desc.getValue();
                        if ("text/html".equals(desc.getType())) {
                            Document doc = new Document();
                            text = extractText(text);
                            Field fieldTitle = new StringField("title", syndEntry.getTitle(), Field.Store.YES);
                            doc.add(fieldTitle);
                            Field pathField = new StringField("path", url, Field.Store.YES);
                            doc.add(pathField);
                            doc.add(new TextField("contents", text, Field.Store.YES));

                            // New index, so we just add the document (no old document can be there):
                            writer.addDocument(doc);
                        }
                    }
                }

            }

        } finally {

            // closeIndexWriter();
        }
        Date end = new Date();
        log.info(end.getTime() - start.getTime() + " total milliseconds");
    }

    public IndexWriter getIndexWriter() throws IOException {

        if (indexWriter == null) {
            Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_47);

            log.info("Indexing to directory '" + indexPath + "'...");
            Directory dir = FSDirectory.open(new File(indexPath));
            IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_47, analyzer);

            config.setOpenMode(OpenMode.CREATE_OR_APPEND);
            indexWriter = new IndexWriter(dir, config);
        }
        return indexWriter;
    }

    @PreDestroy
    public void closeIndexWriter() throws IOException {
        if (indexWriter != null) {
            System.out.println("Done with indexing ...");
            indexWriter.close();
        }
    }

}

我知道问题可能是由config.setOpenMode(OpenMode.CREATE_OR_APPEND)引起的;,但我不知道该怎么解决。


共1个答案

匿名用户

好吧,我想到了检查目录之前是否为空的想法,如果不是,则删除之前的索引,然后每次在OpenModel. Create中进行索引:

File path = new File(System.getProperty("java.io.tmpdir")+"\\index");
        Directory dir = FSDirectory.open(path);

        Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_47);
        IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_47, analyzer);

        if (path.list() != null) {
            log.info("Delete previous indexes ...");
            FileUtils.cleanDirectory(path);
        }
        config.setOpenMode(OpenMode.CREATE);

然后我简单的使用addDocument():

if ("text/html".equals(desc.getType())) {
                        ...
                        // New index, so we just add the document (no old document can be there):
                        indexWriter.addDocument(doc);
                    }