提问者:小点点

Lucene IndexWriter内存不足异常


我在一个目录中有两个大文件(约200 MB),并想在它们上建立索引,所以这是我的代码:

public class LuceneUtil {
      private void indexDoc(IndexWriter indexWriter, Path file, long lastModified) throws IOException{
        try (InputStream stream = Files.newInputStream(file)) {
            Document document = new Document();

            Field pathField = new StringField("path", file.toString(), Field.Store.YES);
            document.add(pathField);
            document.add(new LongField("modified", lastModified, Field.Store.NO));
            document.add(new TextField("contents", new BufferedReader(new InputStreamReader(stream, StandardCharsets.UTF_8))));

            if (indexWriter.getConfig().getOpenMode() == IndexWriterConfig.OpenMode.CREATE_OR_APPEND) {
                // new index
                indexWriter.addDocument(document);
            } else {
                // update existing index
                indexWriter.updateDocument(new Term("path", file.toString()), document);
            }
        }
    }

    private void indexDocs(final IndexWriter indexWriter, Path path) throws ExecutionException, InterruptedException, IOException {
        if (Files.isDirectory(path)) {
            ForkJoinPool FJ_POOL = new ForkJoinPool(3);
            List<Path> files = FSUtils.findAllFiles(path.toString());

            FJ_POOL.submit(() -> files.parallelStream().forEach(t -> {
                try {

                    indexDoc(indexWriter, t, FSUtils.getFileBasicAttribute(t).lastModifiedTime().toMillis());
                } catch (Exception e) {
                    logger.error(e.getMessage(), e);
                }
            })).get();
            FJ_POOL.shutdown();
//            Files.walkFileTree(path, new SimpleFileVisitor<Path>() {
//               @Override
//               public FileVisitResult visitFile (Path file, BasicFileAttributes attrs) throws IOException {
//                   try {
//
//                    indexDoc(indexWriter, file, attrs.lastModifiedTime().toMillis());
//                   } catch (IOException ex) {
//                       ex.printStackTrace();
//                   }
//                   return FileVisitResult.CONTINUE;
//               }
//            });
        } else {
            indexDoc(indexWriter, path, Files.getLastModifiedTime(path).toMillis());
        }
    }

    public void buildIndex(String pathToDocsDir, String pathToIndexDir) throws ExecutionException, InterruptedException, IOException{
        Path docPath = Paths.get(pathToDocsDir);
        Path indexPath = Paths.get(pathToIndexDir);
        long start = System.currentTimeMillis();

        try(Directory dir = FSDirectory.open(indexPath.toFile());
            Analyzer analyzer = new StandardAnalyzer()) {

            IndexWriterConfig iwc = new IndexWriterConfig(Version.LATEST, analyzer);
            iwc.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
            try (IndexWriter indexWriter = new IndexWriter(dir, iwc)) {
                indexDocs(indexWriter, docPath);
            }
        }
    }
    public static void main(String[] args) throws ExecutionException,   InterruptedException, IOException{
      LuceneUtils luceneUtils = new LuceneUtils();

      String docPath = "/home/TestFolder";
      String indexPath = "/home/IndexFolder";
      try {
          luceneUtils.buildIndex(docPath, indexPath);
      } catch (IOException ex) {
          ex.printStackTrace();
      }
   }

}

因此,从我的代码中可以看到,我对两个文件都使用了一个IndexWriter对象,并尝试并行构建索引文件。程序启动几分钟后,我得到了下一个异常:

线程“main”java.util.concurrent.ExecutionException:java.lang.OutOfMemoryError,位于com.service.utils.LuceneUtils.indexDocs(Lucene utils.java:70),位于com.services.utils.lucene utils.buildIndex(Lucene utils.java:100),位置为com.service.uutils.Lucene utils.main(LuceneUtils.java:138),位于sun.reflect.NNativeMethodAccessorImpl.invoke0sun.relect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)位于sun.relect.DelegatingMethodAccessorIntl.invoke(DelegatingMethodAccessorImpl.java:43)位于com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)处的java.lang.OutOfMemoryError位于sun.reflex.NativeConstructorAccessorImpl.newInstance0(本机方法)位于sun.reflek.NativeConstructor AccessorImpl.newInstance(NativeConstructtorAccessorImpl:java:62)位于sun/reflex.DelegatingConstructorAccessor Impl.newInstance(DelegatingConstructorAccessorImpl.java:45)位于java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)位于java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)

是否可以在并行模式下使用一个IndexWriter?我如何解决我的问题?


共2个答案

匿名用户

Lucene有一个很好的函数来并行化索引过程。如果您已在RAMDirectory或FSDirectory中索引了文件,则可以将它们合并为一个索引。您必须使用 addIndexes 进行准备,并使用 forceMerge 来完成合并。因此,您可以将文件拆分为单独的部分,并行索引它们,最后合并它们。

匿名用户

您可以在启动程序时使用-Xmx标志为JVM分配更多内存。例如,标志-Xmx4G将为JVM分配4 GB的RAM。如果您还有足够的额外内存,这很可能会解决您的错误。

如果您使用的是Eclipse,则可以从Run设置标志-