我在一个目录中有两个大文件(约200 MB),并想在它们上建立索引,所以这是我的代码:
public class LuceneUtil {
private void indexDoc(IndexWriter indexWriter, Path file, long lastModified) throws IOException{
try (InputStream stream = Files.newInputStream(file)) {
Document document = new Document();
Field pathField = new StringField("path", file.toString(), Field.Store.YES);
document.add(pathField);
document.add(new LongField("modified", lastModified, Field.Store.NO));
document.add(new TextField("contents", new BufferedReader(new InputStreamReader(stream, StandardCharsets.UTF_8))));
if (indexWriter.getConfig().getOpenMode() == IndexWriterConfig.OpenMode.CREATE_OR_APPEND) {
// new index
indexWriter.addDocument(document);
} else {
// update existing index
indexWriter.updateDocument(new Term("path", file.toString()), document);
}
}
}
private void indexDocs(final IndexWriter indexWriter, Path path) throws ExecutionException, InterruptedException, IOException {
if (Files.isDirectory(path)) {
ForkJoinPool FJ_POOL = new ForkJoinPool(3);
List<Path> files = FSUtils.findAllFiles(path.toString());
FJ_POOL.submit(() -> files.parallelStream().forEach(t -> {
try {
indexDoc(indexWriter, t, FSUtils.getFileBasicAttribute(t).lastModifiedTime().toMillis());
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
})).get();
FJ_POOL.shutdown();
// Files.walkFileTree(path, new SimpleFileVisitor<Path>() {
// @Override
// public FileVisitResult visitFile (Path file, BasicFileAttributes attrs) throws IOException {
// try {
//
// indexDoc(indexWriter, file, attrs.lastModifiedTime().toMillis());
// } catch (IOException ex) {
// ex.printStackTrace();
// }
// return FileVisitResult.CONTINUE;
// }
// });
} else {
indexDoc(indexWriter, path, Files.getLastModifiedTime(path).toMillis());
}
}
public void buildIndex(String pathToDocsDir, String pathToIndexDir) throws ExecutionException, InterruptedException, IOException{
Path docPath = Paths.get(pathToDocsDir);
Path indexPath = Paths.get(pathToIndexDir);
long start = System.currentTimeMillis();
try(Directory dir = FSDirectory.open(indexPath.toFile());
Analyzer analyzer = new StandardAnalyzer()) {
IndexWriterConfig iwc = new IndexWriterConfig(Version.LATEST, analyzer);
iwc.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
try (IndexWriter indexWriter = new IndexWriter(dir, iwc)) {
indexDocs(indexWriter, docPath);
}
}
}
public static void main(String[] args) throws ExecutionException, InterruptedException, IOException{
LuceneUtils luceneUtils = new LuceneUtils();
String docPath = "/home/TestFolder";
String indexPath = "/home/IndexFolder";
try {
luceneUtils.buildIndex(docPath, indexPath);
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
因此,从我的代码中可以看到,我对两个文件都使用了一个IndexWriter
对象,并尝试并行构建索引文件。程序启动几分钟后,我得到了下一个异常:
线程“main”java.util.concurrent.ExecutionException:java.lang.OutOfMemoryError,位于com.service.utils.LuceneUtils.indexDocs(Lucene utils.java:70),位于com.services.utils.lucene utils.buildIndex(Lucene utils.java:100),位置为com.service.uutils.Lucene utils.main(LuceneUtils.java:138),位于sun.reflect.NNativeMethodAccessorImpl.invoke0sun.relect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)位于sun.relect.DelegatingMethodAccessorIntl.invoke(DelegatingMethodAccessorImpl.java:43)位于com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)处的java.lang.OutOfMemoryError位于sun.reflex.NativeConstructorAccessorImpl.newInstance0(本机方法)位于sun.reflek.NativeConstructor AccessorImpl.newInstance(NativeConstructtorAccessorImpl:java:62)位于sun/reflex.DelegatingConstructorAccessor Impl.newInstance(DelegatingConstructorAccessorImpl.java:45)位于java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)位于java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
是否可以在并行模式下使用一个IndexWriter
?我如何解决我的问题?
Lucene有一个很好的函数来并行化索引过程。如果您已在RAMDirectory或FSDirectory中索引了文件,则可以将它们合并为一个索引。您必须使用 addIndexes 进行准备,并使用 forceMerge 来完成合并。因此,您可以将文件拆分为单独的部分,并行索引它们,最后合并它们。
您可以在启动程序时使用-Xmx
标志为JVM分配更多内存。例如,标志-Xmx4G
将为JVM分配4 GB的RAM。如果您还有足够的额外内存,这很可能会解决您的错误。
如果您使用的是Eclipse,则可以从Run设置标志-