How do I use an index in memory?
Author: Deron Eriksson
Description: This Java tutorial shows how to use a Lucene index in memory rather than in the file system.
Tutorial created using:
Windows XP || JDK 1.5.0_09 || Eclipse Web Tools Platform 2.0 (Eclipse 3.3.0)
In another tutorial, we created a Lucene index in the file system and searched that index. Depending on the application, you may require an index to be stored in-memory rather than in the file system. With Lucene, this can be accomplished easily through the use of the RAMDirectory class, which the javadocs describe as "A memory-resident Directory implementation". This example will utilize the following project structure: ![]() Two text files in the "filesToIndex" directory will be indexed. The first one, deron-foods.txt, lists some foods that I like. deron-foods.txtHere are some foods that Deron likes: hamburger french fries steak mushrooms artichokes The second text file, nicole-foods.txt, lists some foods that Nicole likes. nicole-foods.txtHere are some foods that Nicole likes: apples bananas salad mushrooms cheese Now, let's examine the LuceneMemoryIndexDemo class, which creates an in-memory index and then searches it. The LuceneMemoryIndexDemo class is a slight modification of the LuceneDemo class from the other tutorial. The other tutorial describes the indexing and searching processes, so I won't describe them here. LuceneMemoryIndexDemo.javapackage avajava; import java.io.File; import java.io.FileReader; import java.io.IOException; import java.io.Reader; import java.util.Iterator; import org.apache.lucene.analysis.Analyzer; import org.apache.lucene.analysis.standard.StandardAnalyzer; import org.apache.lucene.document.Document; import org.apache.lucene.document.Field; import org.apache.lucene.index.CorruptIndexException; import org.apache.lucene.index.IndexReader; import org.apache.lucene.index.IndexWriter; import org.apache.lucene.queryParser.ParseException; import org.apache.lucene.queryParser.QueryParser; import org.apache.lucene.search.Hit; import org.apache.lucene.search.Hits; import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.search.Query; import org.apache.lucene.store.Directory; import org.apache.lucene.store.LockObtainFailedException; import org.apache.lucene.store.RAMDirectory; public class LuceneMemoryIndexDemo { public static final String FILES_TO_INDEX_DIRECTORY = "filesToIndex"; public static final String FIELD_PATH = "path"; public static final String FIELD_CONTENTS = "contents"; private static Directory ramDirectory = new RAMDirectory(); public static void main(String[] args) throws Exception { createIndex(); searchIndex("mushrooms"); searchIndex("steak"); searchIndex("steak AND cheese"); searchIndex("steak and cheese"); searchIndex("bacon OR cheese"); } public static void createIndex() throws CorruptIndexException, LockObtainFailedException, IOException { Analyzer analyzer = new StandardAnalyzer(); IndexWriter indexWriter = new IndexWriter(ramDirectory, analyzer); File dir = new File(FILES_TO_INDEX_DIRECTORY); File[] files = dir.listFiles(); for (File file : files) { Document document = new Document(); String path = file.getCanonicalPath(); document.add(new Field(FIELD_PATH, path, Field.Store.YES, Field.Index.UN_TOKENIZED)); Reader reader = new FileReader(file); document.add(new Field(FIELD_CONTENTS, reader)); indexWriter.addDocument(document); } indexWriter.optimize(); indexWriter.close(); } public static void searchIndex(String searchString) throws IOException, ParseException { System.out.println("Searching for '" + searchString + "'"); IndexReader indexReader = IndexReader.open(ramDirectory); IndexSearcher indexSearcher = new IndexSearcher(indexReader); Analyzer analyzer = new StandardAnalyzer(); QueryParser queryParser = new QueryParser(FIELD_CONTENTS, analyzer); Query query = queryParser.parse(searchString); Hits hits = indexSearcher.search(query); System.out.println("Number of hits: " + hits.length()); Iterator<Hit> it = hits.iterator(); while (it.hasNext()) { Hit hit = it.next(); Document document = hit.getDocument(); String path = document.get(FIELD_PATH); System.out.println("Hit: " + path); } } } The LuceneMemoryIndexDemo class utilizes a Directory that is a RAMDirectory, which is memory-resident: private static Directory ramDirectory = new RAMDirectory(); The ramDirectory variable is utilized by the IndexWriter object (which creates the index): IndexWriter indexWriter = new IndexWriter(ramDirectory, analyzer); The ramDirectory variable is also specified for the IndexReader, which is used to search the index: IndexReader indexReader = IndexReader.open(ramDirectory); Making these slight modifications results in a memory-based index that is created via IndexWriter and searched via IndexReader/IndexSearcher. That's all there is to it! The console output of executing LuceneMemoryIndexDemo is shown here: Console OutputSearching for 'mushrooms' Number of hits: 2 Hit: C:\projects\workspace\demo\filesToIndex\nicole-foods.txt Hit: C:\projects\workspace\demo\filesToIndex\deron-foods.txt Searching for 'steak' Number of hits: 1 Hit: C:\projects\workspace\demo\filesToIndex\deron-foods.txt Searching for 'steak AND cheese' Number of hits: 0 Searching for 'steak and cheese' Number of hits: 2 Hit: C:\projects\workspace\demo\filesToIndex\nicole-foods.txt Hit: C:\projects\workspace\demo\filesToIndex\deron-foods.txt Searching for 'bacon OR cheese' Number of hits: 1 Hit: C:\projects\workspace\demo\filesToIndex\nicole-foods.txt |