Examples of org.apache.lucene.analysis.TokenStream.end()

Class org.apache.lucene.analysis.TokenStream

Examples of org.apache.lucene.analysis.TokenStream.end()

org.apache.lucene.analysis.TokenStream.end()
This method is called by the consumer after the last token has been consumed, after {@link #incrementToken()} returned false(using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.
This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a {@link WhitespaceTokenizer} was used. @throws IOException

      stream.reset();
      while (stream.incrementToken()) {
        String text = termAtt.toString();
        assertFalse(stopWordsSet.contains(text));
      }
      stream.end();
    } finally {
      IOUtils.closeWhileHandlingException(stream);
    }
  }

View Full Code Here

      while (stream.incrementToken()) {
        String text = termAtt.toString();
        assertFalse(stopWordsSet.contains(text));
        assertEquals(expectedIncr[i++],posIncrAtt.getPositionIncrement());
      }
      stream.end();
    } finally {
      IOUtils.closeWhileHandlingException(stream);
    }
  }

View Full Code Here

                  fragments.add(new LookupHighlightFragment(text.substring(startOffset+prefixToken.length(), startOffset+token.length()), false));
                }
                upto = endOffset;
              }
            }
            ts.end();
            int endOffset = offsetAtt.endOffset();
            if (upto < endOffset) {
              fragments.add(new LookupHighlightFragment(text.substring(upto), false));
            }

View Full Code Here

    try {
      ts.reset();
      while (ts.incrementToken()) {
      
      }
      ts.end();
    } finally {
      IOUtils.closeWhileHandlingException(ts);
    }
  }

View Full Code Here

      TokenStream ts = analyzer.tokenStream("foo", s);
      try {
        ts.reset();
        while (ts.incrementToken()) {
        }
        ts.end();
      } finally {
        IOUtils.closeWhileHandlingException(ts);
      }
    }
  }

View Full Code Here

        CharTermAttribute termAtt = ts.addAttribute(CharTermAttribute.class);
        ts.reset();
        while (ts.incrementToken()) {
          assertTrue(UnicodeUtil.validUTF16String(termAtt));
        }
        ts.end();
      } finally {
        IOUtils.closeWhileHandlingException(ts);
      }
    }
  }

View Full Code Here

  public void testOnlyPunctuation() throws IOException {
    TokenStream ts = analyzerNoPunct.tokenStream("foo", "。、。。");
    try {
      ts.reset();
      assertFalse(ts.incrementToken());
      ts.end();
    } finally {
      IOUtils.closeWhileHandlingException(ts);
    }
  }

View Full Code Here

  public void testOnlyPunctuationExtended() throws IOException {
    TokenStream ts = extendedModeAnalyzerNoPunct.tokenStream("foo", "......");
    try {
      ts.reset();
      assertFalse(ts.incrementToken());
      ts.end();
    } finally {
      IOUtils.closeWhileHandlingException(ts);
    }
  }

View Full Code Here

      for(String reading : readings) {
        assertTrue(ts.incrementToken());
        assertEquals(reading, readingAtt.getReading());
      }
      assertFalse(ts.incrementToken());
      ts.end();
    } finally {
      IOUtils.closeWhileHandlingException(ts);
    }
  }

View Full Code Here

      for(String pronunciation : pronunciations) {
        assertTrue(ts.incrementToken());
        assertEquals(pronunciation, readingAtt.getPronunciation());
      }
      assertFalse(ts.incrementToken());
      ts.end();
    } finally {
      IOUtils.closeWhileHandlingException(ts);
    }
  }

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.