Examples of EncodingDetector

org.apache.nutch.util.EncodingDetector
A simple class for detecting character encodings.
Broadly this encompasses two functions, which are distinctly separate:
1. Auto detecting a set of "clues" from input text.
2. Taking a set of clues and making a "best guess" as to the "real" encoding.
A caller will often have some extra information about what the encoding might be (e.g. from the HTTP header or HTML meta-tags, often wrong but still potentially useful clues). The types of clues may differ from caller to caller. Thus a typical calling sequence is:
- Run step (1) to generate a set of auto-detected clues;
- Combine these clues with the caller-dependent "extra clues" available;
- Run step (2) to guess what the most probable answer is.
org.vietspider.html.parser.EncodingDetector
Author : Nhu Dinh Thuan nhudinhthuan@yahoo.com Apr 27, 2009

Examples of org.apache.nutch.util.EncodingDetector

   */
  public ParseResult getParse(Content content) {
    SyndFeed feed = null;
    ParseResult parseResult = new ParseResult(content.getUrl());


    EncodingDetector detector = new EncodingDetector(conf);
    detector.autoDetectClues(content, true);
    String encoding = detector.guessEncoding(content, defaultEncoding);
    try {
      InputSource input = new InputSource(new ByteArrayInputStream(content
          .getContent()));
      input.setEncoding(encoding);
      SyndFeedInput feedInput = new SyndFeedInput();

View Full Code Here

Examples of org.apache.nutch.util.EncodingDetector

    try {
      ByteBuffer contentInOctets = page.getContent();
      InputSource input = new InputSource(new ByteArrayInputStream(contentInOctets.array(),
          contentInOctets.arrayOffset() + contentInOctets.position(), contentInOctets.remaining()));


      EncodingDetector detector = new EncodingDetector(conf);
      detector.autoDetectClues(page, true);
      detector.addClue(sniffCharacterEncoding(contentInOctets), "sniffed");
      String encoding = detector.guessEncoding(page, defaultCharEncoding);


      metadata.set(Metadata.ORIGINAL_CHAR_ENCODING, encoding);
      metadata.set(Metadata.CHAR_ENCODING_FOR_CONVERSION, encoding);


      input.setEncoding(encoding);

View Full Code Here

Examples of org.apache.nutch.util.EncodingDetector

   */
  public ParseResult getParse(Content content) {
    SyndFeed feed = null;
    ParseResult parseResult = new ParseResult(content.getUrl());


    EncodingDetector detector = new EncodingDetector(conf);
    detector.autoDetectClues(content, true);
    String encoding = detector.guessEncoding(content, defaultEncoding);
    try {
      InputSource input = new InputSource(new ByteArrayInputStream(content
          .getContent()));
      input.setEncoding(encoding);
      SyndFeedInput feedInput = new SyndFeedInput();

View Full Code Here

Examples of org.apache.nutch.util.EncodingDetector

   */
  public ParseResult getParse(Content content) {
    SyndFeed feed = null;
    ParseResult parseResult = new ParseResult(content.getUrl());


    EncodingDetector detector = new EncodingDetector(conf);
    detector.autoDetectClues(content, true);
    String encoding = detector.guessEncoding(content, defaultEncoding);
    try {
      InputSource input = new InputSource(new ByteArrayInputStream(content
          .getContent()));
      input.setEncoding(encoding);
      SyndFeedInput feedInput = new SyndFeedInput();

View Full Code Here

Examples of org.apache.nutch.util.EncodingDetector

   */
  public ParseResult getParse(Content content) {
    SyndFeed feed = null;
    ParseResult parseResult = new ParseResult(content.getUrl());


    EncodingDetector detector = new EncodingDetector(conf);
    detector.autoDetectClues(content, true);
    String encoding = detector.guessEncoding(content, defaultEncoding);
    try {
      InputSource input = new InputSource(new ByteArrayInputStream(content
          .getContent()));
      input.setEncoding(encoding);
      SyndFeedInput feedInput = new SyndFeedInput();

View Full Code Here

Examples of org.apache.nutch.util.EncodingDetector

    DocumentFragment root;
    try {
      byte[] contentInOctets = page.getContent().array();
      InputSource input = new InputSource(new ByteArrayInputStream(contentInOctets));


      EncodingDetector detector = new EncodingDetector(conf);
      detector.autoDetectClues(page, true);
      detector.addClue(sniffCharacterEncoding(contentInOctets), "sniffed");
      String encoding = detector.guessEncoding(page, defaultCharEncoding);


      metadata.set(Metadata.ORIGINAL_CHAR_ENCODING, encoding);
      metadata.set(Metadata.CHAR_ENCODING_FOR_CONVERSION, encoding);


      input.setEncoding(encoding);

View Full Code Here

Examples of org.apache.nutch.util.EncodingDetector

   */
  public ParseResult getParse(Content content) {
    SyndFeed feed = null;
    ParseResult parseResult = new ParseResult(content.getUrl());


    EncodingDetector detector = new EncodingDetector(conf);
    detector.autoDetectClues(content, true);
    String encoding = detector.guessEncoding(content, defaultEncoding);
    try {
      InputSource input = new InputSource(new ByteArrayInputStream(content
          .getContent()));
      input.setEncoding(encoding);
      SyndFeedInput feedInput = new SyndFeedInput();

View Full Code Here

Examples of org.vietspider.html.parser.EncodingDetector

    if(decode) chars = new RefsDecoder().decode(chars);
    return createDocument(chars);
  }
  
  public String detectCharset(byte [] bytes) {
    EncodingDetector encodingDetector = new EncodingDetector();
    String codeCharset = encodingDetector.detect(bytes);
    if(codeCharset == null) codeCharset = "utf-8";
    try {
      HTMLDocument document = createDocument(bytes, codeCharset);
      String docCharset = getCharset(document);
      if(docCharset == null

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.