Examples of net.htmlparser.jericho.Tag

net.htmlparser.jericho.Tag
Represents either a {@link StartTag} or {@link EndTag} in a specific {@linkplain Source source} document.
Take the following HTML segment as an example:
<p>This is a sample paragraph.</p>
The "<p>" is represented by a {@link StartTag} object, and the "</p>" is represented by an {@link EndTag} object,both of which are subclasses of the Tag class. The whole segment, including the start tag, its corresponding end tag and all of the content in between, is represented by an {@link Element} object.
Tag Parsing Process
The following process describes how each tag is identified by the parser:
1. Every '<' character found in the source document is considered to be the start of a tag. The characters following it are compared with the {@linkplain TagType#getStartDelimiter() start delimiters}of all the {@linkplain TagType#register() registered} {@linkplain TagType tag types}, and a list of matching tag types is determined.
2. A more detailed analysis of the source is performed according to the features of each matching tag type from the first step, in order of precedence, until a valid tag is able to be constructed.
  The analysis performed in relation to each candidate tag type is a two-stage process:
  1. The position of the tag is checked to determine whether it is {@linkplain TagType#isValidPosition(Source,int,int[]) valid}. In theory, a {@linkplain TagType#isServerTag() server tag} is valid in any position, but a non-server tag is not valid inside any other tag,nor inside elements with CDATA content such as {@link HTMLElementName#SCRIPT SCRIPT} and {@link HTMLElementName#STYLE STYLE} elements.Theory dictates therefore that {@linkplain StartTagType#COMMENT comments} and explicit {@linkplain StartTagType#CDATA_SECTION CDATA sections}inside script elements should not be recognised as tags. The behaviour of the parser however does not always strictly adhere to the theory, to maintain compatability with major browsers and also for efficiency reasons.
    The {@link TagType#isValidPosition(Source, int pos, int[] fullSequentialParseData)} method is responsible for this checkand has a common default implementation for all tag types (although custom tag types can override it if necessary). Its behaviour differs depending on whether or not a {@linkplain Source#fullSequentialParse() full sequential parse} is peformed.See the documentation of the {@link TagType#isValidPosition(Source,int,int[]) isValidPosition} method for full details.
  2. A final analysis is performed by the {@link TagType#constructTagAt(Source, int pos)} method of the candidate tag type.This method returns a valid {@link Tag} object if all conditions of the candidate tag type are met, otherwise it returnsnull and the process continues with the next candidate tag type.
3. If the source does not match the start delimiter or syntax of any registered tag type, the segment spanning it and the next '>' character is taken to be an {@linkplain #isUnregistered() unregistered} tag.Some tag search methods ignore unregistered tags. See the {@link #isUnregistered()} method for more information.
See the documentation of the {@link TagType} class for more details on how tags are recognised.
Tag Search Methods

Methods that get tags in a source document are collectively referred to as Tag Search Methods. They are found mostly in the {@link Source} and {@link Segment} classes, and can be generally categorised as follows:
Open Search:
These methods search for tags of any {@linkplain #getName() name} and {@linkplain #getTagType() type}.
- {@link Tag#getNextTag()}
- {@link Tag#getPreviousTag()}
- {@link Segment#getAllElements()}
- {@link Segment#getFirstElement()}
- {@link Source#getTagAt(int pos)}
- {@link Source#getPreviousTag(int pos)}
- {@link Source#getNextTag(int pos)}
- {@link Source#getEnclosingTag(int pos)}
- {@link Segment#getAllTags()}
- {@link Segment#getAllStartTags()}
- {@link Segment#getFirstStartTag()}
- {@link Source#getPreviousStartTag(int pos)}
- {@link Source#getNextStartTag(int pos)}
- {@link Source#getPreviousEndTag(int pos)}
- {@link Source#getNextEndTag(int pos)}
Named Search:
These methods include a parameter called name which is used to specify the {@linkplain #getName() name} of the tag to search for.Specifying a name that ends in a colon (:) searches for all elements or tags in the specified XML namespace.
- {@link Segment#getAllElements(String name)}
- {@link Segment#getFirstElement(String name)}
- {@link Segment#getAllStartTags(String name)}
- {@link Segment#getFirstStartTag(String name)}
- {@link Source#getPreviousStartTag(int pos,String name)}
- {@link Source#getNextStartTag(int pos,String name)}
- {@link Source#getPreviousEndTag(int pos,String name)}
- {@link Source#getNextEndTag(int pos,String name)}
- {@link Source#getNextEndTag(int pos, String name, EndTagType)}
Tag Type Search:
These methods typically include a parameter called tagType which is used to specify the {@linkplain #getTagType() type} of the tag to search for.In some methods the search parameter is restricted to the {@link StartTagType} or {@link EndTagType} subclass of TagType.
- {@link Segment#getAllElements(StartTagType)}
- {@link Segment#getAllTags(TagType)}
- {@link Segment#getAllStartTags(StartTagType)}
- {@link Segment#getFirstStartTag(StartTagType)}
- {@link Source#getPreviousTag(int pos, TagType)}
- {@link Source#getPreviousStartTag(int pos, StartTagType)}
- {@link Source#getPreviousEndTag(int pos, EndTagType)}
- {@link Source#getNextTag(int pos, TagType)}
- {@link Source#getNextStartTag(int pos, StartTagType)}
- {@link Source#getNextEndTag(int pos, EndTagType)}
- {@link Source#getEnclosingTag(int pos, TagType)}
- {@link Source#getNextEndTag(int pos, String name, EndTagType)}
Attribute Search:
These methods perform the search based on an attribute name and value.
- {@link Segment#getAllElements(String attributeName,String value,boolean valueCaseSensitive)}
- {@link Segment#getFirstElement(String attributeName,String value,boolean valueCaseSensitive)}
- {@link Segment#getAllStartTags(String attributeName,String value,boolean valueCaseSensitive)}
- {@link Segment#getFirstStartTag(String attributeName,String value,boolean valueCaseSensitive)}
- {@link Segment#getAllElements(String attributeName,Pattern valueRegexPattern)}
- {@link Segment#getFirstElement(String attributeName,Pattern valueRegexPattern)}
- {@link Segment#getAllStartTags(String attributeName,Pattern valueRegexPattern)}
- {@link Segment#getFirstStartTag(String attributeName,Pattern valueRegexPattern)}
- {@link Segment#getAllElementsByClass(String className)}
- {@link Segment#getFirstElementByClass(String className)}
- {@link Segment#getAllStartTagsByClass(String className)}
- {@link Segment#getFirstStartTagByClass(String className)}
- {@link Source#getElementById(String id)}
- {@link Source#getNextElement(int pos,String attributeName,Pattern valueRegexPattern)}
- {@link Source#getNextElement(int pos,String attributeName,String value,boolean valueCaseSensitive)}
- {@link Source#getNextElementByClass(int pos,String className)}
- {@link Source#getNextStartTag(int pos,String attributeName,Pattern valueRegexPattern)}
- {@link Source#getNextStartTag(int pos,String attributeName,String value,boolean valueCaseSensitive)}
- {@link Source#getNextStartTagByClass(int pos,String className)}

        // print and set the title
        LOG.info("KML Element: " + kmlElementNameClean);
        kmlField.setName(kmlElementName);
        kmlField.setNameClean(kmlElementNameClean);


        Tag currentTag = htmlElement.getStartTag().getNextTag().getNextTag();
        while (!currentTag.getName().equals("h2")) {
          if (currentTag.getName().equals("h3")) {
            // here all the magic happens! an
            checkForPossibleSubElements(currentTag, lookForTheseTags, kmlField, kmlElements);
          }


          /*
           * Each element is represented by two tags (start- and end-tag). And because there is no method like nextElement, we have to skip
           * two tags, to get to the next element.
           */
          if ((currentTag.getNextTag() == null) || (currentTag.getNextTag().getNextTag() == null)) {
            LOG.info("-> break");
            break;
          }
          currentTag = currentTag.getNextTag().getNextTag();
        }


        // put current element into the map of kmlElements (but only if a description is set)
        if (kmlField.getDescription().size() == 0) {
          continue;

View Full Code Here

   * @return
   */
  private static Tag checkForPossibleSubElements(final Tag ownerTag, final ArrayList<String> lookForTheseTags, final KMLReferenceField kmlField,
      final HashMap<String, KMLReferenceField> kmlElements) {


    Tag currentTag = ownerTag.getElement().getEndTag().getNextTag();
    final String h3Header = ownerTag.getElement().getTextExtractor().toString().trim().toLowerCase();


    // if current tag is a h3-tag, check of element-properties
    while (!currentTag.getName().equals("h3")) {


      for (final String foundpossibleSubTag : lookForTheseTags) {
        // if possible substring iss found and is not annotated as "back-to-top" then examine the element-kind
        if (currentTag.getName().equals(foundpossibleSubTag) && !"backtotop".equals(currentTag.getElement().getAttributeValue("class"))) {


          final String textFromCurrentElement = currentTag.getElement().getTextExtractor().toString().trim();
          // if there is no text in the current element -> skip to the next tag
          if (textFromCurrentElement.length() == 0) {
            continue;
          }


          /*
           * <h3>Syntax</h3>-Element found? The syntax-element dows usually contain one <pre>-tag containing the basic syntax of the
           * kml-element
           */
          if (h3Header.startsWith("syntax")) {
            // LOG.info("Syntax:     \t" + textFromCurrentElement);
            for (final Element elementPre : currentTag.getChildElements()) {
              kmlField.addToSyntax(elementPre.toString());
            }
          }


          /*
           * <h3>Description</h3>-Element found? The description-element contains one or more <p>-tags, which descripe the usage of the
           * kml-element
           */
          else if (h3Header.startsWith("description") || foundpossibleSubTag.equals("h4")) {
            LOG.info("Description: \t" + textFromCurrentElement);
            kmlField.addToDescription(textFromCurrentElement);
          }


          /*
           * <h3>Extends</h3>-Element found? if only one element is listed it is encapsulated in a <p>-tag and if it extends more elements
           * it's encapsulated in an <ul>-tag as unordered list
           */
          else if (h3Header.startsWith("extends")) {
            if (currentTag.getName().equals("p")) {
              kmlField.addToExtend(textFromCurrentElement);
              LOG.info("Extends:     \t" + textFromCurrentElement);
            } else if (currentTag.getName().equals("ul")) {
              for (final Element elementUL : currentTag.getElement().getChildElements()) {


                for (final Element elementLI : elementUL.getChildElements()) {
                  kmlField.addToExtend(elementLI.getTextExtractor().toString());
                  LOG.info("Extends:     \t" + elementLI.getTextExtractor().toString());
                }
              }
            } else {
              // LOG.info("Extends EL:     \t----------- damn");
            }
          }


          /*
           * <h3>Extended By</h3>-Element found? if only one element is listed it is encapsulated in a <p>-tag and if it extends more
           * elements it's encapsulated in an <ul>-tag as unordered list
           */
          else if (h3Header.startsWith("extended by")) {
            if (currentTag.getName().equals("p")) {
              kmlField.addToExtendedBy(textFromCurrentElement);
              LOG.info("Extended By: \t" + textFromCurrentElement);
            } else if (currentTag.getName().equals("ul")) {
              for (final Element elementUL : currentTag.getElement().getChildElements()) {
                // text in ul we want is encapsulated in <a>- or
                // in <strong>-tag
                LOG.info("Extended By: \t" + elementUL.getStartTag().getNextTag().getElement().getTextExtractor().toString());
                kmlField.addToExtendedBy(elementUL.getStartTag().getNextTag().getElement().getTextExtractor().toString());
              }
            } else {
              // LOG.info("Extended By EL: \t----------- damn ");
            }
          }


          /*
           * <h3>Elements Specific to</h3>-Element found? the complex case! "elements specific to"-could define new elements or explains
           * elements, which are only use as fields for existings kml-fields
           */
          else if (h3Header.startsWith("elements specific to")) {
            if ((currentTag.getName().equals("p") || currentTag.getName().equals("dl") || currentTag.getName().equals("dt") || currentTag
                .getName().equals("ul"))) {
              final Tag endTag = currentTag.getElement().getEndTag();


              KMLReferenceField foundSpecificToField = null;
              while (currentTag.getEnd() < endTag.getEnd()) {
                // abort condition !h3 needed by <IconStyle> because of wrong nested h3-element (<dl><h3></h3>...)
                if (currentTag.getName().equals("h3")) {
                  LOG.info("!! abort condition !h3 needed by <IconStyle> because of wrong nested h3-element (<dl><h3></h3>...)");
                  break;
                }

View Full Code Here

        }


        superNode.getChildren().add(new HtmlBlockNode(text.substring(pos, markdownElement.getBegin())));


        String indent = getIndent(text, markdownElement);
        Tag endTag = getEndTag(text, source, markdownElement, indent);
        String innerText = getInnerText(text, markdownElement, endTag, indent);
        List<Node> children = parseInnerText(innerText);


        superNode.getChildren().add(new MarkdownInsideHtmlBlockNode(
            markdownElement.getStartTag().toString(),
            children,
            endTag.toString()
        ));


        return createMarkdownInsideHtmlBlockNode(text, endTag.getEnd(), source, superNode);
    }

View Full Code Here

    }


  private Field extractFieldByDetectingTagWrapper(Element liElement) {
    Field found = null;
    if (liElement.getAllTags().size() == 4) {
      Tag enclosingTag = liElement.getAllTags().get(1);
      log.info("enclosing tag: {}", enclosingTag);
      log.info("first element of enclosing tag: {}", enclosingTag.getElement().getTextExtractor().toString());
      String tagText = enclosingTag.getElement().getTextExtractor().toString();
      String allText = liElement.getTextExtractor().toString();
      log.info("enclosing tag text starts at: {}", allText.indexOf(tagText));
      log.debug("tagText: {} alltext: {}", tagText, allText);
      if (allText.startsWith(tagText)) {
        found = new ScrapedField(tagText, allText.substring(tagText.length() + 1));

View Full Code Here

      if (parts.length == 2) {
        Field field = new ScrapedField(parts[0], parts[1]);
        extractedFields.add(field);
        log.debug("found <li> to process: {}, added field: {}", li, field);
      } else if (tagsWithSpecificTagRemoved(HTMLElementName.BR, li.getAllTags()).size() == 4) {
        Tag enclosingTag = li.getAllTags().get(1);
        log.debug("enclosing tag: {}", enclosingTag);
        log
            .debug("first element of enclosing tag: {}", enclosingTag
                .getElement()
                .getTextExtractor()
                .toString());
        String tagText = enclosingTag
            .getElement()
            .getRenderer()
            .setMaxLineLength(Integer.MAX_VALUE)
            .toString()
            .trim()

View Full Code Here

  }


  private List<Tag> tagsWithSpecificTagRemoved(String tagNameToRemove, List<Tag> tags) {
    log.debug("Tag name to remove = {}, tags to operate on = {}", tagNameToRemove, tags);
    for (int i = 0; i < tags.size(); i++) {
      Tag currentTag = tags.get(i);
      log.debug("Current tag name = {}", currentTag.getName());
      if (currentTag.getName() == tagNameToRemove) {
        tags.remove(currentTag);
        i--;
      }
    }
    log.debug("returning tags = {}", tags);

View Full Code Here

TOP

Related Classes of net.htmlparser.jericho.Tag

com.ontometrics.scraper.extraction.DefaultFieldExtractor

com.ontometrics.scraper.extraction.DefaultFieldExtractorTest

de.micromata.javaapiforkml.PrepareDocumentation

org.pegdown.GaidenParser

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of net.htmlparser.jericho.Tag

Tag Parsing Process

Tag Search Methods

Related Classes of net.htmlparser.jericho.Tag