Examples of org.archive.wayback.surt.SURTTokenizer

org.archive.wayback.surt.SURTTokenizer
provides iterative Url reduction for prefix matching to find ever coarser grained URL-specific configuration. Assumes that a prefix binary search is being attempted for each returned value. First value is the entire SURT url String, with TAB appended. Second removes CGI ARGs. Then each subsequent path segment ('/' separated) is removed. Then the login:password, if present is removed. Then the port, if not :80 or omitted on the initial URL. Then each subsequent authority segment(. separated) is removed. the nextSearch() method will return null, finally, when no broader searches can be attempted on the URL. @author brad @version $Date: 2007-05-29 22:15:55 +0100 (Tue, 29 May 2007) $, $Revision: 1748 $

    exclusionMap = map;
  }
  
  protected boolean isExcluded(String url) {
    try {
      SURTTokenizer st = new SURTTokenizer(url);
      while(true) {
        String nextSearch = st.nextSearch();
        if(nextSearch == null) {
          break;
        }
        LOGGER.fine("EXCLUSION-MAP:Checking " + nextSearch);
        if(exclusionMap.containsKey(nextSearch)) {

View Full Code Here

  /* (non-Javadoc)
   * @see org.archive.wayback.accesscontrol.ExclusionAuthority#checkExclusion(java.lang.String, java.lang.String, java.lang.String)
   */
  public ExclusionResponse checkExclusion(String userAgent, String urlString, 
      String captureDate) throws Exception {
    SURTTokenizer tokenizer;
    try {
      tokenizer = new SURTTokenizer(urlString);
    } catch (URIException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
      throw new Exception(e);
    }


    while(true) {
      String surtPrefix = tokenizer.nextSearch();
      if(surtPrefix == null) {
        break;
      }
      AdministrativeExclusionRule rule = getRuleFor(surtPrefix,captureDate);
      if(rule != null) {

View Full Code Here

    }
  }
  
  protected boolean isExcluded(String url) {
    try {
      SURTTokenizer st = new SURTTokenizer(url, canonicalizer.isSurtForm());
      while(true) {
        String nextSearch = st.nextSearch();
        if(nextSearch == null) {
          break;
        }
        LOGGER.fine("EXCLUSION-MAP:Checking " + nextSearch);
        if(exclusionMap.containsKey(nextSearch)) {

View Full Code Here

TOP

Related Classes of org.archive.wayback.surt.SURTTokenizer

org.archive.net.UURI

org.archive.url.UsableURI

org.archive.wayback.accesscontrol.staticmap.StaticMapExclusionFilter

org.archive.wayback.accesscontrol.ui.AdministrativeExclusionAuthority

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.