Package org.commoncrawl.util.shared

Examples of org.commoncrawl.util.shared.GoogleURL$GoogleURLComponent

353
354
355
356
357
358
359
360
361
362
363
364
365
   */

  public static String canonicalizeURL(String incomingURL,
      boolean stripLeadingWWW) throws MalformedURLException {

    GoogleURL urlObject = new GoogleURL(incomingURL);

    if (!urlObject.isValid()) {
      throw new MalformedURLException("URL:" + incomingURL + " is invalid");
    }

    return canonicalizeURL(urlObject, stripLeadingWWW);
  }
View Full Code Here
475
476
477
478
479
480
481
482
483
484
485
486
487
488
        if (colonEnd != -1) {
          hostLength = colonEnd;
          host = urlString.substring(hostStart, hostStart + hostLength);
        }

        GoogleURL urlObject = new GoogleURL("http://" + host);

        if (urlObject.isValid()) {
          return urlObject.getHost();
        }
      }
    }
    return null;
  }
View Full Code Here
TOP

Related Classes of org.commoncrawl.util.shared.GoogleURL$GoogleURLComponent

  • org.commoncrawl.util.shared.URLUtils
    • Copyright © 2018 www.massapicom. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.