Package net.vidageek.crawler.component

Examples of net.vidageek.crawler.component.WebDownloader


    }
    if (!Pattern.compile("(?s)^http[s]?://.*$").matcher(beginUrl).matches()) {
      throw new IllegalArgumentException("beginUrl must start with http:// or https://");
    }
    this.beginUrl = beginUrl;
    downloader = new WebDownloader();
    normalizer = new DefaultLinkNormalizer(beginUrl);
    minPoolSize = 30;
    maxPoolSize = 30;
    keepAliveMilliseconds = 30000;
    requestDelayMilliseconds = 1000;
View Full Code Here

TOP

Related Classes of net.vidageek.crawler.component.WebDownloader

Copyright © 2018 www.massapicom. All rights reserved.
All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.