Package websphinx

Examples of websphinx.RobotExclusion

88
89
90
91
92
93
94
     */
    public IterativeHTMLCrawler(String url_list_file, String html_dump_directory, String userAgent) {
        this.url_list_file = url_list_file;
        this.html_dump_directory = html_dump_directory;

        robot = new RobotExclusion(userAgent);
    }
View Full Code Here
103
104
105
106
107
108
109
110
111
112
113

        this.url_list_file = ce.resolvePath(ce.getURIList());
        this.html_dump_directory = ce.resolvePath(ce.getHTDocsDumpDir());

        robot = new RobotExclusion(ce.getUserAgent());

        String robots_file = ce.getRobotsFile();
        String robots_domain = ce.getRobotsDomain();
        if (robots_file != null && robots_domain != null) {
            log.debug(robots_file + " " + robots_domain);
View Full Code Here
TOP

Related Classes of websphinx.RobotExclusion

  • org.apache.lenya.search.crawler.IterativeHTMLCrawler
  • java.io.PushbackInputStream
  • java.net.URLConnection
  • java.io.BufferedInputStream
  • java.net.URL
    • Copyright © 2018 www.massapicom. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.