Examples of us.codecraft.webmagic.downloader.Downloader

Package us.codecraft.webmagic.downloader

Examples of us.codecraft.webmagic.downloader.Downloader

us.codecraft.webmagic.downloader.Downloader
Downloader is the part that downloads web pages and store in Page object.
Downloader has {@link #setThread(int)} method because downloader is always the bottleneck of a crawler,there are always some mechanisms such as pooling in downloader, and pool size is related to thread numbers. @author code4crafter@gmail.com
@since 0.1.0


            @Override
            public Site getSite() {
                return Site.me().setSleepTime(0);
            }
        }).setDownloader(new Downloader() {
            @Override
            public Page download(Request request, Task task) {
                return new Page().setRawText("");
            }

View Full Code Here

TOP

Related Classes of us.codecraft.webmagic.downloader.Downloader

us.codecraft.webmagic.SpiderTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.