Scrapy框架下第一个爬虫-白红宇

Scrapy框架下第一个爬虫

阅读量：7194 次

发布时间：2019-06-29

本文共 3780 字，大约阅读时间需要 12 分钟。

1. 安装scrapy

pip install scrapy

2. 第一个爬虫代码myspider.py

import scrapyclass BlogSpider(scrapy.Spider):    name = 'blogspider'    start_urls = ['https://blog.scrapinghub.com']    def parse(self, response):        for title in response.css('h2.entry-title'):            yield {'title': title.css('a ::text').extract_first()}        for next_page in response.css('div.prev-post > a'):            yield response.follow(next_page, self.parse)

3. 运行

scrapy runspider myspider.py -o result.json

4. 运行结果

017-08-06 17:44:56 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: scrapybot)2017-08-06 17:44:56 [scrapy.utils.log] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True}2017-08-06 17:44:56 [scrapy.middleware] INFO: Enabled extensions:['scrapy.extensions.memusage.MemoryUsage',...2017-08-06 17:44:56 [scrapy.core.engine] INFO: Spider opened2017-08-06 17:44:56 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)2017-08-06 17:44:56 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:60232017-08-06 17:45:01 [scrapy.core.engine] DEBUG: Crawled (200) 
    
      (referer: None)2017-08-06 17:45:01 [scrapy.core.scraper] DEBUG: Scraped from <200 https://blog.scrapinghub.com>{'title': u'Scraping the Steam Game Store with Scrapy'}2017-08-06 17:45:01 [scrapy.core.scraper] DEBUG: Scraped from <200 https://blog.scrapinghub.com>{'title': u'Do Androids Dream of Electric Sheep?'}...2017-08-06 17:45:12 [scrapy.core.scraper] DEBUG: Scraped from <200 https://blog.scrapinghub.com/page/11/>{'title': u'Hello, world'}2017-08-06 17:45:12 [scrapy.core.engine] INFO: Closing spider (finished)2017-08-06 17:45:12 [scrapy.statscollectors] INFO: Dumping Scrapy stats:{'downloader/request_bytes': 2933, 'downloader/request_count': 11, 'downloader/request_method_count/GET': 11, 'downloader/response_bytes': 123915, 'downloader/response_count': 11, 'downloader/response_status_count/200': 11, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2017, 8, 6, 9, 45, 12, 168728), 'item_scraped_count': 105, 'log_count/DEBUG': 117, 'log_count/INFO': 7, 'memusage/max': 43909120, 'memusage/startup': 43909120, 'request_depth_max': 10, 'response_received_count': 11, 'scheduler/dequeued': 11, 'scheduler/dequeued/memory': 11, 'scheduler/enqueued': 11, 'scheduler/enqueued/memory': 11, 'start_time': datetime.datetime(2017, 8, 6, 9, 44, 56, 752503)}2017-08-06 17:45:12 [scrapy.core.engine] INFO: Spider closed (finished)

生成结果的文件result.json

[{"title": "Scraping the Steam Game Store with Scrapy"},{"title": "Do Androids Dream of Electric Sheep?"},{"title": "Deploy your Scrapy Spiders from GitHub"},{"title": "Looking Back at 2016"},{"title": "How to Increase Sales with Online Reputation Management"},{"title": "How to Build your own Price Monitoring Tool"},...{"title": "Spoofing your Scrapy bot IP using tsocks"},{"title": "Hello, world"}]

解析运行过程

当你运行下面命令时，Scrapy框架会启动爬虫引擎，根据myspider.py中的逻辑进行抓取网页，然后把结果存到result.json中。

scrapy runspider myspider.py -o result.json

第一步：爬虫先请求start_urls中定义到URLs。本例中，只有一个URL。请求该URL返回内容，如下所示。

...	Scraping the Steam Game Store with Scrapy
...  Interview: How Up Hail uses Scrapy to Increase Transparency
...
    		  
     OLDER POST				
    
...

第二步：调用callback方法parse，解析返回到内容。在回调函数parse中，爬虫循环使用CSS选择器 h2.entry-title，找出网页中的<h2 class="entry-title">元素，并这些元素中的text找出来，生成Python的dict格式数据。

当前页中所有的h2.entry-title元素找完以后，爬虫会执行下面的代码

for next_page in response.css('div.prev-post > a'):            yield response.follow(next_page, self.parse)

它会找div.prev-post元素，然后再次递归执行parse方法。从网页上可以看到，div.prev-post元素是翻页链接，所以，爬虫会不断翻页，知道抓取所有的网页。

第三步：生成的字典数据存到result.json文件中

scrapy的任务是异步执行的，也就是说，它不用等一个请求返回以后才发送另一个请求，而是可以同时进行的。这可以加快运行速度。我们也可以对Scrapy进行设置，比如每一个请求延迟一段时间，等等。

转载于:https://my.oschina.net/stanleysun/blog/1503588

你可能感兴趣的文章