Overview

发布时间：2020-12-16 23:58:56 所属栏目：Python 来源：网络整理

导读：原文地址：https://doc.scrapy.org/en/latest/topics/architecture.html This document describes the architecture of Scrapy and how its components interact. Overview The following diagram shows an overview of the Scrapy architecture with its co

原文地址：https://doc.scrapy.org/en/latest/topics/architecture.html

This document describes the architecture of Scrapy and how its components interact.

Overview

The following diagram shows an overview of the Scrapy architecture with its components and an outline of the data flow that takes place inside the system (shown by the red arrows). A brief description of the components is included below with links for more detailed information about them. The data flow is also described below.

Data flow

The data flow in Scrapy is controlled by the execution engine,and goes like this:

The??schedules the Requests in the??and asks for the next Requests to crawl.
The??returns the next Requests to the?.
The??sends the Requests to the?,passing through the??(see?).
Once the page finishes downloading the??generates a Response (with that page) and sends it to the Engine,passing through the??(see?).
The??receives the Response from the??and sends it to the??for processing,passing through the??(see?).
The??processes the Response and returns scraped items and new Requests (to follow) to the?,passing through the??(see?).
The??sends processed items to?,then send processed Requests to the??and asks for possible next Requests to crawl.
The process repeats (from step 1) until there are no more requests from the?.

Components

Scrapy Engine

The engine is responsible for controlling the data flow between all components of the system,and triggering events when certain actions occur. See the??section above for more details.

Scheduler

The Scheduler receives requests from the engine and enqueues them for feeding them later (also to the engine) when the engine requests them.

Downloader

The Downloader is responsible for fetching web pages and feeding them to the engine which,in turn,feeds them to the spiders.

Spiders

Spiders are custom classes written by Scrapy users to parse responses and extract items (aka scraped items) from them or additional requests to follow. For more information see?.

Item Pipeline

The Item Pipeline is responsible for processing the items once they have been extracted (or scraped) by the spiders. Typical tasks include cleansing,validation and persistence (like storing the item in a database). For more information see?.

Downloader middlewares

Downloader middlewares are specific hooks that sit between the Engine and the Downloader and process requests when they pass from the Engine to the Downloader,and responses that pass from Downloader to the Engine.

Use a Downloader middleware if you need to do one of the following:

change received response before passing it to a spider;
send a new Request instead of passing received response to a spider;
pass response to a spider without fetching a web page;
silently drop some requests.

For more information see?.

Spider middlewares

Spider middlewares are specific hooks that sit between the Engine and the Spiders and are able to process spider input (responses) and output (items and requests).

Use a Spider middleware if you need to

post-process start_requests;
handle spider exceptions;
call errback instead of callback for some of the requests based on response content.

For more information see?.

Event-driven networking

Scrapy is written with?,a popular event-driven networking framework for Python. Thus,it’s implemented using a non-blocking (aka asynchronous) code for concurrency.

（编辑：李大同）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!