python – 如何将参数传递给scrapy管道对象

发布时间：2020-12-20 11:56:21 所属栏目：Python 来源：网络整理

导读：用scrapy spider抓取一些数据后： class Test_Spider(Spider): name = "test" def start_requests(self): for i in range(900,902,1): ........ yield item 我将数据传递给管道对象,使用sqlalchemy写入sqllite表： class SQLlitePipeline(object): def __ini

用scrapy spider抓取一些数据后：

class Test_Spider(Spider):

    name = "test"
    def start_requests(self):
        for i in range(900,902,1):
            ........
            yield item

我将数据传递给管道对象,使用sqlalchemy写入sqllite表：

class SQLlitePipeline(object):

    def __init__(self):
        _engine = create_engine("sqlite:///data.db")
        _connection = _engine.connect()
        _metadata = MetaData()
        _stack_items = Table("table1",_metadata,Column("id",Integer,primary_key=True),Column("detail_url",Text),_metadata.create_all(_engine)
        self.connection = _connection
        self.stack_items = _stack_items

    def process_item(self,item,spider):
        is_valid = True

我希望能够将表名设置为变量,而不是像现在一样硬编码(“table1”).如何才能做到这一点？

解决方法

假设您通过命令行传递此参数(例如-s table =“table1”),请定义from_crawler方法.

@classmethod
def from_crawler(cls,crawler):
    # Here,you get whatever value was passed through the "table" parameter
    settings = crawler.settings
    table = settings.get('table')

    # Instantiate the pipeline with your table
    return cls(table)

def __init__(self,table):
    _engine = create_engine("sqlite:///data.db")
    _connection = _engine.connect()
    _metadata = MetaData()
    _stack_items = Table(table,_metadata.create_all(_engine)
    self.connection = _connection
    self.stack_items = _stack_items

（编辑：李大同）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!