python – 非阻塞Scrapy管道到数据库

我在Scrapy中有一个Web抓取器,可以获取数据项.我想异步地将它们插入到数据库中.

例如,我有一个事务,使用SQLAlchemy Core将一些项目插入到我的数据库中：

def process_item(self,item,spider):
    with self.connection.begin() as conn:
        conn.execute(insert(table1).values(item['part1'])
        conn.execute(insert(table2).values(item['part2'])

我知道可以与Twisted with alchimia异步使用SQLAlchemy Core.alchimia的文档代码示例如下.

我不明白的是如何在alchimia框架中使用我的上述代码.如何设置process_item以使用反应器？

我可以这样做吗？

@inlineCallbacks
def process_item(self,spider):
    with self.connection.begin() as conn:
        yield conn.execute(insert(table1).values(item['part1'])
        yield conn.execute(insert(table2).values(item['part2'])

我怎么写反应堆部分？

或者是否有更简单的方法在Scrapy管道中进行非阻塞数据库插入？

供参考,以下是alchimia文档中的代码示例：

from alchimia import TWISTED_STRATEGY

from sqlalchemy import (
    create_engine,MetaData,Table,Column,Integer,String
)
from sqlalchemy.schema import CreateTable

from twisted.internet.defer import inlineCallbacks
from twisted.internet.task import react


@inlineCallbacks
def main(reactor):
    engine = create_engine(
        "sqlite://",reactor=reactor,strategy=TWISTED_STRATEGY
    )

    metadata = MetaData()
    users = Table("users",metadata,Column("id",Integer(),primary_key=True),Column("name",String()),)

    # Create the table
    yield engine.execute(CreateTable(users))

    # Insert some users
    yield engine.execute(users.insert().values(name="Jeremy Goodwin"))
    yield engine.execute(users.insert().values(name="Natalie Hurley"))
    yield engine.execute(users.insert().values(name="Dan Rydell"))
    yield engine.execute(users.insert().values(name="Casey McCall"))
    yield engine.execute(users.insert().values(name="Dana Whitaker"))

    result = yield engine.execute(users.select(users.c.name.startswith("D")))
    d_users = yield result.fetchall()
    # Print out the users
    for user in d_users:
        print "Username: %s" % user[users.c.name]

if __name__ == "__main__":
    react(main,[])