如何访问Scrapy CrawlSpider中的特定start_url？

但是你可以覆盖make_requests_from_url方法并将开始网址放入meta.然后在解析中你可以从那里提取它(如果你在后面的请求中产生了解析方法,不要忘记在它们中转发那个开始网址).

我没有使用CrawlSpider,也许Maxim建议你可以使用它,但请记住,response.url在可能的重定向后有url.

这是我将如何做的一个例子,但它只是一个例子(取自scrapy教程)并且未经过测试：

class MySpider(CrawlSpider):
    name = 'example.com'
    allowed_domains = ['example.com']
    start_urls = ['http://www.example.com']

    rules = (
        # Extract links matching 'category.php' (but not matching 'subsection.php')
        # and follow links from them (since no callback means follow=True by default).
        Rule(SgmlLinkExtractor(allow=('category.php',),deny=('subsection.php',))),# Extract links matching 'item.php' and parse them with the spider's method parse_item
        Rule(SgmlLinkExtractor(allow=('item.php',)),callback='parse_item'),)

    def parse(self,response): # When writing crawl spider rules,avoid using parse as callback,since the CrawlSpider uses the parse method itself to implement its logic. So if you override the parse method,the crawl spider will no longer work.
        for request_or_item in CrawlSpider.parse(self,response):
            if isinstance(request_or_item,Request):
                request_or_item = request_or_item.replace(meta = {'start_url': response.meta['start_url']})
            yield request_or_item

    def make_requests_from_url(self,url):
        """A method that receives a URL and returns a Request object (or a list of Request objects) to scrape. 
        This method is used to construct the initial requests in the start_requests() method,and is typically used to convert urls to requests.
        """
        return Request(url,dont_filter=True,meta = {'start_url': url})

    def parse_item(self,response):
        self.log('Hi,this is an item page! %s' % response.url)

        hxs = HtmlXPathSelector(response)
        item = Item()
        item['id'] = hxs.select('//td[@id="item_id"]/text()').re(r'ID: (d+)')
        item['name'] = hxs.select('//td[@id="item_name"]/text()').extract()
        item['description'] = hxs.select('//td[@id="item_description"]/text()').extract()
        item['start_url'] = response.meta['start_url']
        return item

问你是否有任何问题.顺便说一句,使用PyDev的“转到定义”功能,您可以看到scrapy源并了解Request,make_requests_from_url和其他类和方法所期望的参数.进入代码可以帮助并节省您的时间,尽管一开始可能看起来很困难.