I am attempting to learn how to use scrapy, and am trying to do what I think is a simple project. I am attempting to pull 2 pieces of data from a single webpage - crawling additional links isn't needed. However, my code seems to be returning zero results. I have tested the xpaths in Scrapy Shell, and both return the expected results.
My item.py is:
import scrapy class StockItem(scrapy.Item): quote = scrapy.Field() time = scrapy.Field()
My spider, named stockscrapy.py, is:
import scrapy class StockSpider(scrapy.Spider): name = "ugaz" allowed_domains = ["nasdaq.com"] start_urls = ["http://www.nasdaq.com/symbol/ugaz/"] def parse(self, response): stock = StockItem() stock['quote'] = response.xpath('//*[@id="qwidget_lastsale"]/text()').extract() stock['time'] = response.xpath('//*[@id="qwidget_markettime"]/text()').extract() return stock
To run the script, I use the command line:
scrapy crawl ugaz -o stocks.csv
Any and all help is greatly appreciated.