Running multiple spiders in the same process, one spider at a time
web-scraping,scrapy,scrapy-spider
You can rely on the spider_closed signal to start crawling for the next postal code/category. Here is the sample code (not tested) based on this answer and adopted for your use case: from scrapy.crawler import Crawler from scrapy import log, signals from scrapy.settings import Settings from twisted.internet import reactor #...
Extracting links with scrapy that have a specific css class
python,web-scraping,scrapy,screen-scraping,scrapy-spider
From what I understand, you want something similar to restrict_xpaths, but provide a CSS selector instead of an XPath expression. This is actually a built-in feature in Scrapy 1.0 (currently in a release candidate state), the argument is called restrict_css: restrict_css a CSS selector (or list of selectors) which defines...
Using scrapy's FormRequest no form is submitted
python-2.7,web-scraping,web-crawler,scrapy,scrapy-spider
The main problem is in how you are passing the booking type, country, pickup and dropoff. You need to pass the corresponding "id"s instead of literal strings. The following would work in your case: return FormRequest.from_response( response, formxpath="//form[@id='transfer_search']", formdata={ 'bookingtypeid': '1', 'airportgroupid': '14', 'pickup': '121', 'dropoff': '1076', 'arrivaldate': '12-07-2015', 'arrivalhour':...
Scrapy creating XML feed wraps content in “value” tags
python,xml,scrapy,scrapy-spider
The extract() method will always return a list of values, even if there is only a single value as a result, for example: [4], [3,4,5] or None. To avoid this, if you know there is only one value, you can select it like: item['to'] = selector.xpath('//to/text()').extract()[0] Note: Be aware that...
While scraping getting error instance method has no attribute '__getitem__'
python,web-scraping,scrapy,web-crawler,scrapy-spider
You need to call extract() and then get the first item: item['main_title'] = site.xpath('.//li[@id="nav_cat_0"]/text()').extract()[0] # HERE ^ If you want to have a separate category for each item, iterate over them: for title in site.xpath('.//li[starts-with(@id, "nav_cat_")]/text()').extract(): item = AmazonItem() item['main_title'] = title items.append(item) ...
Pass argument to scrapy spider within a python script
python,python-2.7,web-scraping,scrapy,scrapy-spider
You need to modify your __init__() constructor to accept the date argument. Also, I would use datetime.strptime() to parse the date string: from datetime import datetime class MySpider(CrawlSpider): name = 'tw' allowed_domains = ['test.com'] def __init__(self, *args, **kwargs): super(MySpider, self).__init__(*args, **kwargs) date = kwargs.get('date') if not date: raise ValueError('No date...
How to exclude a particular html tag(without any id) from several tags while using scrapy?
python,html,web-scraping,scrapy,scrapy-spider
I would use starts-with() XPath function to get the div element's text that starts with "Timings": sel.xpath('.//div[starts-with(., "Timings")]/text()').extract() Note that the HTML structure of the page doesn't make it easy to distinguish locations between each other - there is no location-specific containers that you can iterate over. In this case,...
Scrapy Limit Requests For Testing
python,python-2.7,web-scraping,scrapy,scrapy-spider
You are looking for the CLOSESPIDER_PAGECOUNT setting of the CloseSpider extension: An integer which specifies the maximum number of responses to crawl. If the spider crawls more than that, the spider will be closed with the reason closespider_pagecount. If zero (or non set), spiders won’t be closed by number of...
Scrapy not giving individual results of all the reviews of a phone?
python,xpath,web-scraping,scrapy,scrapy-spider
First of all, your XPath expressions are very fragile in general. The main problem with your approach is that site does not contain a review section, but it should. In other words, you are not iterating over review blocks on a page. Also, the model name should be extracted outside...
Scrapy: Extract links and text
python,web-scraping,scrapy,scrapy-spider
There is a simple mistake inside the xpath expressions for the item fields. The loop is already going over the a tags, you don't need to specify a in the inner xpath expressions. In other words, currently you are searching for a tags inside the a tags inside the td...
Scrapy keep all unique pages based on a list of start urls
python,web-scraping,scrapy,scrapy-spider
Set the default parse callback to spin off all the links. By default Scrapy does not visit the same page twice. def parse(self, response): links = LinkExtractor().extract_links(response) return (Request(url=link.url, callback=self.parse_page) for link in links) def parse_page(self, response): # name = manipulate response.url to be a unique file name with open(name,...
Remove first tag html using python & scrapy
python,xpath,scrapy,scrapy-spider
You can get all the child tags except the div with class="needremove": response.xpath('//div[contains(@class, "abc")]/div[contains(@class, "xyz")]/*[local-name() != "div" and not(contains(@class, "needremove"))]').extract() Demo from the shell: $ scrapy shell index.html In [1]: response.xpath('//div[contains(@class, "abc")]/div[contains(@class, "xyz")]/*[local-name() != "div" and not(contains(@class, "needremove"))]').extract() Out[1]: [u'<p>text</p>', u'<p>text</p>', u'<p>text</p>',...
Scrapy - Scrape multiple URLs using results from the first URL
I think what you're looking for is the yield statement: def parse(self, response): # Get the list of URLs, for example: list = ["http://a.com", "http://b.com", "http://c.com"] for link in list: request = scrapy.Request(link) yield request ...
Scrapy command in shell script not executing when called from java
java,bash,shell,scrapy,scrapy-spider
Your are missing forward slash on your 'PATH' declaration, and path should be to the directory, not the program You have export PATH=usr/local/bin/scrapy:$PATH Should be export PATH=/usr/local/bin:$PATH ...
Scrapy collect data from first element and post's title
python,web-scraping,web-crawler,scrapy,scrapy-spider
For posting title, get all the text nodes from the span tag and join them: $ scrapy shell http://denver.craigslist.org/bik/5042090428.html In [1]: "".join(response.xpath("//span[@class='postingtitletext']//text()").extract()) Out[1]: u'Tonka double shock boys bike - $10 (Denver)' Note that the "Scrapy-way" to do this would be to use an ItemLoader and the Join() processor. Second is...
How to identify a request's crucial information that needs to be sent?
web-scraping,scrapy,session-cookies,scrapy-spider
You do not even need the cookies to send with your request. The problem is with body=urllib.urlencode(payload), This encodes the body to URL-Format however if you look at the body of the request of your browser you will see that a JSON is the body. So the solution is to...
Why isn't XMLFeedSpider failing to iterate through the designated nodes?
python,xml,rss,scrapy,scrapy-spider
You need to handle namespaces: class PLoSSpider(XMLFeedSpider): name = "plos" namespaces = [('atom', 'http://www.w3.org/2005/Atom')] itertag = 'atom:entry' iterator = 'xml' # this is also important See also: how do I use empty namespaces in an lxml xpath query? Working example: from scrapy.contrib.spiders import XMLFeedSpider class PLoSSpider(XMLFeedSpider): name = "plos" namespaces...
Scrapy - generating items outside of parse callback
I'm not sure if there is a way to push items directly to the engine, but what you could do is push dummy requests with the items in the meta variable and just yield them in the callback. def _file_thread_loop(self): while True: #... read files... for file in files: response...
Scrapy - Issue with xpath on an xml crawl
python,xml,xpath,scrapy,scrapy-spider
The signature of parse_node() is incorrect. There should be selector argument given which you should call xpath() method on, example: def parse_node(self, response, selector): to = selector.xpath('//to/text()').extract() who = selector.xpath('//from/text()').extract() print to, who Prints: [u'Tove'] [u'Jani'] ...
Is there a way using scrapy to export each item that is scrapped into a separate json file?
web-scraping,scrapy,scrapy-spider
You can use scrapy-pipeline and from there you can insert each item into seperate files. I have set a counter in my spider so that it increments on each item yield and added that value to item. Using that counter value I'm creating file names. Test_spider.py class TestSpider(Spider): # spider...
Distinguishing between HTML and non-HTML pages in Scrapy
python,html,web-crawler,scrapy,scrapy-spider
Nevermind, I found the answer. type() only gives information on the immediate type. It tells nothing of inheritance. I was looking for isinstance(). This code works: if isinstance(response, TextResponse): links = response.xpath("//a/@href").extract() ... http://stackoverflow.com/a/2225066/1455074, near the bottom...
Regular expression for Scrapy rules
For the 1-100 range, you can use r"com/vessels\?page=(?:[1-9][0-9]?|100)\b" See demo In case you need any number, just use \d+: r"com/vessels\?page=\d+" See demo 2...
Scrapy crawl and follow links within href
python,web-scraping,scrapy,scrapy-spider
From what I see, I can say that: URLs to product categories always end with .kat URLs to products contain id_ followed by a set of digits Let's use this information to define our spider rules: from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.contrib.linkextractors import LinkExtractor class CodeCheckspider(CrawlSpider): name = "code_check"...
Multiple inheritance in scrapy spiders
python,regex,scrapy,multiple-inheritance,scrapy-spider
You are on the right track, the only thing left is at the end of your parse_product function, you have to yield all the urls extracted by the crawler like so def parse_product(self, response): loader = FlipkartItemLoader(response=response) loader.add_value('pid', 'value of pid') loader.add_xpath('name', 'xpath to name') yield loader.load_item() # CrawlSpider defines...
Scrap a huge site with scrapy never completed
I don't know if CrawlSpider is supposed to be deterministic, but a site with millions of pages has probably not always the same links on a given page and may deliver responses not in the same order. No, I would not expect a CrawlSpider to take the same path. But...
Scrapy simulate XHR request - returning 400
python,ajax,web-scraping,scrapy,scrapy-spider
The key problem is in missing quotes around the filters in the body: url = 'https://www.tele2.no/Services/Webshop/FilterService.svc/ApplyPhoneFilters' req = scrapy.Request(url, method='POST', body='{"filters": []}', headers={'X-Requested-With': 'XMLHttpRequest', 'Content-Type': 'application/json; charset=UTF-8'}, callback=self.parser2) yield req Or, you can define it as a dictionary and then call json.dumps() to dump it to a string: params =...
scrapy itemloaders return list of items
You need to instantiate a new ItemLoader in the loop providing an item argument: l = MytemsLoader() l.add_value('main1', some xpath) l.add_value('main2', some xpath) l.add_value('main3', some xpath) item = l.loaditem() rows = response.xpath("table[@id='BLAH']/tbody[contains(@id, 'BLOB')]") for row in rows: l = MytemsLoader(item=item) l.add_value('table1', some xpath based on rows) l.add_value('table2', some xpath based...
Spider returns only “[” in the items.json file
python,json,web-scraping,scrapy,scrapy-spider
The output file would contain only [ if there was an error while crawling or, there were no items returned. In your case, it is because of the indentation, parse_item() should be indented: class CaptchaSpider(CrawlSpider): name = "CaptchaSpider" allowed_domains = ["*****.ac.in"] start_urls = [ "https://*****.ac.in/*****.asp" ] def parse_item(self, response): item...
xpath preceding child issue
python,xpath,web-scraping,scrapy,scrapy-spider
Let's make it simpler. Each division is represented with div with a mod-teams-list-medium class. Each division div consist of 2 parts: div with class="mod-header" containing the division name div with class="mod-content" containing the list of teams Inside your spider it would be reflected this way: for division in response.xpath('//div[@id="content"]//div[contains(@class, "mod-teams-list-medium")]'):...
How to read xml directly from URLs with scrapy/python
python,xml,web-scraping,scrapy,scrapy-spider
You need to yield Request instances to the other URLs: def check_login_response(self, response): # check login succeed before going on if "incorrect" in response.body: self.log("Login failed", level=scrapy.log.ERROR) return for url in list_or_urls: yield Request(url, callback=self.parse_other_url) def parse_other_url(self, response): # ... ...
Scrapy-Scraper Does Not Run
python-2.7,web-scraping,scrapy,screen-scraping,scrapy-spider
The problem is that you are putting your spider into the items.py. Instead, create a package spiders, inside it create a dmoz.py and put your spider into it. See more at Our first Spider paragraph of the tutorial....
scrapy append to linkextractor links
python,web-scraping,scrapy,scrapy-spider
This is a perfect use case for using process_value argument: from scrapy.contrib.linkextractors import LinkExtractor addition = "?pag_sortorder=0&pag_perPage=999" LinkExtractor(process_value=lambda x: x + addition) ...
Scrapy: catch responses with specific HTTP server codes
python,web-scraping,scrapy,scrapy-spider
By default, Scrapy only handles responses with status codes 200-300. Let Scrapy handle 500 and 502: class Spider(...): handle_httpstatus_list = [500, 502] Then, in the parse() callback, check response.status: def parse(response): if response.status == 500: # logic here elif response.status == 502: # logic here ...
How to crawl classified websites [closed]
web-crawler,scrapy,scrapy-spider
import scrapy from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.selector import Selector from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor from urlparse import urljoin class CompItem(scrapy.Item): name = scrapy.Field() price = scrapy.Field() location = scrapy.Field() class criticspider(CrawlSpider): name = "craig" allowed_domains = ["newyork.craigslist.org"] start_urls = ["http://newyork.craigslist.org/search/cta"] def parse(self, response): sites = response.xpath('//div[@class="content"]') items = []...
Scrapy only crawling given page
python,web-scraping,scrapy,scrapy-spider
You need to make a couple changes to make it work: inherit from CrawlSpider instead of Spider provide a callback different from parse() Here's the code of the spider that follows every link: class LoomSpider(CrawlSpider): name = "loom" allowed_domains = ["2loom.com"] start_urls = [ "http://2loom.com", ] rules = [Rule(SgmlLinkExtractor(), callback='parse_page',...
Pass Selenium HTML string to Scrapy to add URLs to Scrapy list of URLs to scrape
python,selenium,web-scraping,scrapy,scrapy-spider
Just yield Request instances from the method and provide a callback: class AB_Spider(CrawlSpider): ... def parse_abcs(self, response): ... all_URLS = hxs.xpath('a/@href').extract() for url in all_URLS: yield Request(url, callback=self.parse_page) def parse_page(self, response): # Do the parsing here ...
Scrapy: If key exists, why do I get a KeyError?
python,list,key,scrapy,scrapy-spider
Scrapy Item class provides a dictionary-like interface for storing the extracted data. There are no default values set for item fields. To check whether the field was set or not, simply check for the field key in the item instance: if 'link' not in item: print "link has not been...
Using ItemLoader but adding XPath, values etc. in Scrapy
python,xpath,web-scraping,scrapy,scrapy-spider
This is because you are setting a default input and output processors which are applied for all item fields including time which is a float. You have multiple options: instead of default processors, use field-specific processors: l.name_in = MapCompose(lambda v: v.split(), replace_escape_chars) l.name_out = Join() convert/format the time into string:...
Scrapy returning zero results
You need to indent the parse block. import scrapy class StockSpider(scrapy.Spider): name = "ugaz" allowed_domains = ["nasdaq.com"] start_urls = ["http://www.nasdaq.com/symbol/ugaz/"] # Indent this block def parse(self, response): stock = StockItem() stock['quote'] = response.xpath('//*[@id="qwidget_lastsale"]/text()').extract() stock['time'] = response.xpath('//*[@id="qwidget_markettime"]/text()').extract() return stock ...
AttributeError: 'module' object has no attribute 'Spider'
You are using old Scrapy (0.14.4) with the most latest documentation. Solution: upgrade to the latest version of Scrapy or read old docs, that suit currently installed version...
How to use regex in Portia visual scrapy?
python-2.7,web-crawler,scrapy-spider,portia
You need to use capture groups to extract the data so in this case: Location: (.*) This tells portia to extract all data following the Location: string. If for example you only wanted to extract all of the data between Location: and the , you could use the following: Location:...
Python scrapy to extract specific Xpath fields
python,web-scraping,scrapy,scrapy-spider
The problems you have in the code: yield item should be inside the loop since you are instantiating items there the xpath you have is pretty messy and not quite reliable since it heavily relies on the elements location inside parent tags and starts from almost the top parent of...
Scrapy prints fields but doesn't populate XML file
python,xml,xpath,scrapy,scrapy-spider
You need to instantiate a CrawlerItem instance in your parse_node() method: def parse_node(self, response, selector): item = CrawlerItem() item['to'] = selector.xpath('//to/text()').extract() item['who'] = selector.xpath('//from/text()').extract() item['heading'] = selector.xpath('//heading/text()').extract() item['body'] = selector.xpath('//body/text()').extract() return item ...
Simple scrapy XML spider syntax error [closed]
python,xml,scrapy,scrapy-spider
Change word from to something different, in python it is a keyword.
Web scraping error: exceptions.MemoryError
python,web-scraping,scrapy,scrapy-spider
Collect the phone details in the list and join() them after the loop: def parse(self, response): phone = gsmArenaDataItem() details = [] for tableRows in response.css("div#specs-list table"): phone['phoneName'] = tableRows.xpath(".//th/text()").extract()[0] for ttl in tableRows.xpath(".//td[@class='ttl']"): ttl_value = " ".join(ttl.xpath(".//text()").extract()) nfo_value = " ".join(ttl.xpath("following-sibling::td[@class='nfo']//text()").extract()) details.append('{title}: {info}'.format(title=ttl_value.encode("utf-8"),...
Error using scrapy
python,web-scraping,scrapy,scrapy-spider
The XPath configured inside restict_xpaths should point to an element, not an attribute. Replace: //*[@id="content-inhoud"]/div/div/table/tbody/tr/td/h3/a/@href with: //*[@id="content-inhoud"]/div/div/table/tbody/tr/td/h3/a ...
Scrapy: collect retry messages
You can subclass scrapy.contrib.downloadermiddleware.retry.RetryMiddleware and override _retry() to do whatever you want with the request than is given up on. from scrapy.contrib.downloadermiddleware.retry import RetryMiddleware from scrapy import log class CustomRetryMiddleware(RetryMiddleware): def _retry(self, request, reason, spider): retries = request.meta.get('retry_times', 0) + 1 if retries <= self.max_retry_times: log.msg(format="Retrying %(request)s (failed %(retries)d times):...
