Scrapy simulate XHR request - returning 400
python,ajax,web-scraping,scrapy,scrapy-spider
The key problem is in missing quotes around the filters in the body: url = 'https://www.tele2.no/Services/Webshop/FilterService.svc/ApplyPhoneFilters' req = scrapy.Request(url, method='POST', body='{"filters": []}', headers={'X-Requested-With': 'XMLHttpRequest', 'Content-Type': 'application/json; charset=UTF-8'}, callback=self.parser2) yield req Or, you can define it as a dictionary and then call json.dumps() to dump it to a string: params =...
Pass argument to scrapy spider within a python script
python,python-2.7,web-scraping,scrapy,scrapy-spider
You need to modify your __init__() constructor to accept the date argument. Also, I would use datetime.strptime() to parse the date string: from datetime import datetime class MySpider(CrawlSpider): name = 'tw' allowed_domains = ['test.com'] def __init__(self, *args, **kwargs): super(MySpider, self).__init__(*args, **kwargs) date = kwargs.get('date') if not date: raise ValueError('No date...
How to use regex in Portia visual scrapy?
python-2.7,web-crawler,scrapy-spider,portia
You need to use capture groups to extract the data so in this case: Location: (.*) This tells portia to extract all data following the Location: string. If for example you only wanted to extract all of the data between Location: and the , you could use the following: Location:...
Recursive crawling over a page
python,web-scraping,scrapy,scrapy-spider
You need to pass the item instantiated in parse() inside the meta to the secondPage callback: def parse(self, response): for sel in response.xpath('//*[@id="conteudoInternas"]/ul/li'): item = CinemasItem() item['title'] = 'title' item['room'] = 'room' item['mclass'] = 'mclass' item['minAge'] = 'minAge' item['cover'] = 'cover' item['sessions'] = 'sessions' secondUrl = sel.xpath('p[1]/a/@href').extract()[0] # see: we...
scrapy LxmlLinkExtractor and relative urls
The problem is that the HTML has an incorrect HTML base element, which is supposed to specify the base url for all the relative links in the page: <base href="http://www.lecture-en-ligne.com/"/> Scrapy is respecting that, that's why the links are being formed that way....
Web scraping error: exceptions.MemoryError
python,web-scraping,scrapy,scrapy-spider
Collect the phone details in the list and join() them after the loop: def parse(self, response): phone = gsmArenaDataItem() details = [] for tableRows in response.css("div#specs-list table"): phone['phoneName'] = tableRows.xpath(".//th/text()").extract()[0] for ttl in tableRows.xpath(".//td[@class='ttl']"): ttl_value = " ".join(ttl.xpath(".//text()").extract()) nfo_value = " ".join(ttl.xpath("following-sibling::td[@class='nfo']//text()").extract()) details.append('{title}: {info}'.format(title=ttl_value.encode("utf-8"),...
How to crawl classified websites [closed]
web-crawler,scrapy,scrapy-spider
import scrapy from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.selector import Selector from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor from urlparse import urljoin class CompItem(scrapy.Item): name = scrapy.Field() price = scrapy.Field() location = scrapy.Field() class criticspider(CrawlSpider): name = "craig" allowed_domains = ["newyork.craigslist.org"] start_urls = ["http://newyork.craigslist.org/search/cta"] def parse(self, response): sites = response.xpath('//div[@class="content"]') items = []...
Remove first tag html using python & scrapy
python,xpath,scrapy,scrapy-spider
You can get all the child tags except the div with class="needremove": response.xpath('//div[contains(@class, "abc")]/div[contains(@class, "xyz")]/*[local-name() != "div" and not(contains(@class, "needremove"))]').extract() Demo from the shell: $ scrapy shell index.html In [1]: response.xpath('//div[contains(@class, "abc")]/div[contains(@class, "xyz")]/*[local-name() != "div" and not(contains(@class, "needremove"))]').extract() Out[1]: [u'<p>text</p>', u'<p>text</p>', u'<p>text</p>',...
Scrapy keep all unique pages based on a list of start urls
python,web-scraping,scrapy,scrapy-spider
Set the default parse callback to spin off all the links. By default Scrapy does not visit the same page twice. def parse(self, response): links = LinkExtractor().extract_links(response) return (Request(url=link.url, callback=self.parse_page) for link in links) def parse_page(self, response): # name = manipulate response.url to be a unique file name with open(name,...
Crawl spider not crawling ~ Rule Issue
python,web-scraping,scrapy,scrapy-spider
I don't think that you need two rules, you can declare one and do it to follow links and parse each page. In the rule I restrict the xpath to the last link of the list, because otherwise you could be parsing some links multiple times. I use parse_start_url as...
Scrapy python error - Missing scheme in request URL
python,web-crawler,scrapy,scrapy-spider
You need to add scheme for the URL: ftp://ftp.site.co.uk The FTP URL syntax is defined as: ftp://[<user>[:<password>]@]<host>[:<port>]/<url-path> Basically, you do this: yield Request('ftp://ftp.site.co.uk/feed.xml', ...) Read more about schemas at Wikipedia: http://en.wikipedia.org/wiki/URI_scheme...
Can I specify any method as the callback when constructing a Scrapy Request object?
python,callback,web-scraping,scrapy,scrapy-spider
Theoretically, it is doable since callback is just a callable that has a response as it's argument. Though, Items are just containers of the fields, they are for storing data, you should not put logic there. Better create a method in the spider and pass the item instance inside meta:...
Scrapy extracting from Link
For rules to work, you need to use CrawlSpider not the general scrapy Spider. Also, you need to rename your first parsing function to a name other than parse. Otherwise, you will be overwriting an important method of the CrawlSpider and it will not work. See the warning in the...
Parsing the urls in sitemap with different url format using sitemap spider in scrapy, python
python,web-scraping,scrapy,sitemap,scrapy-spider
I think the nicest and cleanest solution would be to add a downloader middleware which changes the malicious URLs without the spider noticing. import re import urlparse from scrapy.http import XmlResponse from scrapy.utils.gz import gunzip, is_gzipped from scrapy.contrib.spiders import SitemapSpider # downloader middleware class SitemapWithoutSchemeMiddleware(object): def process_response(self, request, response, spider):...
Distinguishing between HTML and non-HTML pages in Scrapy
python,html,web-crawler,scrapy,scrapy-spider
Nevermind, I found the answer. type() only gives information on the immediate type. It tells nothing of inheritance. I was looking for isinstance(). This code works: if isinstance(response, TextResponse): links = response.xpath("//a/@href").extract() ... http://stackoverflow.com/a/2225066/1455074, near the bottom...
Scrapy, only follow internal URLS but extract all links found
python,scrapy,web-crawler,scrape,scrapy-spider
You can also use the link extractor to pull all the links once you are parsing each page. The link extractor will filter the links for you. In this example the link extractor will deny links in the allowed domain so it only gets outside links. from scrapy.contrib.spiders import CrawlSpider,...
Scrapy not giving individual results of all the reviews of a phone?
python,xpath,web-scraping,scrapy,scrapy-spider
First of all, your XPath expressions are very fragile in general. The main problem with your approach is that site does not contain a review section, but it should. In other words, you are not iterating over review blocks on a page. Also, the model name should be extracted outside...
Rename output file after scrapy spider complete
python,scrapy,scrapy-spider,scrapyd
I got a working solution after trying different approaches. Since in my particular case I dump the output into files, specifically bz2 files. I customized a FileFeedStorage to do the job before opening and after closing the file. See code below: from scrapy.contrib.feedexport import FileFeedStorage import os import bz2 MB...
scrapy crawling at depth not working
There is a Warning box in the CrawlSpider documentation. It says: When writing crawl spider rules, avoid using parse as callback, since the CrawlSpider uses the parse method itself to implement its logic. So if you override the parse method, the crawl spider will no longer work. Your code does...
Scrapy command in shell script not executing when called from java
java,bash,shell,scrapy,scrapy-spider
Your are missing forward slash on your 'PATH' declaration, and path should be to the directory, not the program You have export PATH=usr/local/bin/scrapy:$PATH Should be export PATH=/usr/local/bin:$PATH ...
Scrapy: If key exists, why do I get a KeyError?
python,list,key,scrapy,scrapy-spider
Scrapy Item class provides a dictionary-like interface for storing the extracted data. There are no default values set for item fields. To check whether the field was set or not, simply check for the field key in the item instance: if 'link' not in item: print "link has not been...
Scrapy only crawling given page
python,web-scraping,scrapy,scrapy-spider
You need to make a couple changes to make it work: inherit from CrawlSpider instead of Spider provide a callback different from parse() Here's the code of the spider that follows every link: class LoomSpider(CrawlSpider): name = "loom" allowed_domains = ["2loom.com"] start_urls = [ "http://2loom.com", ] rules = [Rule(SgmlLinkExtractor(), callback='parse_page',...
Scrapy creating XML feed wraps content in “value” tags
python,xml,scrapy,scrapy-spider
The extract() method will always return a list of values, even if there is only a single value as a result, for example: [4], [3,4,5] or None. To avoid this, if you know there is only one value, you can select it like: item['to'] = selector.xpath('//to/text()').extract()[0] Note: Be aware that...
How to exclude a particular html tag(without any id) from several tags while using scrapy?
python,html,web-scraping,scrapy,scrapy-spider
I would use starts-with() XPath function to get the div element's text that starts with "Timings": sel.xpath('.//div[starts-with(., "Timings")]/text()').extract() Note that the HTML structure of the page doesn't make it easy to distinguish locations between each other - there is no location-specific containers that you can iterate over. In this case,...
Scrapy returning zero results
You need to indent the parse block. import scrapy class StockSpider(scrapy.Spider): name = "ugaz" allowed_domains = ["nasdaq.com"] start_urls = ["http://www.nasdaq.com/symbol/ugaz/"] # Indent this block def parse(self, response): stock = StockItem() stock['quote'] = response.xpath('//*[@id="qwidget_lastsale"]/text()').extract() stock['time'] = response.xpath('//*[@id="qwidget_markettime"]/text()').extract() return stock ...
Error in scrapy screen scraper - Cant find what's wrong for the life of me
python,python-2.7,scrapy,screen-scraping,scrapy-spider
Please, show position of items.py in project structure. You should have smth like this: craig (folder) craig.py items (folder) __ init __.py items.py ...
Cannot download image with relative URL Python Scrapy
python,web-crawler,scrapy,scrapy-spider
Wrap your image url in a list like so: item['image_urls'] = [self.page_name + imageStr[3:-2]] ...
