Rename output file after scrapy spider complete
python,scrapy,scrapy-spider,scrapyd
I got a working solution after trying different approaches. Since in my particular case I dump the output into files, specifically bz2 files. I customized a FileFeedStorage to do the job before opening and after closing the file. See code below: from scrapy.contrib.feedexport import FileFeedStorage import os import bz2 MB...
Scrapy - scraped website authentication token expires while scraping
python,authentication,scrapy,scrapy-spider
I was able to figure this out. I had to override the Request object in order to set a new authorization token into the header when the token expires. I made the the token a global variable. # override Request object in order to set new authorization token into the...
delete spiders from scrapinghub
delete,web-crawler,scrapy,scrapy-spider,scrapinghub
You just need to remove the spider from your project, and deploy the project again, via shub deploy, or scrapyd-deploy.
Spider returns only “[” in the items.json file
python,json,web-scraping,scrapy,scrapy-spider
The output file would contain only [ if there was an error while crawling or, there were no items returned. In your case, it is because of the indentation, parse_item() should be indented: class CaptchaSpider(CrawlSpider): name = "CaptchaSpider" allowed_domains = ["*****.ac.in"] start_urls = [ "https://*****.ac.in/*****.asp" ] def parse_item(self, response): item...
Remove first tag html using python & scrapy
python,xpath,scrapy,scrapy-spider
You can get all the child tags except the div with class="needremove": response.xpath('//div[contains(@class, "abc")]/div[contains(@class, "xyz")]/*[local-name() != "div" and not(contains(@class, "needremove"))]').extract() Demo from the shell: $ scrapy shell index.html In [1]: response.xpath('//div[contains(@class, "abc")]/div[contains(@class, "xyz")]/*[local-name() != "div" and not(contains(@class, "needremove"))]').extract() Out[1]: [u'<p>text</p>', u'<p>text</p>', u'<p>text</p>',...
scrapy from script output in json
python,json,web-scraping,scrapy,scrapy-spider
You need to set FEED_FORMAT and FEED_URI settings manually: settings.overrides['FEED_FORMAT'] = 'json' settings.overrides['FEED_URI'] = 'result.json' If you want to get the results into a variable you can define a Pipeline class that would collect items into the list. Use the spider_closed signal handler to see the results: import json from...
scrapy append to linkextractor links
python,web-scraping,scrapy,scrapy-spider
This is a perfect use case for using process_value argument: from scrapy.contrib.linkextractors import LinkExtractor addition = "?pag_sortorder=0&pag_perPage=999" LinkExtractor(process_value=lambda x: x + addition) ...
Pass argument to scrapy spider within a python script
python,python-2.7,web-scraping,scrapy,scrapy-spider
You need to modify your __init__() constructor to accept the date argument. Also, I would use datetime.strptime() to parse the date string: from datetime import datetime class MySpider(CrawlSpider): name = 'tw' allowed_domains = ['test.com'] def __init__(self, *args, **kwargs): super(MySpider, self).__init__(*args, **kwargs) date = kwargs.get('date') if not date: raise ValueError('No date...
Scrapy only crawling given page
python,web-scraping,scrapy,scrapy-spider
You need to make a couple changes to make it work: inherit from CrawlSpider instead of Spider provide a callback different from parse() Here's the code of the spider that follows every link: class LoomSpider(CrawlSpider): name = "loom" allowed_domains = ["2loom.com"] start_urls = [ "http://2loom.com", ] rules = [Rule(SgmlLinkExtractor(), callback='parse_page',...
Scrapy extracting from Link
For rules to work, you need to use CrawlSpider not the general scrapy Spider. Also, you need to rename your first parsing function to a name other than parse. Otherwise, you will be overwriting an important method of the CrawlSpider and it will not work. See the warning in the...
Scrapy - generating items outside of parse callback
I'm not sure if there is a way to push items directly to the engine, but what you could do is push dummy requests with the items in the meta variable and just yield them in the callback. def _file_thread_loop(self): while True: #... read files... for file in files: response...
Scrapy creating XML feed wraps content in “value” tags
python,xml,scrapy,scrapy-spider
The extract() method will always return a list of values, even if there is only a single value as a result, for example: [4], [3,4,5] or None. To avoid this, if you know there is only one value, you can select it like: item['to'] = selector.xpath('//to/text()').extract()[0] Note: Be aware that...
