Iterate over all links/sub-links with Scrapy run from script
python,windows,python-2.7,web-scraping,scrapy
Your spider is being blocked from visiting pages after the start page by your allowed_domains specification. The value should include just the domain, not the protocol. Try allowed_domains = ["www.WebStore.com"] Also the line desc_out = Join() in your WebStoreItemLoader definition may give an error as you have no desc field....
'NoneType' object is not iterable - looping w/returned value
prog is returning None. The error you get is when you try to unpack the result into the tuple (s, best) You need to fix your logic so that prog is guaranteed to not return None. It will return None if your code never executes the else clause in the...
Use NamedTemporaryFile to read from stdout via subprocess on Linux
python-2.7,subprocess,temporary-files
Unless stdout=PIPE; p[0] will always be None in your code. To get output of a command as a string, you could use check_output(): #!/usr/bin/env python from subprocess import check_output result = check_output("date") check_output() uses stdout=PIPE and .communicate() internally. To read output from a file, you should call .read() on the...
Keep strings that occur N times or more
[s for s, c in counts.iteritems() if c >= 2] # => ['a', 'c', 'b'] ...
Syntax Error (FROM) in Python, I do not want to use it as function but rather use it as to print something
print getattr(i, 'from').username ...
Python root logger messages not being logged via handler configured with fileConfig
python,python-2.7,logging,configuration-files
A potential problem is that you are importing the module before you set up the logger configuration. That way, the module requests a logger before the logging is setup. Looking a fileConfig()'s documentation, the reason subsequent logging to the pre-obtained loggers fails is the default value for its disable_existing_loggers argument:...
Replacing string in script from another script
The following replaceString.py should do it: with open('script.py') as myFile: content = myFile.readlines() with open('script.py', 'w') as myFile: for line in content: newLine = line.replace("string1 = 'OriginalValue'", "string1 = 'newValue'") myFile.write(newLine) ...
Operational error while using stripe with profile app
django,python-2.7,django-templates,stripe-payments
It's kind of weird. But when I followed "No Such Table error" and tested DB it was still empty. So this time I deleted DB but also migrations. Problem is fixed and its just working fine.
Open a file from user input in python 2.7
python,python-2.7,user-interface,user-input
The issue is in the lines - selectfile = file(raw_input("Enter Filename: "), 'r') with open(selectfile, 'r') as inF: You need to open the filename (which is inputted from user) directly, as below - with open(raw_input("Enter Filename: "),'r') as inF: Also, there seems to be an indentation issue in your code,...
Identify that a string could be a datetime object
python,regex,algorithm,python-2.7,datetime
What about fuzzyparsers: Sample inputs: jan 12, 2003 jan 5 2004-3-5 +34 -- 34 days in the future (relative to todays date) -4 -- 4 days in the past (relative to todays date) Example usage: >>> from fuzzyparsers import parse_date >>> parse_date('jun 17 2010') # my youngest son's birthday datetime.date(2010,...
What is the difference between <> and == in python?
The operator "<>" means 'not equal to', and the operator "==" means 'equal to'. The former evaluates to true if two things being compared are not equal, and the latter evaluates to true if two things being compared are equal. http://www.tutorialspoint.com/python/comparison_operators_example.htm...
Python do a lookup between 2 dictionaries
Just iterate the keys and values of the first dict and add the values from the second dict corresponding to the same key. mydict = {41100: 'Health Grant', 50050: 'Salaries', 50150: 'Salaries', 50300: 'Salaries'}; mytb = {'': '', 41100: -3450200.40, 50050: 1918593.96, 50150: 97.50, 50300: 8570.80} result = {} for...
Installing Python to a home directory
It is possible to install any Python version assuming your server has a compiler (GCC) installed you can use. Download Python source code here. Apply basic UNIX command knowledge to download and extract Gzip archive to a folder inside your home holder. Follow instructions to compile Python. Please note that...
python class factory inherit random parent
python,python-2.7,inheritance,factory
In my original answer (which I deleted because it was just plain wrong) I said I would consider doing it like this: class RandomPerson(Person): def __init__(self): rand_person = random.choice((John, Kyle))() self.__dict__ = rand_person.__dict__ This way is an adaptation of the Python Borg idiom; the idea was that everything that matters...
Adding time/duration from CSV file
python,python-2.7,csv,datetime
Use datetime.timedelta() objects to model the durations, and pass in the 3 components as seconds, minutes and hours. Parse your file with the csv module; no point in re-inventing the character-separated-values-parsing wheel here. Use a dictionary to track In and Out values per user; using a collections.defaultdict() object will make...
How to skip a function
If I you're asking what I think you're asking, then yes. In fact, that's the whole point. Functions are sections of reusable code that you define first (they don't run when they're defined!) and then you call that function later. For example, you can define a function, helloworld like this:...
delete data from mysql table based on multiple conditions
Delete from table_xyz where right(lower(source),4) not in ('.txt','.xls','xlsx') ...
random.randint not generating random values
You seem to be confused between defining a function and calling it and how parameters work. def blackjack(A,B): print "Welcome to Blackjack!" print "Your cards are",name[A-1],"&",name[B-1] total = value[A-1] + value[B-1] print "Your card total is",total In your function A and B are place holders. They will be replaced by...
How can I resolve my variable's unexpected output?
Remove the comma on your first line of code, this turns it into a tuple optional_message = form.cleaned_data['optional_message'], should be optional_message = form.cleaned_data['optional_message'] ...
Scrapy Memory Error (too many requests) Python 2.7
python,django,python-2.7,memory,scrapy
You can process your urls by batch by only queueing up a few at time every time the spider idles. This avoids having a lot of requests queued up in memory. The example below only reads the next batch of urls from your database/file and queues them as requests only...
Python initialize strings as variable
#!/usr/bin/python m = 2 k = m*2 calc = "((k+m+46)/2)" result = eval(calc) print result Result 26 ...
Sort List of Numbers according to Custom Number Sequence
This matches your input/output examples, but I had to use descending numbers to get the example answers. Are you sure your explanation is correct? If not, just use 0123456789 instead of 9876543210 in the code below. The algorithm is to provide a sorting key based on translating the digits of...
Count function counting only last line of my list
I don't know what you are exactly trying to achieve but if you are trying to count R and K in the string there are more elegant ways to achieve it. But for your reference I had modified your code. N = int(raw_input()) s = [] for i in range(N):...
How to find Unused Security Groups of all AWS Security Groups?
python-2.7,amazon-web-services,amazon-ec2,amazon-s3,boto
This is a slightly difficult request because Security Groups are used by many different resources, including: Amazon EC2 instances Amazon RDS instances VPC Elastic Network Interfaces (ENIs) Amazon Redshift clusters Amazon ElastiCache clusters Amazon Elastic MapReduce clusters Amazon Workspaces ...and most probably other services, too To obtain a list of...
Problems with tk entry and optionmenu widget in currency converter
My answer didn't satisfy me so I solved your problem. Changes: Added self keyword to currAmount1, currAmount2, and reflected this change everywhere in the script. (currAmount1 -> self.currAmount1) Called .get() on the Tkinter variables at the start of curr_search(): cur1 = self.curr_start1.get() cur2 = self.curr_start2.get() amount = self.currAmount1.get() Instead of...
Compare 2 seperate csv files and write difference to a new csv file - Python 2.7
What do you mean by difference? The answer to that gives you two distinct possibilities. If a row is considered same when all columns are same, then you can get your answer via the following code: import csv f1 = open ("olddata/file1.csv") oldFile1 = csv.reader(f1) oldList1 = [] for row...
Using OR operator in NumPy Array to Append multiple items
Instead of doing the "OR" inside the append, you'll need to do an if statement: if category == 'bulky item': items.append((Address, x, y, x, y, ReasonCode, SRNumber, SRNumber, FullName, ResolutionCode, HomePhone, CreatedDate, UpdatedDate, BulkyItemInfo, k_bulky_comm, ServiceDate, CYLA_DISTRICT, SCCatDesc, # ServiceNotes, Prior_Resolution_Code, CYLA_DISTRICT, )) elif category == 'e-waste': items.append((Address, x, y,...
Concatenation of strings array issues in Python: Str(ArrayOfString[1])+“ ”+Str(ArrayOfString[2]) doesn't seem to work
string,python-2.7,concatenation,slice,string-length
Just change your first attempt to following: if len(str(row[1])) >= 16: row[2] = str(row[1])+" "+str(row[2]) row[1] = str(row[1][0:16]) My answer is freely adapted from comment by @phylogenesis. Update: The above answer wont work because row is a tuple and hence it is immutable. You will need to assign the values...
Why am I getting a (NameError: name 'self' is not defined ) here using Kivy for Python?
python,python-2.7,kivy,self,nameerror
You've mixed tabs and spaces: When you do that, Python gets confused about how things are supposed to be indented, because it doesn't interpret tabs the same way your editor does. Don't mix tabs and spaces; stick to one or the other, and turn on "show whitespace" in your editor...
Update label's text when pressing a button in Kivy for Python
python,python-2.7,button,label,kivy
Well! there was a real need for improvement in your code. (I understand it as you are not experienced.) Improvement: 1 An application can be built if you return a widget on build(), or if you set self.root.(You shouldn't make all of the gui in build function itself.) def build(self):...
Identifying the nearest grid point
You can use numpy.searchsorted for this: import numpy as np lat=np.linspace(15,30,61) long=np.linspace(91,102,45) def find_index(x,y): xi=np.searchsorted(lat,x) yi=np.searchsorted(long,y) return xi,yi thisLat, thisLong = find_index(16.3,101.6) print thisLat, thisLong >>> 6, 43 # You can then access the `data` array like so: print data[thisLat,thisLong] NOTE: This will find the index of the lat and...
Instagram API Comments Created using Python (ERROR: AttributeError)
python,api,python-2.7,instagram,instagram-api
Unfortunately, Instagram is not very consistent in their API. The creation time of a media object is called created_time, but for a comment it is created_at. Also note there's no need to request the comments separately: they are already available for each media object, in media.comments....
how to fetch a column in browse_record_list in orm browse method in openERP
You can access all the fields of that table from the browsable object. id = browse_record.id name = browse_record.name Similarly you can access all the relational tables data as well, like customer in sale order. partner_id = sale_order_object.partner_id.id partner_name = sale_order_object.partner_id.name You can also update tables data through that browsable...
from x(defined in program) import y(defined in program) python
python,python-2.7,import,filenames
You can't use a variable for an import statement. from x import y will try to find a module literally called y. In your case, that probably means you want a file in the same directory called y.py. It will then try to find a variable (including, eg, a function...
Why does `for lst in lst:` work? [duplicate]
In Python, when you say for identifier in iterable: first an iterator will be created from the iterable. So, the actual object will not be used in the looping. Then the iterator will be iterated and the current value will be bound to the name identifier. Quoting the official documentation...
Python code not executing in order? MySQLdb UPDATE commits in unexpected order
python,python-2.7,threadpool,mysql-python
A finally clause is guaranteed to execute, even if the try clause raises an exception. What is probably happening here is that an exception is being raised, preventing the update from being committed, but the threads are triggered anyway. This doesn't really seen to be an appropriate use of try/finally....
iterate using yield of a dictionary of dictionaries
You forgot to recursively yield. for k,d in dict.items(directories): for e in walk_directory_files(d): yield e Note that Python 3.3 adds additional syntax specifically for this: for k,d in dict.items(directories): yield from walk_directory_files(d) # Python 3.3+ only! ...
Who calls the metaclass
You can find the answer relatively easily. First, lets find the opcode for building a class. >>> def f(): class A(object): __metaclass__ = type >>> import dis >>> dis.dis(f) 2 0 LOAD_CONST 1 ('A') 3 LOAD_GLOBAL 0 (object) 6 BUILD_TUPLE 1 9 LOAD_CONST 2 (<code object A at 0000000001EBDA30, file...
Slicing a Python OrderedDict
python-2.7,slice,ordereddictionary
The ordered dict in the standard library, doesn't provide that functionality. Even though libraries existed for a few years before collections.OrderedDict that have this functionality (and provide essentially a superset of OrderedDict): voidspace odict and ruamel.ordereddict (I am the author of the latter package, which a reimplementation of odict in...
Scrapy running from python script processes only start url
When you override the default parse_start_url for a crawl spider, the method has to yield Requests for the spider to follow, otherwise it can't go anywhere. You are not required to implement this method when subclassing CrawlSpider, and from the rest of your code, it looks like you really don't...
