I'm trying to develop my coding skills, I was trying to design a sort of custom client for viewing google searches.
I don't want to use the google API, mainly because I feel like this will cut out a lot of the interesting programming.
I tried downloading the webpages, with the intent to parse the HTML. However, when I got the search page downloaded, I couldn't find any useful data within like the URLs that I saw on the original page.
Here is an example of a search for the word "file".
Does anyone know a correct way to download the page's html?
Thanks in advance
Best How To :
What you're trying to do is called web scraping, or trying to pull out content from websites by pretending you loaded a page through a browser, and then accessing the loaded content by looking into and picking out bits and pieces of the page's code. This can work quite well sometimes (there are entire businesses dedicated to web scraping!) and other times not so much - this being one of them.
Instead, try to download a page with the search results already processed by Google (rather than the results being loaded dynamically), such as this search for "test" on Google. Notice the difference in URLs between that search, and this one, which loads the results for "test" dynamically.