How to handle when webpage does not load on BeautifulSoup?

Yes and no, that is what I WANT, but soup is never None, even when fed a bad address... – Brad Nov 3 at 18:36 When you just return from getwebpage, it is None. – Chris Morgan Nov 3 at 21:16.

Structure the code so that one function encapsulates the entire process of data retrieval from the url, and another that encapsulates the processing of that data: import urllib2, httplib from BeautifulSoup import BeautifulSoup def append_log(message): print message def get_web_page(address): try: user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)' headers = { 'User-Agent' : user_agent } request = urllib2. Request(address, None, headers) response = urllib2. Urlopen(request, timeout=20) try: return response.read() finally: response.close() except urllib2.

HTTPError as e: error_desc = httplib.responses. Get(e. Code, '') append_log('HTTP Error: ' + str(e.

Code) + ': ' + error_desc + ': ' + address) except urllib2. URLError as e: append_log('URL Error: ' + e. Reason1 + ': ' + address) except Exception as e: append_log('Unknown Error: ' + str(e) + address) def process_web_page(data): if data is not None: print BeautifulSoup(data) else: pass # do something else data = get_web_page('http://doesnotexistblah.com/') process_web_page(data) data = get_web_page('http://docs.python.org/copyright.html') process_web_page(data).

Structure the code so that one function encapsulates the entire process of data retrieval from the url, and another that encapsulates the processing of that data.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.