Downloading data with Python

Here, we go through some of the ways in which we can download files in Python.

Note: Although we can do a lot with the below mentioned modules, we’ll focus only on downloading files.

Let’s say we want to download an image from http://google.com/favicon.ico

Option 1: (for Python 2)

import urllib2

import urllib2
url = "http://google.com/favicon.ico"
dest = "./tmp/icon.ico"
filedata = urllib2.urlopen(url)  # returns an object

data_to_write = filedata.read()

with open(dest) as f:
	f.write(data_to_write)

Option 2: (for Python 3)

import urllib.request

import urllib.request
url = "http://google.com/favicon.ico"
dest = "./tmp/icon.ico"

urllib.request.urlretreive(url, dest)

Option 3: (for both Python 2 and 3)

from six.moves.urllib.request import urlretrieve

from six.moves.urllib.request import urlretrieve
url = "http://google.com/favicon.ico"
dest = "./tmp/icon.ico"

urllib.request.urlretreive(url, dest)

Note: six.moves module makes some modules available in Python 3 to be accessible in Python 2.
If you want to know more about making your script compatible with both Python 2 and 3, check targeting_python_2_3

Option 4: (external library)

import requests

Since it is a third-party library, you need to install the module.
To install it using pip, use:

pip install requests (for Python 2)
pip3 install requests (for Python 3)s

import requests
url = "http://google.com/favicon.ico"
dest = "./tmp/icon.ico"

resp = requests.get(url)  # returns the response

with open(dest) as f:
	f.write(resp.content)

CONCLUSION:

Personally, I prefer to use requests module as it simple and gives us a lot of features. However, there are cases where you might not be allowed to use third-party library. In that case you can use any of the other three options depending on the environment you are targetting.

To know how to use requests module to download archived files and extract them, check Downloading and extracting archived files