Skip to main content
Practice

Collecting Web Page Data with the requests Package

One of the most widely used packages for collecting web data in Python is requests.

This package facilitates sending HTTP requests and processing server responses in a convenient way.

In this material, we will introduce the requests package and demonstrate how to collect data using GET and POST requests.


Introduction to the requests Package​

The requests package allows you to send HTTP requests with great simplicity.

It is primarily used to send GET requests to collect data and POST requests to send data.

To use the requests package on your computer, you need to install it using the command pip install requests.


Collecting Data Using a GET Request​

A GET request allows you to retrieve the HTML of a specific web page.

The following code is a simple example of using a GET request to retrieve the HTML of the www.example.com web page.

Getting Web Page HTML with a GET Request
import requests

# URL to request
url = 'https://www.example.com'

# Send GET request
response = requests.get(url)

# Check server response
if response.status_code == 200:
    # Print HTML content
    print(response.text)
else:
    print(f"Failed to retrieve data: {response.status_code}")

In the above code, requests.get() function is used to fetch data from the specified URL.

Then, response.text is used to print the HTML received from the server.


Sending Data Using a POST Request​

A POST request is used to send data to a server.

For example, you can use a POST request to submit login information to a server or upload new data.

Here's a simple example of sending data to a server using a POST request.

Sending Data with a POST Request
import requests

# URL to request
url = 'https://www.example.com/login'

# Data to send
data = {
    'username': 'your_username',
    'password': 'your_password'
}

# Send POST request
response = requests.post(url, data=data)

# Check server response
if response.status_code == 200:
    print("Login successful!")
    print(response.text)
else:
    print(f"Failed to login: {response.status_code}")

In this code, the requests.post() function is used to send login data to the server.

Upon successful transmission, you can check the server's response (e.g., login success message).

Want to learn more?

Join CodeFriends Plus membership or enroll in a course to start your journey.