Skip to main content
Practice

Essential HTTP Knowledge for Web Crawling

To properly execute web crawling, it is crucial to understand HTTP, which appears in the web browser's address bar as http://.

HTTP is a protocol (communication rules) responsible for data exchange between the web browser and the server. It operates as an agreement for requesting and responding to data.

In this lesson, we will cover the basic concepts of HTTP, including the notions of request and response.


How Does HTTP Work?​

HTTP operates on a model where the client (web browser) sends a request to the server, and the server sends a response back to the client.

Web crawling primarily involves obtaining web page data through HTTP requests, making it essential to understand how HTTP requests and responses work.


HTTP Requests: How to Request Data​

An HTTP Request is a message sent by a web browser (or a crawling program) asking the server for specific information.

Requests generally consist of the following elements:

  • Method: Defines what action to request from the server. The most commonly used methods are GET and POST.

    • GET: Used to retrieve data from the server (e.g., the HTML of a web page).

    • POST: Used to send data to the server (e.g., sending login information).

GET Request Example
GET /index.html HTTP/1.1
Host: www.example.com

  • URL: Indicates the location of the resource being requested. For example, https://www.example.com/index.html is a URL.

  • Header: Contains additional information and allows more detailed control of the request. For instance, the User-Agent header provides information about the client sending the request.


HTTP Responses: Answers to Your Requests​

An HTTP Response is a message sent by the server in reply to a client's request.

Responses typically consist of the following elements:

  • Status Code: Indicates whether the request was successfully processed or if an error occurred. For instance, 200 OK means the request was successful, while 404 Not Found indicates the requested resource could not be found.
HTTP Response Example
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 342

<html>
  <body>
    <h1>Example Page</h1>
  </body>
</html>

  • Header: Provides additional information about the response. For example, the Content-Type header indicates the data format of the response.

  • Body: Contains the actual data of the requested resource, such as the HTML of a web page, images, or JSON data.


In web crawling, GET requests are commonly used to fetch the HTML data of web pages.

Following this, you'll need to check the status code and body of the response returned by the server to determine if the crawling was successful.

Want to learn more?

Join CodeFriends Plus membership or enroll in a course to start your journey.