Web Scraping U.S. Stock Indices with Selenium

In the previous lesson, we introduced the requests and BeautifulSoup libraries, which allow us to extract data by fetching the HTML code of a specific web page.

However, if a web page is dynamically generated, meaning its content updates based on user interactions, simply using requests and BeautifulSoup is not enough to extract the desired data.

Modern websites often receive constantly changing data from the server and display it dynamically to users. Such web pages are called dynamic web pages.

Since requests and BeautifulSoup cannot handle dynamic data updates, we need an alternative method to extract such content.

In these cases, we use the Selenium library to scrape data from dynamic web pages.

Introduction to the Selenium Library

Selenium is a powerful library used for automating and testing web pages.

Since it can directly control a web browser, it enables tasks such as scraping dynamic data, clicking elements, and entering data into forms.

Practical Example: Scraping U.S. Stock Indices with Selenium

In this example, we will use Selenium to scrape real-time U.S. stock indices.

The following code extracts live U.S. stock indices from Yahoo Finance.

U.S. Stock Indices Scraping Code

# Import required libraries
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait

# Launch the Chrome web driver to open a browser window
driver = webdriver.Chrome()

# Navigate to the 'Markets' page on Yahoo Finance
driver.get('https://finance.yahoo.com/markets/')

# Wait until the page is fully loaded (maximum wait time of 10 seconds)
wait = WebDriverWait(driver, 10)

...(truncated)...

More details about Selenium can be found in Chapter 3 of the course Essential Knowledge for Work Automation, which will expedite your workflow.

For more information about Selenium, refer to Chapter 3 of the course Essential Knowledge for Work Automation!

Press the green ▶︎ Run button in the code editor to check real-time U.S. stock indices!