How to Scrape Google Finance with Python and ProxyTee

Data is a valuable asset, especially when it comes to financial insights. Aggregated data, like the kind found on Google Finance, is widely sought after for various applications, from building trading algorithms to generating market reports. However, scraping such data can pose challenges, as Google employs anti-scraping measures to detect and block bots.

In this blog, we’ll walk you through how to extract this data using Python and, more importantly, how ProxyTee can significantly enhance your web scraping efforts by providing rotating residential proxies.

Why Use ProxyTee for Web Scraping?

ProxyTee is a provider of highly efficient rotating residential proxies designed to support a wide range of internet activities, including large-scale web scraping. Unlike traditional datacenter proxies, residential proxies mimic real users, making them much harder to detect and block. This makes ProxyTee an ideal companion for scraping Google Finance efficiently and securely.

Key Features of ProxyTee for Web Scraping

• Unlimited bandwidth: Perform large-scale scraping operations without worrying about exceeding data limits or incurring extra costs.
• Global IP Coverage: Gain access to a vast network of over 20 million IP addresses spread across more than 100 countries. This allows for localized data extraction from different market regions.
• Auto-Rotation: IP addresses automatically rotate at intervals between 3 to 60 minutes, helping avoid detection and IP bans from Google Finance.
• Multiple Protocol Support: Compatible with both HTTP and SOCKS5 protocols, making it easy to integrate into various web scraping tools and frameworks.
• Simple API for Automation: Easily integrate ProxyTee’s API into your web scraping projects to streamline and automate the process.

Scraping Google Finance

Prerequisites

To get started with scraping Google Finance, you’ll need:

• Python: A basic understanding of Python, including variables, functions, and loops.
• Python Requests: Used to make HTTP requests to fetch data.
• BeautifulSoup: A Python library that allows parsing HTML content.

You can install the necessary libraries using these commands:

pip install requests
pip install beautifulsoup4

What to Scrape from Google Finance

Google Finance provides data across different markets, including:

• Gainers: Stocks with the highest percentage price increase.
• Losers: Stocks with the highest percentage price drop.
• Market Indexes: Data on different stock indices.
• Most Active Stocks: The most traded stocks based on volume.
• Cryptocurrencies: Market data on popular cryptocurrencies.

Each of these categories is listed within an unordered list (ul) on Google Finance pages, making it relatively simple to extract data using BeautifulSoup.

Scrape Google Finance Manually with Python

Below are two functions that will help extract data from Google Finance and save it as a CSV file.

import requests
from bs4 import BeautifulSoup
import csv
from pathlib import Path

def write_to_csv(data, filename):
    if type(data) != list:
        data = [data]
    filename = f"google-finance-{filename}.csv"
    mode = "w" if not Path(filename).exists() else "a"
    
    with open(filename, mode, newline='', encoding='utf-8') as file:
        writer = csv.DictWriter(file, fieldnames=data[0].keys())
        if mode == "w":
            writer.writeheader()
        writer.writerows(data)
    
    print(f"Successfully wrote {filename} to CSV.")

def scrape_page(endpoint: str):
    url = f"https://www.google.com/finance/markets/{endpoint}"
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, "html.parser")
    tables = soup.find_all("ul")
    scraped_data = []
    
    for table in tables:
        list_elements = table.find_all("li")
        for list_element in list_elements:
            divs = list_element.find_all("div")
            if len(divs) < 12:
                continue
            asset = {
                "ticker": divs[3].text,
                "name": divs[6].text,
                "currency": divs[8].text[0] if endpoint != "cryptocurrencies" else "n/a",
                "price": divs[8].text,
                "change": divs[11].text
            }
            scraped_data.append(asset)
    write_to_csv(scraped_data, endpoint)

if __name__ == "__main__":
    endpoints = ["gainers", "losers", "indexes", "most-active", "cryptocurrencies"]
    for endpoint in endpoints:
        print(f"Scraping {endpoint}...")
        scrape_page(endpoint)

When executed, this script will create CSV files containing the scraped data. However, you may run into rate limits and blocks as Google has mechanisms to detect bots. This is where ProxyTee becomes useful. You should use ProxyTee‘s rotating residential IPs to prevent detection when scraping multiple pages.

Advanced Techniques

Handling Pagination

The script handles pagination by iterating over the `endpoints` array, but for sites using numbers or other parameters, adjustments would be necessary.

Mitigate Blocking

To mitigate blocking, it’s essential to use residential proxies like ProxyTee’s Residential Proxies, as they are harder to detect. Additional techniques to consider are:

• Fake User Agents: Set a custom user agent in your requests.
• Timed Requests: Add delays between requests to avoid overloading servers.

Web Scraping with ProxyTee

Web scraping can be challenging, especially at scale. While it’s possible to create a scraper with Python and BeautifulSoup, dealing with anti-scraping measures can be complex and time consuming. By incorporating a tool like ProxyTee and taking into consideration things like fake user agents and timed requests, the overall complexity can be simplified.

ProxyTee can make your data scraping experience much more reliable, and affordable. The Unlimited Residential Proxies plan, with its affordable and rotating IPs will help ensure a smooth, interruption-free scraping experience, particularly important for large scale operations that would otherwise lead to blocks. Start your web scraping tasks the right way with the powerful tools offered by ProxyTee.