How to Scrape Google Trends Data with ProxyTee

Google Trends is a powerful platform that offers public data on the top Google Search queries. This tool allows you to analyze the search interest for specific keywords over particular time frames and regions. Scraping Google Trends data is an effective method to track keyword popularity over time. For efficiently gathering large amounts of data while avoiding blocks, utilizing rotating residential proxies from ProxyTee is essential.

Understanding what users are searching for is crucial for various businesses. However, manually collecting extensive Google Trends data is challenging. This article provides a step-by-step guide on how to extract Google Trends data using Python, enhanced by ProxyTee's reliable proxy solutions.

Why Scrape Google Trends Data?

Here are a few use cases for scraped Google Trends data:

Keyword Research

SEO specialists and content marketers widely use Google Trends. By providing insights into the past and current popularity of search terms, these professionals can adjust their strategies to drive more website traffic. With web scraping and the aid of ProxyTee, marketers can gather search volume data for trending topics.

Market Research

Google Trends data is invaluable for market research, helping businesses to grasp consumer interests and preferences over time. For example, e-commerce businesses can use these insights for product development and analyze geographic trends. ProxyTee's global IP coverage ensures that businesses can gather localized insights from diverse markets, crucial for pinpointing regional trends.

Societal Research

Journalists and researchers can utilize Google Trends to gain insights into societal trends and public interest in trending topics. By scraping with the help of ProxyTee, they can monitor popular topics over time and evaluate public sentiment effectively. ProxyTee's Residential Proxies provide a clean and stable connection, ensuring consistent data capture.

These are only some examples; Google Trends data can assist with investment decisions, brand reputation tracking, and other business cases.

How to Scrape Google Trends with Python Using ProxyTee

Below is how to collect Google Trends data using Python with the support of ProxyTee. Using ProxyTee will ensure smooth scraping while avoiding any blocks from Google.

1. Install Libraries

Ensure that you have the necessary libraries. Open your terminal and run:

pip install requests pandas

Then import these in your Python file:

import os
import json
from typing import List
import requests
import pandas as pd

2. Configure Your Request

To access Google Trends data, use ProxyTee’s API. This requires a username and password. Here is the basic structure:

USERNAME = "YourUsername"
password = "YourPassword"
query = "persian cat"

print(f"Getting data from Google Trends for {query} keyword..")

url = "https://api.proxytee.com/v1/queries" #This is sample URL. Please refer to official document for actual one

auth = (USERNAME, password)

payload = {
    "source": "google_trends_explore",
    "query": query,
}

Variables `USERNAME` and `password` require authentication by ProxyTee's API, `payload` configures the API for processing your request. `source` specifies the type of scraper to use. For this, use `google_trends_explore` which is ideal for this scenario. Also, provide the search query. Consult our documentation for additional parameters and rate limits of API requests.

Once you configured your parameters, form and send the request like so:

try:
    response = requests.request("POST", url, auth=auth, json=payload)
except requests.exceptions.RequestException as e:
    print("Caught exception while getting trend data")
    raise e

data = response.json()
content = data["results"][0]["content"]

print(content)

This code will return raw result of your query.

3. Save Results to CSV

Now, adjust the format and save data as a CSV file with `pandas`. This makes it easier to analyze and organize the data. The response contains four main categories: `interest_over_time`, `breakdown_by_region`, `related_topics`, and `related_queries`. Let's put each into a separate CSV file by first turning them into dataframes:

def flatten_topic_data(topics_data: List[dict]) -> List[dict]:
    """Flattens related_topic data"""
    topics_items = []
    for item in topics_data[0]["items"]:
        item_dict = {
            "mid": item["topic"]["mid"],
            "title": item["topic"]["title"],
            "type": item["topic"]["type"],
            "value": item["value"],
            "formatted_value": item["formatted_value"],
            "link": item["link"],
            "keyword": topics_data[0]["keyword"],
        }
        topics_items.append(item_dict)
    return topics_items

trend_data = json.loads(content)

print("Creating dataframes..")

# Interest over time
iot_df = pd.DataFrame(trend_data["interest_over_time"][0]["items"])
iot_df["keyword"] = trend_data["interest_over_time"][0]["keyword"]

# Breakdown by region
bbr_df = pd.DataFrame(trend_data["breakdown_by_region"][0]["items"])
bbr_df["keyword"] = trend_data["breakdown_by_region"][0]["keyword"]

# Related topics
rt_data = flatten_topic_data(trend_data["related_topics"])
rt_df = pd.DataFrame(rt_data)

# Related queries
rq_df = pd.DataFrame(trend_data["related_queries"][0]["items"])
rq_df["keyword"] = trend_data["related_queries"][0]["keyword"]

Because the structure of `related_topics` is multilevel, it is important to first flatten it. After preparing all dataframes, save to files:

CSV_FILE_DIR = "./csv/"
keyword = trend_data["interest_over_time"][0]["keyword"]
keyword_path = os.path.join(CSV_FILE_DIR, keyword)

try:
    os.makedirs(keyword_path, exist_ok=True)
except OSError as e:
    print("Caught exception while creating directories.")
    raise e

print("Dumping to CSV...")

iot_df.to_csv(f"{keyword_path}/interest_over_time.csv", index=False)
bbr_df.to_csv(f"{keyword_path}/breakdown_by_region.csv", index=False)
rt_df.to_csv(f"{keyword_path}/related_topics.csv", index=False)
rq_df.to_csv(f"{keyword_path}/related_queries.csv", index=False)

This will produce separate CSV files categorized by keyword within folders.

4. Comparing the results

The result from request is transformed into dataframes, now we can create a comparison. This can help when you want to analysis on various keyword or regions. Adjust the existing code for multiple keyword and to put information all in one place. Split the code into a reusable functions for it:

First, to request data via API, with input as query:

def get_trend_data(query: str) -> dict:
    """Gets a dictionary of trends based on given query string from Google Trends via Web Scraper API"""
    print(f"Getting data from Google Trends for {query} keyword..")
    url = "https://api.proxytee.com/v1/queries" #This is sample URL. Please refer to official document for actual one
    auth = (USERNAME, password)
    payload = {
        "source": "google_trends_explore",
        "query": query,
    }
    try:
        response = requests.request("POST", url, auth=auth, json=payload)
    except requests.exceptions.RequestException as e:
        print("Caught exception while getting trend data")
        raise e
    data = response.json()
    content = data["results"][0]["content"]
    return json.loads(content)

Second, transform the response to a dataframe, store in a CSV file and then return it:

def dump_trend_data_to_csv(trend_data: dict) -> dict:
    """Dumps given trend data to generated CSV file"""
    CSV_FILE_DIR = "./csv/"
    # Interest over time
    print("Creating dataframes..")
    iot_df = pd.DataFrame(trend_data["interest_over_time"][0]["items"])
    iot_df["keyword"] = trend_data["interest_over_time"][0]["keyword"]

    # Breakdown by region
    bbr_df = pd.DataFrame(trend_data["breakdown_by_region"][0]["items"])
    bbr_df["keyword"] = trend_data["breakdown_by_region"][0]["keyword"]

    # Related topics
    rt_data = flatten_topic_data(trend_data["related_topics"])
    rt_df = pd.DataFrame(rt_data)

    # Related queries
    rq_df = pd.DataFrame(trend_data["related_queries"][0]["items"])
    rq_df["keyword"] = trend_data["related_queries"][0]["keyword"]

    keyword = trend_data["interest_over_time"][0]["keyword"]
    keyword_path = os.path.join(CSV_FILE_DIR, keyword)
    try:
        os.makedirs(keyword_path, exist_ok=True)
    except OSError as e:
        print("Caught exception while creating directories")
        raise e

    print("Dumping to csv..")
    iot_df.to_csv(f"{keyword_path}/interest_over_time.csv", index=False)
    bbr_df.to_csv(f"{keyword_path}/breakdown_by_region.csv", index=False)
    rt_df.to_csv(f"{keyword_path}/related_topics.csv", index=False)
    rq_df.to_csv(f"{keyword_path}/related_queries.csv", index=False)

    result_set = {}
    result_set["iot"] = iot_df
    result_set["bbr"] = bbr_df
    result_set["rt"] = rt_df
    result_set["rq"] = rq_df
    return result_set

Next, here is function to create comparison based on result of prior function:

def create_comparison(trend_dataframes: dict) -> None:
    comparison = trend_dataframes[0]
    i = 1
    for df in trend_dataframes[1:]:
        comparison["iot"] = pd.merge(comparison["iot"], df["iot"], on="time", suffixes=("", f"_{i}"))
        comparison["bbr"] = pd.merge(comparison["bbr"], df["bbr"], on="geo_code", suffixes=("", f"_{i}"))

        if not df["rt"].empty and "title" in df["rt"].columns:
            comparison["rt"] = pd.merge(comparison["rt"], df["rt"], on="title", how="inner", suffixes=("", f"_{i}"))

        if not df["rq"].empty and "query" in df["rq"].columns:
            comparison["rq"] = pd.merge(comparison["rq"], df["rq"], on="query", how="inner", suffixes=("", f"_{i}"))

        i = i + 1

    comparison["iot"].to_csv("comparison_interest_over_time.csv", index=False)
    comparison["bbr"].to_csv("comparison_breakdown_by_region.csv", index=False)
    comparison["rt"].to_csv("comparison_related_topics.csv", index=False)
    comparison["rq"].to_csv("comparison_related_queries.csv", index=False)

Now we will have comparison result stored in csv files for easier use and analyze.

Here is the whole logic put together:

import os
import json
from typing import List
import requests
import pandas as pd

def get_trend_data(query: str) -> dict:
    USERNAME = "YourUsername"
    password = "YourPassword"
    print(f"Getting data from Google Trends for {query} keyword...")
    url = "https://api.proxytee.com/v1/queries" #This is sample URL. Please refer to official document for actual one
    auth = (USERNAME, password)
    payload = {
        "source": "google_trends_explore",
        "query": query
    }
    try:
        response = requests.request("POST", url, auth=auth, json=payload)
    except requests.exceptions.RequestException as e:
        print("Caught exception while getting trend data")
        raise e
    data = response.json()
    content = data["results"][0]["content"]
    return json.loads(content)

def flatten_topic_data(topics_data: List[dict]) -> List[dict]:
    topics_items = []
    for item in topics_data[0]["items"]:
        item_dict = {
            "mid": item["topic"]["mid"],
            "title": item["topic"]["title"],
            "type": item["topic"]["type"],
            "value": item["value"],
            "formatted_value": item["formatted_value"],
            "link": item["link"],
            "keyword": topics_data[0]["keyword"]
        }
        topics_items.append(item_dict)
    return topics_items

def dump_trend_data_to_csv(trend_data: dict) -> dict:
    print("Creating dataframes...")
    iot_df = pd.DataFrame(trend_data["interest_over_time"][0]["items"])
    iot_df["keyword"] = trend_data["interest_over_time"][0]["keyword"]
    bbr_df = pd.DataFrame(trend_data["breakdown_by_region"][0]["items"])
    bbr_df["keyword"] = trend_data["breakdown_by_region"][0]["keyword"]
    rt_data = flatten_topic_data(trend_data["related_topics"])
    rt_df = pd.DataFrame(rt_data)
    rq_df = pd.DataFrame(trend_data["related_queries"][0]["items"])
    rq_df["keyword"] = trend_data["related_queries"][0]["keyword"]
    CSV_FILE_DIR = "./csv/"
    keyword = trend_data["interest_over_time"][0]["keyword"]
    keyword_path = os.path.join(CSV_FILE_DIR, keyword)
    try:
        os.makedirs(keyword_path, exist_ok=True)
    except OSError as e:
        print("Caught exception while creating directories.")
        raise e
    print("Dumping to CSV...")
    iot_df.to_csv(f"{keyword_path}/interest_over_time.csv", index=False)
    bbr_df.to_csv(f"{keyword_path}/breakdown_by_region.csv", index=False)
    rt_df.to_csv(f"{keyword_path}/related_topics.csv", index=False)
    rq_df.to_csv(f"{keyword_path}/related_queries.csv", index=False)
    result_set = {}
    result_set["iot"] = iot_df
    result_set["bbr"] = bbr_df
    result_set["rt"] = rt_df
    result_set["rq"] = rq_df
    return result_set

def create_comparison(trend_dataframes: dict) -> None:
    comparison = trend_dataframes[0]
    i = 1
    for df in trend_dataframes[1:]:
        comparison["iot"] = pd.merge(comparison["iot"], df["iot"], on="time", suffixes=("", f"_{i}"))
        comparison["bbr"] = pd.merge(comparison["bbr"], df["bbr"], on="geo_code", suffixes=("", f"_{i}"))
        if not df["rt"].empty and "title" in df["rt"].columns:
            comparison["rt"] = pd.merge(comparison["rt"], df["rt"], on="title", how="inner", suffixes=("", f"_{i}"))
        if not df["rq"].empty and "query" in df["rq"].columns:
            comparison["rq"] = pd.merge(comparison["rq"], df["rq"], on="query", how="inner", suffixes=("", f"_{i}"))
        i = i + 1
    comparison["iot"].to_csv("comparison_interest_over_time.csv", index=False)
    comparison["bbr"].to_csv("comparison_breakdown_by_region.csv", index=False)
    comparison["rt"].to_csv("comparison_related_topics.csv", index=False)
    comparison["rq"].to_csv("comparison_related_queries.csv", index=False)

def main():
    keywords = ["cat", "cats"]
    results = []
    for keyword in keywords:
        trend_data = get_trend_data(keyword)
        df_set = dump_trend_data_to_csv(trend_data)
        results.append(df_set)
    create_comparison(results)

if __name__ == "__main__":
    main()

This final version of the code creates CSV files with combined information across categories for the keywords provided: `interest_over_time`, `breakdown_by_region`, `related_topics`, and `related_queries`.

Scale Your Google Trends Scraping with ProxyTee

Web scraping with the support of ProxyTee allows you to effortlessly collect information to improve business strategies. You can achieve consistent data collection with ProxyTee's Unlimited Residential Proxies which provides unlimited bandwidth and automatic rotation, ideal for handling Google Trends’ geo restrictions.

Additionally, ProxyTee offers multiple protocol support for versatility. If you are aiming to refine your scraping, make sure to explore other options such as rotating Residential Proxies, and Datacenter Proxies provided by ProxyTee. These options are easy to integrate with an intuitive platform, supported by comprehensive API access.

Wrapping Up

Utilize these instructions with ProxyTee to gather the data for trend research. Remember to consult our technical documentation for specific Google Trends API variables. Explore the blog for more tutorials and tools. And If you have any further questions, our professionals are ready to help. Start your data extraction today and propel your business.