How to Scrape Google’s ‘People Also Ask’ Using Python with ProxyTee

Google’s ‘People Also Ask’ (PAA) section is a goldmine of search intent data. It offers valuable insights into user behavior, providing a list of related questions based on an initial search query. This data is crucial for content strategy, SEO research, and market analysis.

In this guide, you’ll learn how to scrape the PAA section using Python while ensuring seamless data extraction with ProxyTee, a provider of high-quality rotating residential proxies.

Why Scrape ‘People Also Ask’?

The PAA box on Google provides a goldmine of information, including:

• Content Ideas: Understand what users are interested in related to your topic.
• SEO Enhancement: Discover relevant keywords to target.
• Competitive Insights: Identify information gaps you can fill with your content.

Introduction to ProxyTee

ProxyTee is a leading provider of rotating residential proxies. We offer a reliable, affordable, and easy-to-use service designed for various online activities, including web scraping. Here’s what makes ProxyTee stand out:

• Unlimited Bandwidth: Scrape as much data as you need without worrying about data overages.
• Global IP Coverage: Access over 20 million IP addresses from 100+ countries for accurate location-based data.
• Multiple Protocol Support: Utilize both HTTP and SOCKS5 protocols for compatibility with different applications.
• User-Friendly Interface: Get started quickly with our simple and intuitive GUI, with no technical expertise needed.
• Auto Rotation: Rotate your IP address automatically at intervals from 3 to 60 minutes, crucial for avoiding detection and blocks, especially while scraping.
• API Integration: Seamlessly integrate ProxyTee with your applications via our simple API.
• Affordable Pricing: Our plans are competitively priced, ensuring value for every budget, up to 50% cheaper than the competition. Especially our Unlimited Residential Proxies.

Step-by-Step Guide to Scraping Google’s PAA Using Python

Now, let’s get to the core of the matter—scraping Google’s PAA section. We’ll be using Python, along with the requests and BeautifulSoup4 libraries.

1️⃣ Step 1: Setting Up the Environment

First, install the necessary libraries using pip:

pip install requests json bs4

2️⃣ Step 2: Connecting to Google

Create a main.py file and make a request to Google:

import requests


# Encode the query for use in a URL
query = "search query"
query = query.replace(' ', '+')


# URL for the Google search
url = f"https://www.google.com/search?q={query}"

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
}


# Send the GET request to Google
response = requests.get(url, headers=headers)


# If the request is successful (status code 200)
if response.status_code == 200:
    print(response.status_code)
else:
    print(f"Error: Unable to fetch the search results. Status code: {response.status_code}")

In the above code, we prepare our query, create the URL, add headers to mimic a browser, and send the request to Google.

3️⃣ Step 3: Finding and Extracting PAA Questions

Now, extract the PAA questions using BeautifulSoup. Inspect the Google page in your browser to find the CSS identifier for the question elements. Here’s the full code to achieve that:

import requests
from bs4 import BeautifulSoup


# Define a function to search Google
def get_soup_from_google_search(query):
    # Encode the query for use in a URL
    query = query.replace(' ', '+')
    
    # URL for the Google search
    url = f"https://www.google.com/search?q={query}"

    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
    }

    # Send the GET request to Google
    response = requests.get(url, headers=headers)

    # If the request is successful (status code 200)
    if response.status_code == 200:
        # Parse the HTML content with BeautifulSoup
        soup = BeautifulSoup(response.text, "html.parser")
        return soup
    else:
        print(f"Error: Unable to fetch the search results. Status code: {response.status_code}")
        return None

query = "how do you get diabetes"
soup = get_soup_from_google_search(query)

if soup:
    for question in soup.select('span.CSkcDe'):
        print(question.text)

This script sends a query to Google and extracts and displays the ‘People Also Ask’ questions.

4️⃣ Step 4: Storing the Data

It’s important to save collected data. We’ll use a JSON file to do so:

import requests
import json
import os
from bs4 import BeautifulSoup
from datetime import datetime

def save_results(query, questions):
    results = {
        "date": datetime.now().strftime("%Y-%m-%d"),
        "query": query,
        "questions": questions
    }
    
    if os.path.exists("results.json"):
        with open("results.json", "r", encoding="utf-8") as file:
            data = json.load(file)
    else:
        data = []
    
    data.append(results)

    with open("results.json", "w", encoding="utf-8") as file:
        json.dump(data, file, indent=4)

    print("Results saved to results.json")

def google_search(query):
    query = query.replace(' ', '+')
    url = f"https://www.google.com/search?q={query}"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
    }
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        soup = BeautifulSoup(response.text, "html.parser")
        return soup
    else:
        print(f"Error: Unable to fetch the search results. Status code: {response.status_code}")
        return None

def extract_questions(soup):
    titles = []
    if soup:
        for question in soup.select('span.CSkcDe'):
            titles.append(question.get_text())
    return titles   

query = "how do you get diabetes"
soup = google_search(query)
questions = extract_questions(soup)
save_results(query, questions)

This complete script will search, extract, and save ‘People Also Ask’ questions into a JSON file. Use our Unlimited Residential Proxies, that ensures you can scrape a large volume of Google Search data with our massive pool of rotating IP addresses.

Other Considerations

1. Avoiding Google Blocks with ProxyTee: Google actively detects and blocks scraping activities. This is where ProxyTee becomes invaluable. Using residential proxies from our ProxyTee service can help you overcome these blocks with IP rotation and by distributing your scraping requests.
2. Track ‘People Also Ask’ Data: The ‘People Also Ask’ data changes frequently, which is why scheduling regular runs of your scraping script will allow you to track shifts in user interests over time.