How to Scrape Google Hotels Using Python and ProxyTee

In the realm of data-driven decision-making, extracting information from platforms like Google Hotels provides a significant advantage. Imagine having access to current hotel data—including prices, availability, reviews, and locations—easily accessible, ready to support market research or aid in price comparisons. This tutorial will guide you through scraping data from Google Hotels using Python and how ProxyTee can enhance your scraping efforts.

By using a web scraper for Google Hotels, you can efficiently monitor pricing trends and analyze hotel rates across various regions. These insights are crucial for conducting market research within the hospitality sector, giving you a competitive edge by understanding customer preferences. While a Google Hotels API might offer a direct method for data retrieval, scraping combined with ProxyTee's reliable network offers a powerful and flexible solution, particularly when direct API access is not feasible or has limitations.

Note: This tutorial is for educational purposes. Before implementing these techniques, please review Google's Terms of Service to ensure compliance and ethical data collection practices.

Step-by-step guide on how to scrape Google Hotels using Python

We'll employ Python with Selenium and BeautifulSoup for scraping data from Google Hotels. Selenium interacts with the site's dynamic content, while BeautifulSoup extracts data using CSS selectors, making it easier to handle automated scraping tasks effectively. For those new to this, you may want to consult foundational Python web scraping resources to enhance your understanding.

Advanced users may find value in using a ProxyTee for tasks that may require additional capacity or locations. Similar to scraping Google News or other dynamic content platforms, the right approach can greatly enhance data collection.

Prerequisites

Make sure that Python is installed. This guide utilizes Python 3.11, but any version 3.8 and above should work effectively. You will also need these libraries:

• BeautifulSoup
• Requests
• Selenium
• Pandas
• lxml

These will help you extract details like hotel prices, ratings, and locations. This data is invaluable for market research, price comparisons, and trend analysis. Install them via:

pip install requests bs4 selenium pandas lxml

Step 1: Setting up the development environment

After installing all necessary libraries, create a new Python file. Start by importing required libraries, such as Selenium for browser automation, BeautifulSoup for parsing data, pandas for saving data, and time for adding delays.

import pandas as pd
import os
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from bs4 import BeautifulSoup
from selenium.webdriver import ChromeOptions

Initialize the Selenium WebDriver object, and set the Google Hotels link for navigation. Adjust location and date as per the data you want to extract.

Step 2: Initialize variables and make a request

# Step 1: Set up Selenium WebDriver
driver = webdriver.Chrome()

options = ChromeOptions()
options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)

location_name = "New York"  # Set the location for hotel search
base_url = f"https://www.google.com/travel/hotels/{location_name}/?q={location_name}¤cy=USD"

driver.get(base_url)  # Load the Google Hotels page for the specified location

The location_name variable sets the location for which hotel data is to be extracted. This can be adjusted according to your needs.

If a cookies page appears, you can use the following code to accept cookies.

# Accept cookies if the button exists
try:
    driver.find_element(By.XPATH, '//button[contains(@aria-label, "Accept")]').click()
except NoSuchElementException:
    pass

Step 3: Scraping data

On the main Google Hotels page, elements include the hotel name, price, rating, and amenities. We'll extract the name, price, and rating. For the location, navigate to the hotel details page. Google’s SERP elements like carousels and ads use similar dynamic features for various content types. The detail page is:

Using Selenium, we will navigate through the main page, extract main data, and then access the detail pages to retrieve hotel addresses. We can view the corresponding HTML by using the browser's developer tools.

Step 4: Extract the data

Extract the data using Beautiful Soup. To prevent storing duplicate listings, use the set() function to keep only unique entries:

# Step 2: Initialize a list to store hotel data
hotel_data = []

# Initialize a set to track unique entries
unique_hotels = set()

# Parse the page with BeautifulSoup using lxml
soup = BeautifulSoup(driver.page_source, "lxml")

# Find all hotel cards on the page
hotel_cards = soup.find_all("div", class_="BcKagd")  # Adjust class if necessary

for hotel in hotel_cards:
    # Extract hotel details
    name = (
        hotel.find("h2", class_="BgYkof").text
        if hotel.find("h2", class_="BgYkof")
        else "N/A"
    )  # Hotel name
    price = (
        hotel.find("span", class_="qQOQpe prxS3d").text
        if hotel.find("span", class_="qQOQpe prxS3d")
        else "N/A"
    )  # Price
    rating = (
        hotel.find("span", class_="KFi5wf lA0BZ").text
        if hotel.find("span", class_="KFi5wf lA0BZ")
        else "N/A"
    )  # Rating

    hotel_link = hotel.find("a", class_="PVOOXe").get("href")  # Hotel link

    # Visit the hotel page to get more details
    driver.get("https://www.google.com" + hotel_link)

    WebDriverWait(driver, 20).until(
        EC.presence_of_element_located((By.CLASS_NAME, "gJGKuf"))
    )

    # Try to get the location of the hotel
    try:
        location = driver.find_element(By.XPATH, '//div[@class="K4nuhf"]/span[1]').text
        contact = driver.find_element(By.XPATH, '//div[@class="K4nuhf"]/span[3]').text
    except NoSuchElementException:
        location = "N/A"  # Default if location not found
        contact = "N/A"

    # Go back to the previous page
    driver.back()

    # Define a unique identifier for the hotel
    hotel_id = (name, price, rating, location, contact)

To handle pagination, implement a loop that processes 5 pages, or until the Next button disappears. This loop scrapes through the data until the conditions are met:

hotel_data = []

# Initialize a set to track unique entries
unique_hotels = set()

no_of_pages = 0  # Counter for the number of pages scraped
next_button_clickable = True  # Flag to track if the next button is clickable

# Step 3: Scrape hotel data page by page
while next_button_clickable and no_of_pages < 5:
    # Parse the page with BeautifulSoup using lxml
    soup = BeautifulSoup(driver.page_source, "lxml")

    # Find all hotel cards on the page
    hotel_cards = soup.find_all("div", class_="BcKagd")  # Adjust class if necessary

    for hotel in hotel_cards:
        # Extract hotel details
        name = (
            hotel.find("h2", class_="BgYkof").text
            if hotel.find("h2", class_="BgYkof")
            else "N/A"
        )  # Hotel name
        price = (
            hotel.find("span", class_="qQOQpe prxS3d").text
            if hotel.find("span", class_="qQOQpe prxS3d")
            else "N/A"
        )  # Price
        rating = (
            hotel.find("span", class_="KFi5wf lA0BZ").text
            if hotel.find("span", class_="KFi5wf lA0BZ")
            else "N/A"
        )  # Rating
        hotel_link = hotel.find("a", class_="PVOOXe").get("href")  # Hotel link

        # Visit the hotel page to get more details
        driver.get("https://www.google.com" + hotel_link)

        WebDriverWait(driver, 20).until(
            EC.presence_of_element_located((By.CLASS_NAME, "gJGKuf"))
        )

        # Try to get the location of the hotel
        try:
            location = driver.find_element(By.XPATH, '//div[@class="K4nuhf"]/span[1]').text
            contact = driver.find_element(By.XPATH, '//div[@class="K4nuhf"]/span[3]').text
        except NoSuchElementException:
            location = "N/A"  # Default if location not found
            contact = "N/A"

        # Go back to the previous page
        driver.back()

        # Define a unique identifier for the hotel
        hotel_id = (name, price, rating, location, contact)

        # Only add the hotel if it hasn't been seen before
        if hotel_id not in unique_hotels:
            unique_hotels.add(hotel_id)
            hotel_data.append(
                {
                    "Name": name,
                    "Price": price,
                    "Rating": rating,
                    "Location": location,
                    "Contact": contact,
                    "Link": "https://www.google.com" + hotel_link,
                }
            )

    # Clear the hotel_data list to store the next page's data
    hotel_data.clear()

    # Step 5: Check for the "Next" button and navigate to the next page
    try:
        WebDriverWait(driver, 20).until(
            EC.presence_of_element_located((By.CLASS_NAME, "K1smNd"))
        )

        # Locate the "Next" button based on the current page number
        if no_of_pages == 0:
            next_button = driver.find_element(
                By.XPATH, '//div[@class="eGUU7b"]/button'
            )
        else:
            next_button = driver.find_element(
                By.XPATH, '//div[@class="eGUU7b"]/button[2]'
            )

        # Click the "Next" button
        next_button.click()
        no_of_pages += 1  # Increment the page counter

    except NoSuchElementException:
        print("No more pages to navigate.")
        next_button_clickable = False  # Stop if no "Next" button is found

# Step 6: Close the Selenium driver
driver.quit()

The code iterates through the first five pages (or less if there are fewer than five pages). The XPATH is specific to the first page vs the rest as the additional buttons are different.

Step 5: Save the data in DataFrame

Save extracted hotel data into a CSV file, adding it to each loop iteration as well:

# Step 4: Save the current page's data to CSV after scraping
df = pd.DataFrame(hotel_data)

# Write header only if the file does not exist
df.to_csv("hotels_data.csv", mode="a", header=not os.path.exists("hotels_data.csv"), index=False)

# Clear the hotel_data list to store the next page's data
hotel_data.clear()

The header is written only once, when the CSV is first created, to ensure all subsequent pages add only rows of the table, not the headers again.

Full code sample

Here’s the complete code:

# Import necessary libraries
import pandas as pd
import os
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from bs4 import BeautifulSoup
from selenium.webdriver import ChromeOptions


# Step 1: Set up Selenium WebDriver
driver = webdriver.Chrome()

options = ChromeOptions()
options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)

location_name = "New York"  # Set the location for hotel search
base_url = f"https://www.google.com/travel/hotels/{location_name}/?q={location_name}¤cy=USD"

# Delete the CSV file if it exists
if os.path.exists("hotels_data.csv"):
    os.remove("hotels_data.csv")

driver.get(base_url)  # Load the Google Hotels page for the specified location

# Accept cookies if the button exists
try:
    driver.find_element(By.XPATH, '//button[contains(@aria-label, "Accept")]').click()
except NoSuchElementException:
    pass

WebDriverWait(driver, 20).until(
    EC.presence_of_element_located((By.CLASS_NAME, "K1smNd"))
)


# Step 2: Initialize a list to store hotel data
hotel_data = []

# Initialize a set to track unique entries
unique_hotels = set()

no_of_pages = 0  # Counter for the number of pages scraped
next_button_clickable = True  # Flag to track if the next button is clickable

# Step 3: Scrape hotel data page by page
while next_button_clickable and no_of_pages < 5:
    # Parse the page with BeautifulSoup using lxml
    soup = BeautifulSoup(driver.page_source, "lxml")

    # Find all hotel cards on the page
    hotel_cards = soup.find_all("div", class_="BcKagd")  # Adjust class if necessary

    for hotel in hotel_cards:
        # Extract hotel details
        name = (
            hotel.find("h2", class_="BgYkof").text
            if hotel.find("h2", class_="BgYkof")
            else "N/A"
        )  # Hotel name
        price = (
            hotel.find("span", class_="qQOQpe prxS3d").text
            if hotel.find("span", class_="qQOQpe prxS3d")
            else "N/A"
        )  # Price
        rating = (
            hotel.find("span", class_="KFi5wf lA0BZ").text
            if hotel.find("span", class_="KFi5wf lA0BZ")
            else "N/A"
        )  # Rating
        hotel_link = hotel.find("a", class_="PVOOXe").get("href")  # Hotel link

        # Visit the hotel page to get more details
        driver.get("https://www.google.com" + hotel_link)

        WebDriverWait(driver, 20).until(
            EC.presence_of_element_located((By.CLASS_NAME, "gJGKuf"))
        )

        # Try to get the location of the hotel
        try:
            location = driver.find_element(By.XPATH, '//div[@class="K4nuhf"]/span[1]').text
            contact = driver.find_element(By.XPATH, '//div[@class="K4nuhf"]/span[3]').text
        except NoSuchElementException:
            location = "N/A"  # Default if location not found
            contact = "N/A"

        # Go back to the previous page
        driver.back()

        # Define a unique identifier for the hotel
        hotel_id = (name, price, rating, location, contact)

        # Only add the hotel if it hasn't been seen before
        if hotel_id not in unique_hotels:
            unique_hotels.add(hotel_id)
            hotel_data.append(
                {
                    "Name": name,
                    "Price": price,
                    "Rating": rating,
                    "Location": location,
                    "Contact": contact,
                    "Link": "https://www.google.com" + hotel_link,
                }
            )

    # Step 4: Save the current page's data to CSV after scraping
    df = pd.DataFrame(hotel_data)

    # Write header only if the file does not exist
    df.to_csv("hotels_data.csv", mode="a", header=not os.path.exists("hotels_data.csv"), index=False)

    # Clear the hotel_data list to store the next page's data
    hotel_data.clear()

    # Step 5: Check for the "Next" button and navigate to the next page
    try:
        WebDriverWait(driver, 20).until(
            EC.presence_of_element_located((By.CLASS_NAME, "K1smNd"))
        )

        # Locate the "Next" button based on the current page number
        if no_of_pages == 0:
            next_button = driver.find_element(
                By.XPATH, '//div[@class="eGUU7b"]/button'
            )
        else:
            next_button = driver.find_element(
                By.XPATH, '//div[@class="eGUU7b"]/button[2]'
            )

        # Click the "Next" button
        next_button.click()
        no_of_pages += 1  # Increment the page counter

    except NoSuchElementException:
        print("No more pages to navigate.")
        next_button_clickable = False  # Stop if no "Next" button is found

# Step 6: Close the Selenium driver
driver.quit()

With the above code, the extracted hotel data will be in a CSV format and we can move forward.

Potential Challenges while Scraping Google Hotels

As you start your own project, you can find that there are challenges to avoid or manage:

CAPTCHA: Google uses CAPTCHAs to stop bots. You might be flagged with frequent queries. To reduce CAPTCHAs, try to include delays (using time.sleep) between requests.

Dynamic Content: Google Hotels uses JavaScript to load content dynamically. This content is often not fully available in a simple request, use Selenium for a full page load. This way dynamically loaded elements can be processed effectively.

IP blocks: Making numerous requests quickly can cause IP blocks. Set delays, vary access, or, use proxies like ProxyTee's rotating residential proxies to prevent this. ProxyTee offers Unlimited Residential Proxies with global coverage and auto-rotation, which help you avoid detection while extracting data.

Conclusion

This tutorial covered scraping Google Hotels using Python. It included the setup of Python and environment, navigating Google Hotels with Selenium, extracting data with BeautifulSoup, saving this information, and going through known challenges such as CAPTCHAs, dynamic content loading, and potential IP blocking.

If you want to scale, improve accuracy, and stability of scraping processes, using services like ProxyTee for proxy service, is a valuable approach.

For specialized cases, exploring further techniques can give extra advantage. Remember, with the right approach, accessing comprehensive data is a key step in making well-informed decisions.