How to Scrape Hotels from Google Travel with ProxyTee

Google Travel compiles travel data from across the web, including flights, vacation packages, and hotel rooms. Navigating the sheer volume of information and sponsored listings can be overwhelming. This post explores how to gather hotel data from Google Travel, along with pricing and amenities, using web scraping techniques or convenient APIs, all while emphasizing the reliability and efficiency of ProxyTee.

If you prefer to bypass the technical details, consider exploring pre-made datasets, such as those available at ProxyTee, to get your data without having to write any code. But, if you are ready to start scraping, read on.

Introduction to ProxyTee

ProxyTee is your reliable partner in web data extraction. Specializing in rotating residential proxies, ProxyTee ensures you have the anonymity and global coverage needed to gather data effectively. Known for its affordability, unlimited bandwidth, and an intuitive user interface, ProxyTee stands out as an ideal solution for web scraping.

Key features of ProxyTee:

• Unlimited Bandwidth: ProxyTee offers proxies with unlimited bandwidth, ensuring no limitations while performing data-intensive tasks.
• Global IP Coverage: Access to over 30 million IP addresses from 100+ countries makes ProxyTee an excellent choice for those targeting specific regions.
• Multiple Protocol Support: ProxyTee supports both HTTP and SOCKS5 protocols for seamless integration with different tools.
• User-Friendly Interface: Get started quickly with minimal setup time and no required technical skills.
• Auto Rotation: Automatic IP rotation between 3 to 60 minutes ensures high anonymity and prevents blocks when scraping.
• API Integration: Simple API supports integration with multiple applications, ideal for developers and businesses looking for automation.

Getting Started with Web Scraping

To begin scraping travel data, you'll need Python along with libraries like Selenium, Requests, or AIOHTTP. Selenium allows direct data extraction from Google Travel, while Requests and AIOHTTP can be used alongside the ProxyTee’s APIs. Make sure you have installed the required packages:

pip install selenium
pip install requests
pip install aiohttp

Extracting Data from Google Travel Using Selenium

When scraping Google Travel directly, the data is housed in c-wiz elements. Identifying and isolating this element is critical for data extraction. You'll need a CSS selector to find hotel listings (e.g. c-wiz > div > a), within these you'll find hotel names in h2 elements, prices in span, and amenities in li elements.

Here's a sample of how to use Selenium for scraping hotel data:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
import json
from time import sleep

OPTIONS = webdriver.ChromeOptions()
OPTIONS.add_argument("--headless")
OPTIONS.add_argument("--window-size=1920,1080")

def scrape_hotels(location, pages=5):
    driver = webdriver.Chrome(options=OPTIONS)
    actions = ActionChains(driver)
    url = f"https://www.google.com/travel/search?q={location}"
    driver.get(url)
    done = False

    found_hotels = []
    page = 1
    result_number = 1
    while page <= pages:
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        sleep(5)
        hotel_links = driver.find_elements(By.CSS_SELECTOR, "c-wiz > div > a")
        print(f"-----------------PAGE {page}------------------")
        print("FOUND ITEMS: ", len(hotel_links))
        for hotel_link in hotel_links:
            hotel_card = hotel_link.find_element(By.XPATH, "..")
            try:
                info = {}
                info["url"] = hotel_link.get_attribute("href")
                info["rating"] = 0.0
                info["price"] = "n/a"
                info["name"] = hotel_card.find_element(By.CSS_SELECTOR, "h2").text
                price_holder = hotel_card.find_elements(By.CSS_SELECTOR, "span")
                info["amenities"] = []
                amenities_holders = hotel_card.find_elements(By.CSS_SELECTOR, "li")
                for amenity in amenities_holders:
                    info["amenities"].append(amenity.text)
                if "DEAL" in price_holder[0].text or "PRICE" in price_holder[0].text:
                    if price_holder[1].text[0] == "$":
                        info["price"] = price_holder[1].text
                else:
                    info["price"] = price_holder[0].text
                rating_holder = hotel_card.find_elements(By.CSS_SELECTOR, "span[role='img']")
                if rating_holder:
                    info["rating"] = float(rating_holder[0].get_attribute("aria-label").split(" ")[0])
                info["result_number"] = result_number

                if info not in found_hotels:
                    found_hotels.append(info)
                result_number+=1

            except:
                continue
        print("Scraped Total:", len(found_hotels))
        
        next_button = driver.find_elements(By.XPATH, "//span[text()='Next']")
        if next_button:
            print("next button found!")
            sleep(1)
            actions.move_to_element(next_button[0]).click().perform()
            page+=1
            sleep(5)
        else:
            done = True

    driver.quit()

    with open("scraped-hotels.json", "w") as file:
        json.dump(found_hotels, file, indent=4)

if __name__ == "__main__":
    PAGES = 2
    scrape_hotels("miami", pages=PAGES)

This script uses custom window size options to prevent loading errors, and an ActionChains instance to interact with page elements, dynamically loading content through multiple pages.

When scraping Google Travel, blocking is not common, but using residential proxies or the unlimited residential proxies and a rotating IP from ProxyTee are recommended to ensure smooth operation.

Extracting Data with ProxyTee’s Travel API

If manual scraping seems complex, using the ProxyTee Travel API is a viable option. This API streamlines the extraction of hotel data by sending HTTP requests, with ProxyTee handling the rest of the scraping.

Requests Example

Here is an example using requests to obtain data via ProxyTee:

import requests
import json
import time

def get_bookings(PROXY_TOKEN, location, dates):
    url = "https://api.brightdata.com/datasets/v3/trigger"

    dataset_id = "gd_m4bf7a917zfezv9d5"

    endpoint = f"{url}?dataset_id={dataset_id}&include_errors=true"
    auth_token = PROXY_TOKEN

    headers = {
        "Authorization": f"Bearer {auth_token}",
        "Content-Type": "application/json"
    }

    payload = [
        {
            "url": "https://www.booking.com",
            "location": location,
            "check_in": dates["check_in"],
            "check_out": dates["check_out"],
            "adults": 2,
            "rooms": 1
        }
    ]

    response = requests.post(endpoint, headers=headers, json=payload)

    if response.status_code == 200:
        print("Request successful. Response:")
        print(json.dumps(response.json(), indent=4))
        return response.json()["snapshot_id"]
    else:
        print(f"Error: {response.status_code}")
        print(response.text)

def poll_and_retrieve_snapshot(PROXY_TOKEN, snapshot_id, output_file="snapshot-data.json"):
    snapshot_url = f"https://api.brightdata.com/datasets/v3/snapshot/{snapshot_id}?format=json"
    headers = {
        "Authorization": f"Bearer {PROXY_TOKEN}"
    }

    print(f"Polling snapshot for ID: {snapshot_id}...")

    while True:
        response = requests.get(snapshot_url, headers=headers)
        
        if response.status_code == 200:
            print("Snapshot is ready. Downloading...")
            snapshot_data = response.json()
            with open(output_file, "w", encoding="utf-8") as file:
                json.dump(snapshot_data, file, indent=4)
            print(f"Snapshot saved to {output_file}")
            break
        elif response.status_code == 202:
            print("Snapshot is not ready yet. Retrying in 10 seconds...")
        else:
            print(f"Error: {response.status_code}")
            print(response.text)
            break
        
        time.sleep(10)

if __name__ == "__main__":
    
    PROXY_TOKEN = "your-bright-data-api-key"
    LOCATION = "Miami"
    CHECK_IN = "2025-02-01T00:00:00.000Z"
    CHECK_OUT = "2025-02-02T00:00:00.000Z"
    DATES = {
        "check_in": CHECK_IN,
        "check_out": CHECK_OUT
    }
    snapshot_id = get_bookings(PROXY_TOKEN, LOCATION, DATES)
    poll_and_retrieve_snapshot(PROXY_TOKEN, snapshot_id)

In this example, get_bookings() sends the initial API request and returns the snapshot_id which is needed for data retrieval. The poll_and_retrieve_snapshot() polls the server every ten seconds until the data is ready for download.

AIOHTTP Example

To achieve faster data gathering, use AIOHTTP, which enables asynchronous processing, allowing for parallel dataset downloads.

import aiohttp
import asyncio
import json


async def get_bookings(PROXY_TOKEN, location, dates):
    url = "https://api.brightdata.com/datasets/v3/trigger"
    dataset_id = "gd_m4bf7a917zfezv9d5"
    endpoint = f"{url}?dataset_id={dataset_id}&include_errors=true"
    headers = {
        "Authorization": f"Bearer {PROXY_TOKEN}",
        "Content-Type": "application/json"
    }
    payload = [
        {
            "url": "https://www.booking.com",
            "location": location,
            "check_in": dates["check_in"],
            "check_out": dates["check_out"],
            "adults": 2,
            "rooms": 1
        }
    ]

    async with aiohttp.ClientSession(headers=headers) as session:
        async with session.post(endpoint, json=payload) as response:
            if response.status == 200:
                response_data = await response.json()
                print(f"Request successful for location: {location}. Response:")
                print(json.dumps(response_data, indent=4))
                return response_data["snapshot_id"]
            else:
                print(f"Error for location: {location}. Status: {response.status}")
                print(await response.text())
                return None


async def poll_and_retrieve_snapshot(PROXY_TOKEN, snapshot_id, output_file):
    snapshot_url = f"https://api.brightdata.com/datasets/v3/snapshot/{snapshot_id}?format=json"
    headers = {
        "Authorization": f"Bearer {PROXY_TOKEN}"
    }

    print(f"Polling snapshot for ID: {snapshot_id}...")

    async with aiohttp.ClientSession(headers=headers) as session:
        while True:
            async with session.get(snapshot_url) as response:
                if response.status == 200:
                    print(f"Snapshot for {output_file} is ready. Downloading...")
                    snapshot_data = await response.json()
                    with open(output_file, "w", encoding="utf-8") as file:
                        json.dump(snapshot_data, file, indent=4)
                    print(f"Snapshot saved to {output_file}")
                    break
                elif response.status == 202:
                    print(f"Snapshot for {output_file} is not ready yet. Retrying in 10 seconds...")
                else:
                    print(f"Error polling snapshot for {output_file}. Status: {response.status}")
                    print(await response.text())
                    break

            await asyncio.sleep(10)


async def process_location(PROXY_TOKEN, location, dates):
    snapshot_id = await get_bookings(PROXY_TOKEN, location, dates)
    if snapshot_id:
        output_file = f"snapshot-{location.replace(' ', '_').lower()}.json"
        await poll_and_retrieve_snapshot(PROXY_TOKEN, snapshot_id, output_file)


async def main():
    PROXY_TOKEN = "your-bright-data-api-key"
    locations = ["Miami", "Key West"]
    dates = {
        "check_in": "2025-02-01T00:00:00.000Z",
        "check_out": "2025-02-02T00:00:00.000Z"
    }

    tasks = [process_location(PROXY_TOKEN, location, dates) for location in locations]
    await asyncio.gather(*tasks)


if __name__ == "__main__":
    asyncio.run(main())

With AIOHTTP, multiple datasets can be handled concurrently, increasing data gathering efficiency by avoiding unnecessary waits. When using multiple locations to gather data, it is very recommended.

ProxyTee Alternative Solutions

ProxyTee goes beyond simple proxy services. We also provide pre-collected, ready-to-use datasets, catering to different requirements. Our most sought-after travel datasets are available including hotel data, specific location such as Expedia or more comprehensive information such as tourism dataset.

Conclusion

Google Travel holds extensive information which can be very helpful. If you’re choosing between custom scrapers or automated APIs for collecting data, remember that ProxyTee's proxies or integrated data APIs provide you reliable and efficient results. Whether it’s analyzing historical pricing or finding better accommodations, mastering the scraping process will help to obtain valuable market insight. With ProxyTee, get reliable access to web data. Sign up now to try ProxyTee for yourself.

Scale Your Web Scraping with ProxyTee

Ready to eliminate IP bans and per-gigabyte bandwidth overages? Explore ProxyTee Unlimited Residential Proxies, learn how our unlimited bandwidth residential proxy ports work, check our proxies for travel fare aggregation, or test endpoints on our free proxy list.