How to Scrape TripAdvisor Hotel Data with ProxyTee

For many years, TripAdvisor has been a go-to resource for travelers seeking information on destinations. Today, we'll explore how to extract hotel data from TripAdvisor effectively using ProxyTee.

TripAdvisor employs several anti-scraping techniques such as JavaScript challenges, browser fingerprinting, and dynamic page content. Traditional scraping methods using Python Requests and BeautifulSoup are often insufficient. We'll address these challenges head-on with a more robust approach.

Prerequisites

To effectively scrape TripAdvisor, you need tools that can handle:

• JavaScript Challenges: TripAdvisor uses CAPTCHAs to verify that a real browser is in use, and not an automated bot.
• Browser Fingerprinting: Cookies are used to track and identify user behavior.
• Dynamic Content: The site uses APIs to dynamically load content, meaning a simple HTML download won't provide all data.

Selenium, along with a webdriver (like ChromeDriver) becomes necessary. Selenium allows us to simulate a browser and interact with dynamic pages properly. Make sure that your ChromeDriver version matches your version of Chrome. You can verify your Chrome version by inputting google-chrome --version on your terminal.

Install Selenium with the command: pip install selenium.

What to Scrape from TripAdvisor

When you search on TripAdvisor, you get various types of results. For our scraping purpose, we want hotel data only. By using a specific query parameter, we can achieve this. Here's a breakdown of the relevant URLs:

• All Results: https://www.tripadvisor.com/Search?q=miami&geo=1&ssrc=a
• Hotels: https://www.tripadvisor.com/Search?q=miami&geo=1&ssrc=h

The ssrc parameter is how we filter the results. We can extract information from each result card (with a data-test-attribute="location-results-card" attribute). We can then find every element with CSS selector: div[data-test-attribute='location-results-card'] to start extraction.

Scrape TripAdvisor With Vanilla Selenium

We'll develop a Python script that uses Selenium to handle our scraping and CSV for storing the extracted results.

Individual Functions

We need two main functions: write_to_csv() and scrape_page().

The write_to_csv() function:

def write_to_csv(data, page_number):
    if type(data) != list:
        data = [data]
    print("Writing to CSV...")
    filename = f"tripadvisor-{page_number}.csv"
    mode = "w"
    if Path(filename).exists():
        mode = "a"
    print("Writing data to CSV File...")
    with open(filename, mode) as file:
        writer = csv.DictWriter(file, fieldnames=data[0].keys())
        if mode == "w":
            writer.writeheader()
        writer.writerows(data)
    print(f"Successfully wrote {page} to CSV...")

• Checks if data is a list; if not, it converts it into a list.
• Builds filename dynamically for each page scraped.
• Writes to an existing file with append mode, or creates a new file with the write mode.
• Initializes CSV DictWriter.
• Write the CSV header and content.

The scrape_page() function uses Selenium and a variety of techniques to get past Tripadvisor blocks:

def scrape_page(page_number: int): 
    options = webdriver.ChromeOptions()
    options.add_argument("--headless")
    options.add_argument(f"--user-agent={USER_AGENT}")
    print("Connecting to Scraping Browser...")  
    scraped_data = []
    print("-------------------------------")
    driver = webdriver.Chrome(options=options)  
    driver.get(f"https://www.tripadvisor.com/Search?q=Miami&geo=1&ssrc=h&offset={page_number*30}")
    print("Connected! Scraping page...")  
    sleep(5)
        
    hotel_cards = driver.find_elements(By.CSS_SELECTOR, "div[data-test-attribute='location-results-card']")
    if not hotel_cards:
        driver.save_screenshot("error.png")
        driver.quit()
        return
    for index, card in enumerate(hotel_cards):
        score = None
        divs = card.find_elements(By.CSS_SELECTOR, "div")
        for div in divs:
            aria_label = div.get_attribute("aria-label")
            if aria_label:
                if "bubbles" in aria_label:
                    score = aria_label
                    break
        data_array = card.text.split("\n")
        hotel_dict = {
            "name": data_array[1],
            "reviews": int(data_array[2].replace(",", "")),
            "score": float(aria_label[0:3]),
            "location": data_array[3],
            "location_mentions": data_array[4].split(" ")[0],
            "review_summary": data_array[5]
        }
        scraped_data.append(hotel_dict)
        print(f"Successfully scraped card {index}")
        
    print(f"Scraped page {page_number}")
    write_to_csv(scraped_data, page_number)

• Initializes Chrome options including headless and a user-agent.
• Opens a new browser window via webdriver.
• Navigates to the target page and waits 5 seconds for content to load, simulating human interaction.
• Looks for all the result cards based on the mentioned CSS selector.
• Extracts key data points like name, reviews, score, location, and more, from each hotel card.
• Appends extracted data to a list for writing to CSV.

Putting It All Together

This is the script combining all the above codes.

from selenium import webdriver 
from selenium.webdriver.common.by import By
from time import sleep
import csv
from pathlib import Path

USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"

def write_to_csv(data, page_number):
    if type(data) != list:
        data = [data]
    print("Writing to CSV...")
    filename = f"tripadvisor-{page_number}.csv"
    mode = "w"
    if Path(filename).exists():
        mode = "a"
    print("Writing data to CSV File...")
    with open(filename, mode) as file:
        writer = csv.DictWriter(file, fieldnames=data[0].keys())
        if mode == "w":
            writer.writeheader()
        writer.writerows(data)
    print(f"Successfully wrote {page} to CSV...")

  
def scrape_page(page_number: int): 
    options = webdriver.ChromeOptions()
    options.add_argument("--headless")
    options.add_argument(f"--user-agent={USER_AGENT}")
    print("Connecting to Scraping Browser...")  
    scraped_data = []
    print("-------------------------------")
    driver = webdriver.Chrome(options=options)  
    driver.get(f"https://www.tripadvisor.com/Search?q=Miami&geo=1&ssrc=h&offset={page_number*30}")
    print("Connected! Scraping page...")  
    sleep(5)
        
    hotel_cards = driver.find_elements(By.CSS_SELECTOR, "div[data-test-attribute='location-results-card']")
    if not hotel_cards:
        driver.save_screenshot("error.png")
        driver.quit()
        return
    for index, card in enumerate(hotel_cards):
        score = None
        divs = card.find_elements(By.CSS_SELECTOR, "div")
        for div in divs:
            aria_label = div.get_attribute("aria-label")
            if aria_label:
                if "bubbles" in aria_label:
                    score = aria_label
                    break
        data_array = card.text.split("\n")
        hotel_dict = {
            "name": data_array[1],
            "reviews": int(data_array[2].replace(",", "")),
            "score": float(aria_label[0:3]),
            "location": data_array[3],
            "location_mentions": data_array[4].split(" ")[0],
            "review_summary": data_array[5]
        }
        scraped_data.append(hotel_dict)
        print(f"Successfully scraped card {index}")
        
    print(f"Scraped page {page_number}")
    write_to_csv(scraped_data, page_number)

  
if __name__ == '__main__':

    PAGES = 1
    for page in range(PAGES):
        scrape_page(page)

Running this script directly might face blocks. Let's discuss how to mitigate those.

Advanced Techniques

This section covers how pagination is handled, as well as how to enhance your script so as to not get detected by TripAdvisor’s anti-bot systems.

Handling Pagination

Pagination in our script relies on the offset parameter of the url like this https://www.tripadvisor.com/Search?q=Miami&geo=1&ssrc=h&offset={page_number*30}. By multiplying the page number by the 30 results per page we move to the next result window in each page scrape. Change PAGES to scrape more pages.

Mitigate Blocking

To avoid blocks, our script includes:

• A User-Agent to simulate real browser.
• The Sleep Function, introducing waits, to make our scraper less like a bot.

Despite those, vanilla selenium can still get blocked by sophisticated anti-bot systems, it would be better to have more advanced scraping methods to obtain data.

Leveraging ProxyTee for Enhanced Scraping

For more reliable results and to evade more aggressive blocking, ProxyTee offers advanced unlimited residential proxies.

With ProxyTee, you get:

• Unlimited Bandwidth: Scrape as much data as you need without data overage concerns. Learn more at our Unlimited Bandwidth page.
• Global IP Coverage: Over 30 million IPs from 100+ countries, providing diverse IPs for accessing geo-specific data. Check Global Coverage.
• Rotating Residential Proxies: Our proxies change IP addresses automatically every 3 to 60 minutes via Auto Rotation.
• Multiple Protocols: Supports HTTP and SOCKS5, compatible with many apps via Multiple Proxy Protocols.
• Affordable Pricing: ProxyTee’s Pricing starts 50% lower than its competitors and you can achieve this without compromising speed and efficiency.
• Simple API: Seamless automation of tasks using API Integration, see Simple API.

Alternative Approach: Datasets

If setting up a scraper is not your preferred way, ProxyTee also has structured TripAdvisor datasets. It is available in formats including JSON, CSV, Parquet and are updated as per your needs. ProxyTee Datasets provide you the ability to concentrate your resources towards data analytics instead of worrying about how you can collect data.

Conclusion

Scraping data from TripAdvisor is quite challenging with traditional tools due to their sophisticated blocking mechanisms. Using ProxyTee can dramatically increase your rate of successful data collection.

With ProxyTee’s unlimited residential proxies you don’t have to worry about blocks, or even complicated CAPTCHAs!

Scale Your Web Scraping with ProxyTee

Ready to eliminate IP bans and per-gigabyte bandwidth overages? Explore ProxyTee Unlimited Residential Proxies, learn how our unlimited bandwidth residential proxy ports work, check our proxies for travel fare aggregation, or test endpoints on our free proxy list.