How to Scrape Yelp Data with ProxyTee

Yelp is a treasure trove of information for businesses looking to understand customer feedback, conduct competitive analysis, and perform market research. It provides detailed profiles of local businesses, including customer reviews, ratings, contact details, and more. In this guide, we will explore how to scrape Yelp using Python and ProxyTee to ensure anonymity and avoid blocks.

Why Scrape Yelp?

Scraping Yelp offers several key advantages:

• Comprehensive Business Data: Access detailed information about local businesses, which can be crucial for understanding market trends and consumer preferences.
• Customer Feedback Insights: Gather real-time user reviews to gain insights into customer opinions and experiences.
• Competitive Benchmarking: Analyze your competitors’ performance, identify strengths and weaknesses, and assess customer sentiment to stay competitive.

While various platforms offer similar services, Yelp’s large user base, diverse business categories, and well-established reputation make it a prime target for data scraping.

Yelp Scraping with Python

Python is an ideal language for web scraping due to its ease of use, clear syntax, and extensive selection of libraries. Let’s dive into setting up a basic Yelp scraper:

Step 1️⃣: Setting Up a Python Project

Before you start, ensure that you have Python 3+ installed on your system and a Python IDE of your choosing. Create a project folder, initialize it with a virtual environment and create a scraper.py to get started

mkdir yelp-scraper
cd yelp-scraper
python -m venv env

Activate the environment (the command depends on your operating system, like for windows env\Scripts\activate.ps1)

And you are ready to proceed to next step and start coding!

Step 2️⃣: Install Required Libraries

The scraping process requires HTTP client and HTML parser libraries. You can install Requests and Beautiful Soup with:

pip install beautifulsoup4 requests

Next you need to import them in your script scraper.py:

import requests
from bs4 import BeautifulSoup

Step 3️⃣: Identify and Download the Target Page

Navigate to the Yelp page you wish to scrape, such as a list of New York’s top-rated Italian restaurants:

url = 'https://www.yelp.com/search?find_desc=Italian&find_loc=New+York%2C+NY'
page = requests.get(url)

Here, we retrieve the content of the page using requests.get(url), that gives access to the page.text storing HTML.

Step 4️⃣: Parse the HTML

Now it’s time to parse the HTML content:

soup = BeautifulSoup(page.text, 'html.parser')

This provides an explorable structure, to be inspected for retrieving desired elements.

Step 5️⃣: Understand the Structure of the Webpage

Using your browser’s developer tools, inspect the page structure and DOM. Be careful when selecting CSS classes, as they are often dynamically generated and unstable, and prefer the use of HTML attributes.

Step 6️⃣: Extract Business Data

Each restaurant is in a card element, use select('[data-testid="serp-ia-card"]') to select the elements, then loop over them to scrape data from them.

You can use select_one() in combination of selectors to extract specific information, and navigate the dom tree as needed:

# inside the for loop
image = html_item_card.select_one('[data-lcp-target-id="SCROLLABLE_PHOTO_BOX"] img').attrs['src']
name = html_item_card.select_one('h3 a').text
url = 'https://www.yelp.com' + html_item_card.select_one('h3 a').attrs['href']
html_stars_element = html_item_card.select_one('[class^="five-stars"]')
stars = html_stars_element.attrs['aria-label'].replace(' star rating', '')
reviews = html_stars_element.parent.parent.next_sibling.text

This method is useful for simple and fast extractions, also remember to clean and convert the data into usable values. For values like tags, which appear multiple times, we need an extra loop:

tags = []
html_tag_elements = html_item_card.select('[class^="priceCategory"] button')
for html_tag_element in html_tag_elements:
    tag = html_tag_element.text
    tags.append(tag)

The other extractions are based on similar approaches.

Step 7️⃣: Implement Crawling Logic

To scrape data from multiple pages, implement a crawling logic.

visited_pages = []
pages_to_scrape = ['https://www.yelp.com/search?find_desc=Italian&find_loc=New+York%2C+NY']

limit = 5
i = 0
while len(pages_to_scrape) != 0 and i < limit:
    # logic for downloading, parsing the page, and extracting data
    # implemented in the previous steps
    
    pagination_link_elements = soup.select('[class^="pagination-links"] a')
    for pagination_link_element in pagination_link_elements:
        pagination_url = pagination_link_element.attrs['href']
        if pagination_url not in visited_pages and pagination_url not in pages_to_scrape:
            pages_to_scrape.append(pagination_url)
    i += 1

The script goes through multiple pages of the results until a limit is reached.

Step 8️⃣: Export Data to CSV

To share the extracted data export it to a CSV file with these few lines of code:

import csv
# ...
with open('restaurants.csv', 'w', newline='', encoding='utf-8') as csv_file:
    writer = csv.DictWriter(csv_file, fieldnames=headers, quoting=csv.QUOTE_ALL)
    writer.writeheader()

    for item in items:
        # transform array fields from "['element1', 'element2', ...]"
        # to "element1; element2; ..."
        csv_item = {}
        for key, value in item.items():
            if isinstance(value, list):
                csv_item[key] = '; '.join(str(e) for e in value)
            else:
                csv_item[key] = value

        writer.writerow(csv_item)

Step 9️⃣: All Together

By completing this step, you’ll have successfully implemented the complete Python script needed for crawling and scraping the desired data from Yelp. Remember that by combining these techniques with ProxyTee‘s  Unlimited Residential Proxies, you will be able to do your research privately and reliably.

ProxyTee: The Ideal Solution for Web Scraping

Web scraping, especially at scale, can expose your IP address, leading to blocks or restrictions. This is where ProxyTee comes in. Here’s why ProxyTee is the perfect solution:

• Unlimited Bandwidth: With unlimited bandwidth, you can scrape large amounts of data without worrying about overage fees.
• Extensive Global Coverage: ProxyTee’s vast pool of 20 million+ IPs from over 100 countries ensures you can access data from specific locations with ease.
• Automatic IP Rotation: Auto-rotation changes your IP at regular intervals (3 to 60 minutes), minimizing the risk of being detected or blocked by target websites.
• Flexibility and Support: Supporting both HTTP and SOCKS5 protocols, ProxyTee can integrate seamlessly with all your existing tools.
• Affordable Pricing: ProxyTee offers very competitive pricing, as much as 50% lower than competitors for similar features.
• User Friendly: The user-friendly interface of ProxyTee and the simple API will allow you to have a smooth and effective scraping experience
• Unlimited Residential Proxies: Especially valuable, our Unlimited Residential Proxies product will guarantee high anonymity, and avoid blocking, since you will be seen as a regular user.

Combining ProxyTee with Python gives you a potent mix for collecting online data anonymously and without restrictions. This can bring huge value in terms of data, market knowledge, competitive analysis and research.