How to Scrape eBay in Python for Price Monitoring with ProxyTee

In the world of e-commerce, staying ahead requires having access to timely and accurate data. This post will guide you through building an eBay scraper using Python, focusing on price monitoring and comparison. We'll also introduce how ProxyTee can enhance your scraping activities with its reliable and efficient residential proxy network.

Why Scrape E-Commerce Data?

Web scraping of e-commerce data provides invaluable insights for a variety of business needs:

• Price Monitoring: Track real-time price fluctuations, helping to identify trends and optimize your pricing strategies or find the best deals as a consumer.
• Competitor Analysis: Gain data-driven insights into competitor strategies by monitoring their pricing, promotions, and product offerings.
• Market Research: Uncover trends and understand customer behavior through the analysis of market data.
• Sentiment Analysis: By collecting and analysing customer reviews to better understand product feedback.

eBay presents a prime scraping opportunity due to its vast array of products, its auction/bidding system, and the presence of multiple price points for the same items.

Why Use ProxyTee for eBay Scraping?

ProxyTee is an excellent choice for your scraping tasks, especially when dealing with dynamic sites like eBay. Here are some key benefits:

• Unlimited Bandwidth: ProxyTee offers unlimited bandwidth, enabling data-intensive operations without concerns about data overages or extra costs.
• Global IP Coverage: Access over 30 million IPs from more than 100 countries through ProxyTee, ensuring comprehensive coverage.
• Residential IPs: ProxyTee's rotating residential proxies make it more difficult for websites to detect and block your scraper.
• Rotating Proxies: Automatic IP rotation is available in intervals from 3 to 60 minutes which reduce the chances of being blocked by target websites when scraping large data volume.
• Affordability: With prices as low as 50% compared to competitors and no hidden fees, ProxyTee's service provides the best value.
• Easy Integration: ProxyTee offers simple APIs to enable easy integration with other systems.

eBay Scraping Libraries and Tools

Python stands out as a top choice for web scraping due to its versatility, readability, and rich ecosystem of libraries. Here are the essential Python libraries for this project:

• Requests: Used to make HTTP requests for retrieving web page content.
• Beautiful Soup: A powerful tool for parsing HTML and XML, facilitating data extraction.

Scraping eBay Product Data With Beautiful Soup

Here’s a step-by-step guide to create your eBay web scraper using Python:

Step 1: Getting Started

Ensure you have Python 3+ installed on your computer, alongside an IDE like Visual Studio Code with the Python extension or PyCharm. To set up your scraping project:

mkdir ebay-scraper
cd ebay-scraper
python -m venv env

Create a scraper.py file inside your project folder. Put in following code to start a project.

print('Hello, World!')

Execute the python script by typing:

python scraper.py

Step 2: Install the Scraping Libraries

Install Beautiful Soup and Requests:

pip install beautifulsoup4 requests

Add imports in scraper.py:

import requests
from bs4 import BeautifulSoup
# scraping logic...

Step 3: Download the Target Web Page

eBay product URLs follow the format:

https://www.ebay.com/itm/<ITEM_ID>

Update scraper.py to read item IDs from command-line arguments using the `sys` module, and construct the eBay URL accordingly.

import requests
from bs4 import BeautifulSoup
import sys

# if there are no CLI parameters
if len(sys.argv) <= 1:
    print('Item ID argument missing!')
    sys.exit(2)

# read the item ID from a CLI argument
item_id = sys.argv[1]

# build the URL of the target product page
url = f'https://www.ebay.com/itm/{item_id}'

# scraping logic...

Run the scraper using a sample product ID with following command:

python scraper.py 225605642071

Add code in scraper.py to use the requests library to download target URL:

page = requests.get(url)

Step 4: Parse the HTML Document

Use Beautiful Soup to parse the HTML content. Here is the snippet code:

soup = BeautifulSoup(page.text, 'html.parser')

soup now stores a tree structure which can be used to get specific HTML elements from the DOM. These are several helpful methods:

• find(): Returns the first element that matches the conditions
• find_all(): Returns a list of elements matching conditions
• select_one(): Returns the first element matched the CSS selector
• select(): Returns a list of elements matched the CSS selector

Step 5: Inspect the Product Page

Explore eBay product pages using your browser's DevTools to understand the DOM structure. You'll see that certain product information may vary by category, but prices are generally consistent, this will help you choose the correct selector when you're going to fetch the data.

Step 6: Extract the Price Data

Start by initializing a dictionary to store scraped data.

item = {}

Find the product price using:

price_html_element = soup.select_one('.x-price-primary span[itemprop="price"]')
price = price_html_element['content']

currency_html_element = soup.select_one('.x-price-primary span[itemprop="priceCurrency"]')
currency = currency_html_element['content']

Extract shipping costs using the sibling element

label_html_elements = soup.select('.ux-labels-values__labels')

for label_html_element in label_html_elements:
    if 'Shipping:' in label_html_element.text:
        shipping_price_html_element = label_html_element.next_sibling.select_one('.ux-textspans--BOLD')
        # if there is a shipping price HTML element
        if shipping_price_html_element is not None:
            # extract the float number of the price from
            # the text content
            shipping_price = re.findall("\\d+[.,]\\d+", shipping_price_html_element.text)[0]
        break

Then assign these price into item dictionary:

item['price'] = price
item['shipping_price'] = shipping_price
item['currency'] = currency
print(item)

Step 7: Retrieve the Item Details

To extract “Item specifics” or “About this product” data:

section_title_elements = soup.select('.section-title')

for section_title_element in section_title_elements:
    if 'Item specifics' in section_title_element.text or 'About this product' in section_title_element.text:
        # get the parent element containing the entire section
        section_element = section_title_element.parent

        for section_col in section_element.select('.ux-layout-section-evo__col'):
            col_label = section_col.select_one('.ux-labels-values__labels')
            col_value = section_col.select_one('.ux-labels-values__values')
            # if both elements are present
            if col_label is not None and col_value is not None:
                item[col_label.text] = col_value.text

Step 8: Export Scraped Data to JSON

Export scraped data into JSON format by the following code.

import json

# scraping logic...

with open('product_info.json', 'w') as file:
    json.dump(item, file, indent=4)

Step 9: Put It All Together

Below is the completed scraper.

import requests
from bs4 import BeautifulSoup
import sys
import re
import json

# if there are no CLI parameters
if len(sys.argv) <= 1:
    print('Item ID argument missing!')
    sys.exit(2)

# read the item ID from a CLI argument
item_id = sys.argv[1]

# build the URL of the target product page
url = f'https://www.ebay.com/itm/{item_id}'

# download the target page
page = requests.get(url)

# parse the HTML document returned by the server
soup = BeautifulSoup(page.text, 'html.parser')

# initialize the object that will contain
# the scraped data
item = {}

# price scraping logic
price_html_element = soup.select_one('.x-price-primary span[itemprop="price"]')
price = price_html_element['content']

currency_html_element = soup.select_one('.x-price-primary span[itemprop="priceCurrency"]')
currency = currency_html_element['content']

shipping_price = None
label_html_elements = soup.select('.ux-labels-values__labels')
for label_html_element in label_html_elements:
    if 'Shipping:' in label_html_element.text:
        shipping_price_html_element = label_html_element.next_sibling.select_one('.ux-textspans--BOLD')
        # if there is not a shipping price HTML element
        if shipping_price_html_element is not None:
            # extract the float number of the price from
            # the text content
            shipping_price = re.findall("\\d+[.,]\\d+", shipping_price_html_element.text)[0]
        break
item['price'] = price
item['shipping_price'] = shipping_price
item['currency'] = currency

# product detail scraping logic
section_title_elements = soup.select('.section-title')
for section_title_element in section_title_elements:
    if 'Item specifics' in section_title_element.text or 'About this product' in section_title_element.text:
        # get the parent element containing the entire section
        section_element = section_title_element.parent

        for section_col in section_element.select('.ux-layout-section-evo__col'):
            col_label = section_col.select_one('.ux-labels-values__labels')
            col_value = section_col.select_one('.ux-labels-values__values')
            # if both elements are present
            if col_label is not None and col_value is not None:
                item[col_label.text] = col_value.text

# export the scraped data to a JSON file
with open('product_info.json', 'w') as file:
    json.dump(item, file, indent=4)

This complete script can now retrieve comprehensive data from any given eBay product, export this into JSON. Run this script with a valid item ID, a `product_info.json` will be created in your current folder.

Conclusion

This guide provides you with a comprehensive understanding of scraping product prices and details from eBay with Python and introduces ProxyTee as a powerful tool for your proxy requirements. With ProxyTee's Unlimited Residential Proxies, you can reliably gather large datasets from eBay with a variety of advanced features including auto-rotation, global IP coverage, and unlimited bandwidth for a high throughput with affordable pricing.

To scale this workflow in production without bandwidth throttling, explore our E-Commerce Price Monitoring Proxies, Python Requests & HTTPX Integration, France Residential Proxies, and ProxyTee vs. IPRoyal Benchmark.