How to Scrape AliExpress Data with ProxyTee: A Comprehensive Guide

AliExpress is a leading global e-commerce platform, offering a vast range of products, from electronics to fashion. The data available on AliExpress, including product pricing, availability, and customer reviews, makes it an invaluable resource for businesses and researchers. In this post, we’ll explore how you can leverage ProxyTee's powerful residential proxies to scrape AliExpress effectively.

Using Python with ProxyTee's capabilities, you can access crucial information that will enable you to conduct extensive market research and make strategic decisions.

What Data Can You Scrape From AliExpress?

With ProxyTee and Python, extracting the following data from AliExpress becomes manageable:

• Current Prices
• Original Prices
• Product Specifications
• Number of Sold Items
• Customer Ratings
• Product Reviews
• Product Images
• Delivery Information
• Related Products
• Seller Information and Products

Leveraging ProxyTee for Effective AliExpress Scraping

While web scraping offers significant benefits, it often encounters challenges such as CAPTCHAs and IP blocks. This is where ProxyTee shines, providing a robust infrastructure with auto-rotating residential proxies to avoid detection and ensure seamless scraping.

ProxyTee is known for its affordable and efficient rotating residential proxies, ideal for various online activities, including web scraping. ProxyTee provides unlimited bandwidth, a large global pool of IPs, and is very easy to use.

Key features that makes ProxyTee suitable for web scraping

• Unlimited Bandwidth: Scrape data without limits.
• Extensive IP pool: Over 20 million residential IP addresses, allows geo-targeting as you need, which is ideal for gathering data for location specific products or customer analysis.
• Auto Rotation: The IP addresses can automatically rotate between 3 to 60 minutes, avoiding blocks from targeted websites.
• Multiple Protocols: Supports both HTTP and SOCKS5, adapting with different tools for tasks such as web scraping and geo-bypassing.
• User-friendly interface: Simple design to allows users to quickly adapt the platform with very little to no setup time, as well as little to no technical skills required.
• Simple API: Enables seamless integrations and automation of the proxy related tasks, specially convenient for developers.

Step-by-Step Guide: Scraping AliExpress Search Results with Python and ProxyTee

Let’s walk through scraping AliExpress search results. This will fetch product details such as prices, ratings, and the number of sales, which you can utilize to further gather product-specific information.

1. Set Up Python and Libraries: Make sure you have Python set up on your system. Install the 'requests' library via pip.
2. python -m pip install requests
3. Store Your API Credentials: Assign your API username and password for accessing the ProxyTee API.
4. Define Your Search Query: Pick a keyword to use for your AliExpress search.
5. Construct the API Payload: Define a JSON object that will contain your scraping instructions.The payload will also specify scrolling instructions and use ProxyTee’s auto-rotating feature to effectively gather data, which simulates human browsing to prevent detection by anti-scraping algorithms.
6. Form the Search URLs Dynamically: Utilize the ProxyTee API to dynamically create AliExpress search page URLs based on your needs, allowing you to scrape a vast variety of pages without limitation.
7. Send requests: Submit your search parameters along with your credentials. ProxyTee’s extensive residential proxy network helps ensure uninterrupted access to data, avoiding blocks and maintaining speed, which will collect all the required information by parsing the HTML response and extracting product information.
8. Save to CSV: Convert the gathered results into an easily digestible CSV format.

The Code for Scraping AliExpress Search Results

Here’s the Python code that encapsulates all the points listed above.

import requests, json, csv

API_credentials = ('USERNAME', 'password')

keyword = 'desktop cpu'
number_of_pages = 3

payload = {
    'source': 'universal',
    'url': None,
    'geo_location': 'United States',
    'locale': 'en-us',
    'user_agent_type': 'desktop',
    'render': 'html',
    'browser_instructions': [
        {
            'type': 'scroll',
            'x': 0,
            'y': 3000,
            'wait_time_s': 2
        }
    ] * 3,
    'parse': True,
    'parsing_instructions': {
        'products': {
            '_fns': [
                {
                    '_fn': 'xpath',
                    '_args': ['//div[@id="card-list"]/div']
                }
            ],
            '_items': {
                'Title': {
                    '_fns': [
                        {
                            '_fn': 'xpath_one',
                            '_args': ['.//h3/text()']
                        }
                    ]
                },
                'Price current': {
                    '_fns': [
                        {
                            '_fn': 'xpath',
                            '_args': ['.//div[contains(@class, "price-sale")]']
                        }
                    ],
                    '_items': {
                        '_fns': [
                            {'_fn': 'xpath', '_args': ['.//span/text()']},
                            {'_fn': 'join', '_args': ''}
                        ]
                    }
                },
                'Price original': {
                    '_fns': [
                        {
                            '_fn': 'xpath',
                            '_args': ['.//div[contains(@class, "price-original")]/span/text()']
                        }
                    ]
                },
                'URL': {
                    '_fns': [
                        {
                            '_fn': 'xpath_one',
                            '_args': ['.//a/@href']
                        },
                        {
                            '_fn': 'regex_find_all',
                            '_args': [r'^\/\/(.*?)(?=\?)']
                        }
                    ]
                }
            }
        }
    }
}

page = f'https://www.aliexpress.us/w/wholesale-{keyword.replace(" ", "-")}.html'
urls = [f'{page}?page={page_num}' for page_num in range(1, number_of_pages + 1)]

data = []

for url in urls:
    payload['url'] = url
    response = requests.request(
        'POST',
        'https://realtime.oxylabs.io/v1/queries',
        auth=API_credentials,
        json=payload
    )
    data.extend(response.json()['results'][0]['content']['products'])

fieldnames = [key for key in data[0].keys() if key]

with open(f'search_{keyword.replace(" ", "-")}.csv', 'w') as f:
    writer = csv.DictWriter(f, fieldnames=fieldnames)
    writer.writeheader()
    for item in data:
        cleaned_item = {key: ', '.join(map(str, value)) if isinstance(value, list) else value for key, value in item.items()}
        filtered_item = {key: value for key, value in cleaned_item.items() if key}
        writer.writerow(filtered_item)

Scraping AliExpress Product Pages

Here are steps to help you further scrape product pages:

1. Import libraries: start by importing the necessary Python libraries.
2. Store Your API Credentials: Assign your API username and password for accessing the ProxyTee API.
3. Add product URLs to a list: Input all the links that you intend to scrape
4. Create the API Payload: In order to collect product specifications such as title, price, sold amount, reviews and specifications. Use the “click” instruction in the ProxyTee platform, so the tool can collect more specification.
5. Send requests: ProxyTee’s easy to use API allows submitting the product parameters, so you can obtain the data while bypassing restrictions that can affect your data gathering.
6. Save to CSV: Output all the data collected into a CSV document for further processing.

Code to Scrape AliExpress Product Pages

Below is the comprehensive code for scraping AliExpress product pages using ProxyTee and Python:

import requests, csv

API_credentials = ('USERNAME', 'password')

products = [
    'https://www.aliexpress.us/item/3256806291837346.html',
    'https://www.aliexpress.us/item/2251832704771713.html',
    'https://www.aliexpress.us/item/3256805974680622.html'
]

payload = {
    'source': 'universal',
    'url': None,
    'geo_location': 'United States',
    'locale': 'en-us',
    'user_agent_type': 'desktop',
    'render': 'html',
    'browser_instructions': [
        {
            'type': 'click',
            'selector': {
                'type': 'xpath',
                'value': '//div[@data-pl="product-specs"]//button'
            }
        }
    ],
    'parse': True,
    'parsing_instructions': {
        'Title': {
            '_fns': [
                {
                    '_fn': 'xpath_one',
                    '_args': ['//h1[@data-pl="product-title"]/text()']
                }
            ]
        },
        'Price current': {
            '_fns': [
                {
                    '_fn': 'xpath',
                    '_args': ['//div[contains(@class, "product-price-current")]']
                }
            ],
            '_items': {
                '_fns': [
                    {'_fn': 'xpath', '_args': ['.//span/text()']},
                    {'_fn': 'join', '_args': ''}
                ]
            }
        },
        'Price original': {
            '_fns': [
                {
                    '_fn': 'xpath_one',
                    '_args': ['//span[contains(@class, "price--original")]/text()']
                }
            ]
        },
        'Discount': {
            '_fns': [
                {
                    '_fn': 'xpath_one',
                    '_args': ['//span[contains(@class, "price--discount")]/text()']
                }
            ]
        },
        'Sold': {
            '_fns': [
                {
                    '_fn': 'xpath_one',
                    '_args': ['//div[@data-pl="product-reviewer"]//span[contains(text(), "sold")]/text()']
                },
                {'_fn': 'amount_from_string'}
            ]
        },
        'Rating': {
            '_fns': [
                {
                    '_fn': 'xpath_one',
                    '_args': ['//div[@data-pl="product-reviewer"]//strong/text()']
                },
                {'_fn': 'amount_from_string'}
            ]
        },
        'Reviews count': {
            '_fns': [
                {
                    '_fn': 'xpath_one',
                    '_args': ['//a[@href="#nav-review"]/text()']
                },
                {'_fn': 'amount_from_string'}
            ]
        },
        'Delivery': {
            '_fns': [
                {
                    '_fn': 'xpath_one',
                    '_args': ['//div[contains(@class, "dynamic-shipping")]//strong/text()']
                }
            ]
        },
        'Specifications': {
            '_fns': [
                {
                    '_fn': 'xpath',
                    '_args': ['//ul[contains(@class, "specification--list")]//li/div']
                }
            ],
            '_items': {
                'Title': {
                    '_fns': [
                        {
                            '_fn': 'xpath_one',
                            '_args': ['.//div[contains(@class, "title")]//text()']
                        }
                    ]
                },
                'Description': {
                    '_fns': [
                        {
                            '_fn': 'xpath_one',
                            '_args': ['.//div[contains(@class, "desc")]//text()']
                        }
                    ]
                }
            }
        }
    }
}

data = []

for url in products:
    payload['url'] = url
    response = requests.request(
        'POST',
        'https://realtime.oxylabs.io/v1/queries',
        auth=API_credentials,
        json=payload
    )
    result = response.json()['results'][0]['content']
    specifications = []
    for spec in result['Specifications']:
        string = f'{spec["Title"]}: {spec["Description"]}';
        specifications.append(string)
    result['Specifications'] = ';\n '.join(specifications)
    result['URL'] = url
    data.append(result)

fieldnames = [key for key in data[0].keys() if key]

with open('products.csv', 'w') as f:
    writer = csv.DictWriter(f, fieldnames=fieldnames)
    writer.writeheader()
    for item in data:
        cleaned_item = {key: ', '.join(map(str, value)) if isinstance(value, list) else value for key, value in item.items()}
        writer.writerow(cleaned_item)

Scraping AliExpress Product Reviews

Here are the simplified steps on gathering all product reviews

1. Import libraries: Start by importing libraries such as request, re (regular expressions) and json
2. Store your credentials: Include your ProxyTee user name and password.
3. Form the Reviews URL: This involves generating a request for all product reviews, by extracting product id from product URL using regex and combining the needed parameters.
4. Send a request: Submit a request via the ProxyTee API, including product URL with parameters, using the global proxy coverage.
5. Parse reviews from JSON response: This processes the reviews data, making them available for download into a csv file
6. Save to CSV: Save parsed information to an easy-to-read csv file for further processing.

Python Code for Scraping AliExpress Reviews

The following code demonstrates scraping of product reviews from AliExpress:

import requests, re, json, csv

API_credentials = ('USERNAME', 'password')

url = 'https://www.aliexpress.us/item/3256805974680622.html'

max_reviews = 100

product_id = re.match(r'.*/(\d+)\.html$', url).group(1)

reviews_url = f'https://feedback.aliexpress.com/pc/searchEvaluation.do?productId={product_id}&lang=en_US&country=US&pageSize={max_reviews}&filter=all&sort=complex_default'

response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=API_credentials,
    json={
        'source': 'universal',
        'url': reviews_url,
        'geo_location': 'United States',
        'user_agent_type': 'desktop'
    }
)

results = response.json()['results'][0]['content']
data = json.loads(results)

parsed_reviews = []

for review in data['data']['evaViewList']:
    parsed_review = {
        'Rating': review.get('buyerEval', ''),
        'Date': review.get('evalDate', ''),
        'Feedback_translated': review.get('buyerTranslationFeedback', ''),
        'Feedback': review.get('buyerFeedback', ''),
        review.get('reviewLabel1', ''): review.get('reviewLabelValue1', ''),
        review.get('reviewLabel2', ''): review.get('reviewLabelValue2', ''),
        review.get('reviewLabel3', ''): review.get('reviewLabelValue3', ''),
        'Name': review.get('buyerName', ''),
        'Country': review.get('buyerCountry', ''),
        'Upvotes': review.get('upVoteCount', ''),
        'Downvotes': review.get('downVoteCount', '')
    }
    parsed_reviews.append(parsed_review)

fieldnames = [key for key in parsed_reviews[0].keys() if key]

with open('reviews.csv', 'w') as f:
    writer = csv.DictWriter(f, fieldnames=fieldnames)
    writer.writeheader()
    for item in parsed_reviews:
        filtered_item = {key: value for key, value in item.items() if key}
        writer.writerow(filtered_item)

Scraping Top Selling Products on AliExpress

You may want to know how to collect product from top-selling pages. Here’s how to get it done.

1. Import libraries: Begin by importing essential libraries like 'requests' and 'csv' to make the data scraping more organized
2. Store Your API Credentials: Input your username and password from the ProxyTee dashboard to use its web scraping features.
3. Create API Payload: Configure settings, and scroll features so that ProxyTee's automatic rotation proxy tool can work smoothly to deliver the data without being affected by the site’s anti scraping measures.
4. Send a request to API: Make a request with the configured parameters using ProxyTee API’s, making sure that all products are returned as output.
5. Save to CSV: Output all data in the easy to read csv format.

Code to scrape Top Selling Products

Here is the full Python code that combines all steps listed above for the top-selling products section.

import requests, csv

API_credentials = ('USERNAME', 'password')

payload = {
    'source': 'universal',
    'url': 'https://www.aliexpress.com/p/calp-plus/index.html?&categoryTab=us_phones_%2526_telecommunications',
    'geo_location': 'United States',
    'locale': 'en-us',
    'user_agent_type': 'desktop',
    'render': 'html',
    'browser_instructions': [
        {
            "type": "scroll",
            "x": 0,
            "y": 2400,
            "wait_time_s": 2
        }
    ] * 19,
    'parse': True,
    'parsing_instructions': {
        'products': {
            '_fns': [
                {
                    '_fn': 'xpath',
                    '_args': ['//div[@data-spm="prodcutlist"]/div']
                }
            ],
            '_items': {
                'Title': {
                    '_fns': [
                        {
                            '_fn': 'xpath_one',
                            '_args': ['.//h3/text()']
                        }
                    ]
                },
                'Price current': {
                    '_fns': [
                        {
                            '_fn': 'xpath',
                            '_args': ['.//div[@class="U-S0j"]']
                        }
                    ],
                    '_items': {
                        '_fns': [
                            {'_fn': 'xpath', '_args': ['.//span/text()']},
                            {'_fn': 'join', '_args': ''}
                        ]
                    }
                },
                'Price original': {
                    '_fns': [
                        {
                            '_fn': 'xpath_one',
                            '_args': ['.//div[@class="_1zEQq"]/span/text()']
                        }
                    ]
                },
                'Sales amount': {
                    '_fns': [
                        {
                            '_fn': 'xpath_one',
                            '_args': ['.//span[@class="Ktbl2"]/text()']
                        }
                    ]
                },
                'URL': {
                    '_fns': [
                        {
                            '_fn': 'xpath_one',
                            '_args': ['.//a/@href']
                        },
                        {
                            '_fn': 'regex_find_all',
                            '_args': [r'^\/\/(.*?)(?=\?)']
                        }
                    ]
                }
            }
        }
    }
}

data = []

response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=API_credentials,
    json=payload,
)
data.extend(response.json()['results'][0]['content']['products'])

fieldnames = [key for key in data[0].keys()]

with open('top_selling.csv', 'w') as f:
    writer = csv.DictWriter(f, fieldnames=fieldnames)
    writer.writeheader()
    for item in data:
        cleaned_item = {key: ', '.join(map(str, value)) if isinstance(value, list) else value for key, value in item.items()}
        writer.writerow(cleaned_item)

Conclusion

This tutorial has taken you through all the steps needed to extract data from different pages on the AliExpress website. All of this can be achieved by implementing ProxyTee’s feature rich service.

To fully resemble human traffic during scraping activities and keep your data gathering practices stable, using reliable residential proxies from ProxyTee is crucial. ProxyTee delivers a large residential IP pool to meet all of your scraping needs at an affordable price and with ease of integration. Check out ProxyTee’s pricing for great deals, compared to all its competitors.

For more information about ProxyTee’s feature-set check the website or get in contact for immediate support.