Boost Your Web Scraping with Asynchronous Requests Using Python and AIOHTTP

In today's fast-paced digital landscape, businesses rely heavily on data to gain a competitive edge. Web scraping is often the go-to method for gathering this crucial information. However, traditional scraping methods can be slow and inefficient, especially when dealing with numerous web pages. This is where asynchronous web scraping comes in, allowing you to extract data from multiple sources at once. In this post, we'll explore how asynchronous web scraping works using Python and the AIOHTTP library, and why it is more efficient than the traditional synchronous approach. We will use ProxyTee to enhance our scraping efforts, taking advantage of their Unlimited Residential Proxies.

What is Asynchronous Web Scraping?

Asynchronous web scraping is a method that allows you to send multiple requests to web servers simultaneously without waiting for each request to complete before sending the next one. Unlike synchronous scraping, where tasks are executed sequentially, asynchronous scraping executes tasks concurrently, maximizing the speed and efficiency of your data extraction process. This approach is particularly useful when you're dealing with large datasets that require processing of multiple URLs simultaneously.

With libraries like asyncio and AIOHTTP, Python becomes an ideal choice for asynchronous operations, giving you robust tools to handle HTTP requests efficiently.

Sync vs. Async: What's the Difference?

Synchronous web scraping is a method in which one URL is scraped at a time. The code processes one request completely before moving on to the next. This approach can be very time-consuming if you are dealing with a large number of pages.

Asynchronous web scraping, on the other hand, uses non-blocking code. It allows the program to start multiple web scraping processes simultaneously, which means you can send multiple requests without waiting for one to complete before sending another. This concurrent approach significantly speeds up the data extraction process, making it far more time-efficient.

With ProxyTee's unlimited bandwidth and vast global IP coverage, you can ensure that your asynchronous web scraping tasks are both quick and reliable. In the next section, we'll demonstrate how to set up the asynchronous web scraping method using Python and AIOHTTP.

Sending Asynchronous HTTP Requests

Below, we provide the necessary steps to demonstrate the use of the asynchronous method with aiohttp. Each step is shown below:

1. Create an empty python file with a main function

To begin, create an async function which runs until the main task is completed. Here's the code snippet for that:

import asyncio


async def main():
    print('Saving the output of extracted information')


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

2. Track script execution time

Tracking your code execution time is very helpful. You can do this by measuring the time from start and the difference in time after execution.

import asyncio
import time


async def main():
    start_time = time.time()
    print('Saving the output of extracted information')
    time_difference = time.time() - start_time
    print(f'Scraping time: %.2f seconds.' % time_difference)


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

3. Create a loop

For this, we will load the URLs from a csv file into an array. Each url then has an associated task.

import asyncio
import csv
import time


async def main():
    start_time = time.time()
    with open('urls.csv') as file:
        csv_reader = csv.DictReader(file)
        for csv_row in csv_reader:
            print(csv_row['url'])
    print('Saving the output of extracted information')
    time_difference = time.time() - start_time
    print(f'Scraping time: %.2f seconds.' % time_difference)


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

4. Complete the scraping tasks

Here we use the gather function that waits for all scraping tasks to complete.

import asyncio
import csv
import time


async def main():
    start_time = time.time()
    tasks = []
    with open('urls.csv') as file:
        csv_reader = csv.DictReader(file)
        for csv_row in csv_reader:
            task = asyncio.create_task(scrape(csv_row['url']))
            tasks.append(task)
    print('Saving the output of extracted information')
    await asyncio.gather(*tasks)
    time_difference = time.time() - start_time
    print(f'Scraping time: %.2f seconds.' % time_difference)


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

5. Create a scrape functionality

The scrape function makes a request to a specific URL loaded from the CSV file and extracts necessary data:

import asyncio
import csv
import time
import aiohttp as aiohttp
from bs4 import BeautifulSoup


async def scrape(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            body = await resp.text()
            soup = BeautifulSoup(body, 'html.parser')
            book_name = soup.select_one('.product_main').h1.text
            rows = soup.select('.table.table-striped tr')
            product_info = {row.th.text: row.td.text for row in rows}

6. Add save_product function

Once the data is extracted it needs to be saved somewhere. The code below does just that. For this the function needs the book name, which will be used for the name of a JSON file, and the product info.

import asyncio
import csv
import json
import time
import aiohttp
from bs4 import BeautifulSoup


async def save_product(book_name, product_info):
    json_file_name = book_name.replace(' ', '_')
    with open(f'data/{json_file_name}.json', 'w') as book_file:
        json.dump(product_info, book_file)


async def scrape(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            body = await resp.text()
            soup = BeautifulSoup(body, 'html.parser')
            book_name = soup.select_one('.product_main').h1.text
            rows = soup.select('.table.table-striped tr')
            product_info = {row.th.text: row.td.text for row in rows}
            await save_product(book_name, product_info)


async def main():
    start_time = time.time()
    tasks = []
    with open('urls.csv') as file:
        csv_reader = csv.DictReader(file)
        for csv_row in csv_reader:
            task = asyncio.create_task(scrape(csv_row['url']))
            tasks.append(task)
    print('Saving the output of extracted information')
    await asyncio.gather(*tasks)
    time_difference = time.time() - start_time
    print(f'Scraping time: %.2f seconds.' % time_difference)


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

7. Run the script

With the implementation of each part, the scraping process is now complete and data should be scraped and saved as JSON files.

Sending Synchronous HTTP Requests

Now, let's walk through the steps to show the scraping using synchronous requests. This example uses the requests module. Similar to asynchronous requests above, it's split up into several sections, each with an important role.

1. Create a Python file with a main function

Here's an empty main function:

def main():
    print('Saving the output of extracted information')


main()

2. Track script execution time

Add this code snippet to see how long the code takes:

import time


def main():
    start_time = time.time()
    print('Saving the output of extracted information')
    time_difference = time.time() - start_time
    print(f'Scraping time: %.2f seconds.' % time_difference)


main()

3. Create a loop

Similar to the async example, we will read and store each url:

import csv
import time


def main():
    start_time = time.time()
    print('Saving the output of extracted information')
    with open('urls.csv') as file:
        csv_reader = csv.DictReader(file)
        for csv_row in csv_reader:
            print(csv_row['url'])
    time_difference = time.time() - start_time
    print(f'Scraping time: %.2f seconds.' % time_difference)


main()

4. Create a scrape function

In this section the requests module will be used to scrape all of the data needed:

import csv
import time
import requests as requests
from bs4 import BeautifulSoup


def scrape(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')
    book_name = soup.select_one('.product_main').h1.text
    rows = soup.select('.table.table-striped tr')
    product_info = {row.th.text: row.td.text for row in rows}


def main():
    start_time = time.time()
    print('Saving the output of extracted information')
    with open('urls.csv') as file:
        csv_reader = csv.DictReader(file)
        for csv_row in csv_reader:
            scrape(csv_row['url'])
    time_difference = time.time() - start_time
    print(f'Scraping time: %.2f seconds.' % time_difference)


main()

5. Add save_product function

The final piece is to save the information. Below we use json for saving:

import csv
import json
import time
import requests
from bs4 import BeautifulSoup


def save_product(book_name, product_info):
    json_file_name = book_name.replace(' ', '_')
    with open(f'data/{json_file_name}.json', 'w') as book_file:
        json.dump(product_info, book_file)


def scrape(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')
    book_name = soup.select_one('.product_main').h1.text
    rows = soup.select('.table.table-striped tr')
    product_info = {row.th.text: row.td.text for row in rows}
    save_product(book_name, product_info)


def main():
    start_time = time.time()
    print('Saving the output of extracted information')
    with open('urls.csv') as file:
        csv_reader = csv.DictReader(file)
        for csv_row in csv_reader:
            scrape(csv_row['url'])
    time_difference = time.time() - start_time
    print(f'Scraping time: %.2f seconds.' % time_difference)


main()

6. Run the script

With everything in place, the code can now be executed and the data should be scraped and saved to the designated json files.

Comparing the Performance of Sync and Async

Upon executing both scripts, a significant performance difference will be noticeable. While the asynchronous web scraping code efficiently completes tasks in just a few seconds, the synchronous one will take a much longer time. The results emphasize the importance of asynchronous HTTP requests, due to their noticeable efficiency, especially with multiple URLs that need to be scraped.

To enhance the performance of your web scraping further, consider using ProxyTee's Residential Proxies. By rotating IP addresses frequently through our auto-rotation feature and using affordable plans, you can prevent blocks and achieve high reliability during web scraping. ProxyTee's residential proxies, featuring a large pool of global IPs and excellent speeds, will definitely help to improve your web scraping needs. By using Unlimited Residential Proxies, you can get high speed and unlimited bandwidth.

Wrapping Up

Collecting high-quality public data requires effective scraping techniques. Asynchronous web scraping can drastically improve scraping speeds and performance and should be considered. As this post has shown, by making use of libraries like asyncio and aiohttp, the web scraping process is greatly enhanced and leads to a more performant program. With additional solutions such as those provided by ProxyTee, you can guarantee even greater speeds and performance in web scraping.

To scale this workflow in production without bandwidth throttling, explore our High-Concurrency Data Scraping Proxies, Python Async & HTTPX Integration, India Residential Proxies, and ProxyTee vs. Rayobyte Benchmark.