Serverless Web Scraping with Scrapy and AWS Lambda using ProxyTee

In today's data-driven world, effective web scraping is crucial for various tasks, from market research to data analysis. However, managing a fleet of servers to handle these operations can be both cumbersome and expensive. This is where serverless scraping frameworks come into play, and integrating a web crawling framework like Scrapy with a serverless platform like AWS Lambda can streamline the process while making it more cost effective. This article explores how to use Scrapy, a popular Python framework for web scraping, with AWS Lambda, a serverless computing service by Amazon. Additionally, we'll showcase how ProxyTee can greatly enhance your web scraping projects, providing you with a powerful toolkit for robust, scalable, and cost-effective web scraping solutions.

What is Serverless Web Scraping?

Serverless web scraping uses serverless computing resources (such as AWS Lambda) with a web crawling framework (like Scrapy) to efficiently extract data from the web. By combining them, developers can create robust, scalable, and cost-effective web scraping pipelines without the overhead of server management. It eliminates the need to manage any servers and the associated costs of idle time. This setup enables the applications to dynamically adjust to different load, scaling up during periods of high demand and reducing when it's not needed, which makes it a smart choice for scraping projects with unpredictable traffic or large data extractions. Furthermore, leveraging ProxyTee’s vast pool of residential IPs, unlimited bandwidth, and easy integration, makes this approach even more efficient for handling various data collection needs.

What is AWS Lambda?

AWS Lambda is a serverless computing service from Amazon that allows you to run code without managing servers. Lambda automatically manages all the underlying infrastructure required to execute code and handles all the allocation and utilization of resources, ensuring that it performs at the best level possible. This means that the infrastructure scales based on needs, scaling up during demand and scaling back down when it's not required, which allows for efficient resource management and is great for projects that have erratic demand. To optimize your scraping projects, consider using ProxyTee alongside AWS Lambda for reliable and secure operations.

What is Scrapy?

Scrapy is an open-source and collaborative web crawling framework built in Python. It's made to handle various web scraping tasks and process the extracted data. It includes functionality to extract and store information in structured formats, making Scrapy a comprehensive framework for all things web scraping. For complex tasks that need data from various sites, it pairs up well with the advanced proxy solution that ProxyTee offers. This helps avoid being blocked by websites while scraping.

To enhance web scraping with Scrapy, ProxyTee’s Residential Proxies or Datacenter Proxies can be used. By using proxies, your actual IP is hidden from the sites you’re scraping. This protects your privacy and prevents bans from websites which often try to block automated tools used to extract their data.

How to Run Scrapy in AWS Lambda

This section details the necessary steps to run a Scrapy crawler with ProxyTee proxies, as an AWS Lambda function.

Step 1: Creating a Scrapy Spider

First, set up a Scrapy crawler, using the following steps:

1. Create a new file named games.py, then copy and paste the following code that will scrape the game products from a demo e-commerce site:

Install Scrapy for your Python environment via the command line:

pip install scrapy

import scrapy

class GamesSpider(scrapy.Spider):
    name = "games"
    allowed_domains = ["sandbox.oxylabs.io"]
    start_urls = ["https://sandbox.oxylabs.io/products"]

    def parse(self, response):
        for card in response.css(".product-card"):
            yield {
                "title": card.css("h4::text").get(),
                "price": card.css(".price-wrapper::text").get(),
            }
        next_page = response.css("li.next > a::attr(href)").get()
        if next_page:
            yield scrapy.Request(response.urljoin(next_page))

This spider starts at https://sandbox.oxylabs.io/products and extracts the title and price of every game and moves to the next page until the final result of around 3,000 scraped game products is extracted. If running the spider on your local machine, you can use the -o switch to direct output to a file using this command:

scrapy runspider games.py -o games.json

Step 2: Modifying Scrapy Spider for AWS

Since the file system is not accessible with Scrapy running as an AWS Lambda function, the output must be sent to an S3 bucket. To make this change, you should have an AWS account already set up. Here’s the code you’ll need, with the S3 bucket included:

class GamesSpider(scrapy.Spider):
    name = "games"
    allowed_domains = ["sandbox.oxylabs.io"]
    start_urls = ["https://sandbox.oxylabs.io/products"]
    custom_settings = {
        "FEED_URI": "s3://YOUR-BUCKET-NAME/items.json",
        "FEED_FORMAT": "json",
    }
    # rest of the code

Replace YOUR-BUCKET-NAME with the actual name of your S3 bucket.

Step 3: Preparing the Environment for Lambda Function

Before you can setup your local environment for AWS lambda, you should have installed the following executable:

• Docker
• AWS CLI
• Serverless Framework
• boto3 package

AWS Lambda requires a Docker image of your Scrapy spider which needs to be uploaded to the AWS platform. You’ll use Docker to package an app with all of its dependencies, which ensures the app will work the same regardless of the environment it's running on. For setup, download Docker Desktop. AWS CLI is a command line interface tool to work with the AWS ecosystem, install it by following the official AWS documentation page. The Serverless Framework can be installed by using npm which is already installed if you have node.js installed. Install it using node.js website.. Use the following command once it's done.

npm install -g serverless

The botocore library is also necessary to enable interaction with AWS services, to install it execute the following commands. You need to activate the virtual env first before using the following commands.

(venv) $ pip install boto3
(venv) $ pip install scrapy

Step 4: Prepare the Code for Lambda Function Deployment

To maintain all necessary dependencies of Scrapy, a requirements.txt file must be created. It may look similar to the following example:

requests==2.31.0
requests-file==1.5.1
Scrapy==2.6.0
boto3==1.28.14
service-identity==21.1.0
cryptography==38.0.4
# other packages

Next you need to create the Docker file. Name it Dockerfile, paste the following into the file:

FROM public.ecr.aws/lambda/python:3.10

# Required for lxml
RUN yum install -y gcc libxml2-devel libxslt-devel

COPY . ${LAMBDA_TASK_ROOT}

RUN pip3 install -r requirements.txt

CMD ["lambda_function.handler"]

Here you can see that this is where the python image, all files from app, python dependencies, and command is being set up. A python file for lambda execution also needs to be created called lambda_function.py which contains the following code:

import sys

def handler(event, context):
    # Run the Scrapy spider
    import subprocess
    subprocess.run(["scrapy", "runspider", "games.py"])
    return {
        'statusCode': '200',   # a valid HTTP status code
        'body': 'Lambda function invoked',
    }

The last step is a YML configuration file needed for the deployment. Create serverless.yml file and copy this content in.

service: scrapy-lambda

provider:
  name: aws
  runtime: python3.9
  stage: dev
  region: us-east-1
  environment:
    BUCKET: my-bucket
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "s3:*"
      Resource: "arn:aws:s3:::${self:provider.environment.BUCKET}/*"

functions:
  scrapyFunction:
    image: YOUR_REPO_URI:latest
    events:
      - http:
          path: scrape
          method: post
          cors: true

Note that we'll update YOUR_REPO_URI in the next steps and that scrapy-lambda is simply the name of the image.

Step 5: Deploying Docker Image

Use AWS IAM to generate user access credentials which are needed to push to ECR. Execute the command below and paste the access keys that were generated earlier:

aws configure

Create ECR by executing this command.

$ aws ecr create -repository --repository-name YOUR_REPO_NAME

Copy the repositoryUri which looks like 76890223446.dkr.ecr.us-east-1.amazonaws.com/scrapy-images and replace YOUR_REPO_URI with it in serverless.yml. Use the console if you still have not created a S3 bucket.

Once that is done update YOUR-BUCKET-NAME in games.py with your s3 bucket name.

Now that that's done you should build your docker image using the command

$ docker build -t scrapy-lambda

Tag and push using these commands.

$ aws ecr get-login-password --region region | docker login --username AWS --password-stdin YOUR_REPO_URI
$ docker tag scrapy-lambda:latest YOUR_REPO_URI:latest
$ docker push YOUR_REPO_URI:latest

Replace region with the AWS region and YOUR_REPO_NAME with the Amazon ECR repository. Once this is complete, the images can be deployed with this command.

$ sls deploy

Step 6: Running the Lambda Function

When you run the deploy command you will see the generated service URL in the output. Send a POST request to the url shown with this command:

curl -X POST https://abcde.execute-api.us-east-1.amazonaws.com/dev/scrape

This will start the execution of the Lambda function and output will be put in your S3 bucket.

How to Debug Python Code in AWS Lambda

You can use these ways to help with debugging the python code in AWS:

• Deploy after each adjustment: Each step during the integration or changes in configuration should be followed up with sls deploy, that way issues will be caught earlier. To speed things up local testing can be done with sls invoke local.
• Using logging: Set up logging using print or the logging module in your python script. The outputs from here will automatically be streamed to CloudWatch, under the /aws/lambda/<function-name>. Inspecting this area is needed as more logs will mean more costs.
• Invoking Lambda function with logs: With logging enabled, sls invoke -f <function-name> --log can be used to run functions, retrieve, and display logs.

ProxyTee: Your Partner in Efficient Web Scraping

For those who are involved in heavy duty web scraping projects, choosing a good proxy service is as crucial as the framework chosen. ProxyTee stands out in its ability to offer unlimited bandwidth, a wide selection of over 20 million residential IP addresses and the ability to specify which geo you're targeting, something that is difficult to accomplish with our competitors. This allows the usage to make sure no restrictions or detection is done when scraping any kind of web resources. For more information on Residential Proxies and how it will help your web scraping efforts visit ProxyTee to find out how it will elevate your web scraping experience.

Integrating a robust proxy provider like ProxyTee with your serverless Scrapy setups ensures a robust, reliable, and scalable solution for all your data collection tasks. ProxyTee will handle all of your heavy needs by utilizing its unlimited bandwidth feature, coupled with its diverse set of IP addresses around the globe at a low price, compared to our competitors. It has an easy-to-use and clean GUI for non-tech-savvy people, and API integration, as seen in our simple API which allows flexibility for every type of project.

With our Unlimited Residential Proxies you can expect high-level quality with a rotating list of residential proxies that can target specific countries, as opposed to the common competitor trend of random IPs or continent selection. Our solutions at ProxyTee are cost effective too at almost 50% compared to some other providers and help with efficient and low-cost web scraping efforts. You can see all the plans in our Pricing page. This all makes ProxyTee the preferred partner when it comes to serverless scraping needs.

Conclusion

In this post, we've delved into serverless web scraping using Scrapy and AWS Lambda, covering necessary prerequisites, Scrapy crawler setup, and AWS Lambda configuration for serverless scraping along with S3 storage setup. These steps will enable you to collect data efficiently with a cost effective approach.

For additional topics, be sure to check our other blog posts related to serverless web scraping and using other technologies in your projects such as Splash and Puppeteer on AWS Lambda. Feel free to reach out via live chat or by emailing hello@proxytee.com for any further assistance.