Scraping Dynamic Websites with Scrapy and Playwright Using ProxyTee

Web scraping is a powerful technique for gathering data from websites, and Scrapy has been a go-to Python framework for many. However, when it comes to scraping dynamic websites that heavily rely on JavaScript, Scrapy alone often falls short. That’s where Playwright comes in. Playwright is a headless browser automation tool that excels at rendering JavaScript, making it perfect for scraping modern websites. When combined with Scrapy, you get the best of both worlds: the robust scraping capabilities of Scrapy and the dynamic rendering power of Playwright.

In this post, we’ll walk you through how to effectively use Scrapy with Playwright for scraping JavaScript-heavy sites, all while ensuring your web scraping operations are smooth and undetectable by leveraging ProxyTee’s rotating residential proxies.

Getting Started with Scrapy Playwright

First things first, let's install the necessary libraries:

pip install scrapy-playwright
playwright install

The first command installs the Scrapy Playwright package, while the second installs the necessary browser engines like Chromium, Firefox, and WebKit.

Setting Up a Scrapy Project

Next, let's create a basic Scrapy project and a spider. Run these commands in your terminal:

scrapy startproject scrapy_book_crawler
cd scrapy_book_crawler
scrapy genspider books https://books.toscrape.com/js/

This will generate a new Scrapy project called scrapy_book_crawler and a spider called books for scraping books.toscrape.com, a practice site for web scraping.

Configuring Scrapy Settings

To make Scrapy work with Playwright, we need to make some adjustments in settings.py:

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = (30 * 1000)
PLAYWRIGHT_BROWSER_TYPE = "chromium"

These configurations set Playwright as the download handler, define the timeout for page loading (30 seconds), and specify Chromium as the browser to use.

Creating an Item

In items.py, let’s define a basic item for our scraped data:

import scrapy


class Book(scrapy.Item):
    title = scrapy.Field()
    price = scrapy.Field()

This simple model defines that we will be collecting book titles and prices from the website.

Implementing the Spider Logic

Here’s the core logic for scraping the book details, implemented in spiders/books.py:

import scrapy
from scrapy_playwright.page import PageMethod
from scrapy_book_crawler.items import Book
from scrapy_book_crawler.enums import BookCategory


class BooksSpider(scrapy.Spider):
    name = "books"
    category_number = BookCategory.CLASSICS

    def start_requests(self):
        url = "https://books.toscrape.com/"
        yield scrapy.Request(
            url,
            meta=dict(
                playwright=True,
                playwright_include_page=True,
                playwright_page_methods=[
                    PageMethod(
                        "click",
                        selector=f"div.side_categories > ul > li > ul > li:nth-child({self.category_number.value}) > a",
                    ),
                    PageMethod("wait_for_selector", "article.product_pod"),
                ],
                errback=self.errback,
            ),
        )

    async def parse(self, response):
        page = response.meta["playwright_page"]
        await page.screenshot(path=f"books-{self.category_number.name}.png")
        await page.close()

        for book in response.css("article.product_pod"):
            book = Book(
                title=book.css("h3 a::attr(title)").get(),
                price=book.css("p.price_color::text").get(),
            )
            yield book

    async def errback(self, failure):
        page = failure.request.meta["playwright_page"]
        await page.close()

This spider now uses Playwright to handle dynamic content and also takes a screenshot before parsing book information.

Navigating Through Multiple Pages

For a more complete scrape, it’s helpful to traverse multiple pages. Let’s modify the parse function:

 async def parse(self, response):
        page = response.meta["playwright_page"]
        await page.screenshot(path=f"books-{self.category_number.name}.png")
        await page.close()

        for book in response.css("article.product_pod"):
            book = Book(
                title=book.css("h3 a::attr(title)").get(),
                price=book.css("p.price_color::text").get(),
            )
            yield book
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            next_page_url = response.urljoin(next_page)
            yield scrapy.Request(
                next_page_url,
                meta=dict(
                    playwright=True,
                    playwright_include_page=True,
                    errback=self.errback,
                ),
            )

This modification allows the spider to identify and move to the next page and continues scraping through them all until the end.

Interacting with the Webpage

Playwright not only renders JavaScript, but it also allows for user interactions, for example, you can click on a category on the page. Below is the definition of categories for reference

from enum import Enum


class BookCategory(int, Enum):
    """All category li nth:child numbers for books.toscrape.com"""

    TRAVEL = 1
    MYSTERY = 2
    HISTORICAL_FICTION = 3
    SEQUENTIAL_ART = 4
    CLASSICS = 5
    PHILOSOPHY = 6
    ROMANCE = 7
    WOMENS_FICTION = 8
    FICTION = 9
    CHILDRENS = 10
    RELIGION = 11
    NONFICTION = 12
    MUSIC = 13
    DEFAULT = 14
    SCIENCE_FICTION = 15
    SPORTS_AND_GAMES = 16
    ADD_A_COMMENT = 17
    FANTASY = 18
    NEW_ADULT = 19
    YOUNG_ADULT = 20
    SCIENCE = 21
    POETRY = 22
    PARANORMAL = 23
    ART = 24
    PSYCHOLOGY = 25
    AUTOBIOGRAPHY = 26
    PARENTING = 27
    ADULT_FICTION = 28
    HUMOR = 29
    HORROR = 30
    HISTORY = 31
    FOOD_AND_DRINK = 32
    CHRISTIAN_FICTION = 33
    BUSINESS = 34
    BIOGRAPHY = 35
    THRILLER = 36
    CONTEMPORARY = 37
    SPIRITUALITY = 38
    ACADEMIC = 39
    SELF_HELP = 40
    HISTORICAL = 41
    CHRISTIAN = 42
    SUSPENSE = 43
    SHORT_STORIES = 44
    NOVELS = 45
    HEALTH = 46
    POLITICS = 47
    CULTURAL = 48
    EROTICA = 49
    CRIME = 50

We can enhance the spider to do more. For example, it can now select specific categories via Playwright PageMethods in the `start_requests` function and take screenshots of the rendered content.  This shows how you can take full control of the scraping process.

Using ProxyTee Proxies with Scrapy Playwright

To enhance your anonymity and avoid detection while scraping, using a proxy service is highly recommended. ProxyTee provides rotating residential proxies which are perfect for tasks like this.

ProxyTee offers an affordable, reliable and user-friendly solution. It features include unlimited bandwidth, a global IP pool and support for multiple protocols. Furthermore, ProxyTee offers user-friendly GUI, auto-rotation, and simple API integration. This combination can significantly improve scraping success and reduce the chance of your IP being blocked by the website. Using ProxyTee is ideal for tasks such as web scraping, streaming, data gathering and many more

Here are three ways to integrate proxies:

PLAYWRIGHT_CONTEXTS in settings.py: Configure proxies within the PLAYWRIGHT_CONTEXTS setting:

PLAYWRIGHT_CONTEXTS = {
    "default": {
        "proxy": {
            "server": "127.0.0.1:60000",
            "username": "username",
            "password": "password",
        }
    },
    "alternative": {
        "proxy": {
            "server": "127.0.0.1:60001",
            "username": "username",
            "password": "password",
        }
    },
}

Meta dictionary in start_requests: Use meta to set proxies on a per-request basis:

def start_requests(self):
        url = "https://books.toscrape.com/"
        yield scrapy.Request(
            url,
            meta=dict(
                playwright=True,
                playwright_include_page=True,
                playwright_context_kwargs={
                    "proxy": {
                        "server": "127.0.0.1:60000",
                        "username": "username",
                        "password": "password",
                    }
                },
                errback=self.errback,
            ),
        )

Spider class custom_settings: Define proxy settings within the custom_settings parameter. Example of Datacenter Proxy can be setup here:

custom_settings = {
    "PLAYWRIGHT_LAUNCH_OPTIONS": {
        "proxy": {
            "server": "127.0.0.1:60000",
            "username": "username",
            "password": "password",
        }
    }
}

When selecting proxies, ProxyTee's Unlimited Residential Proxies are an outstanding choice. It features unlimited bandwidth with rotating IPs, ensuring your scraping process will not be interrupted. You can set up to target specific geographical areas or countries, which is much better than simply randomly rotating or rotating by continents. Also, you’ll find their pricing will pleasantly surprise you, making them the most economical of your options as they're often around 50% less expensive than the competition.

Final Notes

By combining Scrapy with Playwright, you can handle almost any web scraping scenario, from simple static sites to very complex, dynamic websites. Also, by using ProxyTee residential proxies, you ensure your scrapers remain undetected. With this set up, you are able to effectively gather public data that’s necessary for analysis and reporting.

Frequently Asked Questions

Is Playwright Good for Scraping?

Yes, especially for JavaScript-heavy websites. It provides excellent support for multiple browser engines with user-friendly APIs for various web automation tasks.

Is Playwright Better than Selenium?

For web scraping, Playwright is often preferred due to its improved speed, user-friendliness and capabilities for modern web automation scenarios

Can Scrapy Parse JavaScript?

No, Scrapy can’t parse JavaScript natively, it depends on integrations like Playwright to handle this. 

What Is the Difference Between Playwright and Scrapy Splash?

Playwright uses a full browser engine, while Splash relies on a webkit embedded browser engine, thus giving Playwright a slight advantage on reliability and performance.

Scale Your Web Scraping with ProxyTee

Ready to eliminate IP bans and per-gigabyte bandwidth overages? Explore ProxyTee Unlimited Residential Proxies, learn how our unlimited bandwidth residential proxy ports work, check our proxies for travel fare aggregation, or test endpoints on our free proxy list.