How to Scrape Reddit with Python: A Complete Guide for 2026

In this comprehensive guide, you'll discover how to scrape Reddit using Python, bypassing the often expensive and restrictive Reddit API. We'll explore the advantages of web scraping over API usage and provide a step-by-step tutorial for setting up your own Reddit scraper.

Introducing ProxyTee: Your Solution for Reddit Data Extraction

Before diving into the technical details, let's briefly introduce ProxyTee. As a leading provider of rotating residential proxies, ProxyTee offers an affordable, reliable, and easy-to-use solution for all your web scraping needs. With features like unlimited bandwidth, a vast global IP pool, and multiple protocol support, ProxyTee enables efficient and anonymous data collection.

Key features of ProxyTee include:

• Unlimited Bandwidth: Scrape without worrying about data caps, ideal for extensive Reddit scraping tasks.
• Global IP Coverage: Access over 30 million IPs from more than 100 countries for geo-targeted scraping.
• Auto Rotation: Avoid detection with IP auto-rotation, which can be customized to change every 3 to 60 minutes, essential for preventing blocks when scraping platforms like Reddit.
• Multiple Protocol Support: Utilize both HTTP and SOCKS5 protocols for versatile scraping needs.
• Simple API: Easily integrate proxy features into applications and automate proxy-related tasks, which allows for seamless integration in the provided guide below.
• User-Friendly Interface: Get started quickly with an intuitive GUI, no technical expertise required.

Why Web Scraping Over the Reddit API?

Reddit's official API, while straightforward, has become increasingly expensive, especially after the 2023 policy changes. Web scraping, on the other hand, provides several advantages:

• Cost-Effectiveness: Build a Reddit scraper in Python without recurring costs associated with API calls. ProxyTee’s affordable pricing makes it the better choice to support your business needs. Check out the ProxyTee pricing for best deals.
• Customization: Tailor data extraction to precisely match your requirements, overcoming the limitations of the API.
• Access to More Data: Get access to all publicly available data on Reddit, not just the information provided by the API.

How to Build a Reddit Scraper in Python

In this step-by-step tutorial, you will see how to build a Reddit web scraping Python script.

Step 1: Project Setup

Before starting, make sure you have Python 3+ installed. You'll also need a Python IDE such as PyCharm Community Edition or Visual Studio Code with the Python extension.

Let's initialize a Python project and create a virtual environment:

mkdir reddit-scraper
cd reddit-scraper
python -m venv env

Now, open your project in the IDE, create a file called scraper.py, and initialize it:

print('Hello, World!')

Run this to verify that it works. You should see “Hello, World!” in the terminal.

Step 2: Install Scraping Libraries

Since Reddit loads data dynamically with JavaScript, you'll need a tool that can render pages in a browser. We will use Selenium for this:

pip install selenium webdriver-manager

The `webdriver-manager` library will automatically handle the web driver installation process, removing the manual work.

Now integrate Selenium into your `scraper.py`:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options

# enable the headless mode
options = Options()
options.add_argument('--headless=new')

# initialize a web driver to control Chrome
driver = webdriver.Chrome(
    service=ChromeService(ChromeDriverManager().install()),
    options=options
)
# maxime the controlled browser window
driver.fullscreen_window()

# scraping logic...

# close the browser and free up the Selenium resources
driver.quit()

This code initializes a Chrome WebDriver, which controls the browser. We're setting the browser to headless mode using the `--headless=new` option to conserve system resources.

Step 3: Connect to Reddit

Let's target the r/Technology subreddit's top posts of the week:

url = 'https://www.reddit.com/r/technology/top/?t=week'
driver.get(url)

The `driver.get(url)` will instruct the browser to connect to the Reddit page.

Step 4: Inspect the Target Page

Before coding the scraper, you should inspect the structure of Reddit in the browser's dev tools and identify the data attributes you would need. Since class names are often randomly generated, look for unique attributes that can be relied upon.

Step 5: Scrape Subreddit Main Info

Let’s scrape the subreddit’s main information into a dictionary. First, declare an empty dictionary `subreddit = {}`. Then, retrieve the name of the subreddit from the <h1> element. Use the `find_element()` method to target that h1 element. Retrieve the subreddit’s description, creation date, and member count from the right-side panel on the site using attributes like “data-testid=no-edit-description-block”, `icon-cake`, and subscriber ids.

subreddit = {}

name = driver \
    .find_element(By.TAG_NAME, 'h1') \
    .text

description = driver \
    .find_element(By.CSS_SELECTOR, '[data-testid="no-edit-description-block"]') \
    .get_attribute('innerText')

creation_date = driver \
    .find_element(By.CSS_SELECTOR, '.icon-cake') \
    .find_element(By.XPATH, "following-sibling::*[1]") \
    .get_attribute('innerText') \
    .replace('Created ', '')

members = driver \
    .find_element(By.CSS_SELECTOR, '[id^="IdCard--Subscribers"]') \
    .find_element(By.XPATH, "preceding-sibling::*[1]") \
    .get_attribute('innerText')

subreddit['name'] = name
subreddit['description'] = description
subreddit['creation_date'] = creation_date
subreddit['members'] = members

print(subreddit) # Test the retrieval of subreddit info

Step 6: Scrape Subreddit Posts

Next, let's collect information about individual posts, starting by defining an empty list named `posts = []`. We’ll select all post container elements via the CSS attribute selector `[data-testid="post-container"]`. Within the for loop, we iterate through each of the posts and create a post dictionary. To get individual data points (e.g. upvotes), we will access elements relative to each of the post container. The retrieved data like author, title, number of comments are then appended to the post array.

posts = []

post_html_elements = driver \
    .find_elements(By.CSS_SELECTOR, '[data-testid="post-container"]')

for post_html_element in post_html_elements:
    post = {}

    upvotes = post_html_element \
        .find_element(By.CSS_SELECTOR, '[data-click-id="upvote"]') \
        .find_element(By.XPATH, "following-sibling::*[1]") \
        .get_attribute('innerText')

    author = post_html_element \
        .find_element(By.CSS_SELECTOR, '[data-testid="post_author_link"]') \
        .text

    title = post_html_element \
        .find_element(By.TAG_NAME, 'h3') \
        .text

    try:
        outbound_link = post_html_element \
            .find_element(By.CSS_SELECTOR, '[data-testid="outbound-link"]') \
            .get_attribute('href')
    except NoSuchElementException:
        outbound_link = None

    comments = post_html_element \
        .find_element(By.CSS_SELECTOR, '[data-click-id="comments"]') \
        .get_attribute('innerText') \
        .replace(' Comments', '')

    post['upvotes'] = upvotes
    post['title'] = title
    post['outbound_link'] = outbound_link
    post['comments'] = comments

    if title:
        posts.append(post)

subreddit['posts'] = posts

Step 7: Export to JSON

Finally, you should save scraped data in a standardized and commonly used file format such as JSON, rather than a python dict object for improved portability.

import json

with open('subreddit.json', 'w', encoding='utf-8') as file:
    json.dump(subreddit, file, indent=4, ensure_ascii=False)

Step 8: Complete Code

Here is the full `scraper.py` script:

from selenium import webdriver
from selenium.common import NoSuchElementException
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
import json

# enable the headless mode
options = Options()
options.add_argument('--headless=new')

# initialize a web driver to control Chrome
driver = webdriver.Chrome(
    service=ChromeService(ChromeDriverManager().install()),
    options=options
)
# maxime the controlled browser window
driver.fullscreen_window()

# the URL of the target page to scrape
url = 'https://www.reddit.com/r/technology/top/?t=week'
# connect to the target URL in Selenium
driver.get(url)

# initialize the dictionary that will contain
# the subreddit scraped data
subreddit = {}

# subreddit scraping logic
name = driver \
    .find_element(By.TAG_NAME, 'h1') \
    .text

description = driver \
    .find_element(By.CSS_SELECTOR, '[data-testid="no-edit-description-block"]') \
    .get_attribute('innerText')

creation_date = driver \
    .find_element(By.CSS_SELECTOR, '.icon-cake') \
    .find_element(By.XPATH, "following-sibling::*[1]") \
    .get_attribute('innerText') \
    .replace('Created ', '')

members = driver \
    .find_element(By.CSS_SELECTOR, '[id^="IdCard--Subscribers"]') \
    .find_element(By.XPATH, "preceding-sibling::*[1]") \
    .get_attribute('innerText')

# add the scraped data to the dictionary
subreddit['name'] = name
subreddit['description'] = description
subreddit['creation_date'] = creation_date
subreddit['members'] = members

# to store the post scraped data
posts = []

# retrieve the list of post HTML elements
post_html_elements = driver \
    .find_elements(By.CSS_SELECTOR, '[data-testid="post-container"]')

for post_html_element in post_html_elements:
    # to store the data scraped from the
    # post HTML element
    post = {}

    # subreddit post scraping logic
    upvotes = post_html_element \
        .find_element(By.CSS_SELECTOR, '[data-click-id="upvote"]') \
        .find_element(By.XPATH, "following-sibling::*[1]") \
        .get_attribute('innerText')

    author = post_html_element \
        .find_element(By.CSS_SELECTOR, '[data-testid="post_author_link"]') \
        .text

    title = post_html_element \
        .find_element(By.TAG_NAME, 'h3') \
        .text

    try:
        outbound_link = post_html_element \
            .find_element(By.CSS_SELECTOR, '[data-testid="outbound-link"]') \
            .get_attribute('href')
    except NoSuchElementException:
        outbound_link = None

    comments = post_html_element \
        .find_element(By.CSS_SELECTOR, '[data-click-id="comments"]') \
        .get_attribute('innerText') \
        .replace(' Comments', '')

    # populate the dictionary with the retrieved data
    post['upvotes'] = upvotes
    post['title'] = title
    post['outbound_link'] = outbound_link
    post['comments'] = comments

    # to avoid adding ad posts 
    # to the list of scraped posts
    if title:
        posts.append(post)

subreddit['posts'] = posts

# close the browser and free up the Selenium resources
driver.quit()

# export the scraped data to a JSON file
with open('subreddit.json', 'w', encoding='utf-8') as file:
    json.dump(subreddit, file, indent=4, ensure_ascii=False)

Run this code, and the file `subreddit.json` will be generated.

Conclusion

Congratulations! You now know how to scrape Reddit with Python using Selenium. This technique gives you much more flexibility over using Reddit's API. However, as Reddit may update their policies to prevent scraping, utilizing tools like ProxyTee can guarantee reliable and consistent data.

With ProxyTee's Unlimited Residential Proxies, you benefit from unlimited bandwidth, a global IP pool, auto-rotation, and more. This is especially important for tasks such as scraping large sites such as Reddit where getting detected and banned from the site is quite common. These features can save you time and allow you to concentrate on data analysis instead of dealing with the complications associated with web scraping. Feel free to check out all the products and features ProxyTee offers.

To scale this workflow in production without bandwidth throttling, explore our Reddit Scraping & Multi-Account Proxies, Python Requests & HTTPX Integration, South Korea Residential Proxies, and ProxyTee vs. Proxy-Seller Benchmark.