How to Scrape Indeed Job Postings with Python Using ProxyTee

Indeed is a valuable resource for job seekers and recruiters alike, offering a vast database of job listings. For those in need of large-scale data, manually collecting this information can be extremely time-consuming. This guide will show you how to scrape job listings from Indeed using Python with ProxyTee, and how to effectively navigate anti-bot challenges. ProxyTee offers unlimited residential proxies with global coverage and auto-rotation, ensuring seamless and efficient web scraping.

What Is an Indeed Scraper?

An Indeed scraper is a tool that automates the extraction of job postings and related information from the Indeed website. This involves simulating human interactions on the site, navigating through search pages, and pinpointing the key data points such as job titles, company names, locations, and descriptions. Once extracted, this data can be organized and used for various purposes such as market analysis, competitive intelligence, and recruitment.

Data You Can Find on Indeed

Indeed provides access to a range of crucial job-related data, valuable for anyone looking to understand the employment landscape. Key data points you can scrape include:

• Job titles: The role being advertised.
• Company names: Details about the hiring companies.
• Locations: The city, state, or country where the job is located.
• Job descriptions: Details of job responsibilities and requirements.
• Salary ranges: The pay scale, if provided.
• Job types: Such as full-time, part-time, contract, or internship.
• Posting dates: When the job was originally listed.
• Tags and attributes: Keywords such as “Remote” or “Urgently Hiring”.
• Ratings and reviews: Employer ratings and employee feedback.
• Application options: If an "Easy Apply" option is available.

How to Scrape Indeed: Step-By-Step Guide

This tutorial will walk you through building a Python script to scrape job listings from Indeed. We'll focus on extracting data from the "data scientist" job postings page, while leveraging the power of ProxyTee to ensure reliable access.

Step #1: Project Setup

Ensure that you have Python 3 installed. Create a project directory:

mkdir indeed_scraper

Move into the new directory and create a virtual environment:

cd indeed_scraper
python -m venv env

Activate the environment, using:

./env/bin/activate # On Linux/macOS
env\Scripts\activate # On Windows

Finally, create a scraper.py file in your project.

Step #2: Choose the Right Scraping Library

Indeed dynamically loads content using JavaScript, so you'll need a tool that can handle JavaScript rendering, such as Selenium. ProxyTee's residential proxies can help bypass anti-scraping measures when used with Selenium, ensuring smooth data collection. You can also leverage ProxyTee's unlimited residential proxies which are perfect for high volume web scraping and provides a wide array of geo-locations to choose from.

Step #3: Install and Configure Selenium

Install Selenium using pip:

pip install -U selenium

Now, add the following code to your scraper.py to set up the WebDriver:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

# Set up a controllable Chrome instance
driver = webdriver.Chrome(service=Service())

Do not forget to close the WebDriver at the end:

driver.quit()

Step #4: Visit the Target Page

Use Selenium's get() method to load the Indeed page:

driver.get("https://www.indeed.com/jobs?q=data+scientist&l=New+York%2C+NY&from=searchOnHP%2Cwhatautocomplete&vjk=45d1ba700870fbef")

Here is how scraper.py should look like:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

# Set up a controllable Chrome instance
driver = webdriver.Chrome(service=Service())

# Open the target page in the browser
driver.get("https://www.indeed.com/jobs?q=data+scientist&l=New+York%2C+NY&from=searchOnHP%2Cwhatautocomplete&vjk=45d1ba700870fbef")

# Scraping logc...

# Close the web driver
driver.quit()

Step #5: Select the Job Posting Elements

Initialize an empty list to hold job data:

jobs = []

Select the container of job postings and the individual job elements:

jobs_container_element = driver.find_element(By.CSS_SELECTOR, "#mosaic-provider-jobcards")
job_elements = jobs_container_element.find_elements(By.CSS_SELECTOR, "[data-testid=\"slider_item\"]")

Remember to import By:

from selenium.webdriver.common.by import By

Step #6: Scrape the Job Main Info

Now, loop through each job element and extract the job title, URL, company name, and location:

for job_element in job_elements:
    title_element = job_element.find_element(By.CSS_SELECTOR, "h2.jobTitle")
    title = title_element.text

    url_element = title_element.find_element(By.CSS_SELECTOR, "a")
    url = url_element.get_attribute("href")

    company_element =job_element.find_element(By.CSS_SELECTOR, "[data-testid=\"company-name\"]")
    company = company_element.text

    location_element = job_element.find_element(By.CSS_SELECTOR, "[data-testid=\"text-location\"]")
    location = location_element.text

Step #7: Scrape the Job Details

Next, extract the job tags, easy apply availability, and description:

    tags = []
    tags_container_element = job_element.find_element(By.CSS_SELECTOR, ".jobMetaDataGroup")
    tag_elements = tags_container_element.find_elements(By.CSS_SELECTOR, "[data-testid=\"attribute_snippet_testid\"]")
    for tag_element in tag_elements:
        tag = tag_element.text
        tags.append(tag)

    # Check whether the \"Easy Apply\" element is on the page
    try:
        job_element.find_element(By.CSS_SELECTOR, "[data-testid=\"indeedApply\"]")
        easily_apply = True
    except NoSuchElementException:
        easily_apply = False

    description = []
    description_container_element = job_element.find_element(By.CSS_SELECTOR, "[role=\"presentation\"]")
    description_elements = description_container_element.find_elements(By.CSS_SELECTOR, "ul li")
    for description_element in description_elements:
        description_item_text = description_element.text
        # Ignore empty description strings
        if (description_item_text != ""):
            description.append(description_item_text)

Be sure to handle the NoSuchElementException and to create a data structure to contain your data:

from selenium.common import NoSuchElementException

    job = {
        "title": title,
        "url": url,
        "company": company,
        "location": location,
        "tags": tags,
        "easily_apply": easily_apply,
        "description": description
    }
    jobs.append(job)

Step #8: Collect the Scraped Data

The scraped data from each job is now stored in a job dictionary which will be appended to the jobs array.

Step #9: Export the Scraped Data to CSV

Export the scraped data to a CSV file for analysis and portability:

import csv

csv_file = "scraped_jobs.csv"
csv_headers = ["title", "url", "company", "location", "tags", "easily_apply", "description"]

with open(csv_file, mode="w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=csv_headers)
    writer.writeheader()
    for job in jobs:
        writer.writerow({
            "title": job["title"],
            "url": job["url"],
            "company": job["company"],
            "location": job["location"],
            "tags": ";".join(job["tags"]),
            "easily_apply": "Yes" if job["easily_apply"] else "No",
            "description": ";".join(job["description"])
        })

Remember to import csv from the standard library.

Step #10: Put It All Together

Here's the complete scraper.py:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.common import NoSuchElementException
import csv

# Set up a controllable Chrome instance
driver = webdriver.Chrome(service=Service())

# Open the target page in the browser
driver.get("https://www.indeed.com/jobs?q=data+scientist&l=New+York%2C+NY&from=searchOnDesktopSerp")

# A data structure where to store the scraped job openings
jobs = []

# Select the job opening elements on the page
jobs_container_element = driver.find_element(By.CSS_SELECTOR, "#mosaic-provider-jobcards")
job_elements = jobs_container_element.find_elements(By.CSS_SELECTOR, "[data-testid=\"slider_item\"]")

# Scrape each job opening on the page
for job_element in job_elements:
    title_element = job_element.find_element(By.CSS_SELECTOR, "h2.jobTitle")
    title = title_element.text

    url_element = title_element.find_element(By.CSS_SELECTOR, "a")
    url = url_element.get_attribute("href")

    company_element =job_element.find_element(By.CSS_SELECTOR, "[data-testid=\"company-name\"]")
    company = company_element.text

    location_element = job_element.find_element(By.CSS_SELECTOR, "[data-testid=\"text-location\"]")
    location = location_element.text

    tags = []
    tags_container_element = job_element.find_element(By.CSS_SELECTOR, ".jobMetaDataGroup")
    tag_elements = tags_container_element.find_elements(By.CSS_SELECTOR, "[data-testid=\"attribute_snippet_testid\"]")
    for tag_element in tag_elements:
        tag = tag_element.text
        tags.append(tag)

    # Check whether the \"Easy Apply\" element is on the page
    try:
        job_element.find_element(By.CSS_SELECTOR, "[data-testid=\"indeedApply\"]")
        easily_apply = True
    except NoSuchElementException:
        easily_apply = False

    description = []
    description_container_element = job_element.find_element(By.CSS_SELECTOR, "[role=\"presentation\"]")
    description_elements = description_container_element.find_elements(By.CSS_SELECTOR, "ul li")
    for description_element in description_elements:
        description_item_text = description_element.text
        # Ignore empty description strings
        if (description_item_text != ""):
            description.append(description_item_text)

    # Store the scraped data
    job = {
        "title": title,
        "url": url,
        "company": company,
        "location": location,
        "tags": tags,
        "easily_apply": easily_apply,
        "description": description
    }
    jobs.append(job)

# Export the scraped data to an output CSV file
csv_file = "jobs.csv"
csv_headers = ["title", "url", "company", "location", "tags", "easily_apply", "description"]

with open(csv_file, mode="w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=csv_headers)
    writer.writeheader()
    for job in jobs:
        writer.writerow({
            "title": job["title"],
            "url": job["url"],
            "company": job["company"],
            "location": job["location"],
            "tags": ";".join(job["tags"]),
            "easily_apply": "Yes" if job["easily_apply"] else "No",
            "description": ";".join(job["description"])
        })

# Close the web driver
driver.quit()

Run the scraper:

python3 script.py # On Linux/macOS
python script.py # On Windows

A CSV file called jobs.csv should appear with the extracted data.

Unlock Indeed Data With Ease

Indeed is known for employing anti-bot techniques, including CAPTCHAs, which can disrupt web scraping projects and even result in IP bans. Bypassing these measures is essential to conduct effective web scraping on Indeed. ProxyTee helps to overcome these challenges and ensure uninterrupted data collection.

While this guide has provided insights into scraping Indeed, the platform often introduces new protective measures. This can be very frustrating when you only want to gather job market information.

Consider using ProxyTee to automate your proxy handling, ensuring reliable access and smooth data collection while helping you maintain an effective web scraping setup. ProxyTee offers affordable access to residential proxies and flexible rotation, with simple API integration (Simple API) to help you customize how you use its service. This not only helps to reduce cost, but allows you to be compliant by being respectful of robots.txt at all times.

Conclusion

In this tutorial, you learned what an Indeed scraper is and how to build one using Python and Selenium. While manual scraping is feasible, scaling these operations without robust proxy management can be time-consuming, tedious and frustrating. ProxyTee offers an array of features, including a user-friendly interface that ensures a seamless web scraping experience, whether you are collecting market research or conducting competitive analysis.

Whether you are looking for job openings, or wanting to analyze job market trends, make use of ProxyTee today. Get access to high quality residential proxies that rotate automatically with Auto rotation and a wide array of location targets(Global coverage) . We offer competitive pricing and ensure you receive a service that is value for your money!

To scale this workflow in production without bandwidth throttling, explore our LinkedIn & Job Board Scraping Proxies, Python Requests & HTTPX Integration, United States Residential Proxies, and ProxyTee vs. Bright Data Benchmark.