How to Scrape Crunchbase with Python Using ProxyTee in 2026

Crunchbase is a treasure trove of business information, offering insights into companies, funding rounds, and key personnel. However, accessing this data requires overcoming sophisticated anti-scraping measures. This is where ProxyTee comes into play, providing the robust infrastructure you need for reliable data extraction. In this guide, we will explore how to build a Crunchbase scraper using Python and highlight how ProxyTee enhances the process.

What Is a Crunchbase Scraper?

A Crunchbase scraper is an automated tool designed to extract data from Crunchbase web pages. It navigates the site, identifies the desired information, and collects it using web scraping techniques. Given Crunchbase’s anti-bot and anti-scraping measures, a successful scraper needs to handle JavaScript rendering, CAPTCHAs, and browser fingerprinting effectively. This is where a powerful proxy service like ProxyTee becomes essential, ensuring your scraper remains undetected.

What Data To Scrape From Crunchbase

Here is some data you can retrieve from Crunchbase through web scraping:

• Company Information: Names, descriptions, industry, headquarters, founding date, and status.
• Funding Data: Total funding, funding rounds, and investors.
• Key People: Founders, executives, roles, and titles.
• Products and Services: Descriptions and categories.
• Acquisitions and Mergers: Details, dates, and terms.
• Market and Financial Data: Revenue estimates, employee numbers.
• News and Events: Press releases and milestones.
• Competitors: Lists of competing companies.

How to Build a Crunchbase Scraper in Python

Let's dive into creating a Python script to scrape Crunchbase using Selenium and ProxyTee for enhanced anonymity.

Step #1: Set Up Your Python Environment

Ensure you have Python 3+ installed. Create a project directory and a scraper.py file within it. Then, initialize a virtual environment:

mkdir crunchbase-scraper
cd crunchbase-scraper
python3 -m venv env
source env/bin/activate  # On macOS/Linux
env\Scripts\activate  # On Windows

Now you can install project-specific libraries without interfering with other applications.

Step #2: Install Necessary Libraries

Install Selenium to automate browser interactions. Then install a python library to use ProxyTee API, for instance `requests`:

pip install -U selenium requests

Import these libraries at the beginning of the scraper.py file.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import json
import requests

Step #3: Visit the Target Page with Proxies

Normally, a simple request would be blocked, and it's even more critical when working with headless browsers. To make a request with residential proxies, you would need an active Unlimited Residential Proxies subscription. After getting proxy info, start by launching the browser using Selenium, add ProxyTee info, and navigate to the desired page:

#Get proxy settings from ProxyTee API
proxy_api_url = "https://proxytee.com/api/get_proxy"
response = requests.get(proxy_api_url)
if response.status_code == 200:
  proxy_data = response.json()
  proxy_host = proxy_data["proxy_host"]
  proxy_port = proxy_data["proxy_port"]
  proxy_username = proxy_data["proxy_username"]
  proxy_password = proxy_data["proxy_password"]
else:
  print("Failed to retrieve proxy data")
  exit()

# Configure Chrome options to use the proxy with authentication
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(f'--proxy-server=http://{proxy_username}:{proxy_password}@{proxy_host}:{proxy_port}')

# initialize the driver to control a Chrome instance
driver = webdriver.Chrome(options=chrome_options)
url = "https://www.crunchbase.com/organization/brightdata"
driver.get(url)

With ProxyTee’s Unlimited Residential Proxies, your requests are routed through real user IP addresses, enhancing anonymity and reducing the chance of detection. Additionally, ProxyTee provides auto-rotation to periodically change IP, which is crucial to prevent bans and ensure smooth operation of scraping.

Step #4: Handle the Cookie Popup

Websites often use cookie popups, which you have to accept before you continue with browsing or scraping. This is also true on Crunchbase. Here's the Python function to handle it:

def handle_cookie_popup(driver, seconds=60):
    try:
        accept_button = WebDriverWait(driver, seconds).until(
            EC.element_to_be_clickable((By.CSS_SELECTOR, "#onetrust-accept-btn-handler"))
        )
        driver.execute_script("arguments[0].click();", accept_button)
        print("'Accept All' button clicked")
    except:
        print("'Accept All' button not found within {seconds} seconds")

handle_cookie_popup(driver)

This function waits for the "Accept All" button to appear and clicks it, thus allowing interaction with the webpage.

Step #5: Scrape Data From the Webpage

Crunchbase structures its data, but it uses dynamically generated ids/class names. Here's how to get around that issue with `find_parent_node_based_on_child_node_text`, which searches based on specific string in given parent node:

def find_parent_node_based_on_child_node_text(parent_nodes_selector, child_node_selector, text):
    parent_nodes = driver.find_elements(By.CSS_SELECTOR, parent_nodes_selector)
    for parent_node in parent_nodes:
        try:
            child_node = parent_node.find_element(By.CSS_SELECTOR, child_node_selector)
            if text.upper() in child_node.text.upper():
                return parent_node
        except:
            continue
    return None

Now, use this method along with CSS selectors, and scrape all the data. The complete code, is included at the end.

Step #6: Export the Scraped Data

Store the extracted data in a Python dictionary, which will be exported into a JSON format.

# Export to a JSON File
company = {
    "about": about,
    "industries": industries,
    "founded_date": founded_date,
    "company_type": company_type,
    "operating_status": operating_status,
    "headquarters": headquarters,
    "founders": founders,
    "email": contact_email,
    "phone": phone_number,
    "description": description,
    "products": products
}
with open("company.json", "w") as json_file:
  json.dump(company, json_file, indent=4)

This saves all gathered company data into a JSON file named company.json, and will allow to use for analysis purposes.

Step #7: Full Scraper Script

Here’s the complete script that integrates all the above steps, you may need to add logic to detect empty list or null variables:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import json
import requests

def find_parent_node_based_on_child_node_text(parent_nodes_selector, child_node_selector, text):
    parent_nodes = driver.find_elements(By.CSS_SELECTOR, parent_nodes_selector)
    for parent_node in parent_nodes:
        try:
            child_node = parent_node.find_element(By.CSS_SELECTOR, child_node_selector)
            if text.upper() in child_node.text.upper():
                return parent_node
        except:
            continue
    return None

def handle_cookie_popup(driver, seconds=60):
    try:
        accept_button = WebDriverWait(driver, seconds).until(
            EC.element_to_be_clickable((By.CSS_SELECTOR, "#onetrust-accept-btn-handler"))
        )
        driver.execute_script("arguments[0].click();", accept_button)
        print("'Accept All' button clicked")
    except:
        print("'Accept All' button not found within {seconds} seconds")

#Get proxy settings from ProxyTee API
proxy_api_url = "https://proxytee.com/api/get_proxy"
response = requests.get(proxy_api_url)
if response.status_code == 200:
  proxy_data = response.json()
  proxy_host = proxy_data["proxy_host"]
  proxy_port = proxy_data["proxy_port"]
  proxy_username = proxy_data["proxy_username"]
  proxy_password = proxy_data["proxy_password"]
else:
  print("Failed to retrieve proxy data")
  exit()

# Configure Chrome options to use the proxy with authentication
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(f'--proxy-server=http://{proxy_username}:{proxy_password}@{proxy_host}:{proxy_port}')

# initialize the driver to control a Chrome instance
driver = webdriver.Chrome(options=chrome_options)

# navigate to the desired Crunchbase page
url = "https://www.crunchbase.com/organization/brightdata"
driver.get(url)

# handle the cookie popup, if present
handle_cookie_popup(driver)

# scraping logic
about_node = driver.find_element(By.CSS_SELECTOR, "profile-section description-card")
about = about_node.text

industries_parent_node = find_parent_node_based_on_child_node_text("fields-card li", "label-with-info", "Industries")
industries_nodes = industries_parent_node.find_elements(By.CSS_SELECTOR, "chips-container a")
industries = []
for industry_node in industries_nodes:
    industries.append(industry_node.text)

founded_date_parent_node = find_parent_node_based_on_child_node_text("fields-card li", "label-with-info", "Founded Date")
founded_date_node = founded_date_parent_node.find_element(By.CSS_SELECTOR, "field-formatter")
founded_date = founded_date_node.text

company_type_parent_node = find_parent_node_based_on_child_node_text("fields-card li", "label-with-info", "Company Type")
company_type_node = company_type_parent_node.find_element(By.CSS_SELECTOR, "field-formatter")
company_type = company_type_node.text

operating_status_parent_node = find_parent_node_based_on_child_node_text("fields-card li", "label-with-info", "Operating Status")
operating_status_node = operating_status_parent_node.find_element(By.CSS_SELECTOR, "field-formatter")
operating_status = operating_status_node.text

headquarters_parent_node = find_parent_node_based_on_child_node_text("fields-card li", "label-with-info", "Headquarters Regions")
headquarters_node = headquarters_parent_node.find_element(By.CSS_SELECTOR, "field-formatter")
headquarters = headquarters_node.text

founders_parent_node = find_parent_node_based_on_child_node_text("fields-card li", "label-with-info", "Founders")
founders_nodes = founders_parent_node.find_elements(By.CSS_SELECTOR, "identifier-multi-formatter a")
founders = []
for founders_node in founders_nodes:
    founders.append(founders_node.text)

legal_name_parent_node = find_parent_node_based_on_child_node_text("fields-card li", "label-with-info", "Legal Name")
legal_name_node = legal_name_parent_node.find_element(By.CSS_SELECTOR, "field-formatter")
legal_name = legal_name_node.text

contact_email_parent_node = find_parent_node_based_on_child_node_text("fields-card li", "label-with-info", "Contact Email")
contact_email_node = contact_email_parent_node.find_element(By.CSS_SELECTOR, "field-formatter")
contact_email = contact_email_node.text

phone_number_parent_node = find_parent_node_based_on_child_node_text("fields-card li", "label-with-info", "Phone Number")
phone_number_node = phone_number_parent_node.find_element(By.CSS_SELECTOR, "field-formatter")
phone_number = phone_number_node.text

description_node = driver.find_element(By.CSS_SELECTOR, "section-card description-card")
description = description_node.text

products_parent_node = find_parent_node_based_on_child_node_text("profile-section", ".section-title", "Products and Services")
if products_parent_node:
    products_table_rows = products_parent_node.find_elements(By.CSS_SELECTOR, "table tbody tr")

    # scrape the product table
    products = []
    for row in products_table_rows:
        # extract the name and description from each row's columns
        name = row.find_element(By.CSS_SELECTOR, "td:nth-child(1)").text
        description = row.find_element(By.CSS_SELECTOR, "td:nth-child(2)").text
        product = {
            "name": name,
            "description": description
        }
        products.append(product)
else:
    products = []

# populate a dictionary with the scraped data
company = {
    "about": about,
    "industries": industries,
    "founded_date": founded_date,
    "company_type": company_type,
    "operating_status": operating_status,
    "headquarters": headquarters,
    "founders": founders,
    "email": contact_email,
    "phone": phone_number,
    "description": description,
    "products": products
}

# export the scraped data to a JSON file
with open("company.json", "w") as json_file:
    json.dump(company, json_file, indent=4)

# close the driver and release the browser resources
driver.quit()

Launch this script via python3 scraper.py, and the script creates a company.json file in your directory. The gathered data will allow you to perform data analysis on desired companies.

Unlocking Crunchbase Data with Ease

While the script demonstrates a DIY approach, keep in mind that as a regular user you may encounter several anti-scraping methods such as CAPTCHAs and browser fingerprinting that can block access to data on a regular basis. This could result in a loss of scraped data, time, resources and frustration.
That's why using a service like ProxyTee makes scraping more robust and scalable. With its wide range of IPs and focus on anonymity, ProxyTee ensures your scrapers consistently deliver data, without blocks.

Conclusion

This guide shows you how to build a basic Crunchbase scraper with Python, Selenium and ProxyTee for overcoming bot detection and IP bans. For more complex needs, dedicated services like ProxyTee's Unlimited Residential Proxies provide robust infrastructure, unlimited bandwidth, and global coverage, which ensures successful data extraction.
With ProxyTee, users can achieve a reliable, efficient, and cost-effective scraping experience.

To scale this workflow in production without bandwidth throttling, explore our B2B Market Intelligence Proxies, Playwright Stealth Proxy Integration, Italy Residential Proxies, and ProxyTee vs. Geonode Benchmark.