How to Scrape YouTube Metadata with Python Using ProxyTee

In this guide, we'll explore how to scrape YouTube video data using Python and ProxyTee. While YouTube offers its official Data API, scraping can provide more flexibility, access to unofficial data, and fewer limitations. Let’s dive into how you can efficiently collect YouTube data for your projects using ProxyTee's powerful residential proxies.

Why Scrape YouTube?

The YouTube Data API is the go-to official method to access data, but here’s why scraping might be a better approach for specific use cases:

• Flexibility: When using a custom scraper, you control which data you collect. You tailor your script to get specific information rather than relying on the predefined data of the API.
• Access to Unofficial Data: The API offers a limited data subset. Web scraping grants you access to any data on the YouTube website, including info not exposed by the API.
• No Limitations: YouTube APIs have rate limits on request frequency and volume. When scraping directly, you can bypass these restrictions.

What Data Can You Scrape?

When scraping YouTube, you can target a wide variety of data, including:

• Video Metadata:
  • Title
  • Description
  • Views
  • Likes
  • Duration
  • Publication date
  • Channel

• User Profiles:
  • Username
  • User Description
  • Subscribers
  • Number of Videos
  • Playlists

• Other:
  • Comments
  • Related Videos

To accomplish this, you need a versatile tool and language like Python.

Why Python?

Python stands out as one of the best languages for web scraping. Its simplicity, numerous libraries, and active community make it easy to use and effective for a variety of web scraping projects. Plus, with ProxyTee's rotating residential proxies, your scraping tasks will be more reliable and efficient, ensuring your data collection remains uninterrupted.

Step-by-Step Guide to Scraping YouTube with Selenium

Follow the guide below to get started:

Step 1: Setup

Start by making sure your system is set up with the following:

• Python 3+: Download and install the latest version of Python.
• Python IDE: Consider using a free option such as PyCharm Community Edition or Visual Studio Code with the Python extension.

To initialize a new Python project, create a virtual environment, open the new project in your IDE, and create a scraper.py file:

mkdir youtube-scraper
cd youtube-scraper
python -m venv env

Add some test code to scraper.py:

print('Hello, World!')

If you run this file, either through your IDE or the command line (python scraper.py) , you should see “Hello, World!” as output. This means you are good to go.

Step 2: Install Scraping Libraries

YouTube is highly interactive, which requires a web scraping tool that can render dynamic content in a browser. Selenium is perfect for this purpose, which allows you to scrape any data on dynamic websites using a headless browser in Python. Together with webdriver-manager, you’ll be able to handle the browser configuration and installation. Use the following commands:

pip install selenium webdriver-manager

Add this code to your scraper.py:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options

# initialize a web driver instance to control a Chrome window in headless mode
options = Options()
options.add_argument('--headless=new')

driver = webdriver.Chrome(
    service=ChromeService(ChromeDriverManager().install()),
    options=options
)

# scraping logic...

# close the browser and free up the resources
driver.quit()

This code sets up a headless Chrome instance that allows you to control a browser without a UI. It will open Chrome to load the content to scrape, then close it.

Step 3: Connect to YouTube

Start by targeting a specific video to extract its data. This example uses the URL for a video about Bright Data's Scraping Browser (https://www.youtube.com/watch?v=kuDuJWvho7Q) but can easily be replaced by any other video:

url = 'https://www.youtube.com/watch?v=kuDuJWvho7Q'
driver.get(url)

Add this to your script to tell Selenium to visit the target YouTube video’s page.

Step 4: Inspect the Page

To scrape correctly, you need to interact with page elements to enable a smooth extraction of information, starting by dismissing the initial cookie consent. Using your browser’s developer tools (DevTools) to “Inspect” elements on the webpage helps to find selectors for specific items. Locate and inspect the consent dialog and “Accept all” button, you’ll find it with these selectors:

• The Dialog ID: dialog
• The “Accept All” Button: .eom-buttons button.yt-spec-button-shape-next

Insert the code below to programmatically handle this consent popup:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common import TimeoutException

try:
    # wait up to 15 seconds for the consent dialog to show up
    consent_overlay = WebDriverWait(driver, 15).until(
        EC.presence_of_element_located((By.ID, 'dialog'))
    )

    # select the consent option buttons
    consent_buttons = consent_overlay.find_elements(By.CSS_SELECTOR, '.eom-buttons button.yt-spec-button-shape-next')
    if len(consent_buttons) > 1:
        # retrieve and click the 'Accept all' button
        accept_all_button = consent_buttons[1]
        accept_all_button.click()
except TimeoutException:
    print('Cookie modal missing')

Add this code to make the script accept YouTube's cookie policy, with a timeout of 15 seconds and some exception handling in case the dialog does not show up.

After handling the cookie policy, the script should also wait until the page is loaded, the title of the video becomes visible ( h1.ytd-watch-metadata), before performing any scrape:

# wait for YouTube to load the page data
WebDriverWait(driver, 15).until(
        EC.visibility_of_element_located((By.CSS_SELECTOR, 'h1.ytd-watch-metadata'))
)

Step 5: Extract Video Data

Now, extract specific data from the YouTube page. Initialize a Python dictionary where to store the result: video = {}. Next, scrape the following data using the right CSS selectors:

• Title: h1.ytd-watch-metadata

title = driver \
    .find_element(By.CSS_SELECTOR, 'h1.ytd-watch-metadata') \
    .text

Find the channel information using the element's “owner” ID, including URL, name, image and subscriber count.

# dictionary where to store the channel info
channel = {}

# scrape the channel info attributes
channel_element = driver \
    .find_element(By.ID, 'owner')

channel_url = channel_element \
              .find_element(By.CSS_SELECTOR, 'a.yt-simple-endpoint') \
              .get_attribute('href')
channel_name = channel_element \
              .find_element(By.ID, 'channel-name') \
              .text
channel_image = channel_element \
              .find_element(By.ID, 'img') \
              .get_attribute('src')
channel_subs = channel_element \
              .find_element(By.ID, 'owner-sub-count') \
              .text \
              .replace(' subscribers', '')

channel['url'] = channel_url
channel['name'] = channel_name
channel['image'] = channel_image
channel['subs'] = channel_subs

Extract video views and publication date using these selectors:

• Video view and publish date: #info-container span

# click to expand the video description to access all content
driver.find_element(By.ID, 'description-inline-expander').click()

info_container_elements = driver \
    .find_elements(By.CSS_SELECTOR, '#info-container span')

views = info_container_elements[0] \
    .text \
    .replace(' views', '')
publication_date = info_container_elements[2] \
    .text

And extract the video description using:

• Video description: #description-inline-expander .ytd-text-inline-expander span

description = driver \
    .find_element(By.CSS_SELECTOR, '#description-inline-expander .ytd-text-inline-expander span') \
    .text

Extract the number of likes using:

• Number of Likes: #segmented-like-button

likes = driver \
    .find_element(By.ID, 'segmented-like-button') \
    .text

Finally, add these values to the main video dictionary:

video['url'] = url
video['title'] = title
video['channel'] = channel
video['views'] = views
video['publication_date'] = publication_date
video['description'] = description
video['likes'] = likes

Step 6: Export the Scraped Data to JSON

Use the Python json library to serialize your scraped dictionary in the JSON format into a video.json output file:

import json

with open('video.json', 'w') as file:
    json.dump(video, file, indent=4)

Step 7: Full Script

Here is the complete scraper.py script:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common import TimeoutException
import json

# enable the headless mode
options = Options()
options.add_argument('--headless=new')

# initialize a web driver instance to control a Chrome window
driver = webdriver.Chrome(
    service=ChromeService(ChromeDriverManager().install()),
    options=options
)

# the URL of the target page
url = 'https://www.youtube.com/watch?v=kuDuJWvho7Q'
# visit the target page in the controlled browser
driver.get(url)

try:
    # wait up to 15 seconds for the consent dialog to show up
    consent_overlay = WebDriverWait(driver, 15).until(
        EC.presence_of_element_located((By.ID, 'dialog'))
    )

    # select the consent option buttons
    consent_buttons = consent_overlay.find_elements(By.CSS_SELECTOR, '.eom-buttons button.yt-spec-button-shape-next')
    if len(consent_buttons) > 1:
        # retrieve and click the 'Accept all' button
        accept_all_button = consent_buttons[1]
        accept_all_button.click()
except TimeoutException:
    print('Cookie modal missing')

# wait for YouTube to load the page data
WebDriverWait(driver, 15).until(
        EC.visibility_of_element_located((By.CSS_SELECTOR, 'h1.ytd-watch-metadata'))
)

# initialize the dictionary that will contain
# the data scraped from the YouTube page
video = {}

# scraping logic
title = driver \
    .find_element(By.CSS_SELECTOR, 'h1.ytd-watch-metadata') \
    .text

# dictionary where to store the channel info
channel = {}

# scrape the channel info attributes
channel_element = driver \
    .find_element(By.ID, 'owner')

channel_url = channel_element \
              .find_element(By.CSS_SELECTOR, 'a.yt-simple-endpoint') \
              .get_attribute('href')
channel_name = channel_element \
              .find_element(By.ID, 'channel-name') \
              .text
channel_image = channel_element \
              .find_element(By.ID, 'img') \
              .get_attribute('src')
channel_subs = channel_element \
              .find_element(By.ID, 'owner-sub-count') \
              .text \
              .replace(' subscribers', '')

channel['url'] = channel_url
channel['name'] = channel_name
channel['image'] = channel_image
channel['subs'] = channel_subs

# click the description section to expand it
driver.find_element(By.ID, 'description-inline-expander').click()

info_container_elements = driver \
    .find_elements(By.CSS_SELECTOR, '#info-container span')
views = info_container_elements[0] \
    .text \
    .replace(' views', '')
publication_date = info_container_elements[2] \
    .text

description = driver \
    .find_element(By.CSS_SELECTOR, '#description-inline-expander .ytd-text-inline-expander span') \
    .text

likes = driver \
    .find_element(By.ID, 'segmented-like-button') \
    .text

video['url'] = url
video['title'] = title
video['channel'] = channel
video['views'] = views
video['publication_date'] = publication_date
video['description'] = description
video['likes'] = likes

# close the browser and free up the resources
driver.quit()

# export the scraped data to a JSON file
with open('video.json', 'w') as file:
    json.dump(video, file, indent=4)

Why ProxyTee?

While this script provides the code, it’s important to acknowledge the challenges you might encounter with YouTube’s anti-bot mechanisms. These can range from CAPTCHAs to IP blocking, especially when you make several requests from the same IP. This is where ProxyTee's rotating residential proxies become essential.

ProxyTee offers:

• Unlimited Bandwidth: Scrape YouTube without worrying about bandwidth overages, perfect for extracting large quantities of data from various videos and channels.
• Extensive IP Pool: Access over 20 million residential IPs from more than 100 countries. This wide coverage makes it easy to avoid geo-restrictions and collect targeted regional information.
• Auto Rotation: Avoid detection with automatic IP address rotation at customizable intervals.
• Multiple Protocol Support: Compatibility with HTTP and SOCKS5 protocols makes integrating with your scraping tools much easier.
• Simple API Integration: ProxyTee provides a user-friendly API to support automation of all of its proxy service features.
• Affordable Pricing: Get all of these features at a cost-effective rate.

Conclusion

Web scraping YouTube can be useful for various purposes if you follow a structured approach and apply efficient methods. The example above makes the process easier. Remember that websites often change, so having the tools at hand to quickly update your script and improve your method is useful. Using ProxyTee's reliable, affordable, and user-friendly proxy services to handle all these tasks will greatly improve your scraping activity.