Integrating LLMs with Web Scraping Using ProxyTee

The Assistants API from OpenAI offers developers the tools to harness the power of AI and integrate external tools for building intelligent applications. While this API can interpret code, use uploaded files, and call functions, it lacks direct, reliable access to the latest website data. To overcome this limitation, you can integrate web scraping solutions, such as ProxyTee's rotating residential proxies, with the Assistants API. This combination enables your AI tools to operate on real-world, real-time data.

This guide details how to effectively merge scraped data with the Assistants API, using ProxyTee for data extraction and a practical Product Assistant application. By the end of this tutorial, you will be able to use the combined strengths of these tools to analyze data extracted from the web.

Understanding the Functionality of the Assistants API

The workflow of the Assistants API can be summarized in these key steps:

• Defining Purpose: Assistants use specific instructions to establish their role, capabilities, and personality.
• Tool Integration: They can access both built-in and user-defined tools simultaneously.
• Conversation Management: They can initiate or engage in conversation threads.
• File Utilization: They can create and use files provided by users or developers to deliver answers.

Limitations of the Assistants API

Before you start building your own application using this API, consider these critical limitations:

• Code interpreters can only manage a maximum of 20 attached files, while the File search feature is limited to 10,000 files.
• The limit for each file is 512MB and 5,000,000 tokens, with a total default upload limit of 100GB, which can be extended by request.
• Image processing is limited to two models currently supporting the Vision feature.
• API requests are subject to rate limits based on your subscription plan.
• AI responses may occasionally be incorrect, even when the input data is accurate.

Integrating Scraped Data With the Assistants API

To successfully integrate scraped data with the Assistants API, follow these general steps:

1. Use ProxyTee’s rotating residential proxies to extract public data.
2. Transfer this data to the Assistants API for processing.
3. Send prompts to the Assistants API and receive generated responses.

For our specific example, we’ll demonstrate creating a simple Product Assistant using data from a demo e-commerce site. This assistant will answer product-specific questions based on scraped web content. The main focus here will be on showing you how these technologies can effectively be integrated.

Step-by-step Guide for Integrating Web Scraping With the Assistants API

1. Extract data using ProxyTee

Start by setting up a script that extracts product information and stores it in JSON. Supplying the full HTML to the Assistants API might be impractical because of token restrictions and higher API costs. Therefore, data parsing before input is recommended. Using ProxyTee's rotating residential proxies is essential to avoid IP blocks and ensure smooth data collection.

ProxyTee offers easy-to-integrate proxies with unlimited bandwidth, a vast global IP pool, and robust API access, allowing for automated data extraction at scale. Here's why ProxyTee is a perfect fit:

• Unlimited Bandwidth: Avoid extra costs and data limits with unlimited bandwidth.
• Global Coverage: Scrape data from over 100 countries using a pool of more than 20 million IP addresses with our global IP coverage.
• Auto Rotation: Keep your scraping tasks safe from blocks and detection with automatic IP rotation, customizable from 3 to 60 minutes via the auto-rotation feature.
• Multiple Protocol Support: Our multiple proxy protocols that support HTTP and SOCKS5 offer the versatility required for various tasks, including web scraping and geo-block circumvention.
• Simple API: Our simple API allows effortless integration of the service's capabilities into any development workflow.
• User-Friendly Interface: ProxyTee also offers an easy-to-use graphical interface to start the scraping process immediately with no need of a developer.

For more specific extraction use cases, ProxyTee provides different options of proxies for use cases like web scraping. You can leverage Residential Proxies for regular scraping and Datacenter Proxies if you need very fast extraction speeds.

Using ProxyTee's services for web scraping significantly cuts down on the time and technical resources required to collect data, giving you a powerful start for AI integration.

2. Install and Set Up the OpenAI Library

Install the OpenAI Python library using pip:

pip install openaiThen you will also need to securely store your OpenAI API key using system environment variables.

3. Create Your Product Assistant

In a new Python file, define your assistant’s properties. You'll specify things like the name, detailed instructions for the task, the file path of your data file, and the OpenAI model to use:

from openai import OpenAI

file_paths = ["products.json"]

name = "Product assistant"

instructions = (
    "You are a product assistant that analyzes video game data (products.json file) and provides very short information to the user about specific products. "
    "You must analyze each product's information. The 'availability' key specifies whether the product is in stock or out of stock."
    "Provide answers in plain text without markdown. Use this format for listing products: Product Title - Product Price - Product Availability - Product Link. "
    "For product link, use the URL value and add it to https://sandbox.oxylabs.io. For instance, https://sandbox.oxylabs.io/products/1."
)

model = "gpt-4o"

4. Implement the Assistant Class

Develop a ProductAssistant class to create an assistant from this blueprint, manage files, and process conversations:

class ProductAssistant:
    def __init__(self, file_paths, name, instructions, model):
        self.file_paths = file_paths
        self.name = name
        self.instructions = instructions
        self.model = model
        self.client = OpenAI()
        self.assistant = None
        self.thread = None
        self.setup_assistant()

5. Setup Assistant and File Handling

Implement several key functionalities within your class:

• Retrieve Existing Assistant: Check for an existing assistant that matches the specified name and instructions, which helps prevent creating duplicates every time.
• Create Vector Store with Files: Add your scraped data file to a vector store and get the file's ID, ensuring it’s ready to be used by the assistant.
• Create New Assistant: Use the name, instruction, and model, combined with the ability for the assistant to find files, enabling it to respond using scraped content.

Lastly, create a setup function to execute all above methods at once:

    def setup_assistant(self):
        self.assistant = self.retrieve_assistant()

        if not self.assistant:
            vector_store_id = self.create_vector_with_files()
            self.assistant = self.create_assistant(vector_store_id)

        self.thread = self.client.beta.threads.create()

6. Initiate the Conversation

Prepare two additional class methods:

• The add_message_to_thread method that includes user queries in a thread for processing by the AI.
• The get_response method then triggers the AI assistant, gets its response, and sends it back to the user, ensuring a functional two-way communication system.

These two methods are combined into one called ask_question() which returns the AI response.

7. Create the Main Loop

Tie everything together with a main loop that instantiates the assistant and continually accepts questions until 'exit' is typed:

def main():
    assistant = ProductAssistant(file_paths, name, instructions, model)

    while True:
        question = input("\033[1;34mQuestion ->\033[0m ")

        if question.lower() in ["exit"]:
            print("\033[1;31mExiting the assistant.\033[0m")
            break

        answer = assistant.ask_question(question)
        print("\033[1;33mAnswer -> " + f"\033[1;32m{answer}\033[0m")


if __name__ == "__main__":
    main()

By following the guide above, you will have implemented a method that lets the AI handle inquiries by looking for the right data in JSON format in a separate file.

Additional Methods for Data Integration

Besides uploading a file into the vector store, there are multiple ways to transfer your data to the OpenAI Assistants API.

Directly Attach a File

The API allows attaching files directly to user messages, which either creates a new or adds to an existing vector store. This method simplifies the process by embedding all the data needed within one prompt. In order to upload a file and integrate it with an assistant, you should adjust the __init__(), create_assistant(), add_message_to_thread(), and setup_assistant() and include the upload_file() function. While this method could be a good solution for many use cases, bear in mind the limitations of processing massive file sizes or data amounts within a single request.

Inline Data Insertion

For smaller datasets, the data can be passed directly within the user’s message. In other words, instead of working with file IDs and a file upload stage, you will insert a JSON file to the user prompt with each message. However, this method is best for small snippets of data and isn’t suitable for large information sets because it raises API costs, slows data processing and adds complications in terms of overall management. To make use of this implementation, the load_products method should load the data from the file path into a class variable. The add_message_to_thread will then incorporate the new class variable.

Final Thoughts

By integrating ProxyTee's robust proxy solutions with the Assistants API from OpenAI, you open up new possibilities for AI applications that use real-time data. Whether through the more elaborate file uploads or using the leaner option of inline data inclusion, the methods detailed above provide powerful tools for any data-driven AI project. Using an automated method of getting website content that also helps to prevent anti-scraping mechanism is invaluable in building a competitive business.

Want to enhance your scraping and AI integration capabilities? Explore more about ProxyTee’s affordable pricing plans and start leveraging robust rotating residential proxies today.