Introduction
ChatGPT is becoming popular for getting info from websites. To make it easier for you, we’ve made a simple guide on how to use ChatGPT to scrape website. ChatGPT uses GPT-3, a powerful language tool made by OpenAI.
Many companies are now using ChatGPT in their daily work. The graph below shows how much money US companies saved in February 2023 by using it.
This guide will teach you everything from signing up for ChatGPT to writing prompts and checking the code it makes. We even have some pro tips for tricky webpages, to help you improve your scraping and avoid problems that other people have faced.
Let’s explore!
Can Chat GPT Scrape a Website?
ChatGPT can’t gather info from websites automatically like a person would. It can’t browse the internet. Instead, it uses the tons of information it has learned to answer questions.
Even though ChatGPT can’t scrape websites on its own, it can still help a lot. For example, if you want to scrape a website using Python, ChatGPT can give you pieces of code and point you to useful tools like Beautiful Soup or Scrapy.
How to Use ChatGPT to Scrape Website?
Now, let’s go through the steps to scrape data from this webpage using ChatGPT.
Create a ChatGPT Account
Firstly, you need to go to the ChatGPT website, and then click on “Sign-up”. If you want, you can sign up with your Google account. After you sign up, you’ll see the chat window. You can start chatting by typing your question in the box.
Locate Elements for Scraping
Before we ask ChatGPT for help, let’s find the parts of the webpage we want to get information from. Let’s say we only want the names and prices of the video games.
- Right-click on the name of a game and choose “Inspect.” This will open a window with the website’s code.
- Right-click on the highlighted part of the code that has the game’s name and choose “Copy selector.”
- Write down this selector. Now do the same thing to find the selector for the price of a game.
Create a ChatGPT Prompt
To get the best results, give ChatGPT clear instructions. Tell it the programming language (Python), the tools (Beautiful Soup), the selectors (the code bits you copied earlier), what you want the output to look like (a CSV file), and any special rules the code needs to follow.
Here’s an example prompt you can use:
Let’s write a website scraper with Python and BeautifulSoup.
Sample Target: https://sandbox.oxylabs.io/products
Goal: Get the names and prices of all the video games on the page.
CSS selectors are as follows:
1. Title: #__next > main > div > div > div > div:nth-child(2) > div > div:nth-child(1) > a.card-header.css-o171kl.eag3qlw2 > h4
2. Price: #__next > main > div > div > div > div:nth-child(2) > div > div:nth-child(1) > div.price-wrapper.css-li4v8k.eag3qlw4
Output: Save all the titles and prices in a CSV file
Special Instructions: Deal with character encoding and remove any weird symbols in the CSV file.
See how we’ve included the CSS selectors for the titles and prices that we copied earlier.
The scraped data might have some strange characters if we don’t handle the encoding correctly. This happens when the website and the Python code read the characters differently. The “Special Instructions” help avoid this issue.
Review the Code
After ChatGPT gives you the code, it’s a good idea to check it first. Just look it over to see if it seems to be doing what you asked for. Check if it uses any tools or libraries you don’t want. If you find any problems, tell ChatGPT to try again and make new code.
Test the Code
Copy the code that ChatGPT gave you and see if it works. Here’s the code it made:
Python
import requests
from bs4 import BeautifulSoup
import csv
# Website to scrape
url = “https://sandbox.oxylabs.io/products”
# Get the website’s code
response = requests.get(url)
# Make the code easier to read
soup = BeautifulSoup(response.content, “html.parser”)
# Find the parts of the code with the names and prices
title_selector = “a.card-header h4”
price_selector = “div.price-wrapper”
# Get all the names and prices
titles = soup.select(title_selector)
prices = soup.select(price_selector)
# Make a list to store the data
data = []
# Put the names and prices into the list
for title, price in zip(titles, prices):
game_title = title.get_text(strip=True)
game_price = price.get_text(strip=True)
data.append((game_title, game_price))
# Name the file
filename = “game_data.csv”
# Save the data to a CSV file
with open(filename, “w”, newline=””, encoding=”utf-8″) as file:
writer = csv.writer(file)
writer.writerow([“Title”, “Price”]) # Add a header row
for game in data:
writer.writerow(game)
print(f”Data scraped successfully and saved to ‘{filename}’.”)
Note: You need to install the BeautifulSoup and requests tools before running the code. You can do this when you type this command in your terminal:
Bash
pip install requests beautifulsoup4
Useful Tips to Use ChatGPT to Scrape Website
Below are helpful tips and tricks when you use ChatGPT to scrape website.
Use Code Editing
ChatGPT has a really cool feature: it can change code. If the code it gave you isn’t what you want or doesn’t work right, you can ask ChatGPT to change it for you.
Just tell it what you want to change, like getting different info, making the code work better, or changing how it gets the data. ChatGPT can give you more code options or ideas to improve your web scraping.
Linting Code
Linting code makes it easier to read and maintain. ChatGPT can help you with this by suggesting best practices, finding possible mistakes, and making your code easier to understand.
To follow good coding rules, you can ask ChatGPT to look at your code and give you suggestions. You can even copy and paste your code and ask ChatGPT to lint it. Just add the words “lint the code” to your instructions when you ask.
Optimize Code
When you scrape websites, being fast is important, especially if you’re working with a lot of data or tricky websites. ChatGPT can give you tips to make your code run faster.
You can ask for advice on using special tools that speed up scraping, saving data to use later, doing things at the same time, and avoiding unnecessary requests to websites.
Realize Pagination Strategy
To get all the info you need from websites with multiple pages, use these tricks: go through each page, change page numbers or scroll settings to load more data.
By using these pro tips, you’ll get better at scraping and get more accurate and efficient results.
Conclusion
You can use ChatGPT to scrape website. ChatGPT has changed web scraping, making it easier and simpler. But, while ChatGPT makes it easy to make web scrapers, it’s important to know what it can’t do.
Sometimes, ChatGPT might give you unexpected results because of how its AI model works. It also can’t help you get around CAPTCHAs or give you web proxies to scrape more websites.
Read more at: Blog Rpa Cloud
