Skip to main content
Brave Programmer Logo

BraveProgrammer

BraveProgrammer

HomeProjectsBlogsCoursesLessonsAbout

Site footer

BraveProgrammer

Free coding courses, practical tutorials, and real projects from BraveProgrammer. Learn web development with React, Next.js, and TypeScript.

Navigation

  • Home
  • Projects
  • Blogs
  • Courses

Resources

  • About
  • Lessons
  • Teach on BraveProgrammer

© 2026 BraveProgrammer. All rights reserved.

Web Scraping with requests and | Python Fundamentals lesson | BraveProgrammer
  1. Courses
  2. /
  3. Python Fundamentals

Lesson 40 of 55

·Python

Web Scraping with requests and BeautifulSoup

20 minutes

Course progress73%

What you'll learn

Keep these ideas in mind as you work through the lesson.

4 points
  • `requests.get(url)` returns a `Response` object; check `status_code` or use `raise_for_status()`.
  • `BeautifulSoup` parses HTML; use CSS selectors (`.select`) or `.find_all`.
  • Handle pagination by fetching subsequent pages in a loop.
  • Be polite: add a short `time.sleep()` between requests and set a custom `User-Agent`.

Web Scraping with requests and BeautifulSoup

Scraping lets you extract data from web pages. requests fetches the HTML, while BeautifulSoup parses it into a searchable structure.

import requests
from bs4 import BeautifulSoup

url = 'https://quotes.toscrape.com/'
response = requests.get(url)
response.raise_for_status()

soup = BeautifulSoup(response.text, 'html.parser')
quotes = soup.select('.quote .text')
for q in quotes[:5]:
    print(q.get_text())

Always respect a site’s robots.txt and terms of service when you scrape.

Make it stickPractice what you've learned1 quiz · 5 practical tests · Optional activities from Python FundamentalsBrowse activities
Previous lessonData Visualization with MatplotlibContinue to lesson 41Working with APIsBack to Python Fundamentals curriculum