Python Tutorial / Extract Data from HTML with Beautiful Soup

Python Tutorial

Use an HTML parser to select elements and read their text or attributes. Validate expected counts and required fields because changed markup or selectors can otherwise turn a broken extraction into apparently valid empty data.

Begin with a controlled static document

This exercise parses an HTML string you own and sends no website requests. Parsing HTML and collecting external websites are separate steps. External collection also requires appropriate access, service terms, request frequency and treatment of personal data.

Install Beautiful Soup in the project environment:

python -m pip install beautifulsoup4

The distribution name is beautifulsoup4, while its import name is bs4. This example uses Python’s built-in html.parser as the parsing backend.

For the relevant APIs and platform requirements, see Extract Data from HTML with Beautiful Soup documentation.

Run the complete extraction

Save the complete code below as parse_html.py in a separate practice folder, using UTF-8. Run python parse_html.py in a terminal with your project virtual environment selected. Outside a virtual environment on macOS or Ubuntu, use python3 when that is your interpreter command.

from bs4 import BeautifulSoup

html = """<ul id="expenses">
<li data-amount="3500"><span class="name">coffee</span></li>
<li data-amount="8000"><span class="name">lunch</span></li>
</ul>"""
soup = BeautifulSoup(html, "html.parser")
rows = soup.select("#expenses > li")
if len(rows) != 2:
    raise ValueError("unexpected row count")
total = 0
for row in rows:
    name = row.select_one(".name")
    amount_text = row.get("data-amount")
    if name is None or not amount_text or not amount_text.isascii() or not amount_text.isdecimal():
        raise ValueError("missing or invalid field")
    amount = int(amount_text)
    print(name.get_text(" ", strip=True), amount)
    total += amount
print("total:", total)

Expected output:

coffee 3500
lunch 8000
total: 11500

Read selectors, optional matches and text

#expenses > li selects direct li children of the specified ID. .name selects a class. select_one returns None when there is no matching element, so check before accessing text.

get_text’s separator and strip options control spacing between nested text nodes. Data visible in a browser is not necessarily present in the downloaded HTML: JavaScript may create it later. Inspect the actual response and data source rather than assuming a plain download contains the rendered page.

Detect changes instead of accepting empty results

Zero matches can indicate a selector failure rather than a genuinely empty collection. The example expects exactly two rows. A real collector needs a reasonable policy for counts, required fields and collection context.

Change the name class and verify that the required-field check fails.
Set data-amount to -1 or abc and confirm rejection.
Add a nested tag inside a name and inspect the extracted text, then restore the markup.

The regular-expression lesson explains why matching text patterns and parsing a structured format are different tools.

Python Tutorial