Python Tutorial / Remove Duplicates with Sets and Choose Collections
A set stores unique hashable items and supports membership tests and set operations. Converting data to a set loses duplicate counts and positional information, so decide whether those details matter first.
Create a set and read its operations
Use a literal such as {"food", "books"} for a nonempty set and set() for an empty one. Empty braces {} create a dictionary. Lists and dictionaries cannot be set members because they are not hashable. Set iteration order is unsuitable as a meaningful display order or indexing scheme.
| Expression | Meaning |
|---|---|
| a | b | Union |
| a & b | Intersection |
| a - b | Items in a but not b |
| a ^ b | Items present in exactly one set |
For the relevant APIs and platform requirements, see Python: stdtypes.
Identity keys and duplicate validation build on the rules demonstrated in Store Key–Value Data with Dictionaries and Project: Validate CSV Data and Produce Reports.
Compare used and permitted categories
Save the complete code below as sets.py in a separate practice folder, using UTF-8. Run python sets.py in a terminal with your project virtual environment selected. Outside a virtual environment on macOS or Ubuntu, use python3 when that is your interpreter command.
used = {"food", "transport", "food"}
allowed = {"food", "books"}
print(sorted(used))
print(sorted(used & allowed))
print(sorted(used - allowed))
used.discard("missing")
print("food" in used)
Expected output:
['food', 'transport'] ['food'] ['transport'] True
Interpret duplicate removal and difference
Repeated food values collapse into one member. used - allowed identifies transport as an unapproved category. sorted() makes the displayed results deterministic; it returns a sorted list rather than making the set itself ordered.
For hashable items, list(dict.fromkeys(items)) can remove duplicates while retaining their first occurrence order. Still decide whether every removed occurrence represented meaningful data. Duplicate removal can otherwise hide bad input rather than validate it.
Two transactions with the same amount can be distinct transactions. Removing duplicate values is different from validating duplicate transaction IDs. Collapsing two payments of 8,000 into one set element produces the wrong total. Check stable transaction IDs and choose explicitly whether a repeated ID is rejected or updated.
Use missing-item behavior intentionally
remove() raises KeyError when an item is absent; discard() accepts absence. The exception can help reveal a broken requirement when an item must exist, whereas discard is convenient for optional cleanup.
Add food again and confirm that the size does not increase. Then add books to used and compare the intersection. Restore the initial data afterwards. Two expenses of the same amount may be separate transactions: putting their amounts into a set would produce an incorrect total.
Choose a list for order and repeats, a tuple for a short positional record, a dictionary for named lookups and a set for uniqueness and membership. Let the information you need to retain determine the structure.
