Explain shallow copy vs deep copy.

Detailed explanation

import copy

config = {"tables": ["orders", "customers"], "retries": 3}

shallow = copy.copy(config)
deep = copy.deepcopy(config)

shallow["tables"].append("products")   # mutates the shared inner list
shallow["retries"] = 5                 # rebinds a key on the copy only

print(config)
print(deep)

{'tables': ['orders', 'customers', 'products'], 'retries': 3}
{'tables': ['orders', 'customers'], 'retries': 3}

Appending through the shallow copy changed the original’s list, because both dicts point to the same list object. Reassigning retries did not, because that replaced the value on the copy only. The deep copy is unaffected by either change.

Shallow copies you may not notice

list(x), x[:], x.copy(), dict(x) and {**x} all make shallow copies.

When it matters in pipelines

A shared default configuration or a template record that each task “copies” and then modifies. With nested structures and a shallow copy, one task’s change leaks into every other task’s config.

Common mistakes

1. Assuming .copy() makes nested data independent.
2. Using deepcopy on large objects in a hot loop (it is slow), when building fresh objects would be clearer.
3. Forgetting that immutable nested values (strings, numbers, tuples of immutables) are safe to share.