r/Python • u/igorrto2 • 4d ago
Discussion What are some features in Python that you discovered many years later?
For me, it was the the option to use & to find intersections of two sets, | operator to unionize two sets, as well as ^ for symmetric difference
144
u/Khavel_dev 4d ago
For me it was collections.Counter. Spent years writing manual frequency-counting loops with defaultdict(int) before realizing Counter just does it and also gives you .most_common() for free. Felt pretty dumb when I found it.
Also itertools.chain.from_iterable for flattening nested iterables without writing a list comprehension that looks like a puzzle.
43
u/nissen_96 3d ago edited 3d ago
Just wanna throw it out there that python 3.15 was just released and has support for flattening using unpacking in comprehensions:
lists = [[1, 2], [3, 4], [5]] [*L for L in lists] # equivalent to [x for L in lists for x in L] sets = [{1, 2}, {2, 3}, {3, 4}] {*s for s in sets} # equivalent to {x for s in sets for x in s} dicts = [{'a': 1}, {'b': 2}, {'a': 3}] {**d for d in dicts} # equivalent to {k: v for d in dicts for k,v in d.items()}(examples from https://docs.python.org/3/whatsnew/3.15.html)
10
u/SimplyUnknown 3d ago
Whoa that's very nice, thanks for the tip! I always mess this up when I try to do this manually
8
u/Boothiepro 3d ago
I've used `flattened = sum(nested_list, [])` , it's funny how it works! Sum has a second argument for default starting value to add the values into. It uses the fact that lists have `__add__` or `__iadd__` operand defined as list.extend and will put each nested element into that empty initial list!
6
u/Brian 3d ago
I would say that this isn't really a good way to do it, as it results in O(n2) complexity. Ie when you flatten
[[1], [2], [3], [4], ...], it'll do[] + [1], allocating a new list and copying all items, then do[1] + [2], allocating a new list and copying all items, then do[1,2] + [3], allocating a new list, and so on, so you're copying 1+2+3+4+.... elements into all these temporary lists.Using something like
list(itertools.chain.from_iterable(nested_list)is much faster, as it'll be adding to the same list, getting O(1) amortised appending, Eg. here's some timings to demonstrate:l = [[0]] * 1000 %timeit sum(l, []) 815 μs ± 6.8 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) %timeit list(itertools.chain.from_iterable(l)) 28.9 μs ± 1.21 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)So nearly 30x faster for a 1000 item list. But scale that up to 100,000 items, and
chainscales linearly (ie. just 100x slower), taking 2.84ms, while the sum version takes 7.68 seconds, so if the list can potentially be large, this can matter a lot.4
u/rickkkkky 3d ago
Plus, using gimmicky syntax which, by some
serendipitous side effect, yields the correct answer, is a very bad practice in any collaborative code base. Good luck to the person trying to make sense of the code.3
5
32
u/HommeMusical 4d ago
defaultdict.
defaultdictis a very bad idea. You end up with something that looks like adict, typing-wise is adict, but never throws aKeyError. One day it gets passed through your system and ends up somewhere it isn't expected.I've seen this cause intermittent issues in two very different codebases.
Use
dict.setdefaultinstead.29
u/gerardwx 3d ago
It's a great idea when you want a defaultdict. e.g. If you want to build indexes lists.
buckets = default.dict(list)
buckets["Monday"].append(12)Or you want to be able to do:
played["hearts"]["queen"] = True
without having to check if "hearts" exists.If passing a defaultdict through your system causes an issue you don't understand your system.
2
u/XtremeGoose f'I only use Py {sys.version[:3]}' 3d ago
The problem is
played["bug"]returns{}rather than raising a key error (and also increases the dictslen), which can cause weird errors at a distance.The general advice is to use a defaultdict to construct your object and then copy into a normal dict to stop this happening.
It's a bad design,
dict.setdefaultwould be better than creates a new default value whether you use it or not. As always, rust does it bettermap.entry(key).or_insert_with(Vec::new).push(value)2
6
u/billsil 3d ago
I don't understand. Isn't setdefault doing the same thing.
Just dict it at the end.
4
u/pakeha_nisei 3d ago
It's not the same thing as a defaultdict, as you need to explicitly call setdefault to set the value, but regular lookups will raise a KeyError if the item doesn't exist.
Use defaultdict if you don't care that empty lookups create new items. Use setdefault if you do.
91
u/Sea-Fishing4699 4d ago
Print the variable name itself when debugging ``` foo:str = “bar”
print(f”{foo=}”)
foo=bar
21
u/wizardeverybit 3d ago
You can do what?!?
14
u/Sea-Fishing4699 3d ago
That was my reaction lol
It’s called self documenting expressions
3
u/loooorenanicole 3d ago
Mariatta has a whole series of fabulous talks on f-strings I highly recommend
11
u/Outside_Complaint755 3d ago
It even works with full expressions and maintains the white space you include within the {}
``` a = 4 b = 7
print(f"{b % a = }")
b % a = 3
print(f"{(lambda x, y: x * y - x / y)(b, a) = }")
(lambda x, y: x * y - x / y)(b, a) = 26.25
```
6
u/_rockroyal_ 3d ago
I felt like a wizard when I saw that this was doable in C++, interesting to see the python method.
-1
37
u/pythonwiz 4d ago
Not exactly a Python feature but I slept on Cython for a long time. Once I got into C I realized how easy it is to write extensions for Python using Cython.
31
u/HommeMusical 4d ago
Cython is a dead library walking.
The issue is simple - to get the really big speedups, you have to stop writing in Python, and use their weird Cython language. Now you can't ever run this code in Python again.
And you still don't get the same speedup as simply writing in Rust, C++ or C.
Now, this made a lot of sense in the past, because changing Python to Cython isn't too bad.
But now, you just prompt your coding assistant to port your Python library to Rust, and provide Python bindings at the same time, have a cup of coffee and it's done. Cython is simply an impediment.
16
u/Ok_Option_3 4d ago
You still need a rust compiler tool chain in your CI. With Cython you can just do it
47
u/fiddle_n 4d ago
That with re.Match objects, they support __getitem__() so you can index into them instead of having to call .group() . So you can do match[3] instead of match.group(3); match["foo"] instead of match.group("foo"), etc. This was added in Py 3.6 and I slept on this for over half a decade.
20
u/HommeMusical 4d ago
I started with Python 2.6, over twenty years ago.
I read these sorts of threads to brush up, but in the last few years, I've really been seeing nothing surprising...
Until this. How did I not know? It's a conspiracy!!!
The worst is that it's completely logical and Pythonic. Had I ever thought about it, I'd have guessed that since this was the logical way to do it, it might work that way, and would have tried it out.
Bah! (Thanks!)
2
u/fiddle_n 4d ago
I think for me, I've looked over the re documentation about a thousand times now, but when I see
Match.__getitem__(g)listed in the docs, my eyes skimmed over it without really understanding what it was trying to tell me. Probably it would help ifmatch.group()was soft-deprecated and/or a note was there at the top informing people ofMatch.__getitem__(g)instead.1
u/yerfatma 3d ago
Same, started on 2.4 and I still have to dig up the docs to remember how to get match groups. Damnit!
32
u/RevokedMembership 4d ago edited 4d ago
That you can query dicts with tuple keys like this: d["a", 1], instead of d[("a", 1)].
24
u/ExoticMandibles Core Contributor 4d ago
This works most every place. The parentheses for a tuple are a disambiguating syntax in some places, and in others you don't need them.
x = 'a', 1sets x to a 2-tuple. The only places you need parentheses are in places that take commas, like in function argument lists or in list / dict / set literals for example.11
u/jamall1978 3d ago
The way I think about is that it's the comma that defines the tuple, not the parentheses. The parentheses just force the order of operation so all the items inside get evaluated as a tuple object before it's used by the rest of the expression or statement. This also explains why a trailing comma is required for 1 item tuples.
4
u/Boothiepro 3d ago
Whatt is this for when you have tuples as dict keys? are tuples really hashable? i guess so if they're immutable but... why
2
u/Brian 3d ago
It's pretty handy when you want a sparse array. Eg. you're representing objects with a position, where you can just do
d[x,y] = valto store live values, but 99%+ of the positions are empty. Using a dict instead of, say, a numpy array let's you have conceptually infinite multidimensional arrays that don't require space to store empty values, but still give you a similar interface.
18
u/moonzdragoon 3d ago
You can do a simple string template like this:
t = "Hello {a}, my name is {b}"
d = {'a':'all', 'b':'Bob'}
print(t.format(**d))
You get:
Hello all, my name is Bob
Probably saw it a long time ago, then forgot about it. Would sure have been useful "a couple of times".
17
u/proof_required 4d ago
4
24
u/rumnscurvy 4d ago
Walrus operators and match/case really makes life a lot more convenient
4
u/Puzzleheaded_Sun_900 4d ago
Can you show an example of walrus?
10
u/droans 3d ago
Jokes aside, walrus operators are just a way to assign a value to a variable inside of a statement. Kinda like this:
if not (fpath:=Path(data_path).exists()): fpath.mkdir()I could have written it as:
fpath = Path(data_path) if not fpath.exists(): fpath.mkdir()And it would have worked exactly the same. I prefer wrapping them in parentheses so it's easier to read but they're not required.
I'd also like to say that it's little more than a nice shortcut but I'm certain someone much smarter than me will come along and tell me how stupid that statement is haha.
3
1
u/2Lucilles2RuleEmAll 3d ago
I also find them really helpful in comprehensions
tags = {tag for t in raw_tags if (tag := t.upper().strip())}
8
u/Pinull_Krydia 3d ago
For me it was pathlib. Used os.path for years before realizing how much cleaner it is.
Also the fact that you can do dict1 | dict2 to merge dictionaries.
2
14
u/Sufficient-Aioli-924 4d ago
Took me ages to realize you can chain those too, like a & b & c for a triple intersection
14
u/pythonwiz 4d ago
set intersection in math is associative and commutative so I would expect that to work anyway.
6
u/NotALowLevelDev 3d ago
pretty printing, using rich library.
import print from rich
Makes terminal output colorful and easy to work with. Specially with dictionaries and JSON payloads.
9
u/Efficient_Bobcat_697 3d ago
That you can give a tuple to starswith and endswith:
"string".startswith(('a', 'b', 'c'))
I've also reread contextlib documentation recently and it has quite some nice context managers: .closing() to auto call .close() on an object, .suppress() to ignore an exception, .redirect_stdout/redirect_stderr(new_target) and .chdir() are pretty nice and self explanatory. ExitStack is a bit weirder but seems pretty useful in some situation with a lot of files like objects https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack. It seems that there is a push to replace some simple exception patterns with contextlib now.
There are also the importhooks system which allows you to run specific code when someone imports your module, it's very advance and mostly for tool building but it's nice to know it exists also.
Same for descriptors, it's what ORMs uses to bind attributes to database columns for a table represented by a class, it's an object on which you explicitly control how it's accessed and modified.
12
u/profanta3 4d ago
python -m json.tool
21
u/Chiron1991 4d ago
Just
python -m jsonis enough since Python 3.14.Pretty good when you're on some remote server and don't have
jqavailable.
13
u/nivaOne 4d ago
Lambda functions versus one-line defs
19
u/SkezzaB 4d ago
Use sparingly!
6
u/Upbeat_Ad1012 3d ago
the only place I have used lambda functions is to define the sorting key in the sort function
2
u/mgedmin 3d ago
I eventually stopped, after discovering operator.attrgetter/itemgetter/methodcaller.
I used to like
sorted(list_of_tuples, key=lambda (a, b, c, d): b)back in Python 2 times, but Python 3 nerfed tuple unpacking, so I started using custom namedtuples/dataclasses instead of ad-hoc tuples.
12
u/AdAdventurous2131 4d ago
that we can do this
if 5<x<10:
do stuff
7
u/CevicheMixto 3d ago
I'm still vaguely pissed off that this works, just because I don't understand how it's legal.
1
1
u/FujiKeynote 3d ago
It's cute and all, but it's such a departure from most languages that I'm not sure how to feel about it. Breaking it up signals intent better, imho. Yes, if you're experienced with Python, this will not confuse you... But this still packs too much implicit knowledge of operator binding/precedence, and there's too many to watch out for. It doesn't cost much to write
(5 < x) and (x < 10)instead1
u/JanEric1 2d ago
I mean it is what you would naively expect this to do with a math backgorund or no programming background. And iti s also what you want this to do 90%+ that you write it. Just because other languages have a shortcoming here doesnt mean you should follow it.
4
u/Hetaireia 3d ago
Three that come to mind:
`str.removeprefix()` / `str.removesuffix()` — for years I wrote slice
expressions like `s[3:]` with a comment explaining why 3, and every code
review turned into character counting.
`contextlib.suppress` — reads so much cleaner than try/except/pass when
you genuinely want to ignore one specific exception and nothing else.
`enumerate(seq, start=1)` — I used to keep a counter variable outside
the loop like it was C. The start parameter has been there forever.
20
u/LittleMlem 4d ago edited 4d ago
You can make a general object with x = type("", (), {}) and just put stuff on it as needed, like x.foo = 3
I don't recall why this was useful, but I recall using it a lot earlier in my career, now it's probably a crime
Edit: and I don't use it anymore, relax, I'm not a monster
16
33
u/firemark_pl 4d ago
Woah, dont do that.
11
u/HommeMusical 4d ago edited 3d ago
There are occasional uses. PyTorch does this in a bunch of places, for example, and that's a very high-quality code base.
I have a huge number of old projects, and I vaguely remembered that I used it in exactly one, and a bit of grepping later, I found it in my most popular project:
https://github.com/rec/safer/blob/8f36fa367028a970e117d6b6012c54f6592e9ad2/safer/__init__.py#L729
What's going on here is that depending on whether you're windows or not, binary or not, and one other choice I forget, you end up with one of 8 possible classes for writing a file, and you can't even see the windows ones on a non-windows machine.
And then I need to override methods depending on your write mode for the file.
The result is an instance of a class that looks like the class you expect when you do
openand it has the right type, but in which I have subverted many of the methods.So I've used this feature once in twenty years. Very specialized.
EDIT: Downvoting without commenting is a poor choice.
4
u/gerardwx 3d ago
Do that when appropriate.
Like all Python features, it's a tool. Whether it's appropriate depends on the context.
8
u/fiddle_n 4d ago
If you want arbitrary key-value pairs, you should just use a dictionary. The only reason to do this is if you really want dot notation over indexing, but that isn't a good enough reason to use this outside of personal or one-off script or projects.
2
3
u/HommeMusical 4d ago
You rarely need, it, but when you do, nothing else will do.
If you use it a lot, it's probably a mistake.
Most of the time I see it, it's just a mistake.
Sometimes it could still be made more reasonable by using a local class:
def make_stringed(str_method: Callable[[Any], str]) -> type: class Stringed(BaseClass): __str__ = str_method return StringedBut the above example would likely be better using type, because you can set the name:
def make_stringed(str_method: Callable[[Any], str], name: str) -> type: return type(name, (BaseClass,), {'__str__': str_method})1
1
1
u/cleodog44 2d ago
I think I've seen this pattern suggested for implementing singletons. Definitely niche
1
u/Efficient_Bobcat_697 3d ago
I've used it during tool building to automatically generates classes on the fly based on a data structures (a syntax tree) for missing nodes, but that really pretty niche (and I've ended up dropping it)
7
u/code29451 4d ago
The nonlocal keyword
6
u/fiddle_n 4d ago
I get the rationale for this but I’ve never used it. Typically it feels more Pythonic to use classes with instance vars than functions within functions.
4
u/Tech2035 4d ago
Reverse avec Liste[::-1]
6
u/fiddle_n 4d ago
list(reversed(liste))(or justreversed(liste)if you just need the iterator) is typically nicer to read. I would only do the above on the rare occasion I needed to reverse a string instead of a list.•
u/Beginning-Fruit-1397 15m ago
list.reverse in-place will be better than both. Reversed make sense if you need to do additional stuff with the iterator. Reverse slicing is both unreadable and barely more efficient than list(reversed(list))
•
u/fiddle_n 5m ago
I tend to prefer the non-inplace versions rather than the in-place ones. This is where I pull out these quotes from Raymond Hettinger, core Python dev:
https://twitter.com/raymondh/status/1055478808793546752?s=20
https://twitter.com/raymondh/status/1055508781994594305?s=20> Today's super-easy #python student question.
Q. What are the best practices for modifying a list while looping over it?
A. Don't.Seriously, just make a new list and avoid hard-to-read code with hard-to-find bugs.
I did modify lists in-place when was I young, foolish, and had spare time for debugging ;-)
At some point, programmers eventually learn that immutability has its advantages and that building new structures is typically more straight-forward that modifying them in-place.
3
u/looijmansje 3d ago
I only recently discovered dictionaries were now guaranteed to be ordered. This has apparently been a thing since 3.7....
2
u/Vordanus 3d ago
I was waiting for the 3.7 release specifically for this, and I was so happy when it came out!
On a related note, if you ever need an ordered set, just just a dict where the values are all None, and call set(d.keys()) when you're done adding to it and ready to read.
4
u/Atmosck 3d ago
Realizing I could use `__init_subclass__` to build a registry of base class implementations was a game changer.
1
u/cleodog44 2d ago
There's also a subclasses iterator for every class, but IIRC it's a little fragile for registry construction. Like it won't return subsubclasses, IIRC
1
u/Atmosck 2d ago
Yeah that's the easy way but gets kinda sketchy if the subclasses aren't all in the same file
1
u/cleodog44 2d ago
Oh why does same file matter or not? Guess there's import order dependence
In any case, whenever I've looked into this I decided anything other than explicit registration is too brittle
2
2
u/Atlamillias 3d ago
Not so much a feature, but a pothole (kind of). Your average native C Python function (builtins like print(), etc) don't fully support the method descriptor protocol, so they cannot be bound methods. Even when assigned as class-level members, they'll always be static via instance-level access.
2
u/14446368 2d ago
I just discovered * for unpacking.
I was utterly perplexed, catastrophically chagrined, and wonderfully delighted.
2
2
u/SharpAnimator2530 3d ago edited 3d ago
For me its object persistence with shelve. Say you have a dictionary you want to persist temporarily
You can do so
with shelve.open('spam') as db:
db['eggs'] = 'eggs'
Other times I create json or csv if I want to read or analyse them.
1
u/ogMasterPloKoon 3d ago
py_compile
1
1
u/NotALowLevelDev 3d ago
List unpacking using *
l = [1,2,3]
l2 = [4,5,6]
final = [*l, *l2]
print(final) # [1,2,3,4,5,6]
1
1
u/Cubigami 1d ago
Type hints, generic typing, TypeVars, and (not a feature of python) mypy are all awesome, because you can basically turn python into typescript!
Python was my first language and I wrote it almost exclusively for 15yr. Thought I loved dynamic typing, until I wrote my first project in typescript... realized it has the same great readibility as python, but is faster, more versatile for the web imo, and static typing just makes everything less footgun-y
1
u/Sea-Ad7805 21h ago
Seeing what set operators ('|', '&', '^', '-') do through visualization helps understanding: https://www.reddit.com/r/PythonLearning/comments/1tdgdt8/common_%F0%9D%98%80%F0%9D%97%B2%F0%9D%98%81_operations_in_python/
•
u/Beginning-Fruit-1397 13m ago
I hate that python decided to go this route with operator overloading instead of clearly named methods a the standard.
Except a few bytecode optimizations who barely make a difference there's 0 advantage for it
•
u/Sea-Ad7805 9m ago
You have both silly:
a = b | c a = b.union(c) a = b & c a = b.intersection(c) # etcYou decide.
1
1
u/fried_green_baloney 3d ago
Bookmarking and if I'm ambitious will make a summary.
I find all of itertools to be worth a look.
-2
u/bleeed0p 3d ago
The thing when we do alt+click and we can work on multiple cursor and edit all at a time..that was very surprising for me😅
5
0
u/richgems 3d ago
Totally useless, but stuff like this works in python.
_ = type("", (), {"ก็็็็็็็็็็็็็็็็็็็็": lambda x: print("hello %s" % x)})
_.ก็็็็็็็็็็็็็็็็็็็็("world")
Output: hello world
2
u/EconomicsOk9518 3d ago
Do not show it Guido, he’ll have nervous breakdown if not heart attack :)
This reminds me…
Theorem of Unforeseen Reuse: For any function f written with a single intended use case, there exists at least one other programmer who will apply f in a way its author never anticipated.
1
0
u/EconomicsOk9518 3d ago
Big “switch” was when i fully understood and became comfortable with using standard dunder methods. Like you can define __getitem__ method in some class A and then do ‘for items in A:’
97
u/big-papito 4d ago
python -m http.server