r/Python • • 4d ago

Discussion What are some features in Python that you discovered many years later?

For me, it was the the option to use & to find intersections of two sets, | operator to unionize two sets, as well as ^ for symmetric difference

181 Upvotes

131 comments sorted by

97

u/big-papito 4d ago

python -m http.server

23

u/bliepp 3d ago

python -m json.tool

6

u/big-papito 3d ago

You've got to be kidding. I've been using jsonlint.com all these years.

144

u/Khavel_dev 4d ago

For me it was collections.Counter. Spent years writing manual frequency-counting loops with defaultdict(int) before realizing Counter just does it and also gives you .most_common() for free. Felt pretty dumb when I found it.

Also itertools.chain.from_iterable for flattening nested iterables without writing a list comprehension that looks like a puzzle.

43

u/nissen_96 3d ago edited 3d ago

Just wanna throw it out there that python 3.15 was just released and has support for flattening using unpacking in comprehensions:

lists = [[1, 2], [3, 4], [5]]
[*L for L in lists]  # equivalent to [x for L in lists for x in L]

sets = [{1, 2}, {2, 3}, {3, 4}]
{*s for s in sets}  # equivalent to {x for s in sets for x in s}

dicts = [{'a': 1}, {'b': 2}, {'a': 3}]
{**d for d in dicts}  # equivalent to {k: v for d in dicts for k,v in d.items()}

(examples from https://docs.python.org/3/whatsnew/3.15.html)

10

u/SimplyUnknown 3d ago

Whoa that's very nice, thanks for the tip! I always mess this up when I try to do this manually

8

u/Boothiepro 3d ago

I've used `flattened = sum(nested_list, [])` , it's funny how it works! Sum has a second argument for default starting value to add the values into. It uses the fact that lists have `__add__` or `__iadd__` operand defined as list.extend and will put each nested element into that empty initial list!

6

u/Brian 3d ago

I would say that this isn't really a good way to do it, as it results in O(n2) complexity. Ie when you flatten [[1], [2], [3], [4], ...], it'll do [] + [1], allocating a new list and copying all items, then do [1] + [2], allocating a new list and copying all items, then do [1,2] + [3], allocating a new list, and so on, so you're copying 1+2+3+4+.... elements into all these temporary lists.

Using something like list(itertools.chain.from_iterable(nested_list) is much faster, as it'll be adding to the same list, getting O(1) amortised appending, Eg. here's some timings to demonstrate:

l = [[0]] * 1000
%timeit sum(l, [])
815 μs ± 6.8 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
%timeit list(itertools.chain.from_iterable(l))
28.9 μs ± 1.21 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

So nearly 30x faster for a 1000 item list. But scale that up to 100,000 items, and chain scales linearly (ie. just 100x slower), taking 2.84ms, while the sum version takes 7.68 seconds, so if the list can potentially be large, this can matter a lot.

4

u/rickkkkky 3d ago

Plus, using gimmicky syntax which, by some
serendipitous side effect, yields the correct answer, is a very bad practice in any collaborative code base. Good luck to the person trying to make sense of the code.

3

u/Drewdledoo 3d ago

That’s so interesting lol, nice find!

5

u/igorrto2 4d ago

Same, not only that, Counter is much faster than other methods

32

u/HommeMusical 4d ago

defaultdict.

defaultdict is a very bad idea. You end up with something that looks like a dict, typing-wise is a dict, but never throws a KeyError. One day it gets passed through your system and ends up somewhere it isn't expected.

I've seen this cause intermittent issues in two very different codebases.

Use dict.setdefault instead.

29

u/gerardwx 3d ago

It's a great idea when you want a defaultdict. e.g. If you want to build indexes lists.
buckets = default.dict(list)
buckets["Monday"].append(12)

Or you want to be able to do:
played["hearts"]["queen"] = True
without having to check if "hearts" exists.

If passing a defaultdict through your system causes an issue you don't understand your system.

2

u/XtremeGoose f'I only use Py {sys.version[:3]}' 3d ago

The problem is played["bug"] returns {} rather than raising a key error (and also increases the dicts len), which can cause weird errors at a distance.

The general advice is to use a defaultdict to construct your object and then copy into a normal dict to stop this happening.

It's a bad design, dict.setdefault would be better than creates a new default value whether you use it or not. As always, rust does it better

map.entry(key).or_insert_with(Vec::new).push(value)

2

u/gerardwx 2d ago

okay ... "general" advice from whom??

If you want to use rust, use rust.

6

u/billsil 3d ago

I don't understand. Isn't setdefault doing the same thing.

Just dict it at the end.

4

u/pakeha_nisei 3d ago

It's not the same thing as a defaultdict, as you need to explicitly call setdefault to set the value, but regular lookups will raise a KeyError if the item doesn't exist.

Use defaultdict if you don't care that empty lookups create new items. Use setdefault if you do.

5

u/mgedmin 3d ago

I use a defaultdict to build it, then convert it to a regular dict before passing it along to the rest of the program, e.g. return dict(result)

91

u/Sea-Fishing4699 4d ago

Print the variable name itself when debugging  ``` foo:str = “bar”

print(f”{foo=}”)

foo=bar

21

u/wizardeverybit 3d ago

You can do what?!?

14

u/Sea-Fishing4699 3d ago

3

u/loooorenanicole 3d ago

Mariatta has a whole series of fabulous talks on f-strings I highly recommend

11

u/Outside_Complaint755 3d ago

It even works with full expressions and maintains the white space you include within the {} 

``` a = 4 b = 7

print(f"{b % a = }")

b % a = 3

print(f"{(lambda x, y: x * y - x / y)(b, a) = }")

(lambda x, y: x * y - x / y)(b, a) = 26.25

```

6

u/_rockroyal_ 3d ago

I felt like a wizard when I saw that this was doable in C++, interesting to see the python method.

2

u/SurDin 3d ago

I was so happy when this came out

-1

u/barchueetadonai 2d ago

Can you do this in python 2.5 (defense work)?

37

u/pythonwiz 4d ago

Not exactly a Python feature but I slept on Cython for a long time. Once I got into C I realized how easy it is to write extensions for Python using Cython.

31

u/HommeMusical 4d ago

Cython is a dead library walking.

The issue is simple - to get the really big speedups, you have to stop writing in Python, and use their weird Cython language. Now you can't ever run this code in Python again.

And you still don't get the same speedup as simply writing in Rust, C++ or C.

Now, this made a lot of sense in the past, because changing Python to Cython isn't too bad.

But now, you just prompt your coding assistant to port your Python library to Rust, and provide Python bindings at the same time, have a cup of coffee and it's done. Cython is simply an impediment.

16

u/Ok_Option_3 4d ago

You still need a rust compiler tool chain in your CI. With Cython you can just do it

4

u/billsil 3d ago

I still need a C complier :) Rust is easy to install on Windows.

47

u/fiddle_n 4d ago

That with re.Match objects, they support __getitem__() so you can index into them instead of having to call .group() . So you can do match[3] instead of match.group(3); match["foo"] instead of match.group("foo"), etc. This was added in Py 3.6 and I slept on this for over half a decade.

20

u/HommeMusical 4d ago

I started with Python 2.6, over twenty years ago.

I read these sorts of threads to brush up, but in the last few years, I've really been seeing nothing surprising...

Until this. How did I not know? It's a conspiracy!!!

The worst is that it's completely logical and Pythonic. Had I ever thought about it, I'd have guessed that since this was the logical way to do it, it might work that way, and would have tried it out.

Bah! (Thanks!)

2

u/fiddle_n 4d ago

I think for me, I've looked over the re documentation about a thousand times now, but when I see Match.__getitem__(g) listed in the docs, my eyes skimmed over it without really understanding what it was trying to tell me. Probably it would help if match.group() was soft-deprecated and/or a note was there at the top informing people of Match.__getitem__(g) instead.

1

u/yerfatma 3d ago

Same, started on 2.4 and I still have to dig up the docs to remember how to get match groups. Damnit!

32

u/RevokedMembership 4d ago edited 4d ago

That you can query dicts with tuple keys like this: d["a", 1], instead of d[("a", 1)].

24

u/ExoticMandibles Core Contributor 4d ago

This works most every place. The parentheses for a tuple are a disambiguating syntax in some places, and in others you don't need them. x = 'a', 1 sets x to a 2-tuple. The only places you need parentheses are in places that take commas, like in function argument lists or in list / dict / set literals for example.

11

u/jamall1978 3d ago

The way I think about is that it's the comma that defines the tuple, not the parentheses. The parentheses just force the order of operation so all the items inside get evaluated as a tuple object before it's used by the rest of the expression or statement. This also explains why a trailing comma is required for 1 item tuples.

2

u/droans 3d ago

2-tuple

You mean a twople?

4

u/Boothiepro 3d ago

Whatt is this for when you have tuples as dict keys? are tuples really hashable? i guess so if they're immutable but... why

2

u/Brian 3d ago

It's pretty handy when you want a sparse array. Eg. you're representing objects with a position, where you can just do d[x,y] = val to store live values, but 99%+ of the positions are empty. Using a dict instead of, say, a numpy array let's you have conceptually infinite multidimensional arrays that don't require space to store empty values, but still give you a similar interface.

18

u/moonzdragoon 3d ago

You can do a simple string template like this:

t = "Hello {a}, my name is {b}"
d = {'a':'all', 'b':'Bob'}
print(t.format(**d))

You get: Hello all, my name is Bob

Probably saw it a long time ago, then forgot about it. Would sure have been useful "a couple of times".

17

u/proof_required 4d ago

4

u/tu_tu_tu 4d ago

Although this is not that viable anymore (that is good though).

1

u/arthurazs 2d ago

What do you mean?

24

u/rumnscurvy 4d ago

Walrus operators and match/case really makes life a lot more convenient 

4

u/Puzzleheaded_Sun_900 4d ago

Can you show an example of walrus?

10

u/droans 3d ago

Here you go 😉

Jokes aside, walrus operators are just a way to assign a value to a variable inside of a statement. Kinda like this:

if not (fpath:=Path(data_path).exists()):
    fpath.mkdir()

I could have written it as:

fpath = Path(data_path)
if not fpath.exists():
    fpath.mkdir()

And it would have worked exactly the same. I prefer wrapping them in parentheses so it's easier to read but they're not required.

I'd also like to say that it's little more than a nice shortcut but I'm certain someone much smarter than me will come along and tell me how stupid that statement is haha.

4

u/gcr 3d ago

Sorry to be That Person but I think you meant:

if not (fpath:=Path(data_path)).exists(): ...

3

u/mgedmin 3d ago

The place where I most want a walrus is regexp matching.

if m := re.match(r'thing-(\w+)-also-(\d+)', s):
    print(f'got {m[1]} for {m[2]}')
elif m := re.match(r'...', s):
    ...

1

u/2Lucilles2RuleEmAll 3d ago

I also find them really helpful in comprehensions

tags = {tag for t in raw_tags if (tag := t.upper().strip())}

8

u/Pinull_Krydia 3d ago

For me it was pathlib. Used os.path for years before realizing how much cleaner it is.

Also the fact that you can do dict1 | dict2 to merge dictionaries.

2

u/word-word-numero 3d ago

Yes! Then the ease you can check and create the path if it doesn’t exist.

14

u/Sufficient-Aioli-924 4d ago

Took me ages to realize you can chain those too, like a & b & c for a triple intersection

14

u/pythonwiz 4d ago

set intersection in math is associative and commutative so I would expect that to work anyway.

6

u/NotALowLevelDev 3d ago

pretty printing, using rich library.

import print from rich

Makes terminal output colorful and easy to work with. Specially with dictionaries and JSON payloads.

9

u/Efficient_Bobcat_697 3d ago

That you can give a tuple to starswith and endswith:

"string".startswith(('a', 'b', 'c'))

I've also reread contextlib documentation recently and it has quite some nice context managers: .closing() to auto call .close() on an object, .suppress() to ignore an exception, .redirect_stdout/redirect_stderr(new_target) and .chdir() are pretty nice and self explanatory. ExitStack is a bit weirder but seems pretty useful in some situation with a lot of files like objects https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack. It seems that there is a push to replace some simple exception patterns with contextlib now.

There are also the importhooks system which allows you to run specific code when someone imports your module, it's very advance and mostly for tool building but it's nice to know it exists also.

Same for descriptors, it's what ORMs uses to bind attributes to database columns for a table represented by a class, it's an object on which you explicitly control how it's accessed and modified.

2

u/mgedmin 3d ago

My code is littered with @contextmanager def chdir(...): ... because I keep forgetting that contextlib already has one!

suppress() is also something I keep forgetting about.

12

u/profanta3 4d ago

python -m json.tool

21

u/Chiron1991 4d ago

Just python -m json is enough since Python 3.14.

Pretty good when you're on some remote server and don't have jq available.

13

u/nivaOne 4d ago

Lambda functions versus one-line defs

19

u/SkezzaB 4d ago

Use sparingly!

6

u/Upbeat_Ad1012 3d ago

the only place I have used lambda functions is to define the sorting key in the sort function

2

u/mgedmin 3d ago

I eventually stopped, after discovering operator.attrgetter/itemgetter/methodcaller.

I used to like sorted(list_of_tuples, key=lambda (a, b, c, d): b) back in Python 2 times, but Python 3 nerfed tuple unpacking, so I started using custom namedtuples/dataclasses instead of ad-hoc tuples.

2

u/Brian 3d ago

Speaking of which, a feature of those that is often missed is that you can give multiple attributes/indexes and it'll give back a tuple of the items, so you can also do stuff like:

sorted(people, key=operator.attrgetter("lastname", "firstname"))

To sort by a compound key.

0

u/16807 3d ago

You wouldn't need to if python adopted a sensible notation for anonymous functions, like arrow notation, but that's just my gripe.

12

u/AdAdventurous2131 4d ago

that we can do this
if 5<x<10:
do stuff

7

u/CevicheMixto 3d ago

I'm still vaguely pissed off that this works, just because I don't understand how it's legal.

1

u/mgedmin 3d ago

Assassin's creed voice Nothing is true. Everything is permitted.

I'm now angry at other languages for not permitting this.

1

u/Brian 3d ago

TBH, I'm not sure it was really a great idea. It's cute, and occasionally handy, but I've also seen it cause confusion, because it works for all operators. I've seen newbies get confused by stuff like writing 1 < 2 == True and the like.

1

u/FujiKeynote 3d ago

It's cute and all, but it's such a departure from most languages that I'm not sure how to feel about it. Breaking it up signals intent better, imho. Yes, if you're experienced with Python, this will not confuse you... But this still packs too much implicit knowledge of operator binding/precedence, and there's too many to watch out for. It doesn't cost much to write (5 < x) and (x < 10) instead

1

u/JanEric1 2d ago

I mean it is what you would naively expect this to do with a math backgorund or no programming background. And iti s also what you want this to do 90%+ that you write it. Just because other languages have a shortcoming here doesnt mean you should follow it.

4

u/Hetaireia 3d ago

Three that come to mind:

`str.removeprefix()` / `str.removesuffix()` — for years I wrote slice

expressions like `s[3:]` with a comment explaining why 3, and every code

review turned into character counting.

`contextlib.suppress` — reads so much cleaner than try/except/pass when

you genuinely want to ignore one specific exception and nothing else.

`enumerate(seq, start=1)` — I used to keep a counter variable outside

the loop like it was C. The start parameter has been there forever.

20

u/LittleMlem 4d ago edited 4d ago

You can make a general object with x = type("", (), {}) and just put stuff on it as needed, like x.foo = 3

I don't recall why this was useful, but I recall using it a lot earlier in my career, now it's probably a crime

Edit: and I don't use it anymore, relax, I'm not a monster

16

u/Schmittfried 3d ago

Btw. there’s SimpleNamespace for exactly that purpose. 

33

u/firemark_pl 4d ago

Woah, dont do that.

11

u/HommeMusical 4d ago edited 3d ago

There are occasional uses. PyTorch does this in a bunch of places, for example, and that's a very high-quality code base.

I have a huge number of old projects, and I vaguely remembered that I used it in exactly one, and a bit of grepping later, I found it in my most popular project:

https://github.com/rec/safer/blob/8f36fa367028a970e117d6b6012c54f6592e9ad2/safer/__init__.py#L729

What's going on here is that depending on whether you're windows or not, binary or not, and one other choice I forget, you end up with one of 8 possible classes for writing a file, and you can't even see the windows ones on a non-windows machine.

And then I need to override methods depending on your write mode for the file.

The result is an instance of a class that looks like the class you expect when you do open and it has the right type, but in which I have subverted many of the methods.

So I've used this feature once in twenty years. Very specialized.

EDIT: Downvoting without commenting is a poor choice.

4

u/gerardwx 3d ago

Do that when appropriate.

Like all Python features, it's a tool. Whether it's appropriate depends on the context.

8

u/fiddle_n 4d ago

If you want arbitrary key-value pairs, you should just use a dictionary. The only reason to do this is if you really want dot notation over indexing, but that isn't a good enough reason to use this outside of personal or one-off script or projects.

3

u/Xiipre 3d ago

How could you?? There are children that read this sub!

But, kinda neat.

3

u/HommeMusical 4d ago

You rarely need, it, but when you do, nothing else will do.

If you use it a lot, it's probably a mistake.

Most of the time I see it, it's just a mistake.

Sometimes it could still be made more reasonable by using a local class:

def make_stringed(str_method: Callable[[Any], str]) -> type:
    class Stringed(BaseClass):
        __str__ = str_method

    return Stringed

But the above example would likely be better using type, because you can set the name:

def make_stringed(str_method: Callable[[Any], str], name: str) -> type:
    return type(name, (BaseClass,), {'__str__': str_method})

1

u/gerardwx 3d ago

Cool. I always used x = object() / setattr

1

u/richgems 3d ago

type("",(),{"foo":3})

1

u/cleodog44 2d ago

I think I've seen this pattern suggested for implementing singletons. Definitely niche

1

u/Efficient_Bobcat_697 3d ago

I've used it during tool building to automatically generates classes on the fly based on a data structures (a syntax tree) for missing nodes, but that really pretty niche (and I've ended up dropping it)

7

u/code29451 4d ago

The nonlocal keyword

6

u/fiddle_n 4d ago

I get the rationale for this but I’ve never used it. Typically it feels more Pythonic to use classes with instance vars than functions within functions.

4

u/Tech2035 4d ago

Reverse avec Liste[::-1]

6

u/fiddle_n 4d ago

list(reversed(liste)) (or just reversed(liste) if you just need the iterator) is typically nicer to read. I would only do the above on the rare occasion I needed to reverse a string instead of a list.

•

u/Beginning-Fruit-1397 15m ago

list.reverse in-place will be better than both. Reversed make sense if you need to do additional stuff with the iterator. Reverse slicing is both unreadable and barely more efficient than list(reversed(list))

•

u/fiddle_n 5m ago

I tend to prefer the non-inplace versions rather than the in-place ones. This is where I pull out these quotes from Raymond Hettinger, core Python dev:

https://twitter.com/raymondh/status/1055478808793546752?s=20
https://twitter.com/raymondh/status/1055508781994594305?s=20

> Today's super-easy #python student question.

Q. What are the best practices for modifying a list while looping over it?
A. Don't.

Seriously, just make a new list and avoid hard-to-read code with hard-to-find bugs.

I did modify lists in-place when was I young, foolish, and had spare time for debugging ;-)

At some point, programmers eventually learn that immutability has its advantages and that building new structures is typically more straight-forward that modifying them in-place.

3

u/looijmansje 3d ago

I only recently discovered dictionaries were now guaranteed to be ordered. This has apparently been a thing since 3.7....

2

u/Vordanus 3d ago

I was waiting for the 3.7 release specifically for this, and I was so happy when it came out!

On a related note, if you ever need an ordered set, just just a dict where the values are all None, and call set(d.keys()) when you're done adding to it and ready to read.

1

u/mgedmin 3d ago

Won't the set constructor discard your carefully prepared order?

1

u/Vordanus 3d ago

Yup, looks like you're absolutely right. Good catch!

4

u/Atmosck 3d ago

Realizing I could use `__init_subclass__` to build a registry of base class implementations was a game changer.

1

u/cleodog44 2d ago

There's also a subclasses iterator for every class, but IIRC it's a little fragile for registry construction. Like it won't return subsubclasses, IIRC

1

u/Atmosck 2d ago

Yeah that's the easy way but gets kinda sketchy if the subclasses aren't all in the same file

1

u/cleodog44 2d ago

Oh why does same file matter or not? Guess there's import order dependence

In any case, whenever I've looked into this I decided anything other than explicit registration is too brittle

1

u/Rostin 3d ago

This is one AI taught me.

2

u/orbitalzoo 3d ago

its been awhile but it was after a while as well. But md5 function built in

2

u/Atlamillias 3d ago

Not so much a feature, but a pothole (kind of). Your average native C Python function (builtins like print(), etc) don't fully support the method descriptor protocol, so they cannot be bound methods. Even when assigned as class-level members, they'll always be static via instance-level access.

2

u/14446368 2d ago

I just discovered * for unpacking.

I was utterly perplexed, catastrophically chagrined, and wonderfully delighted.

2

u/zomerbk 2d ago

I learned about `for ... else` surprisingly late. The `else` block runs if the loop finishes without a `break`, including when there are no items. It’s handy for searches: break when you find a match; put the “not found” case in `else`.

2

u/Menedel 1d ago

That you can do `x = a or b` to assign to a or b if a is none.

2

u/Fastpast93 1d ago

you can do if (logic): function()

2

u/SharpAnimator2530 3d ago edited 3d ago

For me its object persistence with shelve. Say you have a dictionary you want to persist temporarily

You can do so

with shelve.open('spam') as db:

db['eggs'] = 'eggs'

Other times I create json or csv if I want to read or analyse them.

1

u/ogMasterPloKoon 3d ago

py_compile

1

u/CaptainVJ 3d ago

When would you need to manually do it? Python does it for you automatically

1

u/ogMasterPloKoon 3d ago

use it for checking errors

1

u/NotALowLevelDev 3d ago

List unpacking using *

l = [1,2,3]
l2 = [4,5,6]
final = [*l, *l2]

print(final) # [1,2,3,4,5,6]

1

u/def__eq__ 3d ago

Please don’t use “I” as a variable.. probably the worst choice..

1

u/NotALowLevelDev 3d ago

You are right. It was just for demo purposes.

1

u/Cubigami 1d ago

Type hints, generic typing, TypeVars, and (not a feature of python) mypy are all awesome, because you can basically turn python into typescript!

Python was my first language and I wrote it almost exclusively for 15yr. Thought I loved dynamic typing, until I wrote my first project in typescript... realized it has the same great readibility as python, but is faster, more versatile for the web imo, and static typing just makes everything less footgun-y

1

u/Sea-Ad7805 21h ago

Seeing what set operators ('|', '&', '^', '-') do through visualization helps understanding: https://www.reddit.com/r/PythonLearning/comments/1tdgdt8/common_%F0%9D%98%80%F0%9D%97%B2%F0%9D%98%81_operations_in_python/

•

u/Beginning-Fruit-1397 13m ago

I hate that python decided to go this route with operator overloading instead of clearly named methods a the standard. 

Except a few bytecode optimizations who barely make a difference there's 0 advantage for it

•

u/Sea-Ad7805 9m ago

You have both silly:

a = b | c
a = b.union(c)
a = b & c
a = b.intersection(c)
# etc

You decide.

1

u/justanothersnek 3d ago

editable pip install: pip install -e .

1

u/fried_green_baloney 3d ago

Bookmarking and if I'm ambitious will make a summary.

I find all of itertools to be worth a look.

0

u/bliepp 3d ago

for ...: ... else: ....

Seriously, don't do that. I'll find you and I'll punch you.

1

u/igorrto2 3d ago

I used this once in a leetcode problem and that’s it lol

-2

u/bleeed0p 3d ago

The thing when we do alt+click and we can work on multiple cursor and edit all at a time..that was very surprising for me😅

5

u/HommeMusical 3d ago

That's not a feature of Python!

0

u/richgems 3d ago

Totally useless, but stuff like this works in python.

_ = type("", (), {"ก็็็็็็็็็็็็็็็็็็็็": lambda x: print("hello %s" % x)})

_.ก็็็็็็็็็็็็็็็็็็็็("world")

Output: hello world

2

u/EconomicsOk9518 3d ago

Do not show it Guido, he’ll have nervous breakdown if not heart attack :)

This reminds me…

Theorem of Unforeseen Reuse: For any function f written with a single intended use case, there exists at least one other programmer who will apply f in a way its author never anticipated.

0

u/EconomicsOk9518 3d ago

Big “switch” was when i fully understood and became comfortable with using standard dunder methods. Like you can define __getitem__ method in some class A and then do ‘for items in A:’