Transcript #316: Python 3.11 is here and it's fast (crossover)
Return to episode page view on github00:00 Hey folks, for our final episode of 2022 here on Python Bytes, we're crossing the streams with my
00:05 other show, Talk Python to Me. I present to you one of the more important episodes over there
00:10 for the year, the release of Python 3.11 with its new features and 40% performance improvements.
00:16 Thank you for listening to Python Bytes in 2022. Have a great holiday break,
00:22 and Brian and I will see you next week. Here's that Python 3.11 episode.
00:28 Python 3.11 is here. Keeping with the annual release cycle, the Python Core devs have released
00:34 the latest version of Python, and this one is a big one. It has more friendly error messages and
00:39 is massively faster than 3.10, being between 10 to 60% faster in general, which is a big deal for a
00:47 year-over-year release of a 30-year-old platform. On this episode, we have Erit Katril, Pablo Galindo
00:53 Sogato, Mark Shannon, and Brant Booker, all of whom participated in releasing Python this week.
00:58 The hero on Talk Python to tell us all about that process and some of the highlight features.
01:02 This is Talk Python to Me, episode 388, recorded October 28th, 2022.
01:21 Welcome to Talk Python to Me, a weekly podcast on Python. This is your host, Michael Kennedy. Follow
01:27 me on Twitter, where I'm @mkennedy, and keep up with the show and listen to past episodes at
01:31 talkpython.fm. And follow the show on Twitter via at Talk Python. We've started streaming most of our
01:37 episodes live on YouTube. Subscribe to our YouTube channel over at talkpython.fm/youtube to
01:43 get notified about upcoming shows and be part of that episode.
01:47 Hey, everyone. Welcome to Talk Python to Me. It's great to have you all here. Erit,
01:51 Brant, Pablo, and Mark. It's going to be super fun to speak with all of you about Python 3.11.
01:57 Before we get into it, I guess, just real quickly, I know some of you have been on the show before,
02:02 but not all of you. So let's just do a quick introduction about who you are and how you ended
02:07 up here on the show. Erit, you want to start first?
02:09 Yeah. Hi, I'm Erit, I'm a Python core dev. Earlier in the week, we streamed the release of Python 3.11.
02:16 And on the back of that, Michael just invited us all here for chat.
02:20 Fantastic. Yeah. That was a great live stream. And we'll talk about that for sure in a second.
02:25 But Brant, welcome back.
02:26 Hello. My name is Brant Bucher. I have been a core dev for like two years now. And I work with Mark and
02:33 Erit on the Faster CPython team at Microsoft.
02:35 Right on.
02:35 And I was on the show like a month ago.
02:37 Yeah, you were talking about the Faster Python stuff, which we'll touch on again.
02:41 Hello. I'm Pablo Galindo. I'm the infamous release manager. I released Python 3.11. And you can
02:48 redirect all your complaints to my email address. No, please don't do that. So I'm a cPython core dev.
02:54 I'm also serving this year and the last year on the Python steering council. And I also release,
03:00 I'm the release manager for Python 3.10 and 3.11, which is now the best version of Python. Download it
03:06 today. Apart from that, I do a bunch of parser stuff. But now we are not talking about that.
03:10 Yeah, fantastic. Well, welcome. Mark, welcome back.
03:12 Hi there. I'm Mark Chan. I'm the tech lead of the Faster CPython team. I work with Erit and Brant.
03:17 I've been a core dev for some number of years. I don't recall.
03:20 You've been spending a couple of years working on this Faster CPython thing and very excited to see
03:27 some of the fruits of those labors, you know, starting to show up and get in the hands of
03:31 everyone with this release. Yeah, it's good to have the stuff out actually in public and in people's
03:35 hands. It's really rewarding to know that stuff you're working on is actually used and used by a
03:39 lot of people. Yeah, that is totally true. It's one thing to build software. I mean, just by itself,
03:44 but it's fun. But all of you are working on code that touches so many people. Think about there's
03:51 layers, right? One layer is how many people use Python? Many millions, millions. Does anyone know
03:57 a reasonable estimate of this number? I think some, I don't remember who came with the number,
04:02 but I think they were estimating like 6 million Python developers, something like that. I mean,
04:07 probably is between zero and 10 million, let's say. Yeah, that's a massive impact, but, and also maybe
04:14 nervousness about pushing code out to that group. But then, you know, those people will build software
04:19 for others, right? If you're using Instagram or using YouTube or other things, right? It's also
04:25 having massive knock-on effects there. So thanks for putting all this together. Thanks for improving the
04:30 tools that we all get to use. So yeah, big news. The big news is that Python 3.11 is out. And as
04:39 Iretz had said, you all live streamed that release. So here we're all together, we're having an awesome
04:46 chat about the features and the, what people can do to take advantage of it and why they might care about
04:51 new features and want to learn them. But there you did a little bit of that, but also Pablo, you actually
04:57 step-by-step did the release of CPython mostly live, right? Yeah, I did. It was, except the boring
05:04 parts. This is something that I started last year because apparently I didn't have enough
05:11 things to worry and I decided to make my life even more difficult. I'm an expert on that. Quite
05:16 proficient. I'm also an expert. I'm very bad at doing too many things and yeah. You could be a release
05:21 manager. It's the only requirement. So yeah, the idea is that the kind of releasing Python is a process that
05:28 is quite complicated. It's also quite boring. So it's not like, you know, you need to be,
05:34 have a galaxy brain kind of thing to do it, but it's just a lot of steps and it's very easy to do
05:38 it wrong. And it's very unglamorous. So I said, oh wow, I'm sure people really would like to see
05:45 a very unglamorous process happening in life. And then I said, let's do it. And I asked around and I
05:50 was surprised about how many people enjoy unglamorous processes. And then I did the release of Python 310
05:55 beta 1, which turned out to be much funnier than I thought because we just broke GitHub. That happened live.
06:01 Yes. Was that when you imported all the, you imported all the issues and did that migration or was that
06:07 separate? You will think that that is a good candidate, but no, that was not the thing that broke GitHub.
06:12 We renamed master to main on the CPython repo and the whole GitHub platform was down. What about that?
06:17 Wow. Yeah. You can see those Ruby workers really struggling with the renaming, all those forks.
06:25 I think we were the, I don't know, someone at GitHub may confirm this, but I think we were the first big
06:30 project to do the renaming. Something went wrong. And it was very funny because I literally said,
06:36 how funny will it be if now I get a 500? There you go. A 500 on the screen. Yeah. Yeah. It's recorded.
06:41 There is someone actually recorded that clip. Yeah. Yeah. So I said, wow, man, this has been a
06:46 such excitement thing that I can break such a big project. Let's do it more. So I decided also to
06:51 stream the, the 310 release itself. And I said, well, technically the release, the final release is,
06:57 is, is, is even more boring and longer. So that is actually probably not going to be even like,
07:03 you know, something that someone went to see. So I said, okay, let's, let's not do it alone.
07:06 So I invited a bunch of friends and core developers so they can actually talk about, you know, the things
07:12 that they worked on the Python 3.10 release and a brand that needed were there so that they can,
07:18 they can probably tell you how they found out. But like, apparently it was something that a lot of
07:22 people enjoy because, you know, it's not only a opportunity to see how the sausage is made because,
07:28 you know, I was just explaining all the commands and all the faces and whatnot, but like when something
07:32 became very boring, then, you know, like brand and it were there to save the day and explain the cool
07:37 things they, they work on. So, you know, which is a very good opportunity because, you know, when is
07:42 the last time you could hear the author of the feature that you love, talk about the feature that
07:47 you love. That is fantastic. And it happened.
07:48 Right. And not only did it happen, but as they were explaining the feature that they built,
07:54 the action of it being delivered to the entire world was right. It was like all coming together in a
08:00 pretty awesome way. Exactly. And I could only do that just to be fair also, and, you know,
08:04 create what credit is due. I could only do that because the first time I did the live thing,
08:10 I was also doing all the, you know, pushing all the buttons and at the same time doing all the video
08:14 stuff with, I don't know what is the software to the stream, but like whatever. And the second time we
08:20 use the help of the Python discord team, which are fantastic. And they, they help us a lot. They,
08:26 they, they have this fantastic UI where, you know, all the questions that were asked on the chat,
08:32 there are some on the screen and we couldn't use it. Do you know why? Because Facebook or now Meta
08:38 decided to break DNS globally. What an incredible fit, just in time. I think one.
08:44 So what I'm learning is if we need some sort of like big cloud global outage, you all just need to live.
08:50 Just call Pablo. Yes, exactly. Exactly. Just hire me today. So yes, that, that, that, now we were like
08:58 two big outages on, on Python release. The, you know, there is only a line that passes through two points, but I, you know, it was a, it was a, it was a, it was a, it was a, it was a, it was a good, a good statistic already. So we said, what can, what else can we break? So there you go. We decided to do the three levels.
09:02 release again. Then Mark was there as well, which increases the probability of things being broken by a lot. Sorry, Mark. I had to do the joke. He also fixes them. So, you know, it's fine and nothing broke. So, so kudos to Mark. Everything. Thanks to that. And we did the release. So, so we did the same thing.
09:31 We explained the whole thing. So people could see from the authors themselves, like why all the switches are very cool. And I did the non boring parts of the release. And then we have a bit of some dramas in backstage because my Juviki that I used to sign release broke and I freak out quite a lot, but I thankfully have a backup Juviki. So nobody had, yeah, yeah. So crazy. Because if I didn't have that, then I will have to stop the whole thing, but we didn't have to do that.
10:00 So yeah, it was just backstage. So yeah, quite, quite exciting. Nothing broke except my Juviki. I suppose that's the third thing that broke. It's not a global, you know, software, but I still mourn it. It's here.
10:10 Yeah. It served you well, but now it's, it gave its life for Python 3.11.
10:15 Too much power. Like 3.11 was too powerful.
10:18 It just broke.
10:21 This is a dangerous job that you got.
10:22 Yeah, yeah, yeah.
10:24 But you've handed it off, right? This is your last time, last main release.
10:29 Yeah, yeah. I need to do the security and bug fix releases, but I don't need to do the ones that, you know, you need to chase people down and ask for like cherry picking.
10:37 And there was a bunch of things of the release that were quite boring.
10:40 Like normally we release the previous version, like before the final version, there is something called the release candidate, which is, you know, like the last version that people need to try out before we do the final release.
10:51 And ideally that is the last version that we publish. Normally it means that you publish from that commit, but this is not the case.
10:59 This is the first release that had 130 something commits on top of that, but I have to painstakingly cherry pick and it was not fun, but I did that before the release.
11:08 It's like two hours because you need to fix conflicts and things like that. Yeah. Very, very boring.
11:13 But yeah, I started the stream with that already done. So it was fine.
11:17 Yeah. Fantastic.
11:18 Now, before we get into all the features and I want to maybe just talk a little bit about some of the tools for actually doing the release and maybe start with you is what, what is 3.11 mean for you all getting this out?
11:33 What does that mean for the Python community from your perspective?
11:36 Well, 3.11 is, it is a huge release. There's a lot packed into it compared to the last few releases.
11:42 There are no features. There's the performance work. It's, it's just massive changes internally.
11:48 It's, it's just a huge release. And personally, I've started working on, you know, exception groups about two years ago. So for me, this is, it almost feels like finishing another PhD or something. It's, it's a massive kind of effort. And here it is. It's done.
12:03 Yeah.
12:03 It was a big day Monday. I had a bottle of champagne ready for the stream. It was a celebration.
12:08 Yeah, it was. Brent, how about you?
12:11 I'm really excited about 3.11 because I think there's something for everyone. And I think you'd be hard pressed to find someone who doesn't want their code to run faster and who doesn't want better error messages. And then you have all these other improvements on top of that. It's really nice to see both these like new features, which are something that we get in most Python releases, but also just the stuff that's there for everyone else who just wants to upgrade Python and just have a better experience all around.
12:35 Yeah, I totally agree with that.
12:36 It's cool to see people's responses to that too, because responses have been really, really positive, which is another thing that I liked about the live stream because we did, you know, live Q&A and we had the chat and everything going on. And when you're staring at the same code base for like a year, you're like, okay, I'm pretty sure that what we've done here is really, really cool. But, you know, like, is it actually as awesome as I think it is? You know, or have I just been staring at it for too long?
12:58 Yeah.
12:59 And then release it to the world and people are even more stoked about it than you are. And that's a really good feeling.
13:03 Yeah, it is. Awesome. Mark?
13:06 Yeah, well, I guess I started on trying to get Python faster 15 years ago, I guess, early PhD time.
13:11 Yeah, with Hot Pie, right?
13:12 Yeah, yeah. So that was a long, long. This has been a long time coming. So yeah, it's amazing to have it actually out and starting to see the speed ups. And obviously we're keeping working on it. So it's pretty good.
13:24 Yeah, fantastic. You must be really proud because like you said, you have been proposing this for a really long time. You've had a lot of ideas. And finally, you've got a group of people working on it. And you're all on the same team with Mark and Guido.
13:37 Yeah. And just making legitimate, serious progress here. So it's, you must be really proud to just sort of see this actually go out the door.
13:46 Yes, definitely.
13:47 Especially in main Python too. It's really nice that we're able to, you know, deliver this for everyone.
13:53 Yeah. For me, I see basically three things like, kind of like you said, Brent, I see that obviously there's these new features like exception groups, which are lovely and make the language better.
14:03 But it also gets friendlier for, especially for beginners, but for everyone, of course, with the better error messages and better reporting and tracebacks.
14:11 And it gets faster. And so, I mean, it, and all the axes that seem to matter. It's, it's really fantastic.
14:17 Okay. So let's dive in. I just, Pablo, let's go back just a little bit to the, the release process because people got to watch you do it, but they didn't actually, you know, see exactly what you're typing on your screen the whole time.
14:30 It was more of about a, like an event of it. Sometimes your screen was up, sometimes it wasn't, but there's an official PEP that talks about like, here's the recipe for doing this, right?
14:39 That is correct. It's PEP 101 doing Python releases. And that is a curious document. It's peculiar document.
14:46 It talks about how it's done, but it's like, it's kind of weird. So the document is up to date. Like you can actually, you know, search PEP 101 and it will show you the thing.
14:56 So what is there is the actual process. It's just, it also contains these weird sentences. Like if you search for it, there is a bunch of places that says stop, stop, stop, stop, stop, stop.
15:07 And that was quite funny. And that was a, if I recall correctly, Larry Hastings, that he wrote those things. And the idea is that he could search for those places.
15:16 And he knows that at that stage he needs to wait for something to happen or something. And we left it there. So there's a bunch of like weird artifacts.
15:25 And, you know, it's full of bullet points because you, some stages you need to do some things and some others and things like that. And, you know, it says, okay, if you're running a beta release, then you need to do this bunch of things. And if you're running an alpha release, you need to do this bunch of things. And I have done the, I have done a state machine that goes through the whole thing.
15:43 And I have done the whole thing because like, if you actually write this down is quiet, is it, you know, the, the, how is this called the maintainability index of this process is insane. It is just rejects your thing. Just don't, don't merge it. Right. And I said like, yeah, I'm not doing this reading. So one thing I did, which is the thing that I was using at the, at the stream, my first work as a release manager is say, I'm not going to do this by hand. And that is the vision.
16:08 And then I did this script that is on a, GitHub slash Python slash release tools. And it's a, it's my attempt at automating this process as much as possible, which unfortunately, you know, it still requires a bunch of manual steps because like that's life and things happen, but, it's quite automatic. Like at least things that are not like final releases. So alpha and release candidates are now that we are in backfix releases, it mostly runs automatically, except that, you know, in the final release,
16:38 everything fails because there's the final release for you. And then you need to fix things manually. So you, I think you saw me, you know, executing a bunch of those fixes at some point.
16:47 And I added a division by zero just to know that something was hit. And that was seen on the screen because like, and people were like division by zero. Why do you need that to release Python? I don't know. That's very complicated.
16:58 I could have asserted false. Come on. Anything wouldn't work.
17:01 No, no. We divide by zero. I'm a physicist. So that's what I do.
17:04 Okay.
17:06 You studied black holes, right? Yeah.
17:08 You were looking for some sort of like infinite sort of thing there divided by zero. Yeah.
17:13 I'm too tired for today. Let's just collapse the universe divided by zero.
17:17 Oh, but Python was too friendly. Instead of collapsing the universe, it sold me an exception, you know, quite nice.
17:22 Only in 311. No, no, I'm joking. Anyway. So yes, yes. You can follow this PEP and, you know,
17:28 just enjoy the whole process on its glory or you can see the script, but yeah, it's quite verbose. You can see that it's very, it's lots of places when everything can go wrong and you can panic.
17:38 Now we know one more, apparently your Juviki can break. So that's something that can happen as well. But like, you know, it's quite annoying. And that's the main job of the release manager. Go through this annoying process. So yeah.
17:50 Yeah. I see that there's some parts in here. You should have a few more stops. I should say, stop, stop, stop. Make sure GitHub still works. Stop, stop, stop. Make sure Azure still works. Stop, stop, stop.
17:59 Yeah. Don't cry. Don't cry at this stage. Everyone is looking at you.
18:03 But yeah, the one thing that is not in this PEP is that you also are in charge in theory of this extreme abstract mandate, which is that you're in charge of the stability of the release, whatever that means.
18:14 That translates mostly on chasing people because they broke things. Another unfortunate event that we are trying to also fix a bit for the releases is that most people turn to the release manager to solve problems.
18:27 So they say, Hey, this person says that we should do X while this other person says that we should do Y. We need someone to decide. Let's, let's, let's, let's reach the release manager.
18:36 But the release manager is this guy on the corner. Like he doesn't know shit. So I like, you know, it's not the best, not the best person to fetch it. But everybody was like, what do you think, Pablo?
18:45 So we merged this. I'm like, I don't know, man, this is some enum things. Like, I don't know about this. I have no context whatsoever.
18:51 Your only concern is, will it still build and shift?
18:54 Exactly. Like how, like, yeah, what about these 2000 lines of code that sold this tiny bag? It was like, well, maybe let's not merge that. But yeah, like we are trying to also like, you know, redirect all of this to the steering council, which also I am in the steering council. So apparently I'm not going to get rid of these questions.
19:09 I'm joking. I enjoy all these questions, but as a release manager, I don't. So I like the key here is that the release manager should not take unilateral decisions on the evolution of these things because like it's just the release manager.
19:19 So the reason the steering council is five people.
19:22 But you are the one who delivers the code. You could kind of, you could sneak a feature in there.
19:26 No, no, no. I don't decide important things. I just execute and chase people. And I'm this annoying guy that says, you broke this, fix it.
19:32 But like then if there is some important decisions to be taken, you know, that's the steering council job, which is five people.
19:38 Because, you know, one person shouldn't decide these things. It's like, and this happens. Like sometimes I say, hey, there is this PR when people are asking, what should we do?
19:47 And then this is my opinion as the member of the steering council and the other four members, maybe they say, well, actually, that's not a good opinion.
19:54 So what about this? You know, we ended in a much better place because it was five people, five persons doing a decision instead of one.
20:00 This portion of Talk Python to me is brought to you by Microsoft for Startups Founders Hub.
20:07 Starting a business is hard. By some estimates, over 90% of startups will go out of business in just their first year.
20:14 With that in mind, Microsoft for Startups set out to understand what startups need to be successful and to create a digital platform to help them overcome those challenges.
20:24 Microsoft for Startups Founders Hub was born. Founders Hub provides all founders at any stage with free resources to solve their startup challenges.
20:33 The platform provides technology benefits, access to expert guidance and skilled resources, mentorship and networking connections, and much more.
20:41 Unlike others in the industry, Microsoft for Startups Founders Hub doesn't require startups to be investor backed or third party validated to participate.
20:51 Founders Hub is truly open to all.
20:54 So what do you get if you join them?
20:55 You speed up your development with free access to GitHub and Microsoft Cloud computing resources and the ability to unlock more credits over time.
21:03 To help your startup innovate, Founders Hub is partnering with innovative companies like OpenAI, a global leader in AI research and development, to provide exclusive benefits and discounts.
21:13 Through Microsoft for Startups Founders Hub, becoming a founder is no longer about who you know.
21:19 You'll have access to their mentorship network, giving you a pool of hundreds of mentors across a range of disciplines and areas like idea validation, fundraising, management and coaching, sales and marketing, as well as specific technical stress points.
21:32 You'll be able to book a one-on-one meeting with the mentors, many of whom are former founders themselves.
21:37 Make your idea a reality today with the critical support you'll get from Founders Hub.
21:43 To join the program, just visit talkpython.fm/founders hub, all one word, no links in your show notes.
21:48 Thank you to Microsoft for supporting the show.
21:52 Amazing.
21:53 Okay, so if people want to follow along with the process, they can check out PEP 101.
21:56 Let's keep over here.
21:58 You also talked about the Python build bot that people can check out, but I think maybe we want to jump into our first feature.
22:04 There's, as Arit said, there's a ton of features and things in here, but there's also maybe some top-level ones that'll be really important for a lot of folks.
22:12 And Arit, you want to tell us about your work?
22:15 You mentioned before the exception groups and exception star.
22:18 This is kind of a major new feature that we added, and the idea is that sometimes you'll have a situation where you did several things and maybe more than one of them raised an exception.
22:29 And now you need to report that there was more than one error in whatever you did.
22:33 And what you did could have been a bunch of asynchronous tasks, which is, that was the use case that motivated this whole thing.
22:40 But there are also situations where you just iterate over a few things and repeat them and accumulate exceptions, and you want to kind of report all of them.
22:50 And the PEP lists a bunch of examples of where this can happen.
22:55 So people, typically what they do is they'll take a list of exceptions, wrap it in another exception, multi-error, some other kind of wrapper, and throw that.
23:05 And then you have to catch it.
23:06 And then you have to iterate over the list and look at the exceptions.
23:09 But you don't have a method to handle the exceptions.
23:12 Like you have try accept, like catch these, but not catch exceptions.
23:16 Right, right.
23:16 Because in accept, you might have like accept socket error, or you might have accept like file not found type of thing.
23:23 But if those both happen, neither of those would run in Python 3.10, right?
23:28 Because it's some kind of weird wrapper, and it's not a socket exception.
23:31 It's not a file exception, but it kind of contains both.
23:34 And so in a sense, you can both run?
23:36 I don't know.
23:36 And then if you catch the wrapper, and you do something with some of the exceptions, you better not forget to erase the rest because you're not handling them.
23:44 So yeah, there are a lot of problems when you try to work around this.
23:47 And like what happened with Trio.
23:49 So Trio had multi-error, would raise this wrapper, and it had to do a lot of complicated acrobatics just to have some error handling.
23:58 So the motivation was, yeah, we have task groups in Python 3.11, which are kind of like Trio nurseries.
24:05 Kind of a structured collection of asynchronous tasks.
24:09 And task groups were on the cards.
24:12 He started, like Yuri Selibanov, who was kind of maintaining asyncio in the beginning.
24:17 He wrote a lot of asyncio.
24:18 He wanted to add task groups since 2017, 2018, something like that.
24:23 And what was holding it up was error handling.
24:26 There was no good way to handle errors.
24:29 So now we have accept star, which generalizes accept and works with groups.
24:34 So you can say accept star socket error, and then it will just extract all the socket errors from the group and give you those and automatically re-erase everything else.
24:44 That's basically the idea.
24:45 This is pretty interesting.
24:46 We have try, do your thing, and then accept star, you know, one error type.
24:51 Accept star, another error type.
24:53 Accept star, a set of errors, potentially.
24:55 So what happens if I'm in this situation and, say, the first error type and maybe something from the third error catch clause is thrown in one of these task groups, exception groups?
25:08 Each exception in the group will be handled by at most one of the clauses.
25:12 So the first clause that matches its type will consume it, and each clause executes once.
25:18 So if there are more than one errors of that type, then what gets kind of bound in the accept star full error as e, what gets bound to e is a group of full errors.
25:28 So you get all the full errors in a group, execute that clause, and then move on to the next clause with whatever is not handled yet.
25:35 Interesting.
25:36 So it might run two of the clauses.
25:39 Exactly.
25:40 Whereas in traditional exception handling, it goes from top to bottom, and it looks for an inheritance type of match.
25:45 And the first one that matches, that's it.
25:47 But in this case, with the star, you could get multiple ones.
25:51 I guess the star, to me, when I look at this, the star is reminiscent of Argg's star, where you have...
25:58 Unpacking, yeah.
25:59 Yeah, yeah, exactly.
26:00 It's not exactly unpacking, but it was kind of the intention to make it look a bit like unpacking.
26:06 Nice.
26:06 Yeah, this looks like a really cool feature.
26:08 You talked about the task groups and trio and those things.
26:12 So when I saw this, concurrent errors obviously come to mind.
26:15 Because if I try to both write something to a database and call a web service asynchronously, and I start both of those, and they both crash, or multiple ones crash, which error do you want?
26:25 The database error?
26:25 Or do you want the API error?
26:26 You probably want to know about both of them, right?
26:28 So that's a real natural reason to bring these together.
26:31 But maybe you'll also list out some of the other reasons that you might run into this.
26:36 Maybe give people some other ideas.
26:37 So the example of the socket module, we have the create connection function.
26:41 And that function, I was showing it in the stream.
26:44 It iterates over all the configurations that you could try to connect with.
26:49 And then depending on what's going on on the other side, hopefully one of them works.
26:52 But if none of them work, you have to report errors.
26:55 And what we do in Python 3.10 is we just raise the last exception.
26:59 So you don't know what happened, really.
27:01 You only know why the last attempt failed.
27:04 You don't even know how many attempts were made to connect to.
27:06 How many configurations did we try?
27:08 So that was a long-standing open problem, kind of, can we do better than just for the last error?
27:14 And we closed it.
27:15 We just added it for us to that.
27:17 Give me a demo for the group.
27:18 Another place that comes to mind is, maybe you'll be familiar with some of these retry libraries.
27:23 Yeah.
27:23 Like, retry, but I think there's others as well, where you put a decorator onto some function.
27:28 You say, try this multiple times.
27:31 And if it fails, do, like, some sort of exponential backoff because maybe the server's overloaded.
27:36 Right?
27:36 Those types of things would be really great.
27:38 Like, if it retries all the times it's supposed to and it fails, it'd be good to get all the errors,
27:43 not just the last one or the first one or whatever it decided it was going to give you.
27:47 Yeah.
27:47 Yeah.
27:47 It's the kind of thing.
27:48 Exactly.
27:49 Yeah.
27:49 Nice.
27:50 Okay.
27:50 Well, congratulations on getting that feature out.
27:52 That's great.
27:53 All right.
27:53 What do we got next here?
27:54 I think also related to this, I wanted to talk about this PEP678.
28:01 That's a very small and simple feature that Zach had for Dodds wrote this PEP.
28:07 He was trying out exception groups.
28:09 He was the first kind of user.
28:11 Even before the PR was merged, he was trying it out.
28:13 He was trying to integrate it with the hypothesis library.
28:17 So there you write a test and the library executes it many times with different inputs and you get failures in some of the inputs and you want to report all of them.
28:25 So Zach had an exception group.
28:26 So Zach had an exception group, kind of an exception wrapper, kind of like Trio multi-error.
28:30 He had his own version that he built in his library and he could associate each exception he attached to it, which inputs generated this error, which is very important.
28:41 You need to tell people what the input was and what happened with it.
28:45 And he couldn't do that in a convenient way with exception groups.
28:49 So we added this to base exception.
28:53 This is not a group feature.
28:54 It's any exception.
28:55 You can add strings.
28:57 You call add note, give it a string and you can call it as many times as you want and add notes to the exceptions and they will appear in the default trace pack that the interpreter prints.
29:08 So that's all it is.
29:09 It's a very simple feature, but it was received surprisingly well.
29:13 People kind of like it that you can enrich an exception after you catch it.
29:18 So you have the information that, you know, the error message and the type, you decide that when you raise the exception.
29:23 But then sometimes when you're catching, there's some more information, some context, like what was I trying to do when this error happened?
29:30 Sure.
29:30 Yeah.
29:31 Because often you'll see, except some type, some exception type, you'll deal with what you can, but you can't really handle it there.
29:37 So you got to raise it again.
29:38 And this is a place to add more information without completely wrapping it.
29:43 Right.
29:43 Right.
29:43 Exactly.
29:44 A lot of people have to chain it, say this raised from that.
29:48 So there will be situations where maybe you won't need to do that.
29:52 Yeah.
29:52 I'd love to see that go away.
29:53 I don't sort of template libraries and stuff in the web all the time.
29:57 I see like there's all these different errors and you got to hunt through a bunch of stuff to figure out what happened.
30:02 Yeah.
30:02 Yeah.
30:02 But also think about, for instance, like I think this is super useful actually for end users even like you think about that you're doing some query to the database.
30:11 Right.
30:11 And then I don't know, it may fail for six million reasons.
30:14 And then you want to add what you're asking for.
30:16 Right.
30:17 So you add your query or your user or whatever, because probably the exception that the Postgres thingy that is underneath is not going to contain your actual thing.
30:27 So this actually may save you hours, right?
30:30 Because in many enterprise environments, you don't even have easy access to that.
30:35 Sorry, to prod.
30:36 So you cannot just go there and see what's going on.
30:39 So it would be super cool that you say, if something fails, you know, I was trying to do this with this data and like with these things.
30:47 And like if it fails now, you can know what's going on and you don't even need to log in, which is I think it's a.
30:51 Yeah.
30:52 Yeah.
30:53 It's a great idea.
30:54 Or if you know, look, here's probably why this happened as a library developer.
30:59 You're like, look, this is the error.
31:00 But here's a note.
31:01 This is probably because you didn't initialize the connection before you called this.
31:05 So make sure, you know, like another area where I see this could be useful is I want to raise like the example you have in the docs is type error.
31:15 But it could also, you know, it could be value error or some other built in low level type.
31:19 You're like, really, this is just I want to raise that error, but it doesn't have a place for me to put additional information.
31:25 And so I want to kind of enrich that with more.
31:27 And so not just catch, add the data and then raise it again.
31:30 But actually, I want to use a base error type that doesn't let me put more details in it and then just raise that.
31:35 Right.
31:35 That would also work.
31:36 I think so.
31:37 I mean, I think the intention was there was some discussions about using notes in the interpreter and I pushed back on it because I said this is owned by the application.
31:45 The interpreter shouldn't touch the notes, you know, because people can wipe out the notes.
31:49 They can change the order.
31:50 They can do what they want.
31:51 It's the applications, at least the way I see it.
31:54 The application owns it.
31:55 You put whatever context you want to put.
31:57 Is there only one note?
31:59 When I say add note, does that set the note or can I have a list of notes?
32:03 It's a list of notes.
32:04 Okay, got it.
32:04 Yeah, and you can wipe it out if you want.
32:06 You can, it's just a list.
32:08 It's attached to the exception.
32:10 You can do what you want with it, really.
32:11 Yeah.
32:11 Cool.
32:12 Okay.
32:12 Yeah, it's a great, it's a really great feature.
32:14 I mean, it's, I'm sure it was way less work than accept star, but it's also going to be really valuable, I think.
32:20 It's very simple, but it's, yeah.
32:22 Yeah.
32:22 Brent or Mark, you guys got any thought about this before we move on to the next?
32:25 I think it's really cool.
32:26 I like it.
32:27 Great work here.
32:28 Indeed.
32:28 I think it is as well.
32:30 Actually, I'm really excited about it.
32:31 I want to talk about, let's talk about faster Python.
32:34 for a little bit.
32:35 So, Mark, I had you and Guido on back on, wow, almost to the day a year ago.
32:41 We're off by November 1st, 2021.
32:44 So, not that long ago.
32:47 Let's talk a little bit about the work that you're doing there.
32:50 I guess the headline is that Python 3.11 is 10 to 60, 10 to 50% faster than previous, sort of on a reasonable range of applications.
33:01 Is that the story?
33:02 Yeah.
33:02 It's somewhere between minus a few percent and plus 100, but it varies a huge amount.
33:08 I mean, if you've got some application that basically spends all its time in NumPy or something like that, you're not really going to speed up at all.
33:16 But if it's pure Python, you'd expect it to be a good 40, 50% faster.
33:21 But it depends.
33:22 Right.
33:23 That's a good point because a lot of people do make Python faster by writing C or Rust or other languages.
33:29 And at that point, like, it's out of your hands, right?
33:31 Yeah.
33:32 So, I mean, we're looking, hopefully, for 3.12 to start looking at the sort of interface between Python and C code.
33:38 So, we should speed up code even though there's quite a lot of C code.
33:41 We won't spend up the time spent in the C code in doing the actual work in the C code, but there's still quite a lot of sort of marshalling of data that happens.
33:49 And hopefully, we'll streamline that.
33:50 But the existence of C extensions is sort of, in some ways, limits our opportunity to speed things up.
33:56 But it's also, you know, why Python is so proper in the first place are one of the main reasons.
34:00 So, definitely need to acknowledge it.
34:02 Yeah, absolutely.
34:03 So, Brent, I'll definitely have you talk about the specializing interpreters.
34:06 Mark, maybe give us a rundown of some of the things from your plan that made it in here.
34:11 I know some were aimed for 3.10, but they didn't make it until here, right?
34:15 Yeah.
34:15 So, the whole thing, oh, that original plan I put up, that was more of a just to get the discussion going sort of thing.
34:21 And it's basically, it's more or less a year off.
34:23 So, if you just shift everything one forward.
34:26 I mean, there was a lot of discussion on speeding up the interpreter in the first iteration and then looking more to the data structures in the second thing.
34:34 It's much more jumbled than that.
34:35 We're doing sort of a bit of everything.
34:37 So, obviously, I was planning on, you know, expecting a smaller team.
34:40 So, things are being a bit shuffled.
34:42 So, yeah, there's a specializing interpreter.
34:44 Obviously, that's kind of key.
34:46 There's also quite a lot of stuff we've done with data structures.
34:48 I mean, we shrunk the Python object.
34:51 So, I mean, the Python object, you know, has been shrinking for years.
34:54 I mean, I've got some numbers here.
34:56 So, like in 2.7 and 3.2, like an object with just four attributes would take 352 bytes on a 64-bit machine.
35:05 And for 3.11, we've got it down to 112.
35:07 And for 3.12, it would be 96.
35:09 Well, before you get too excited, there's only 32 in C++.
35:12 So, you know, we've got a bit of a way to go.
35:14 Yeah, but, you know, it's going in the right direction for sure.
35:18 And, you know, I'm sure some people out there listening just say like, okay, well, it's half the size roughly and it's going to be less than that.
35:24 So, yay, we can use less memory.
35:26 But maybe you could talk a little bit about how that affects things like L1, L2, L3 cache hits and other sort of,
35:34 like it's more important than just I need less RAM, right?
35:37 Yeah, yeah.
35:37 So, there's two things that happen.
35:39 Yeah, things are faster because the hardware is just happier.
35:43 If you pack everything together, it's in a high-level cache.
35:46 So, you're not getting these sort of long pauses as you hit main memory.
35:50 And the other thing is just the data structures are, because there's less of them, there's less indirection.
35:54 So, for example, to load an attribute, we've got it down for basically an old, you know, older versions of Python.
36:00 It was sort of effectively five memory reads and they were dependent memory.
36:04 You have to read one before the next one and so on.
36:06 Right, right.
36:06 Go to the class, find its, go to the object, find its dictionary.
36:10 Yeah.
36:10 Then find the pointer that's in the dictionary and then go to that, right?
36:13 Like it's...
36:13 Yeah, yeah.
36:14 Very much that.
36:15 And it's down to more or less two now.
36:17 Okay.
36:18 That's pretty awesome.
36:18 I mean, obviously, there's still interpretive overhead on that.
36:21 So, it's not quite that much faster, but it's getting there.
36:26 So, yeah, there's a data structure.
36:27 And then the frames, the Python frames, every time you call a Python function, we used to just allocate a heap object for the frame and all this stuff would go in there.
36:36 And now they're all basically in a big contiguous sort of block of memory.
36:40 So, it's just bumping a pointer rather than allocating, which is also faster.
36:44 And frames are just smaller anyway because of the zero cost exceptions, which I think we mentioned on the release thing.
36:52 But, yeah, this is...
36:53 Well, let's tell people about zero cost exceptions.
36:55 Okay.
36:56 Well, zero cost...
36:57 You shouldn't have to pay for errors if you're not raising errors, right?
37:00 Yeah.
37:00 That's the idea.
37:01 And that's why they're called zero cost.
37:02 But zero cost is in quotes in this.
37:04 And the reason for that is that's the name it has got.
37:07 They're definitely not zero cost.
37:09 The idea is that they're pretty low cost if you don't have an exception.
37:13 But they tend to be even more expensive if you do get an exception because you have to do more lookup.
37:17 The important thing here is that just there was lots of runtime information we need to maintain and we don't now.
37:23 So, that, again, shrinks the frames and just makes calls faster because calls in Python were notoriously slow.
37:28 So, that's one thing we've sped up significantly.
37:31 Yeah.
37:31 So, the idea was in previous releases of Python, if you just enter a try block, even if it was successful,
37:39 there was a little bit of overhead to set up the mechanisms of potentially handling the errors and the information you needed, right?
37:44 Yeah.
37:45 And this wasn't just the overhead.
37:47 You defer more of that, right?
37:48 Yeah.
37:48 I mean, it's actually not so much that overhead as just the space you had to put that data in had to be allocated every time you made a call in case there was an exception.
37:58 And then we had to, it was massively over allocated to the amount of space anyone ever needed.
38:02 So, just that was the big sort of advantage.
38:04 Nice.
38:05 Yeah.
38:05 This is fantastic.
38:06 You don't want to discourage people from putting proper error handling in their code.
38:11 Yeah.
38:12 What do you think?
38:13 I see your name on this feature here in GitHub.
38:16 What are your thoughts on it?
38:17 Yeah.
38:17 I think, I think it's cool.
38:18 And I was kind of, you know, it was a nice touch that Mark implemented it between when I wrote the prototype for exception groups and when the PEP was approved.
38:29 So, that got in the way a little bit.
38:33 But it was good.
38:34 I got intimately acquainted with zero-cost exceptions through that exercise.
38:39 Well, it's zero-cost for some people.
38:43 Yeah.
38:43 Yeah.
38:43 I tease Mark a lot about that.
38:46 But no, I think it's a cool feature.
38:48 And I mean, I followed up on that.
38:51 We now have, after we removed that, we still had a, I was talking about this on Monday, we had a jump over the exception handler.
38:59 And then I told Mark, wait a minute, there's a jump.
39:01 It's not zero.
39:02 You have to jump over the exception handler if there's no exception.
39:05 So now we have, we did, we identify exception handlers as cold blocks.
39:11 And before we lay out the code of the function, we put all the code blocks in the end.
39:15 So now if there's no exception, there isn't even an exception handler to jump over.
39:20 That would be in 3.12.
39:21 Yeah.
39:22 Excellent.
39:23 You made zero-cost even faster.
39:25 So now it's like, yeah.
39:26 Zero even smaller.
39:27 Yeah.
39:28 It's asymptotically approaching zero.
39:31 Yeah.
39:31 So, but it's kind of nice that we have this notion of cold blocks and hot blocks and we can maybe do other things with it.
39:38 Kind of nice that all the, all the happy, but the fast code is kind of in the beginning of the functions bytecode block.
39:46 And, you know, in terms of caches and all that, you don't have to, I think it would, we bring a few benefits beyond just not having to jump.
39:53 Yeah.
39:53 Yeah.
39:54 This is excellent.
39:54 It's a really great feature and pretty straightforward.
39:56 All right.
39:57 Brent, tell us about the specializing adaptive interpreter.
40:00 That's a big deal.
40:01 You and I spoke about that about six weeks ago, I think.
40:04 Yeah.
40:04 Yeah.
40:05 Basically, the headline is the bytecode changes while it's running to adapt to your code, which is really neat.
40:10 So it's kind of finding places where we can do the same thing, but using less work by like cheating a little bit.
40:18 But cheating in a way that is not visible at all.
40:21 A good example is something like a global load or a load from the built-in.
40:27 So if I'm looking at like the len function, that requires two dictionary lookups.
40:31 Every time I want to look at the len function anywhere, I first need to check the global namespace and that's going to be a failed lookup.
40:37 Then I need to check the built-ins dictionary and that's going to be a successful lookup.
40:40 So every time I want to use len or range or list or any of those built-ins, that's the cost that I have to pay.
40:47 But people don't change the global namespace that often.
40:50 And people change the built-ins namespace even less often, or at least they shouldn't be changing it very often.
40:55 I'm going to make false true and true false.
40:58 Let's see what that is.
41:00 And so you can make these observations where it's like, okay, well, if the set of keys in the global namespace hasn't changed since last time this bytecode instruction ran, then I know that that lookup is going to fail.
41:11 Because if it failed last time and the keys are the same, then it's going to fail this time as well.
41:15 So we can just skip that.
41:16 And same for the built-ins dictionary.
41:19 You know, if we know that the keys in that dictionary haven't changed, that actually means that the internal layout of the dictionary is the same.
41:26 And we don't even need to look up len in the built-ins dictionary.
41:30 We can reach directly to the last location where it was before and give you that instead.
41:35 And so you often see in a lot of code as like an older code as a kind of a micro-optimization.
41:41 Whenever someone was using a built-in in like a very hot Python loop, sometimes you'd see them like do this kind of quark trick where they make it a local variable by saying like len equals len or something like that as part of the function's arguments.
41:54 So that you turn it into a fast local load.
41:56 And what we've essentially done is, you know, made ugly acts like that totally unnecessary.
42:02 Yeah.
42:03 Which is really cool.
42:03 Do that behind the scenes transparently.
42:05 Yeah.
42:05 Exactly.
42:05 And so that's just, you know, one example.
42:08 We've done tons of specializations for all sorts of things, ranging from calls to attribute lookups to attribute stores, etc.
42:15 So, yeah, it's a really, really powerful thing.
42:18 What was it?
42:20 It's the 569?
42:22 Yeah, Mark wrote it.
42:23 It was 659.
42:24 659.
42:25 Almost there.
42:26 Yeah.
42:26 Yeah.
42:26 So this interpreter is Mark's baby.
42:28 He could tell you much more about it than I could.
42:30 Yeah.
42:30 I just want to give you a chance to give a shout out about specialist.
42:33 Yeah.
42:33 Yeah.
42:34 So this is why I was on your show a couple of weeks ago.
42:36 So looking at bytecode disassemblies is not fun.
42:40 And so one thing that's kind of cool is, you know, if you upgrade to Python 3.11, you run your code and you saw it got, you know, 10, 20, 30% faster.
42:46 You might be wondering, like, okay, where did it get faster?
42:49 Like, what is faster about my code?
42:51 And so specialist is basically a package that I made.
42:55 It's pip installable.
42:56 It only works on 3.11.
42:58 And basically, if you run your code using specialist instead of Python, so you just type specialist myproject.py or whatever, it will open a web browser and show you your code, but color highlighted to show you where the interpreter was able to specialize your code, where it wasn't.
43:14 And that's really neat.
43:16 So you can see, like, oh, actually, you know, these are the attribute loads that got faster.
43:19 These are the places where my global loads are being cached.
43:22 That's awesome.
43:23 Yeah, this is a really cool project.
43:24 And it has some proactive features, not just informational aspects, I think, anyway.
43:30 You know, if you run a profiler, it'll show you where your code's spending time, but it doesn't mean you should go change everything to make it faster.
43:38 You should look at, like, oh, this loop or this one function is, like, the thing that maybe we should think about slightly changing the algorithm or the way we do a loop or something.
43:46 And it's a little bit similar here because the specializing adaptive interpreter only specializes some things.
43:52 Like, it doesn't specialize floats interacting with ints or those types of things.
43:58 Or I think division as well.
44:00 And so there's certain ways you might be able to slightly change inside of a really hot loop.
44:04 You know, make something to float ahead of time if you know it's going to be involved in floating point operations.
44:08 Right.
44:09 Yeah.
44:10 The idea is that this is show us how we can fix things so that you don't need to mess with your code.
44:15 I see.
44:17 So this is in the future.
44:19 Okay.
44:19 Yeah.
44:19 Awesome.
44:20 Yeah.
44:20 I would not necessarily encourage people to start tuning individual bytecode instructions in their code due to our implementation details.
44:27 Otherwise, you will end coding in C.
44:29 You all mean I need to, I got to take all those decimal points back out of my code.
44:32 No, just kidding.
44:33 Yeah.
44:33 I want to get every single bytecode instruction green.
44:35 Some things will never specialize.
44:38 And that's just an artifact of programs.
44:40 But, you know, if we can specialize enough, and we typically do, you know, one line, maybe 20 bytecode instructions.
44:45 If, you know, four of them get specialized successfully and two of them don't, generally, that will still be faster.
44:51 Brian, do you know what you should do for April Fool?
44:54 Like, you should do a pytest plugin that shows you the percentage of specialized instructions in your code.
45:00 And people can fix the percentage.
45:02 So they can say, fail my test suite if my code is not specialized more than 50%.
45:06 If you de-specialize it, it's like a performance regression.
45:10 Drop it.
45:11 Yeah.
45:11 Yeah.
45:12 It's like a coverage thing.
45:14 Yeah.
45:14 No, I was kind of thinking about this.
45:16 So Paulo can tell you more about this.
45:17 But his cool new tracebacks, the whole reason Specialist is able to do these cool, you know, column level highlighting of your source code is because we do have that fine-grained position information under the hood.
45:29 So we kind of just piggybacked on that feature in order to give you that.
45:34 But I was kind of thinking another thing, another April Fool's project could be, you know, column level coverage information.
45:41 So to get to 100% coverage, you have to cover every single column.
45:44 Exactly.
45:44 Yeah, yeah.
45:45 I feel like people might take that too seriously.
45:47 Even the white space.
45:48 Oh, this white space is not covered.
45:49 Yeah.
45:51 You think you're intense by having branch coverage turned on.
45:53 Just wait till you have column coverage turned on.
45:55 Yeah.
45:55 You can only cover two white spaces per line.
45:57 So you got to call that a lot.
45:59 All right.
45:59 I think that's a perfect segue over to one of the most tangible contributions from Pablo here.
46:06 Maybe tell us about this new fine-grained error locations and tracebacks.
46:10 This is fantastic.
46:10 This will save people being in debuggers or rewriting their code with tons of parent statements to figure out what's going on, I think.
46:17 Yeah.
46:18 Thank you very much.
46:18 We put a lot of effort into this.
46:21 So this is, man, I don't even remember my pep, so I don't know.
46:24 It's PEP something something, and it has a horrendous name.
46:27 Six, five, seven, and let's see.
46:30 Six, five, seven.
46:31 Thanks.
46:31 Let's include fine-grained error locations and tracebacks.
46:34 Yeah, the worst name.
46:36 I think I was talking with Mark in the Python code of press print, and he was saying, like, what it means, like, fine-grained.
46:42 Like, you know, like, is this very fine-grained?
46:45 Like, so I think we are renaming the PEP to fancy tracebacks.
46:48 I think that's much better.
46:50 Anyway, so this is a project I worked together with Batuhan, Tazkaya, and Omar Azkar.
46:55 So kudos to them as well, because they participated equally on this.
46:59 And the idea is that we were, like, we started this project to make, you know, to improve the error messages in the interpreter and the general experience.
47:08 Not only for, you know, people, because when people talk about this, they normally refer to people starting to learn Python.
47:14 But, like, to be honest, most of these things also affect people that are experts.
47:18 Like, I always say that when I implemented the suggestions, I was the first one benefiting from them because, like, I make a lot of typos.
47:24 And, you know, like, this is odd.
47:26 You mean this?
47:27 So the idea that we have is that most of the time, the lack of, you know, the interpreter shows you kind of the position when the error happens.
47:34 But it's quite limited because most of the time people tend to have, due to Python flexible syntax, a huge amount of, like, complexity, even in a single line.
47:44 In the pep, there is a bunch of examples.
47:46 Like, you access a bunch of keys in a dictionary and some of them doesn't work or it's not there or it's none or something like that.
47:52 Right.
47:52 And then it fails.
47:54 Or sometimes you have, like, several function calls or several additions.
47:58 And, you know, it's quite difficult.
48:00 And most of the time fixing these things involve going into a debugger like PDB and then trying to inspect every single object and say, okay, this dictionary doesn't have this key at this level.
48:11 And, like, you know, that sucks.
48:12 Like, it's not.
48:13 Yeah.
48:13 Because, like, the buggers are cool, but, like, it's cooler not to use them.
48:17 Right.
48:18 And, you know, we thought, what can we do here?
48:21 And we arrived to this idea, actually, also to mention everyone involved.
48:26 This was originally inspired by some kind of prototype that Carl from the PiPi team made very long ago when he saw, like, a kind of minimal version of this.
48:37 And then I said, okay, can we do this?
48:40 And what we do now is that we propagate because the parser, our super cool PEG parser, knows position of all the tokens and things like that.
48:48 So we are propagating that information through the interpreter.
48:52 And we store this information now in code objects.
48:55 So a side effect of this PEG actual is that code objects are slightly bigger.
48:59 Although, you know, because code objects don't tend to be a huge percentage of your application, it doesn't really matter that much.
49:05 Maybe PYC files are a bit bigger.
49:06 But, you know, you have a lot of disk space, I'm sure.
49:10 And the idea is that, you know, we store this information in code objects.
49:13 So when you raise an exception, we say, well, what is the instruction that raised this exception?
49:19 And then once we know which is the instruction that raised the exception, then we go and say, okay, what is the position information that generated this instruction?
49:26 And because we propagated it, we know.
49:28 And then we can say, okay, here is kind of like the lines, the columns that this instruction spans.
49:35 So that kind of allows us to underline the specific location.
49:39 Yeah, but we go a bit further.
49:40 Sorry, sorry, go ahead, Michael.
49:41 I was going to say, this is super valuable.
49:43 The example you have in the PEP is you have a dictionary, you say bracket of key A, and then the thing that comes back is another dictionary.
49:51 So you say bracket B, and then another dictionary, bracket C, and then bracket D.
49:55 And if, you know, in 3.10, the error is just like, if one of those is none, say none type is object is not subscriptable, or maybe, you know, does not contain that key or some weird thing like that.
50:06 But is it A, B, C, or D?
50:09 You have no idea.
50:09 You're in a debugger printing them out separately or something.
50:12 But now it just goes, nope, it's the C one.
50:14 That's, it's the third subscript one.
50:17 And that's just, just jump right to it.
50:19 Oh, okay.
50:19 Yeah, also this error, none type is not subscriptable.
50:22 It's kind of like, thanks for the info.
50:25 Like, it's like, you know, water is wet.
50:27 Okay, thanks.
50:28 It's not super useful.
50:30 No, exactly.
50:30 So tell me when it's going to rain.
50:32 Anyway, we did like the first version of this.
50:34 And then we realized, realized that there was some kind of like, you know, it was cool.
50:39 Like most people really like it, but like, especially for instance, with the example, with the dictionary that has many dictionaries inside.
50:45 There was some confusion because like, you know, it underlines the whole thing.
50:49 And then, you know, the order of operations and, you know, also with complex mathematical expressions, like you do A plus B plus C and the last addition fails.
50:58 It needs to underline A plus B plus C because what happened is that it first added A plus B and that gives you something that then you added to C.
51:05 And what happened is that the last addition failed, but that includes A plus B.
51:10 So you need to underline the whole thing.
51:11 If you know the order of operations and I just underline A plus B plus C, you know that what will fail is the last one.
51:18 Because that's the last one that is executed.
51:20 But it's still confusing because, you know, specifically also with the dictionary, people were saying, yeah, okay, but like you're underlining three keys here, which is the one that failed.
51:28 I mean, you know, you can learn by experience that is the last one, but it's kind of like, it was not a great experience.
51:33 So we went a step farther.
51:35 So what we do is that once we know the kind of range in the line that shows the problem, we reparse that chunk of expression.
51:43 And then we know, okay, so we know now that this expression has this AST.
51:47 And then we analyze the AST.
51:48 And then we say, okay, is this AST something that we can further improve the error message?
51:54 Like, for instance, is this AST a bunch of key access in a dictionary or a bunch of attribute access or a bunch of function calls or maybe binary operations?
52:03 And if it's the case, then we use a specialized, like, you know, underline, I don't know, tildes or squiggles or whatever it's called.
52:10 Yeah.
52:11 And, you know, the dictionary ones have this different one that marks which key access it was known, the same thing for binary operations and things like that.
52:19 So we do this extra step at the end that, you know, does a bunch of extra work, but it tries to improve even upon the kind of just underlining the line just so we can offer even more rich information.
52:30 And I'm quite happy.
52:32 I'm very pleased about this.
52:33 Sorry, Mark, but I think it's the best feature of 3.11.
52:38 Yeah, this is probably the second stream when I said this, but it's true.
52:41 Totally, totally true.
52:43 100% true.
52:44 So I'm very excited about this.
52:45 I literally use this every day.
52:46 Today I was deploying Python 3.11.
52:48 Well, this week, sorry.
52:50 I was deploying Python 3.11 at Bloomberg and something went wrong.
52:53 And literally, this thing saved my day.
52:55 This saved me to just logging into some forsaken machine and understanding what's going on.
52:59 What about that?
53:00 So super cool.
53:01 Very happy.
53:01 I hope that everybody that uses this and is happy reached to us and say, I'm happy.
53:06 Because normally people reach to us when they are not happy.
53:09 And they say, evil core developers, you break everything.
53:12 But instead of that, you should reach to us and say, nice.
53:15 I did this cool thing.
53:18 You should tweet at Pablo or something, though.
53:20 Don't open issues saying you're happy.
53:22 Exactly.
53:24 Just tweet a couple of tildes, a few carrots, and a smiley face at him.
53:28 Exactly.
53:28 Tweet happy at python.org.
53:30 I will take that email address.
53:32 Awesome.
53:33 It is that.
53:34 We improve it a bit further.
53:36 One of the things that happen is that, you know, like sometimes if the whole line is wrong,
53:40 because this example you have there, if you, sorry, for the ones listening to the podcast,
53:44 we have here some, we are seeing some output, but that doesn't matter.
53:47 Don't worry.
53:48 I will describe it.
53:49 So for instance, you're calling a function and that's the whole thing that is in the line.
53:53 We used to underline the whole thing.
53:55 So we'll say, okay, even if the whole line is failing, so there's not like a part of the line
54:00 is failing, the whole thing is failing.
54:02 We used to underline that and that apparently is still on the pep.
54:05 Maybe I should change that because that is not like that anymore.
54:08 Because someone suggested, I mean, come on, like if it's the whole line is failing,
54:12 underline the whole line is actually not that useful.
54:14 And you know, you are, you are spending vertical space.
54:17 So you need to scroll a lot.
54:19 And at the beginning I say, yeah, but it's inconsistent.
54:22 I don't like it.
54:23 And I pushed back a bit, but like then, you know, more people say, Pablo, you're wrong.
54:28 And then I say, okay, okay, I'm wrong.
54:29 We improve this further.
54:31 So you say, but don't take this as an advice.
54:34 Don't tell me that I'm wrong collectively, please.
54:37 But right.
54:38 So now if the whole line is underlined, we don't underline it because it doesn't really
54:43 add any new information, right?
54:45 So, so only if a part of the line is contains the error, not the whole line.
54:49 So this means that we are not going to, you know, consume a lot of vertical space for no reason.
54:54 And the last thing I wanted to say is that, you know, there is some people somewhere in the universe
54:58 that may care about that extra disk space on their PYC files, or they just really, really
55:05 hate squiggles.
55:06 I don't know if that is even physically possible, but you know, there are very different and
55:10 diverse set of people.
55:12 You are one of those.
55:13 There is a collection of different ways you can deactivate this feature.
55:17 There is an environment variable with a super long name.
55:20 And there is minus X option when you launch the interpreter.
55:23 So you can say Python minus X, something, something.
55:26 I don't know how it's called.
55:27 I think it's called no the back ranges.
55:29 What about that?
55:30 What an incredible naming.
55:32 And then you set no the back ranges to one and it deactivates the feature.
55:36 Incredible.
55:37 Like magic.
55:38 It's gone.
55:38 And you can reclaim your PYC files.
55:41 And you can even generate PYC files without this information.
55:44 If when you're compiling PYC files, you set this evil variable.
55:48 But don't do that, listeners.
55:50 Don't do that.
55:50 It's evil.
55:51 Don't do that.
55:51 Just use it.
55:52 It's great.
55:53 Yeah.
55:54 There's another kind of type of errors that I think we're going to get is about edge cases where the compiler doesn't get the line numbers right.
56:03 Because all these kind of fine grained locations, it's all new.
56:07 And, you know, we're still ironing out.
56:09 The front future.
56:10 There is a front future, I think, that you put like a bunch of things with the front future.
56:14 It just complains on a random place.
56:15 Yeah.
56:16 Today I found that one.
56:18 But I've been looking at the compiler and line number, location information.
56:23 And it's a bit off here and there.
56:24 And we have received bug reports from other people as well.
56:27 The range here doesn't look right.
56:29 The range here looks too broad.
56:31 So, yeah, we're going to be ironing that out, I guess, for 312s.
56:36 Yeah.
56:36 It's really nice when people are using betas and release candidates, though, because we were able to catch a lot of those before the release.
56:42 Because there were a couple people, I forget exactly the name of the project, but they were working on like a code animation tool where it animates your code while it's running.
56:50 And they were using these new ranges to identify AST nodes and stuff.
56:54 And so they did this thing, I guess, where they like run their tool in the entire standard library and make sure it's correct.
56:59 And so we got a bunch of bug reports that basically said, oh, you know, this column information is off for this weird multi-line attribute access or something.
57:06 If you recall, I think you fixed an error.
57:09 That was super weird because it was like a method access, like, you know, my instance of full.
57:15 Yeah.
57:15 And if the method access has like some like vowel or something like that, it was wrong.
57:20 And if you added some extra letter, it was fine.
57:23 Yeah, it was like if you split your method access across two lines, if you do like x dot method or x dot method or x dot method on three lines or two lines or something.
57:35 The way we trace those lines, we always trace the method when we're actually loading the method, even if it's on a different line.
57:43 It's like where the actual method load started.
57:45 And then we were doing some weird math to like figure out where the dot is.
57:48 So we would try to put it on the same line as the dot.
57:51 So we would just like subtract one from the length of the name.
57:54 So there's all sorts of crazy stuff.
57:55 And that came from the grave because we fixed that.
57:58 And then it was wrong again because like we were like miscalculating the name.
58:02 It's just that easy.
58:04 Oh my goodness.
58:05 Yeah.
58:05 So all sorts of fun stuff like that.
58:07 Yeah.
58:07 Amazing.
58:08 Well, yeah, this is definitely one of the highlight features for sure.
58:12 And also the performance work that you're all doing.
58:14 All right.
58:14 We're getting very, very short on time.
58:16 So I think maybe a super, super lightning round here.
58:19 Let me just say we also got Tomolib support built in.
58:22 We've got the AsyncIO task groups a la Trio Nurseries.
58:26 We've got new features for atomic grouping and regular expressions.
58:30 A self-type.
58:32 A lot of type things have been added.
58:34 So we've got the self-type, bariatric generics, literal strings, which is very interesting.
58:39 Lukash did a talk about that on the release live stream.
58:43 Stuff for type dict and data class transformations.
58:46 So great stuff.
58:47 Now, let's just really quickly round out.
58:50 What's the Python 3.11 story for PyScript, Pyodide?
58:54 Is there, do you know, have anyone out there know?
58:57 I don't know.
58:58 I suppose it works.
58:59 I think WebAssembly is now a tier two or tier three supported platform, right?
59:03 So he has been making a lot of improvements to the build process, which, you know, is not easy.
59:08 So kudos to Christian Himes.
59:10 If you're listening, you're great.
59:12 I suppose that PyScript can, through Pyodide.
59:16 This is how many layers is this?
59:18 So Pyodide through this can leverage all these improvements because I don't know how the whole
59:24 layer tower thing is working.
59:26 But Pyodide has a bunch of patches that, you know, you need to modify CPython so it builds
59:31 nicely on WebAssembly platforms.
59:33 I don't know the details on that.
59:34 I just know that some of them are okayish.
59:37 Some of them are not okay and quite difficult to maintain.
59:39 And Christian Himes has been making a lot of great effort to, you know, change here and there
59:45 and like put a lot of macros and if devs and things like that.
59:48 So CPython kind of builds easier.
59:50 This probably translates that Pyodide, I hope, kind of, you know, can consume this build in
59:57 an easier way with less patches.
59:58 And I suppose that translates into PyScript, like just using the Pyodide thing easier.
01:00:04 But yeah, I don't think that there is a huge amount of improvements more than, you know,
01:00:09 we are working towards official support as Bram was mentioning.
01:00:13 We have this next year system.
01:00:14 It's super cool.
01:00:15 And as like an unrelated fun fact, Mike Drapu, one of the early developers of Pyodide, is
01:00:21 actually managing our team at Microsoft.
01:00:22 Oh, it's funny how the circle comes back around indeed.
01:00:27 How the dorm tables.
01:00:28 That's right.
01:00:30 All right.
01:00:31 We are out of time, but super exciting.
01:00:34 I wish we had some champagne.
01:00:36 And Pablo, we didn't even bring hats to celebrate Python 3.11.
01:00:39 But I know everyone out there is extremely excited.
01:00:42 I cannot see it, but I have a Python 3.11.
01:00:44 Yes.
01:00:44 Yeah.
01:00:45 That's a great new logo for 3.11 and stuff.
01:00:47 Not for in general, but just for the release.
01:00:49 It's awesome.
01:00:50 All right.
01:00:50 Before we get out of here, let me just ask you one final question and then we'll call it
01:00:54 a show.
01:00:55 Notable PyPI package.
01:00:56 Something you want to give a shout out to.
01:00:57 We'll go top to bottom in the picture here.
01:00:59 Pablo?
01:01:00 Notable PyPI package.
01:01:02 Yeah.
01:01:02 Yeah.
01:01:02 Yeah.
01:01:02 Some library.
01:01:03 And I'm going to say memory.
01:01:05 Use memory.
01:01:06 The one and only Python memory profiler.
01:01:09 Yeah.
01:01:09 Solve your problems on production today with memory.
01:01:12 That and the underlying errors, you'll be all good.
01:01:15 Exactly.
01:01:15 Yeah.
01:01:15 A combination.
01:01:16 Arit, how are you?
01:01:19 Well, I've had some interaction with the author of bytecode recently because I was looking
01:01:23 at things to do in the testing and in the interpreter that are kind of like that.
01:01:28 So this is a library you can kind of from Python write bytecode and it's pretty neat.
01:01:33 And it's struggling with zero cost exceptions, but that's what it is.
01:01:38 It's like an inline assembly, but for Python.
01:01:40 Yeah.
01:01:41 It's like from a Python script, you can kind of write bit of bytecode and get it to, I
01:01:47 don't know, do a lot of interesting stuff.
01:01:49 That's awesome.
01:01:49 Brent, how about you?
01:01:50 Well, I'm partial towards specialists, but if I had to choose something else, I, speaking
01:01:54 of speed, I really like the scaling profiler.
01:01:57 I've been using it a lot of my own projects and it's awesome.
01:02:00 I don't know how it's memory profiling compares to memory.
01:02:03 I'm sure memory is better, but scaling is really nice for measuring the performance across
01:02:09 both Python and C code, which is cool.
01:02:10 Excellent.
01:02:11 Mark?
01:02:11 Well, it's not actually a PyPI package.
01:02:13 I was going to say the sys module, which is like pretty much the most fundamental module
01:02:17 I'm going on.
01:02:18 There's all sorts of fun things in there.
01:02:20 You can change the recursion limit and C, you can muddle it.
01:02:23 If you're interested in how Python works, it's actually quite a sort of fun thing to play
01:02:27 with.
01:02:27 Thank you all for all the hard work.
01:02:29 And I know there are many people who did a ton of work as well who are not on the show
01:02:33 here, but you can represent them as well.
01:02:35 Thanks all for being here.
01:02:37 Final call to action.
01:02:38 People want to get started at 3.11.
01:02:39 What do you tell them?
01:02:40 Is it ready for them to get going?
01:02:42 What do you think?
01:02:43 It's awesome.
01:02:44 It's awesome.
01:02:44 And also now 3.11 comes with a bunch of wheels for all your packages because there has been
01:02:49 a lot of good work in the people of third-party libraries.
01:02:54 And now that people are using CI build wheel, 3.11 was released with wheels for NumPy and
01:03:00 Pandas and a bunch of other things that previously was failing massively because nobody could compile
01:03:04 them on their crappy laptop.
01:03:06 But now you don't need that.
01:03:07 You can just download them and it works.
01:03:09 So just use 3.11.
01:03:10 There is no reason.
01:03:11 Yeah, that's excellent.
01:03:12 That's just boring.
01:03:12 That would be a reason.
01:03:13 If you're boring and you don't want to use 3.11, then don't use it.
01:03:16 You didn't break anything.
01:03:17 Not even a package.
01:03:18 Much less GitHub.
01:03:19 Right.
01:03:20 And we need more benchmarks.
01:03:22 Well, that's true.
01:03:24 Yeah, absolutely.
01:03:25 That's how people can help us make things faster.
01:03:27 There's more benchmarks.
01:03:27 There's a standard Python performance suite, but it's kind of a bunch of toy programs and
01:03:34 so on.
01:03:34 So if you've got something that might make a nice sort of benchmark, self-contained but
01:03:40 sort of realistic program, then let us know.
01:03:42 All right.
01:03:42 Cool.
01:03:43 Well, thanks again.
01:03:44 Great work on it.
01:03:44 Cam Geerlock out in the audience says, yay, CI Buildwheel.
01:03:47 Yeah, absolutely.
01:03:48 Great stuff.
01:03:49 So thanks again, everyone.
01:03:51 I'm super excited to start using 3.11 myself.
01:03:53 Thank you, Michael, for inviting us.
01:03:55 Yeah, it's great to have you here.
01:03:56 Thank you.
01:03:57 Thank you.
01:03:57 Bye, all.
01:03:58 I hope you enjoyed this crossover Talk Python episode.
01:04:01 If you did, please consider subscribing to Talk Python to Me if you don't already do so.
01:04:06 On behalf of myself and Brian Okken, thanks for being part of Python Bytes.
01:04:10 Bye now.