Transcript #206: Python dropping old operating systems is normal!

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to

00:04 your earbuds. This is episode 206. Wow. Recorded October 28th, 2020. I am Brian Okken.

00:11 And I'm Michael Kennedy.

00:12 Yeah. And we have a special guest today, Steve Dower.

00:14 Hi. Thanks for having me on.

00:15 Hey, Steve. Thanks for coming. It's great to have you here.

00:17 I also want to throw out that this is sponsored by Tech Meme Ride Home podcast. Check them out

00:22 at pythonbytes.fm/ride. Steve, I'm sure many listeners know you, but maybe just give us

00:26 the quick rundown. You do cool stuff at Microsoft, getting Python better on Windows and you're a core

00:31 developer. Yeah. So my main day job is for Microsoft, where I work as basically a Python engineer,

00:37 kind of a wide ranging resource to the company. So I haven't shipped anything in my own in a while,

00:44 but I've had my fingers in a lot of Python related things that have gone out recently. So it's a lot

00:50 of fun, a lot of bouncing around between different teams, getting to work with a lot of different

00:53 people. And yeah, as you say, I'm a CPython core developer, one of the Windows experts on the team.

00:58 I'm responsible for the builds that go up on python.org and just generally keeping Python running

01:04 while on Windows.

01:04 Yeah, that's awesome. And you've done some interesting talks like Python is okay on Windows actually,

01:10 and talked about some of the popularity of it and how we as a community shouldn't

01:14 discount it just because we might happen to be on a Mac or use Linux or whatever, right? A lot of

01:19 people do Python on Windows. Yeah, yeah, the estimates vary. And every time I get new numbers,

01:25 they seem to show up slightly different. So it's real hard to get a good fix on how many

01:28 Python developers there even are in the world. I did get some numbers recently that I had a few

01:33 people double check because they were saying there's like 20 million installs of Python on

01:38 Windows in the entire ecosystem, which sounded like too many to me. So I had them double check.

01:43 And then I had someone else double check. And they all came back saying, yeah, it's about that.

01:47 So I'm like, okay, there's a lot of Python on Windows out there. But yeah, it doesn't show up

01:51 in conferences doesn't show up on Twitter that much. And a lot of people just look at the packages that

01:56 don't work and go, well, I guess it doesn't exist on Windows, because otherwise this package would

01:59 work. And so you know, chicken and egg problem, right? Yeah, there's a lot of chicken and egg

02:04 problems in the Python space. I mean, it's a beautiful place. But there are some of these weird chicken

02:08 and egg ones. Yeah, it's weird. I've been using Python on Windows since I started Python.

02:12 So have I.

02:13 But one thing I haven't been using very much is enums. So that was an attempt at a transition.

02:20 Why not, Brian? Tell us more.

02:22 Actually, I've tried many times I've tried to use enums. And I actually just to be honest,

02:28 I don't very much. And partly because I'm used to using enums in C and C++. And they're,

02:34 they just will act like symbols in C and C++. They work pretty good. There is some weirdness

02:40 with enums in Python. And I'm going to highlight an article called Making Enums, as always,

02:46 arguably more Pythonic by Harry Percival. He starts it off by saying, I hate enums. So Harry's a funny

02:53 guy. And this is a fairly hilarious look at why enums are frustrating sometimes. And then he presents a

03:01 semi reasonable workaround, I think, to make them more usable. So what's wrong with enums? Well,

03:06 he gives an example. And just as a simple enum with string values. And you can't directly if you

03:15 try to compare one of the enum elements to the value, like the value gave like a similar string,

03:21 it's not equal, it doesn't compare. But you can use a dot value or in the enum value. But that's,

03:27 it's just weird. He also said, he kind of thinks it'd be neat if you could do a random choice of all

03:32 the enum values, I think that would be neat. And you can't directly convert them to a list. There's

03:37 just interacting with the enum type itself, or the class itself is, has problems. In the documentation,

03:44 there is a suggestion that you can, with you can, instead of strings, use ints, and do an

03:50 int enum, and it works a little better. And if you like it like that, but want strings, you can make

03:54 your own string enum class. I'm not sure why they didn't just build this into the default, or, you know,

04:01 one of the standard types anyway, but string enum is not there. But there's an example. And it sort of

04:09 fixes a lot of stuff, but not everything. It doesn't, still doesn't allow for those direct

04:13 comparisons. So the solution that Harry came up with is just kind of like the solution, the

04:19 documentation says, derived from both enum type and str when you're creating your enum class. But then

04:26 also define this little snippet of a dunder str method, so that the str method works better. And at

04:35 that point, most of the stuff works that he wants to work. It still doesn't do random choice. But

04:41 apparently, he's gotten over that a little bit. So I actually think this is really, this is still

04:46 just a really small snippet, a little, like two lines of extra code to add to your enum types,

04:52 make them a little bit more usable. So I think it's reasonable.

04:55 If you're judging by, like, value add per character, this is awesome, because it makes working with

05:02 the enumeration so much nicer, you can do like the natural things that you would expect,

05:05 especially around testing and comparison. And it's like, also add a drive from str and just

05:11 add a dunder str method. And you're good. Steve, what do you think about this?

05:15 Yeah, I'd just add that the gotcha that you have by doing this is now you can have two values from

05:21 different enums compare as equal, which as I recall from the original discussions was the reason this

05:26 wasn't put in by default, say you've got a color enumeration and a fruit enumeration,

05:30 is orange the same between the two? And I think the decision was made for enums to say no, it's not.

05:37 If it's a fruit orange, it's different from the color orange. Making this change or using an int

05:42 enum is going to make them equal. So as long as you're prepared to deal with that, which to be quite

05:47 honest, every time I've reached for enums, I am much happier with string literals and quite comfortable

05:52 with them matching equal. Yeah, but that is just one thing to watch out for. Usually it's about

05:56 constraining the list of things like I know there's five things I want to make sure I don't like somehow

06:00 mix up what those five are. I just want to go, you know, class dot and here's the five. Let my editor

06:06 tell me which one of those it is. It seems like it's all that. So what do you think about if you overrode

06:11 like if you added dunder EQ, dunder NEQ, so like the comparison would say it has to be this type of

06:18 enumeration and the stir value has to match? Yeah, that would certainly deal with that. Gotcha.

06:22 Again, when these were being designed, basically anything that gets designed and added to Python

06:27 has a very large group of very smart people work through it. And, you know, as a result,

06:32 things always get missed. So it's possible that one was just missed. It's also possible that

06:36 someone did figure out a reason why that was also risky. And, and, you know, risky when you're

06:42 developing one of the most popular languages in the world is just anything that might surprise

06:46 anyone. So someone has deliberately designed around using enums everywhere, they're probably

06:51 not going to be surprised. Someone who is using code and that developer has swapped out all of their,

06:57 you know, they, they had a class with static variables and they turned it into an enum and

07:01 now stuff breaks because of the defaults that were chosen for enum. That's the kind of thing that

07:06 you're trying to avoid. And, you know, in a language that has anywhere between, you know,

07:10 five and 20 million kind of regular users. But as a workaround, I mean, if you know where your enum is

07:16 going to be used, there's a reason you can derive from string and it's exactly for stuff like this.

07:20 Yeah. Thanks Harry for putting out there. That's quite a neat, a little bit of a advisor. And I'm so glad

07:25 that we have Steve here because I picked some sort of semi internal type of pieces and I'm going to make

07:31 some statements about it. And then Steve can correct me to make it more accurate. And also we get the

07:36 core developers, but a perspective, not that you represent all core developers, but at least a

07:41 slice. All right. So this next one I want to cover is that Python 3.10 will be 10% faster. And this is

07:47 the 4.5 years in the making. So Yuri Stelovanov long ago did some work on optimizing, I think,

07:57 what was it? Load attribute or was it load method, load call method. So about some of these load

08:04 operations down at the CPython C eval level. And then Pablo Galindo, who's also a core developer

08:13 and the Python 3.10, 3.11 release manager picked up this work. And now we have load method,

08:20 call method and load global going faster. So basically there's some optimizations around

08:27 those opcodes that make this much faster. And this idea apparently first originated in PyPy.

08:34 PyPy. So I'm pretty excited to see that, you know, some simple internals that don't seem to change a

08:41 whole lot of stuff might make this a lot faster. What do you think, Steve?

08:44 This one is real. Like I was so excited about this when it was first being proposed. The basis of the

08:49 idea is that Python keeps everything in dictionaries, which means every time you look up, you know, dot

08:55 name of anything, it takes that name, makes it a string, turns it into a, like gets the hash value,

09:00 looks it up in a dictionary, finds the value, maybe wraps that up in some extra stuff. Like if it's

09:05 going to be a method, it's not stored as a method. You turn it into a method when you know what it's

09:09 being kind of dotted through to get to, and then returns that. That's a whole lot of work. And if

09:15 you're regularly calling the same method over and over again, why not cache it?

09:19 That's the heart of this, right? It does that cache around load adder, right?

09:22 Yeah, it does that cache. And the insight that Yuri had that he made work, and in fact,

09:26 I think someone else had suggested it earlier and hadn't gotten it to work was what happens when

09:31 things change? Because again, as I say, it's, we're designing language for many, many people do all

09:37 sorts of weird things. And if you cache a method lookup, and then someone tries to monkey patch it,

09:41 you know, we've now broken their code for the sake of an optimization, which, you know, is a no,

09:46 in Pythons, like correctness beats performance. In every case, that's just the trade off that the

09:51 language chooses to make. That's almost always what you want, right? Like, when would you want to be

09:55 faster and wrong rather than slower and right? I'd be happy with faster and no monkey patching.

10:00 But yes, yes, sure. Like faster and fewer restricted capabilities might be a really good trade off,

10:06 but faster and wrong is not a good one. Yeah, we did some benchmarking and basically found that

10:11 there was a way to track all dictionary updates in the entire runtime with a version tag that was not

10:19 going to instantly overflow and not going to break everything. So it became really easy to say,

10:24 has this dictionary changed since I last looked at it with one single value comparison? And so it looks

10:30 at that value. If it has changed, it's going to do the full lockup again. 99.999% of the time it hasn't

10:37 changed so it can give you the cached value and you saved big dictionary lookup, possibly error handling,

10:42 descriptive protocol, all of this extra stuff that just adds so much weight to every operation.

10:49 Yeah. And that's everywhere. I mean, that's everywhere in the language.

10:51 Absolutely everywhere.

10:52 That's fantastic. One of the things when I was first getting into Python that made me sort of have a sad

10:57 face was realizing that having function calls was pretty expensive, right? Like having a big call chain,

11:05 like actually the act of calling a function has a fair amount of overhead. When I wanted to break my

11:10 code into like a bunch of small functions to make it real readable, this part needs to go a little

11:14 faster. Maybe that's not what I want. You know? And so hopefully this helps with that as well.

11:20 Yeah. That and vector call is another optimization that we got in recently. I think that might've been

11:26 the pet 509 actually was vector call also designed to make that faster. Just removing some of the

11:32 the internal steps for doing a function call. Fantastic. And it's like Brian said, this is

11:36 everywhere. So that's everyone's going to benefit. This is fantastic. Yeah. Well, we would like to thank

11:42 our sponsor. So this episode is brought to you by Tech Meme Ride Home podcast. This is a great podcast

11:49 for more than two years and nearly 700 episodes. The Tech Meme Ride Home has been Silicon Valley's

11:55 favorite tech news podcast. The Tech Meme Ride Home is a daily podcast only 15 to 20 minutes long and even

12:02 and every day by 5 p.m. Eastern. It's all the latest tech news, but it's more than just headlines.

12:09 You could get a robot to read your headlines. The Tech Meme Ride Home is all the context around the

12:15 latest news of the day. It's all the top stories, the top posts and tweets and conversations about those

12:20 stories as well as behind the scenes analysis. The Tech Meme Ride Home is the TLDR as a service.

12:26 The folks at Tech Meme are online all day reading everything so that they can catch you up. Search

12:31 your podcast app now for Ride Home and subscribe to the Tech Meme Ride Home podcast or visit

12:38 pythonbytes.fm/ride to subscribe. Yeah. Thanks for sponsoring the show. And Brian,

12:43 every day, like we do this once a week and it's a lot of work. These guys are on it. I could totally do

12:49 this every day. If I didn't have another job, I would not have any problem with catching up on

12:53 Python News Daily. So actually sounds quite lovely. I wonder how that podcast is doing now that not so

12:59 many people are having to, you know, commute to and from work. That sounds like one of the things where

13:06 you hope that you've, you've given people the excuse to tell their employer on my commute between

13:12 four and 5 p.m. I can't do it. You know, I've logged off and go listen to a podcast during that time.

13:17 I listen to podcasts while I'm, I realized I was missing out. So I started listening to podcasts

13:22 while I'm doing my laundry. I do it when I'm, we're doing some doing dishes. I listen when I'm

13:28 doing yard work, stuff like that. Yeah. I recently broke out some other older podcasts just to catch

13:32 up on stuff with a big mess I had around the home. Like it's a long story with a new puppy that's not

13:37 worth going into. Maybe I'll tell you guys after the show. It's pretty outrageous anyway. Yeah. And

13:45 it's so enjoyable, but what I've actually found is shows like Python bites and right home that have

13:50 like a bunch of short little things that you can just drop in and out of match a lot better. Now that

13:55 people are not commuting so much, you can like do that 10 minutes while you're like folding laundry

13:58 and get like a whole segment. And so I think he gets varied, but actually it's pretty interesting.

14:03 And I think that they and us here are well positioned. Like talk Python has more of a dip

14:08 than Python bites. Have you surveyed your listeners to find out what they're doing while they listen to

14:12 you? No, not really. Everyone should tweet at these guys, Twitter. What were you doing when you were

14:17 hearing this podcast? Yeah, that's awesome. I've got a few anecdotes, but nothing like a survey that

14:23 would give me a proper answer. Steve, I think your next item is super, super interesting. And

14:27 people speaking to Twitter, right? Like this whole conversation started as a, Hey, Twitter

14:33 message. Like why did this happen? Well, let's ask Steve. Yeah. As you know, I spend a decent amount

14:38 of time on Twitter. My handle there is Zuba, Z-O-O-B-A, which you'll probably never find in search if

14:44 you're looking for my name, but that's where I am. I really like actually searching for what people

14:50 are saying about Python on windows. It's kind of the most honest feedback you get when they think

14:54 you're not listening. And so I go and listen. And one of these popped up, which was, Oh, I tried to

15:00 install Python 3.9 newest release about a little bit under a month ago on my windows seven machine,

15:07 and I couldn't install it. And since then, I've actually seen a few more posts. Someone managed to

15:12 bypass the installer completely and get it all the way onto their windows seven machine and then found out

15:16 it wouldn't run. Oh man. Yeah. And so the question was, I was like, why would you do this? Windows seven

15:21 is still a fairly big platform. Why would you take it out? And you know, the answer was just a bit too

15:26 long for a tweet, but someone, you know, kindly included Python bites in the reply. And so I said,

15:30 Hey, I'll come on and talk about it. So let's do this topic. And the answer is multiple legal looking

15:38 documents all come together and have to be read in parallel to figure out why we dropped it.

15:45 Yeah. The small business owners know what I'm talking about. So one of those documents is PEP 11,

15:50 one of the lowest number PEPs that we have, and it's titled removing support for little used platforms.

15:56 The title was not originally about windows, but there is a section in that PEP that describes Python's

16:04 policy for supporting windows on release day of 3.x.0, all the supported versions of windows covered by

16:13 Microsoft support lifecycle will be supported by CPython. And that's on the 3.x.0 release date.

16:20 So what that means is then you now have to go and look at Microsoft support lifecycle website and look

16:27 up all of the different versions of windows to see which ones are still covered by support to that date.

16:32 Windows seven fell out of extended support in January. There was quite a bit of noise about that

16:37 because that means no more security patches, except Microsoft did do a couple more security patches

16:43 because some really bad stuff was found, but that's largely stopped. Essentially, it's the point when

16:48 the only people who get windows seven support are paying for it. And I assume they're paying large

16:54 amounts of money for it. I don't actually know how much it costs, but that's the point where, you know,

16:58 you bought it, but you don't get the free support anymore. So CPython follows that because

17:03 no one is paying the core team to support Python on all of these platforms. And so it's,

17:08 it seems like the fairest point to draw that line is at some point we have to say our volunteers can no

17:16 longer keep a windows seven machine running. Even I can't keep a windows seven machine running safely

17:21 because there's no security updates for it. How am I meant to develop Python on it? How am I meant to

17:26 test Python on it? The burden there is too high for volunteers to handle. So we just say that's the

17:33 point where it goes away. So because those two documents lined up, Windows eight actually dropped

17:37 off a couple of years ago because the support lifecycle ended early for that to force everyone

17:42 onto 8.1. Windows 8.1 has about three more years. So I think Python 3.12 will be the last one to support

17:49 8.1. And then it's all Windows 10 or whatever comes out in the future. Yeah. Yeah. Windows 10 X or

17:54 whatever they call it. That's a different one. That's the Xbox. Yeah. So I, you know, I think this makes a ton

18:00 of sense. And two thoughts I had as you were laying out the case here. One is if you're running a Windows

18:08 seven and you can't upgrade to even Windows eight or more recently Windows 10 or one of the server

18:13 equivalents, right? I'm sure there's like a server equivalent, like Windows 2003 server. I don't know how

18:18 long I supported, but whenever it goes out, it probably falls under that banner as well, right? Yeah. Windows

18:22 server is a bit more interesting. Their life cycles tend to last longer, but historically,

18:28 CPython has only kind of tied itself to the client operating systems. Gotcha. Oh, interesting. Okay.

18:34 So to me, I feel like if you're running code or you're running systems that old, you must be running

18:40 it because it's like some super legacy thing. So do you absolutely necessarily need to have the most

18:47 cutting edge Python or whatever language? Like it's probably something that's

18:53 that way because it's calcified and you probably don't need, or you probably shouldn't be putting

18:59 like the newest, shiniest things on it. Right. That's that one. What do you think?

19:03 Yeah. No, I totally agree with that. If that setup that you're running is so critical that you can't

19:09 upgrade the operating system. How can you upgrade a language runtime? It's like, how can you upgrade

19:13 anything on that? I feel like it's in the category of, please just don't touch it. It's over there.

19:18 Just don't even walk by it. Leave it alone. We cannot have it break. Just leave it over there.

19:23 Probably still has a PS2 keyboard plugged into it. Oh, it might with a little blue or pink round thing.

19:28 Yeah, absolutely. The screen has probably got at least 16 colors. Yeah. And the monitor is probably

19:32 really heavy. Windows seven. It's not that, it's not that old. Some of the, seriously, like the stuff is

19:38 old and you probably don't want to touch it, right? Yeah, that's exactly it. It's all the motivation

19:42 that you would have for updating to Python 3.9 from 3.8. And again, we're talking about a version

19:47 that's only one year old. Like Python 3.8 is not that old and you desperately need to upgrade to 3.9.

19:53 You even more desperately need to upgrade Windows. And there's just really, there really is no question

19:59 about that. The same thing applies to early versions of Ubuntu. People running Ubuntu 14 or even 16 at this

20:05 point, like need to be facing the same thing. And we have similar discussions around OpenSSL,

20:11 where occasionally people will be, oh, I need Python 3.9 to run on OpenSSL 0.9. To which our answer is

20:20 basically, that's pretty hot bleed. It's like, okay, I'm going to play the other side. I totally get the

20:27 reasons. But I also get the questions because the users and the developers or the whoever's wanting

20:34 to install Python, they usually don't get to choose what operating system they're using, but they do get

20:41 to choose which version of Python they're using. So I do get said in some cases and in some cases,

20:47 I totally understand where the question is coming. Yeah. We joke about how old these machines are and

20:53 they're really not. Like people are setting up new machines, probably with Windows 7. They certainly

20:57 were within the last year. And there's good legitimate reasons for that. And we're making

21:02 fun of some of the apparent contradictions, but we're definitely not making fun of the people

21:06 who have often been forced into these positions. But the reality is we can't afford as a volunteer team

21:12 to maintain Python against unmaintained operating systems. And so the advice is stay on the previous

21:20 version of Python that the latest version of Python that works for you. It's not going to break. We're

21:25 not changing it. Anything new that comes up, security fixes will still come out. At some point,

21:31 there just has to be a line drawn. And that's the point where we've chosen to draw it. The other thing

21:36 I want to point out that we changed in this release, which people are more excited about is if you go to

21:40 python.org to download Python for Windows, you get this real big obvious button up front that just says

21:46 download for Windows or download now or something. As of Python 3.9, that's now getting the 64-bit

21:51 version rather than the 32-bit version. For a long time, it's been 32-bit. The reason for that was

21:56 compatibility. We knew a 32-bit one would run anywhere. When we put Python in the Windows store,

22:01 that was 64-bit only. We kind of wanted to test the waters and see, hey, will people notice that we

22:07 haven't put a 32-bit version here? Turns out no one did. And so when we got to 3.9, had that change,

22:13 we made it 64-bit by default. So that has a flow and effect to the ecosystem. A lot of particularly

22:19 data science packages would rather just do 64-bit only packages. Some of them certainly get theirs

22:25 done first and not the 32-bit ones. So we expect to see some flow and impact from that just broader use

22:32 of the 64-bit Python throughout the Windows ecosystem.

22:36 Yeah, that's super cool. And just like the final thought I had was, you know, Django dropped Python

22:41 2 and they're like, we were able to remove so much code and it is easier for new people to contribute

22:47 because they don't have to write for two ecosystems. They write for one. NumPy did the same thing. And I

22:52 feel like this is sort of the same story. Like you guys can just not worry about yet another older,

22:57 outdated operating system and stay focused on what most people care about.

23:02 One thing that someone did suggest in one discussion was why not dynamically light stuff up for the newer

23:07 operating system? And the answer is we do that. And when we drop the older operating system,

23:13 we get to delete about a hundred lines of code for each point where we do that. So it is,

23:18 we get to do a cleanup. We get to say, oh, we don't have to dynamically check for this API,

23:22 this API, load this, cache this, store that, call that, call this format.

23:26 We can just condense that into, oh, we know that this API is there so we can use it

23:31 and just reduce a lot of kind of effectively dead code on newer operating systems.

23:37 Nice.

23:38 Is that a pre-compile like hash if death sort of thing, or is it a runtime thing? Does it make a

23:44 performance difference?

23:44 It definitely makes a performance difference, though we try and minimize it. But again, there's,

23:49 there's always some impact. It tends to be in operating system calls anyway. So you expect a bit of

23:54 overhead. And so it's not going to add a significant kind of percentage overhead compared to whatever

23:59 operation you're doing. But it does certainly add a lot of cognitive burden to someone who's reading

24:06 the code. Sure.

24:07 One example that we got to clean up recently, not in the, in a previous version was we had about,

24:12 I think 70 or 80 lines of code to concatenate two path segments. And this is before Python's loaded.

24:18 So we have to do this with the operating system, the API call up until windows seven,

24:24 I think. So pre windows seven was not secure and it would, you know, buffer over runs, all sorts of

24:30 horrible stuff, but it was the best available function there for handling certain cases. So we'd use it,

24:36 but first we dynamically look for the newer safer one and call that soon as we dropped, I think Vista,

24:42 we could delete all of that code and just unconditionally call the one safe path combined

24:47 function. And that code got a whole lot simpler.

24:51 Yeah. Lovely. That's awesome.

24:52 Yeah. Cool.

24:53 Brian, would you say it's more robust now?

24:55 Yes. I think it would be more robust. Actually, I thought I showed up to the bash podcast. Is this

25:02 the Python podcast?

25:03 Yeah, this is not as bash bites.

25:05 Okay. I love Python. Of course, I still use bash regularly. And I know a lot of people that are,

25:12 like sysops people and other people are using bash daily as well. So I wanted to highlight this

25:17 cool article. This is an article by David Paschley called writing robust bash shell scripts.

25:22 And even though I've been writing scripts for decades, I learned a whole bunch in this and I'm

25:28 going to start changing the way I do things right away. The first two tips right away are things I'm

25:33 going to do. First tip is to include a set dash you and never even heard of this. What it does is it

25:40 makes your bash script exit if it encounters an uninitialized variable. So the problem without

25:47 this is like, let's say you're constructing a path name or something or a path, a long path. And one of

25:53 the directories or file names you have in a variable, if that's never set, a bash normally just silently just

26:01 deletes it and it's just not there. And it'll still keep executing anyway, but it's not going to be what you

26:07 want it to do. So yeah, I definitely want to turn this on so I don't use uninitialized variables. Similarly, if

26:16 any of your script statements returns a non true value, so it's a that's usually in scripts or

26:22 shell work, non true value means something bad happened. If you use set dash E, that will make

26:30 your script exit at any point if one of the sub statements returns a non returns an error value.

26:36 So you don't you don't want to just keep rolling with an error condition. So this is good. I hopefully

26:42 I'll cautiously add this to scripts because I want to make sure they keep working. And then a tip just to

26:48 say expect the unexpected. There will be times where you'll have missing files or missing directories or

26:53 directory that you're writing into is not created. So there's ways to make sure it's there before you

26:58 write to it. If you're especially if you're running on Windows, be prepared for spaces and file names.

27:03 And so variable expansion in bash does not really isolate spaces. So you have to put quotes around

27:11 expansion to make sure that it's a single thing. And one of the things right away, the next one is using

27:17 trap. And I've never actually knew how to do this before. So if you've got a bash script that's running,

27:23 and it just something's wrong and it won't exit, you can kill it or other ways to get it to stop.

27:29 But if you have the system in a state that needs some cleanup so that this there's a way to use a trap

27:37 command to exit gracefully and clean up things.

27:40 The last couple of points were be careful of race conditions and be atomic.

27:44 Those are good things to do, but at least a handful of these I'll put to use right away.

27:48 So it's good. Yeah, this is neat. And a lot of the stuff I didn't really

27:52 know about. So yeah, like, continue on if something went wrong, just plowing ahead.

27:58 Yeah, that's cool to know that you can make it stop.

28:00 Steve, do you ever do any bash? You got a WSL thing going on over there?

28:03 I've certainly done a lot more bash since I started using WSL for a lot of things. I was aware that

28:10 using an uninitialized variable would substitute nothing. But I'm very happy to know that there's

28:15 a way to kind of turn that off because that that has certainly caught me out in the past many times.

28:21 And this looks like just a good article that I'm going to have to go read myself now because it has

28:26 everything that you learn from doing scripts in like command prompt or PowerShell now or even Python to

28:32 some extent. I have not personally mapped those to bash equivalents. So it sounds like this would be a

28:38 good place for me to go through that and up my skills a little bit. My favorite thing was the find

28:43 command. And once I got that, that felt as powerful as a reg X. And I'm kind of like, oh, I don't need to write a

28:49 whole script. Now I can just do one excessively long find command.

28:53 Nice.

28:54 Yeah, you find a lot as well.

28:56 All right, this next one, I don't want to spend too much time on because I feel like you could easily

29:01 just go and spend an hour on it. But for time's sake, we don't have a whole lot of time left for

29:06 the episode because we have a bit of a hard stop. So I'm going to go through this and get your guys'

29:11 thoughts on it real quick. There was a tweet about a GitHub repository that was a conversation on

29:19 the Python mailing list. Lots of places. So Anthony Shaw tweeted, calling attention to a roadmap by Mark

29:27 Shannon, called Ideas for Making for Five Times Faster CPython. So he laid out a roadmap and a funding map

29:37 and some interesting ideas. And I'm going to go through them quick. And then especially Steve,

29:42 we'll see what your thoughts are here, how reasonable this might be. So the idea is like,

29:47 there's going to be four different stages. And at each stage, I think you can get 50%. Speed improvement,

29:53 you do that four times, that's, you know, compounding performance interest, you get five.

29:57 So I think it talks about three nines somewhere. But anyway, I think maybe it's got to shift its

30:05 numbers a little. Anyway, so Python 3.10, stage one was to try to improve will be adaptive,

30:10 specialized interpreter that will adapt types and values during execution, exploiting type stability

30:16 without the need for runtime code generation. That almost sounds a little bit like what you're

30:20 talking about with the 10% increase earlier, Steve. And then 3.11, stage two would be improved

30:27 performance for integers, less than one machine word, faster calls and returns to better handling

30:32 of frames and better object memory layout. Stage three, Python 3.12 requires runtime code generation,

30:38 a simple JIT for small regions. Python 13, extending the JIT to do a little bit more.

30:44 And I'm linking to a conversation, a long threaded conversation over on Python dev. There's a whole

30:52 bunch of stuff going on here. So I encourage people to read through it. But there's just like a lot of

30:56 interesting implications about like, how do we pay this if we pay someone to do it? People like Steve

31:01 work on CPython and they don't get paid. Like, how is it fair to pay someone else to do it when other

31:06 people are volunteering their time? There's a lot going on here. Steve, what do you think about this?

31:10 Have you been following this? I read through the original proposal. I haven't had a chance to chat

31:15 with Mark directly about it. I will, I guess, start by saying that Mark is a very smart guy. And he has

31:21 done all of this planning off on his own in secret and kind of come out and shared this plan with us,

31:27 which, you know, is it's not an ideal kind of workflow, certainly when you're part of a team.

31:32 But I have certainly found in the past that when you get a very smart guy or very smart girl goes

31:37 off and disappears for a few weeks and comes back and says, I've solved it. There's a good chance

31:42 they've solved it. So I'm very interested to see where it goes. The part of the discussion that you

31:47 didn't mention is, or that you hinted at is this is kind of a proposal for the Python Software Foundation

31:53 to fund the work. And that part of that funding is conditional on delivery. So the way he's proposed,

32:00 this would work. And the implication seems to be that Mark will do the work himself and be the one

32:05 getting paid for it. Yeah, that seemed like it wasn't clear from his GitHub repo. But if you read

32:10 the conversation, it was like, look, I'm pretty sure I can do these things. This is how much would make

32:14 sense for me to spend the next couple of years working on it and getting paid. How do we do a

32:19 fundraiser so that I can do this for everyone? Yeah. And, you know, I think under those conditions,

32:23 if the PSF is able to put the budget towards it, they are in a bit of a tight spot.

32:29 Since PyCon is normally the big fundraiser for the year, and that didn't happen. On the other hand,

32:33 it's also the big expense. But financially, the PSF is not in their normal place where they'd be for

32:40 for the year, because PyCon didn't happen in the same way. But I think if they're prepared to put

32:47 funding towards this, I guess if the community consensus is that this is the most important

32:53 thing for us to do. And there's certainly potential downsides to doing it. Code complexity is the big

32:58 one. And I don't actually think there's a way that you implement this or even achieve these performance

33:04 gains without making the code much more complex, and hence less accessible to new contributors and,

33:11 people in earlier stages of learning to code, at least on the C side.

33:15 Yeah.

33:15 Yeah.

33:16 So there's trade-offs. I'm very interested to see what would come about. I assume that because 310 is

33:22 targeted for the first pass, that it's already done. And he's already got the code. And he's trying to

33:28 actually... And he's just trying to get confirmation that he can spend the next few years heavily investing

33:33 in it instead of having to go find a full-time job and go back to doing this in the evenings.

33:38 Yeah.

33:38 Which, you know, I'm fully supportive of. Again, it's really just a big open question of,

33:43 is this the most important thing for Python to be funding right now, for CPython to be getting

33:50 in particular? Someone, I forget who, raised the question of what if we put that money towards PyPy

33:55 instead? You know, what could they do with it in that amount of time? And ultimately,

33:59 it's going to come down to someone, probably a small group, presumably the steering council will

34:05 have some involvement from the technical side. The Python Software Foundation Board will no doubt be

34:10 involved in just deciding, is this the best use of the money that we have or can go out and get

34:18 for what benefits it would produce?

34:21 Yeah. When I look at it with the funding side, I see it as very brought with challenges on the

34:26 sort of community funding and the PSF funding. But I know there's so many huge companies out there that

34:31 spend an insane amount of money on compute and infrastructure that make a lot of money. And that

34:36 if they could have a 5x speed up on their code, they could probably save that money right away on

34:43 infrastructure. So it seems like that they could also get funded that way. But we should probably

34:47 move on just because I've got to, I'm going to make sure we have time for everything else before we

34:51 end up running out of time. I just do want to call out, like, you should go check out that

34:55 conversation. There's a very funny excerpt from Larry Hastings. It says, speaking as the gillectomy

34:59 guy, they were talking about borrowed references being a challenge, saying borrowed references are evil.

35:03 The definition of a valid lifetime of borrowed reference doesn't exist because they are a hack

35:07 baked into the API that we mostly get away with because of the GIL. If I still had wishes left on my

35:13 monkey's paw, I'd wish them away. Unfortunately, I used my last wish back in February wishing I could spend more

35:18 time at home. So bad. All right, Steve, let's get a little bit more insight from you on this last one,

35:26 huh? Because you were at the core developer sprints, which recently happened.

35:30 Yeah. So I don't know exactly what day this is going to go out. But last week,

35:33 from recording day, we had the CPython core developer sprints. So this is kind of a get together,

35:41 generally in person event that the core development team has done for five years now. I think this is

35:48 the fifth year. In the past, we've all gone down to Facebook or Microsoft or last year, we hung out

35:54 at Bloomberg in London and basically spent a week in a room together coding, discussing, reviewing things,

36:02 designing things, planning things, and otherwise just getting to actually meet our other contributors

36:08 because we all work online. We all mostly work over email and kind of bug tracker and GitHub

36:14 pull requests throughout the year. And so it's a really good opportunity to get to meet each other,

36:19 get to see who we're dealing with. It's a lot harder to be angry at someone over email when you've met

36:24 them. Yes.

36:25 And so it's been a really good event. This year, because we're obviously not traveling for it,

36:29 we were hosted by Python Discord, which is at pythondiscord.com. There's a server that is really

36:36 well managed. It's really well organized. I was impressed. I have not been there before,

36:40 but it was great. They set up, felt like thousands of channels for us, far too many,

36:45 but it gave us plenty of space to kind of mingle with other core devs while we were discussing and

36:52 working and planning anything. We also did a Q&A. So there'll be a link in the notes for that from

36:57 YouTube that we live streamed. We had people submit questions ahead of time, everything from

37:02 what situations should I use a mangled name in, like a double underscore led name through to,

37:08 you know, what's your least favorite part of Python? What do you most want to replace?

37:12 Did you ever expect Python to get so big? And we had a lot more people involved. We normally do

37:16 a panel for the host company. So we'll, we'll get kind of their employees together. And it's like part

37:21 of the perk for funding the venue and typically meals and coffee and everything for the week.

37:26 This time it was public on YouTube. It was all kind of over video. So everyone got a bit of a turn

37:33 to jump in. So you'll get to see a lot more core developer faces than you've probably ever seen before.

37:37 You'll get to hear from a lot more of us than you have before.

37:41 And a lot of interesting things, the big kind of ideas that came out of the week,

37:46 kind of hard to say. A lot of us did come out feeling like we didn't get as much forward momentum

37:52 on stuff as we normally would in person. But at the same time, a lot of things did move forward.

37:57 I think there were about seven or eight peps passed up to the steering council during the week,

38:02 the various things. One of mine was deprecating disti-tills, which is an entire podcast on its own.

38:08 So I might have to call you guys another time to talk about that one through to a proposal to change

38:15 how we represent 3.10. Because a lot of places we put the version numbers back to back with no

38:21 separator. And so you have, you know, 3.8, 3.9 with no nothing in between. Now we're up to 3.10,

38:26 or is it 3.10? Yeah. Okay. How do we fix that? And we had a lot of discussions about that.

38:33 There was obviously a lot of talk about JIT, about the C API, all the usual things that we talk about.

38:39 But again, because it was online, it was really good to have such a range of people involved

38:43 across different time zones and people who would not normally get to travel.

38:48 Yeah. It makes it more accessible. Yeah. That's awesome.

38:50 Yeah. We have core developers in countries who can't leave. Like they literally cannot leave their

38:54 country either because the populace is just strictly controlled or they know they would not get back in

39:01 it when they tried to go home. And so they were able to participate. And that was great to, you know,

39:07 see and meet some of those people. We had a few mentees come along to interact with the rest of the team.

39:14 And just overall a good week. Awesome. Yeah. Cool. And yeah, people can check it out. The YouTube

39:18 stream. I definitely want to check that out. Sounds neat. All right, Brian, we've got two minutes left.

39:22 Do you think we should do a joke? Yeah, let's do the joke. Oh, let's just cut to the joke. So we,

39:26 we don't miss that. Right. So you and I spoke about Hacktoberfest going wrong and like random PRs to like

39:33 config files and change, changing the spelling and config file settings. So there is a guy who posted on

39:40 Twitter and said, Hey, I let me double check the name. It was Stuart McCroden. And he posted this cool

39:50 t-shirt that he got. It says Hacktoberfest 2020. Any PR is a good R and it's Lua.py. And it has import

39:57 high in Vim. Then the PR just adds hash. This imports a package.

40:05 That's awesome. Steve, did you suffer from any of these?

40:07 I did not. I might've done. My GitHub notifications are a mess.

40:11 So yeah, I don't even know yet.

40:14 Yeah. I don't, I don't see poor requests until I actually go look at the repo myself

40:18 for the most part. Yeah. I got a bunch. I got a whole bunch.

40:22 Yeah, me too. Cool. Okay. I wanted to do one. This should have been a topic, but the

40:26 five most difficult programming languages in the world. This was submitted to us by Troy

40:32 Codil, Codil, I think. It's not really a full topic, but I thought it was hilarious. This

40:37 is an article where the author, Locad, Locate, I guess, actually took five programming languages,

40:46 Malbosh, Intercal, Brain, you know, we all know that one, Cow, and Whitespace, and wrote

40:53 Hello World in that language. And these are hilarious. And my favorite is Whitespace because the entire

40:59 language depends on space tab and line feed for writing the program. And any non-Whitespace

41:05 character is considered a comment. So this is great.

41:08 That's crazy. I don't know why the APL wasn't on there. APL is just fully insane. Let me,

41:15 I'll put it just in the show notes here at the bottom of this, an example of APL. That right

41:22 there that I put on is, I can't try to even speak this, but that is an entire thing that finds

41:27 all prime numbers from one to R in that line. I hope you guys see at the bottom of the notes.

41:32 That's insane, isn't it?

41:33 That's not even intentionally bad, is it?

41:35 No, it's meant to be a real programming language. It's as if the Egyptians who only wrote in Hero

41:41 Graphics decided to write a programming language. That's how I feel. It's insane. But it's a

41:47 legitimate language. People try to use it. They do use it. Anyway.

41:50 Not very long, I expect.

41:52 Only as long as they must. And then they immediately stop.

41:56 I just like that this Intercal example, it's so polite. It's please, please do, please, please,

42:03 oh, please give up.

42:06 Yeah. Apparently you have to sprinkle pleases in it or else it's like error because it's,

42:12 you're not polite enough. But if you do too much, it also errors because you're overly polite.

42:17 I like that. We need more passive aggressive languages like that.

42:20 Lovely.

42:23 Cool. Well, thanks a lot, guys. It was fun.

42:26 Yeah. Yeah. Thanks, Brian, Michael.

42:27 Yeah. Thanks. And thanks for being here, Steve. It's great to have your perspective.

42:30 Thank you for listening to Python Bytes. Follow the show on Twitter at Python Bytes. That's Python

42:36 Bytes as in B-Y-T-E-S. And get the full show notes at Python Bytes.FM. If you have a news item

42:43 you want featured, just visit Python Bytes.FM and send it our way. We're always on the lookout for

42:47 sharing something cool. This is Brian Okken. And on behalf of myself and Michael Kennedy,

42:51 thank you for listening and sharing this podcast with your friends and colleagues.