Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


« Return to show page

Transcript for Episode #271:
CPython: Async Task Groups in Python 3.11

Recorded on Wednesday, Feb 16, 2022.

00:00 Hey there. Thanks for listening. Before we jump into this episode, I just want to remind you that this episode is brought to you by us over at talk Python training, and Brian through his pi test book. So if you want to get hands on and learn something with Python, be sure to consider our courses over at talk Python training, visit them via Python bytes at FM slash courses. And if you're looking to do testing and get better with PI tests, check out Brian's book at Python bytes.fm/pi test. Enjoy the episode.

00:29 Hello, and welcome to python bytes where we deliver Python news and headlines directly to your earbuds. This is episode 271. Really well. Recorded February 16 2022. I'm Brian Aachen.

00:42 I'm Michael Kennedy,

00:43 and I'm saved. Hello. Welcome, Steve.

00:45 So thanks, Steve dour, but saved li Yeah, a

00:48 number of things. Probably most interesting to this audience is I'm a core developer on C Python, one of our windows experts, I spend a lot of my time focusing on making Python run better on Windows. I also work on Microsoft work at Microsoft, where I also spend a lot of my time making Python run better on Windows. So I'm kind of a bit of a one trick pony, I guess. But I feel like it's good work. And it helps a lot of people. So

01:13 so if I have a problem with Python on Windows, it's your fault. I

01:17 if there's solutions for Python on Windows, then it's my fault. I'll let other people own the problems.

01:25 So if I go to the Windows Store, I can now install Python from there. You were you were part of that right? Oh, I should I should

01:33 have had that up on screen, shouldn't I? Yeah, that was that that was actually the the request came from people within Microsoft were like, Hey, why can't Why can't we get Python up on the store? to it? And my response to all of these is like, Well, if the community is willing to do it, which, you know, is half me and as half the people who would have to take over if I stopped doing it, then then yeah, we'll go ahead and do it. And so I you know, I got actual work time for that. That was that was a contribution from Microsoft for that one. But yeah, the community was on board. And, you know, it's, it's going really well, that's, that's also the one that we tied into the default Python dot exe that's on every Windows machine now. So if you go to a brand new machine and just type in Python, you will get straight to the PSF. Python, it's not Microsoft is not doing it anymore. We just contributed the change. And And now, you know, now I switch hat and do it with the other hat on. So you know, it's it's real Python, right? It's exactly the same as what you get from python.org. It's just delivered, you know, easily fast install automatic updates, and a couple of edge issues that were that we're working on bringing down so yeah,

02:40 automatic. I know, this wasn't one of the topics. But now I think I might have to rethink how I'm installing Python on my desktop at work. So that's it, guys.

02:50 I have only had store installs on my own machines since 3.8. I think I haven't, apart from testing. I haven't actually used the regular install. On my demeanor You

03:02 of course, that makes sense. But

03:04 you know, it's always testing right? Every time I'm using Python, I'm testing. So

03:09 Chris May out there says thank you so much for making my work life and Windows easier.

03:14 Anytime. Yeah. Well, Michael, why don't you kick us off with a story or

03:17 tie have got a good one. So I'm a big fan of fast API and fast API being built on starlett. So by the transitive property, I'm also a fan of Starlet. And there's this thing I want to cover called fast API events. So when a request comes in to a particular API endpoint, or if you convert it over to a web map to a web page sort of request or something, you might want to dispatch that out to say, like WebSocket listeners or something along those lines. So there's this cool project called fast API events. It's pretty small and new. So I'm going to try to give it some visibility. It's only got 36 stars, it's pretty new. But the idea is that you can go through and basically create this middleware handler that will let you say, when a request comes in, here's the way when an event is raised, here's the thing that's going to handle it. And then in some API endpoint, you can say dispatch give the event a name, and some dictionary data to be passed along, I suppose it doesn't have to be dictionary could be whatever. And then in other parts of your code, you say I want to just hear about this event that happens, no matter what API endpoint received it no matter where in like, how deep down in the code it was received, and so on. So then way down here, you just put a role handler decorator on there and say, I want to capture all events that start with some substring like cat star for like, category, whatever, or this one is actually literally about cats. And then you can just go through and write these functions that will then handle that and, you know, you could be one you could also pass them off to queues like union can use the SQ s the simple queuing service from AWS, I believe that is as the endpoint instead of it just being your app, right. So if you've got like lots of scale out and stuff like that.

05:01 Wow. Cool. So is this like a neat way to do logging? Or even distributed logging? I guess if you've got forwarding handlers in there, you can just Yeah.

05:08 Yeah, it seems like it right like, or if you want to sort of build up like, here's the request transaction, and here we're at the stage or like, you could maybe do like visibility into long run a workflow is with this kind of thing or something along those lines, I would think. So. Yeah. There's also an echo handler for debugging. I kind of like that. Like, if I just need to see what is happening. It'll just print whatever's happening. Just start printing out all the behaviors that you're logging. So

05:32 and then when you want to stop doing that, you just take away the handler, and you don't have to search the entire code base for print and find everywhere that yes, the bugging.

05:41 Exactly. Alvaro, out there says this. This looks similar to Django events. Yeah, I suspect a suspect it is somewhere. Yeah. Anyway, pretty short and simple. But if you're looking for a way to sort of put notifications in a structured way into a fast API app, well, here you go,

06:01 Oh, I'm thinking of a whole bunch of more abusive ways to use this.

06:07 It's yeah, you can write some really impressive spaghetti code with this.

06:11 Yeah, I'm sure that you can. Cloud involved in everything.

06:16 Yeah. So let's switch gears a little bit and talk about testing. Imagine that I've got a testing topic. So this is I'm pretty excited. This is a, I've been asked a lot about testing, pipelines testing data science stuff. And, and I'm not, that's not something I do day to day. So I'm really glad they find people talking about it. So this, we've got an article from Peter bomb, Gartner ways to use testing ways I use testing as a data scientist. And I actually just really love this article. It's great with that one of the to start with one of the he starts off with what he uses testing for as a data scientist. He uses testing to make sure you make sure things work, the document is understanding and to prevent future errors. Well, that that seems straightforward. But the the reason why he wrote wrote this up is, apparently because there's a lot of software, there's a lot of testing stuff out on the web, but it's not it's like geared towards test engineers, or, or software developers. And he's like, I'm not a software developer. I'm a, I'm a in you know, I'm doing something else. I'm doing analysis. I'm not a software person, even though Yeah, you are. But but to write this up in in a context where data people might understand it better. For instance, he doesn't even start off with right having written tests, is his analysis is like, if you're doing notebooks or other code, just use a cert a lot. So he's using a cert all over the place, including, he says, already have use it for as many intermediate calculations and processes as you can as it makes sense. Because in doing things like checking, obvious stuff, like he's got an example of a table, a table count, we're just counting up all the yeses, well, you can do a little bit of math just to make sure the math works. So like All yeses, and noes and missing should all add up to the same count, let's go ahead and throw an assert in there, because sometimes it doesn't. And in this example, he said that he, he actually caught an error, because he was looking at two different data frames. So they really weren't, they didn't add up to the same. So you can catch things like that. So just double checking yourself on, on things as you go wrong. away, go, as you're developing. One of the cool quotes he has in here is like, as he has a habit of when he's using notebooks to whenever he's visually inspecting the output. If you're visually looking at the data that comes out, maybe write an assert statement to do that analysis, so that it's always checked. And this is a cool use of putting asserts in notebooks. I like this idea. The article goes on, it's pretty extensive. Talking about checking the data using hypothesis two, well, not the date, not the data at this part. But the your your assumptions around the data. So using hypothesis to check your assumptions and hypothesis will think show you things that maybe you didn't consider like Nan's are you handling those correctly? Empty series or empty data structures that are going into your into your code? are you handling those? If I mean, hypothesis does have take some hand holding, but it does make you think about really what is the shape of the data going in? And do can you do you need to eliminate it? What hypothesis is looking at or do you need to change your code to handle more things?

09:51 I thought this is great. I've used that for a couple of pausing projects or combining projects I spent way too long. adding all the strategies to be able to test a URL positive that I was calling into. But it's fantastic for finding kind of things that you would not have thought of.

10:10 Yeah, I mean, it's finding things but it's also in it does make, yeah, that that aspect of it seems like the point of it. But the real value I get out of hypothesis is thinking, making sure I really understand the data that's going to come in. And thinking through those goes on to talk about actually testing your data using things like Pandora, which I wasn't familiar with. And another package called Great Expectations to look at, like putting schemas around the data coming in and making sure that the data always matches the schema going on to talk about arrange, act, assert, and using PI test by test comes in with, he's only really writing formal tests when he's writing libraries for other people, but all these other packages to be able to test with data science, I think this is a great addition to the data science community.

10:59 Yeah. Alvaro talks about how this is often referred to as defensive programming. And then, you know, I feel for most of us that work with this with our Fortran code. So there's that but I do think this is a really interesting way of thinking about defensive code, you know, I just, I think of writing defensive code as like, Oh, I'm gonna have a bunch of a statement to verify this thing's not none, or verify that this is the right type, and that it has like a reasonable value, and raise exceptions. I haven't really thought so much of it for like notebooks. So that's pretty interesting.

11:31 And one of the neat things about like, if you're actually putting asserts in your code, you can actually you can write tests against your code that don't even have any certs in them. And because the search will happen within your code, and the test will still fail and catch it. So it's kind of cool. Yeah. Very cool. Good stuff. Yeah.

11:50 Steve, I am super excited to hear about what you're you got coming up, because this is brand new, being a core developer, I feel it is appropriate that you break this news. Yeah, I

12:00 mean, I'm not gonna lie when it came to, you know, what am I going to talk about? Okay, what's the most recently accepted pep that was somewhat controversial. And I think just as you kind of look down to the section on rejected ideas, which is considerably longer than the accepted ideas, you can probably get a bit of a sense for just what went on with with exception groups. And I know, Michael, you just had a conversation, you learned all about them. So you can you can take over when when I run out here, but I'll share

12:31 my thoughts with it. But yeah, go ahead, I'd love to hear about it. This is sort of inspired by trio right?

12:36 The the end goal kind of is. So this is an interesting path. And we've got a few of these on the go at the moment, it's kind of like a stepping stone towards a better programming model or a stepping stone towards better libraries. So it's, it's something that I think in my opinion, very few kind of application developers kind of the, the last developer in the chain is like, often not going to use them, and they're not going to need them. But as you go further in towards the lower levels of libraries, especially people writing async, schedulers are going to find incredible value out of them. Essentially, what the idea is, is that when you're running multiple tasks in parallel, if some of them fail, we don't currently have a have a neat way to capture the exceptions from all of the ones that failed. There's some approaches that would be like, wait for all of them to complete and wrap it in a list. And then you get some exception that contains a list of exceptions. But that's lost a whole lot of context, you can get just whichever exception happens first, but then you lose all the other exceptions. And there's just been no real way to handle it. So an exception group, essentially does bundle up all the exceptions, internally in some way. But the really interesting thing is the except star syntax, which I'm gonna have to scroll all the way down to find where that comes up. But but this is really clever, because if you're in that situation where, say you're running 10 parallel processes. So here's kind of the first example of it, then exceptions are no longer control flow at this level. Because if you're, if you've run 10 things and you're waiting for 10 things to complete, you're not actually doing control flow with the exceptions anymore, what you're doing is handling the exception, but then the control flow is going to go back to where it was anyway. Because you because you're going to be doing something different. So for example, if a file doesn't open, then you would want to do something different, right? You're going to stop going on and trying to read from the file. But if you've tried to open 10 files, and three of them failed you at the outside level, so at the end, at the end level for each file that may have failed, you'll do something different at the outer level, all you're really going to do is say hey, this task failed because of file couldn't be opened. And maybe you do something else but it's at the outside level. So accept star takes that exception group and it's going to give you a chance to handle each exception. Essentially on its own it will group them together. So in this example, if you know, five tasks report spam error, then you'll get into this accept spam error block with all five of them at once. Which is just,

15:11 what is that a list of spam exception? Exceptions, something like a tuple, something like that. I think it's I

15:17 think it's a tuple. I think with the Stata Center, I believe, something iterable basically, yeah. So it's something you can iterate over to see the exceptions. But it's really just this happened at some point, and you process it. And if the group actually contains multiple types of exceptions, then each handler that matches is going to be called for all the exceptions that match that. So you could have this try block, raise an exception group that has some spam errors, it has some foo errors, it has some bar errors. And all three, except Except style blocks are going to get called with the exceptions that match those, which is a bit, it's, it's definitely going to confuse a lot of people, it confuses me, which is, you know why I was keen to actually spend a bit more time digging into it, and trying to figure out what's really valuable about this. And I do think the most valuable one is really where the error is cancelled error. Because if for whatever reason, five of your tasks have been canceled, then you need to capture that and do something with that outside of it. But it doesn't necessarily mean you want to throw away the five successful results. And so you do kind of want to keep a bit of everything going on it. And like I say it's a building block on its own, this isn't enough to do anything new and useful. The next thing that comes along is task groups. And that's, you know, being worked on by a, you know, a expect a lot of the same people who worked on exception groups, because we're task groups now you can actually start, there we go, great. We don't just merge task groups. Excellent. Then now you can actually, yeah, like run the task group. And if the group raises any errors, then you'll catch them through an exception group. And so that enables a whole lot of new uses, and new ways to use async. IO or just async. Generally, no matter the library, as you say, trio has already had something like this for a while, and some of their nursery thing. Yeah. And so that is now being standardized. So libraries can kind of share their implementations and work together on it.

17:23 So one of the reasons you need this is if I start to web requests, and three database queries, and then I go to wait on them, you know, then the if several of them fail, the the error state captured in totality is a tree of errors, that representable this, this, this task, or this other task, which then had this error, this other one, right. So you need some way to deal with a group of errors that could happen, kind of all at once, right? And one of these task groups that gets kicked off. Yeah, so the new task group thing is super cool. So you say async, with task group as TG. And there's two things that are neat about it. One is right now, if you fire off a bunch of tasks in async, and await style, they're basically unrelated. Like, if one fails, that means nothing for the other, right? They're just like, well, here's a bunch of stuff that happened. And this creates a relationship between them, right, so that if one fails, I think it might not schedule new ones, something to that. Like it's brand new, I'm just seeing the tweet. So I think that that's the story. I believe that was the story of Brio. The other thing that's interesting here that, in this example, which I'll link to from Yuri, that he posted, he tweeted about the news was, notice that the first one says, task group create task for some tasks, and then await something that creates another task. There's nowhere where you say, store all those values into like some lists of tasks, then go to the task and iterate them and wait for them or gather them or whatever the heck it was you had to do before. This now makes tasks fire and forget, I can say run this, run this. And within that, it could do more of those types of things. And then you just block at the width, context manager level to wait for all the tasks to finish, which I think is a real big improvement. Because right now you've got to like constantly juggle, well, I've got to return a task from this. So I can go wait on it later. And all those sorts of oddities. And this cleans up a lot of that.

19:12 Right? And and of course, being Python, I don't know exactly how the syntax works. But being Python that TGW objects, that task group doesn't actually disappear at the end of the width block. So if that's got results stored into it, then you still have access to those and all of the information about the task group even after you've waited for it to complete running.

19:30 Oh, yeah, that's cool. Yeah. So I think this is a nice addition to async IO in Python this this call apparently three Levin is coming.

19:37 Coming in 311. I do see a question from Sam moly in the chat. Is there a way to short circuit so that you don't recatch certain exceptions? My understanding and Michael, if you've got a better one, correct me is that the exception, the Accept blocks work in the same way as regular ones and an the first one that matches a particular exception will handle it and the later ones don't either If they would also match. So if you have, so if the spam era is a subclass of foo arrow, but there's another subclass of foo arrow, then spam arrows will get handled by the spam error handler. The food through our handler will handle all the other ones apart from the spam arrow subclass. Nice. Yeah, I don't

20:19 know much about the except star other than it was basically a requirement for the task group stuff to be implemented properly. So when you one came, and then the other could come in Yeah,

20:28 see, the only feasible way to actually do something as a result of an exception group. Otherwise, you do end up with, you know, a very generic exception. And then you write a for loop over all the exceptions that are handled and try and figure it out yourself. So you'd end up rewriting the code. And it was just not going to be feasible. It needed to be syntax. And so it is, yeah,

20:51 right on. Very, very exciting and very timely. Thanks to you.

20:54 I'm kind of glad that I put off learning how to do async code until 311. Looks easy.

21:00 It's a good band, and a good time for async. Io. Oh, cool. All right, I guess I'm up with the next one hub.

21:05 Right? Yeah, let's see what you got.

21:08 I have got some other interesting things I'm here about showing off the the underappreciated projects are the new projects, just a couple of stars here. And we talked about overloading before, but I thought this was a clean way to do it that people can think about and see, I would definitely love to hear your thoughts on this. So Felix, the cat, created this library called PI overload. And idea is basically once you have type information, then you can have method or function overloading the idea being like, Okay, if a function called foo or whatever, and if it, you can say if it takes an integer, I want this implementation run, if it takes a string, I want some other implementation to run, right? That's sort of the traditional C++ C sharp definition of it, right? But in Python, we don't have that, really, because the language started without type. So how are you going to figure out the type the overload or you know, right, there's, like, doesn't make any sense. So with this one, you could sort of use like, traditionally, you could use instance, we're going to do one thing or another? Is it a single thing? Or is it a list of those things? What are we going to do with this one, you can put just add overload, and then whatever the signature is, if you can say it has no functions, or has no parameters, or it has like two integers, or it has three integers, or it has, like a list of them, whatever. And there's even a way to sort of say, somewhere down, here's a way to say like, if none of them match, call this particular one. So basically, it's, it's just straight function overloading in Python, if that's the thing you want, Steve, does this make you cringe? Or do you like it? Oh, well, both.

22:37 You know, I'm, I'm not gonna lie, I'm not the most into static typing in my Python code, as a lot of other people. And there's a lot of, there's a lot of complicated reasons. But I think for a situation like this, I mean, I know, if I was writing a function that took a string or an int, the very first line would be converted to whichever type I actually want, and then the rest of the function is going to look identical. And that's

23:00 sure and that, in that case, where like, there might be a unparsed type of thing, for sure, I think you wouldn't really do an overload, that would be insane. And my kind of gut feel, and

23:11 you know, I'm always open to two examples proving me wrong, in which case, I, you know, I would write the instance code that's in those examples, you know, my kind of gut feeling is that if you're doing two drastically different things in the function based on the type, you need two functions. And once you've got two separate functions, you know, if the people calling don't know what they're passing you, then that's, you know, they've got a problem. And it's not so much my responsibility to fix it with overloading. That said, overloading is really cool. And, you know, I am the exact opposite person, when it comes to like C and C++, I will do all the craziest possible stuff with overloading in those languages, because I think it fits the language, and it's a lot of fun. And, and there's definitely occasions and value for having it in Python, we do have the single dispatch decorator has been part of Python for a while, which will do this on the very first parameter. This, you know, very trivially extending it to the whole function signature is, is really cool. So, you know, it's, if I needed to do this, I would probably want to use this library like this. What would I you know, I'd probably, I would probably reconsider my API design choices up to that point. But I can understand the the attraction of getting to, you know, to reuse, reuse the name and not make the person calling it think too hard about how you know what's actually going to run.

24:35 Yeah, the place where this sort of seems interesting to me is, you know, there's some lot of like tricks and juggling people do with like, star star star, kW orgs. Where like, okay, depending on how you pass it stuff, we'll do a bunch of things. Yeah. And I'm always looking for a way to like not do that. Yeah. How can I'm not, how can I remove that like, it's completely opaque. I have to do a Google search and read the docs to figure out what is it at all possible here?

25:00 One of these days, I'm going probably going to take all of the kinds of patents for that kind of thing that I've collected and turn it into a book. But writing a book just feels like way too much work. So not anytime soon, sorry. My colleagues at work can ping me at any time, and I'll give them a patent for what they're trying to do. That's, I do have quite a set of Oh, usually, you're trying to make stuff weirdly, working this way. Here's, here's a nice way that you can enable that without having to resort to, you know, type checks and everything. Yeah,

25:31 I've been using. I mean, I've been using Python for a long time. And I do remember, one of the first things that I noticed as I couldn't do overloading and at the time, so this is, you know, many years ago, I was using a lot of overloading in my C and C++ code. And, and I was like, Oh, I can't do overloading but one of the things I've noticed is actually the instead of keeping, wishing that I had overloading in Python, I've noticed that I don't really use it in C and C++ anymore. I've gotten the other way. Yeah, I really, I'd rather be more explicit about the and just have a function that two functions that that are some maybe there's similarly named, but they have an appendix that's, that's different. So that if you have different data, you pass it, and I'm with you, Michael, I'd rather have people go, Well, which one do I need? I'll look it up, then just passing the wrong data type and having me suck, because, you know, sometimes if if they haven't converted the data, like, a String, String versus number is a scary one for me, because I'm often getting getting my numbers from an API or something. And they come in as a string, if you forgot to convert it, and you passed it to the wrong thing, and you're really doing something completely different. That's, that's not a good thing.

26:45 But I got bit by that one, just yesterday, updating one of my, one of my CI builds to use Python 3.1. I mean, three dot 10. Is it a string to the number?

26:58 teresting? Yes.

27:01 But yeah, certainly that conversion would be what you know, would be wearing the other one is, is it a string? Or is it a list of strings? And that's the one that bytes is in Python all the time? And I don't even know how you resolve an overloaded function based on is it a string? Or can I iterate it?

27:17 Well, like in that case, actually, I would rather just have that part be part of the function at the at the top of it, if it can handle both to check the type and, and iterate or not, but you know,

27:28 yeah, well, alright, let me close this out with two quick thoughts. First, I think this is interesting, because it's one of the things that's possible with modern Python, like once we've added typing, now, you could consider this as a thing. Whereas previously, it really was highly impractical, I think, as a way to do do it. So I think that's kind of cool. And then to I think it might be an entryway for people who are not where Brian and I'll put myself in there as well, yet. I've going like, actually, these things I thought I need, I don't need those, right. There's a lot of stuff I thought I needed. And I haven't used it for three years. So maybe I actually don't need it. But that's not how you maybe first approach approach solving your first problem in Python that you're coming from C++ or whatever, C sharp, whatever, this might be a gateway. So anyway, those are my two thoughts. One more

28:15 thought from Dean after Python 311. Do we get Python? 95?

28:21 I there was no, there was a Windows three point 12. So I think Python gets to do three point 12 as well. And I think it was only available in China. Interesting.

28:33 And I believe I like to follow on with that Dean. Very funny. I believe that Windows 10 was named to you let me know if you know different Steve, Windows 10 was in Windows 10. Because there used to be the check Windows nine as the starting string for 95 and 98. So you can't be nine because then you're going to be 95. So we got a kick on past it's kind of

28:53 it was some embarrassingly vague language runtimes out there still doing that check. That really struggled with Windows nine. And showed up in enough places that Yeah, I think it just made sense for everyone to just skip it.

29:14 Not to give him 13 We're skipping nine. It's too unlucky. All right. Awesome. Brian, what you got for us?

29:19 Oh, what do I have next? I have the next generation seaborne interface. So seaborne is a really awesome plotting library built on matplotlib. And I, you know, actually, I don't use it that much. But I've always been intrigued by it, and what kind of watching what plotting libraries do and stuff. And one of the things I was curious about, which I'm really grateful for this article, is some of the history behind it. So the the article starts off next generation cbord interface, talks about the background and goals, but some of the, some of the great things in here, let me grab, grab some notes. This work grew out of a long Running effort to refactor seaborne internals. So that functions, you know anyway, where I wanted to get at was he was developing a refactor of the internals, he's like, wait a second, if I want to refactor it, maybe I should expose more stuff. And some of the background was originally seaborne was originally conceived of as a toolbox to do of domain specific statistical graphics to be used alongside matplotlib. So the intent was people would use both Seabourn and matplotlib together, however, things have people of doing are doing things differently, a lot of people just grab seaborne by itself. Some people even just learn C were in before they even learned matplotlib, which is an interesting thing. And that's how I thought you were supposed to be doing this. But the concept was an end in overtime, there's a whole bunch of features that have been added to seaborne toward so like really slick looking. But to do the same thing by hand, and matplotlib is a lot of work. So there's some things that like if you, if you see bornes, almost there, but you need to tweak it a little bit, and you have to do things manually, well, then you have to just do everything by yourself. And it's a lot of work. So the idea around this, this a rewrite of the API is let's rework some of the internals so that a lot of the little sub components that go inside of a plot are exposed, that way people can get access to it to do more fine tune configuration, within the so they don't really have to just do everything by hand. It's either all or nothing seaborne or matplotlib, you can kind of do both more easily, which is a kind of cool idea. There's a whole bunch of great details in here that talk about some of the API changes, basically exposing the internet, if you create a plot, there's nothing there, and it won't show up, you have to create layers on the plot. And then within the layers, you've got marks and, and different components that go into it. I kind of like this idea of building things up. But what it really like is the public aspect of this, so you've got a, you've got a library that's out in the open. It's being used by a lot of people already and somebody saying, Maybe we should tweak the API and do something different. And just go ahead and doing that in the open saying, Hey, we're gonna do this. There's a note at the top, or I'm thinking about doing this note at the top saying it's a work in progress don't depend on these examples, because things might change. But this is the direction we're trying to go. Trying to get feedback from people. And I think this is a lot of thing, things that a lot of people struggle with, when they're maintaining packages that have been around for a long time is wanted to things a little different. But am I gonna break everybody? And talking through it? So anyway, this is a great read, especially if you're a data plotting kind of person.

32:43 Yeah, very nice. I always want to do more with visualization. And I'm sure that I have some good data I can pull up. Yeah, I ended up basically just writing API's and websites these days. But But I really should be pulling this up and doing some of these graphs. And I'm really happy these are these around. Yeah, Steve, how about you

32:59 see it see bones great. It's always like, back when I first discovered it, one of its major selling points was simply importing, Siebel and would magically make your default matplotlib charts look nicer.

33:11 Which which mantle of it is I love it. It's like the bootstrap of matplotlib.

33:15 It really was, it's like they they just apply that style by default. And every matplotlib charts on they look nice that way, which you know, matplotlib done their own styling work now. So it's less variable for that. I do like this API, it looks good. And and as Dean's pointing out in charts, like matplotlib has an object oriented plotting API, similar to this possibly identical, just like everyone else, I've never learned the object oriented API. But but it is there. And it's, it's, you know, it's, that's the modern one. It's like, I know, a lot of people say matplotlib is impenetrable, and kind of hard to build things up. But it does have a really nice API there. It's just not the pie plot one that kind of imitates Matt labs Oh, API. And so you know, having it there is is really nice and see one, you know, having their own is also great.

34:07 Another nice thing that to read about in this is, is does a hat tip to gg plot or gg plot to or whatever it's called. Saying that, yes, it's good to look, a lot of this is similar to gg plot, but it isn't that I'm trying to copy it or maybe that's that's definitely influence. But it is. That seaborne is a is important because we think about things differently in Python than we do in R. And, and, and just having it would be but also a hat tip to another library that is a wrapper around gg plot if you just want that. You can do that in python two that's available. So it is interesting to these are we think of these as competing libraries, but they're really not competing with each other. They're working together to push the push, plotting forward. So bad

35:00 Nice, Dean out there points out, you can do plot that style that you seaborne or it's PG plot, which let me throw out Oh, yeah, let's see, gg plot, certainly

35:10 the one to copy from. I mean, that's the there's a reason that one is, you know, as universally popular, as you know, any plotting library can possibly be. It's probably even, you know, it's probably competing with Excel for popularity of flooding data. Realistically, it's, it's a really nice API, and it looks good. And everyone's familiar with it. And so you know, there's nothing wrong with copying from gg plot,

35:35 as I got one more shout out to throw into this conversation, the XKCD plotting style, or matplotlib. So you've got I mean, this is fantastic. It looks like the, it really does look like XKCD would, you know, the comic would do for the for these. This is fantastic. I love it.

35:57 What I love is I actually see this, I see this in papers and stuff like that people just go ahead and use the XKCD style, and for serious stuff, and it just is awesome. I love it.

36:07 I think there's actually some value to having like cartoony looking graphics, like UI sketches and graph to say like, look, this is speculative. This is just like, Don't read too much into it. I'm trying to give you an idea rather than the exact thing. And I think sort of UI like cartoony looking sketches. And this also plays into that. Yeah. All right, Steve, you got the last one, held the

36:27 last one. Yeah. So this is another kind of recent delivery from from the C Python core team, we can now compile C Python to WebAssembly. Wow. So and to a lot of people, that probably means very little. But I guess the brief brief summary is web assembly is a kind of what the JavaScript in your browser compiles to before it runs. So it's skipped that initial step of being JavaScript, and it's now ready to run in the browser. So it's it's a low level, there are two chains out there that can compile all sorts of languages directly to WebAssembly. And so in this case, we've taken I believe, we use one of the I don't know the exact tool chain that's used, and it may not matter. But it basically takes the C code and compiles that to web assembly gives you a package that can be brought into an electron app or a no js app or a web browser, modern web browser and be run in the browser. There is so this page is a little bit dated. There's been a bit more work since then. But I found this is the best overview of where things are kind of long list of C extensions that don't work. Probably unsurprising. Like the browser doesn't have a lot of this stuff in it. Yeah.

37:39 Don't have what no ticking all the different API's the wind 32 API underneath. Yeah, whatever it was delegating to. Yeah,

37:45 I don't know TK, enter. What? No.

37:48 Yeah, no, no, TK, enter no sub process. C types, apparently you can do I've heard there is a lib FFI port to, to the Emscripten kind of platform. So how this kind of works is kind of when you take web assembly in into a browser, it has access to nothing like it starts off in a really enclosed kind of box of things that it can do. And that doesn't actually work for Python at all, because they're very early thing that tries to do is search the file system for what files it should be loading. So you wait, we actually build it as part of another platform and scripting is one platform, that kind of polyfills a whole lot of native looking API's. So that code that's compiled on top of Emscripten is able to use it and that this little demo, which are set start repple. This is running on that. So this is a build of Python 311 Alpha four, built with clang and running on Emscripten. And I can I believe I can do this be like Oh, s lista. And it thinks as a filesystem, then now that's not my file system that's in memory, it can be changed to browser storage. But this is entirely in the browser, like there's nothing downloaded. There's nothing running on my machine here. There's nothing running in the cloud. It's literally in the browser. I can probably freeze my browser with this, like I can do an infinite loop and do it. Do it. Let's see if this cuts me off. I'll just let that run. What happens if you hit it again, we'll start again. Now it's done. A refreshing. Yeah. So yeah, and and the, there's a second one that the actual build of those committed supports, which is Waze WSI. That's a slightly different approach to adding all the functionality around a web assembly module. It's so it's a little bit more flexible, a little bit more controlled. Emscripten is really like give me a POSIX system inside my browser all in memory. And so well, and so we have two options. And these are available in the in the main branch. I don't at the moment we're not shipping prebuilt modules. for web assembly, that might be a possibility. If that's something that you'd like to see, then I guess go to discuss stuff item.org and post about it. It's probably a poster, I should look for a poster. But we're not currently doing prebuilt releases, but, but I think we could, I think this is one of these options where the WebAssembly build is totally portable. And so if we build it, we can distribute it. And then, you know, websites that want to do something like this could just download it from our servers and, and run it. So I think there's, there's a lot of potential here. What I'm, yeah, and and it's had the potential stage, right, this is another stepping stone to bigger and better things. Our kind of responsibility as the core team is to enable it. And now we really want people to come in and pick this up and do awesome things with it. Firstly, so we can figure out what gaps still need to be filled. But also just just to just to expand, you know, the growth and the reach of piping to bring it into places that currently doesn't exist or can't work. And, you know, give it new life and new places, open it up to new people.

41:03 This is fantastic. Congratulations. And so the look for

41:06 this primarily done by Katie Bell, Christian Himes and Ethan Smith. So I think Christian got to do all the merge commits, but it's definitely been number of people working on this for a while. Those are the the arm the primary three,

41:21 I'm really excited for the possibility for this. I think one of the things that could be amazing, obviously, running it in the front end is the thing that could be done. I saw the documentation said it was about 10. Meg's to download it, I'm sure you can put that on like a CDN. So you kind of hit it once somewhere for a particular version of Python. That's pretty good. You know, it's we all have pretty fast things these days. Yeah. Yeah. What gets me really excited, though, is putting that into an electron js app. Yeah, absolutely. Right. Because electron js is a really interesting way to bring web technologies cross platform, as much as I like OS and electron app. Still, it's, it's really opened up the possibility for a lot of things. But it really is meant, okay, you're doing TypeScript, you're doing JavaScript, and you just have to go full on in that world. So here, you could still do like your front end and whatever. But having the core logic of that desktop app being and Python running in this, that's exciting, if that's, that can be put together,

42:21 I should also add two things. Pi iodide is a project that people have probably heard of before, which has been working on this for considerably longer than the core team has. And so I think a lot of the patches that needed to happen have come from them. And they now get to spend more time focusing on the data science stack, which because they've got parts of NumPy, and pandas and other libraries, to actually do data science in the browser. And the other interesting thing that I saw was someone from from conda Forge, suggesting that they could elevate was wasn't built to their kind of automated level. And so all of conda forge may suddenly become available to us in the browser on top of a glorified and like this.

43:04 So that that would unlock so much. That'd be incredible. Interesting,

43:08 I imagine initially would unlock a lot of bug reports, but but we need to work through those first.

43:15 Yeah, I was just thinking of, you know, take the top 1000 most popular packages, Oh, could you get 90% of those compiled to like other web assembly, things that then could be included and then imported here, somehow?

43:26 Exactly. And the top 1000 with native code, because it's only the native code, right? The Python code still compiles in the browser, just like it would in the C Python interpreter. It's only the native code that has to be ported and built. And so once that's done, then, you know, we're up and running. So the top 1000 is probably more than you need. Yeah,

43:47 absolutely. All right. Awesome. I'm looking forward to seeing where this goes. So many neat options. There's there's just cool ways to say like ship the Python runtime, the places where maybe it would have been hard to get now you dropped this wasum file plus something that can run wasm. And then now you've got a deployable shippable. Yeah, C Python runtime, without TK enter and a few things but still, you might not miss it. Anyway, it depends what you're doing. I mean, most of apps are not to Gator as all I'm saying. I'm not trying to bang on it,

44:16 though. But I just haven't. Every time it comes up. That is still there. I'm like really? We still love that. Okay.

44:23 Don't ask me what I've been spending my week working on, Brian. It's not gonna make you happy.

44:29 Are you are you creating a TK inter base killer for against textual?

44:35 No, no, no, actually not.

44:40 Awesome. All right. Well, Brian, are we at extras we are

44:44 at extras. Do you want to kick us off?

44:46 I will kick us off. So I've got a couple of things that I think are interesting. Let's start with this one. So we've talked about oh, oh, my Z shell, right. Yeah, a lot. Love it. I just came across realizing that actually this is a Portland company that puts together the sort of core maintainers oh, cool of that. So I just thought it was funny to give a quick shout out to play at Argonne. They're not really in the Python space, but they're in Portland, which I thought was kind of fun. And then what is this? This next one comes to us, I think, via pi coders. That's where I got this. Django just reformatted all of Django with black. And I know, I was just having a discussion with somebody like, Oh, your code doesn't have doesn't follow pep eight. Like, I don't want it to follow pep eight. Yeah, but if people are gonna use your code, you know, like, literally got imported, then it probably should follow. Like, it should not come up with all sorts of warnings. So I thought it was interesting that Django just said everything, make it black. See, what do you think about that?

45:47 I'm totally on board with with just using black on everything. I don't agree. 100% with the style, but I agree 100% with not arguing about it. So yeah. Yeah. Close enough. Yeah. Plus,

46:01 there's enough tweaks like that make it good. Like I really, I'm really grateful that has, you can tweak the line length, for instance. Yeah. Because of it. Here's example. What if I want it really short. So for? No, for seriously, for formatting the code for the PI test book, I want them all quite a bit shorter, so that they fit better in a book format. And I could use black to cover with that and convert everything with with black and make him like that. So it was great.

46:32 Nice. Awesome. Alright, and the final one is I have been doing some stuff with more fun things on YouTube trying to put these little short videos together. So here's a how long the six minutes 44 second video on using time delta to get like, how many weeks or in some time span that you know, the cool tricks you can do there. So people should check that out. That's my latest Python short thing. And yeah, that's, that's it for my extras.

46:58 Okay, so I've got a quick one, just, uh, I've got I don't have a graph some to throw up. But I just I was looking at looking at the Git history of a repo and trying to figure out whether I included one of my co workers branches in it if I merged it yet or not, things like that. And I was on the command line I just learned, I'm like, can I just do this with the command line? Apparently, I didn't know this exists. So apparently, a git log dash dash graph just shows you the Git graph of your branch history or the branch graph on the command line. And I didn't know it was there until today. I started using it tweeted about it. And then a whole bunch of people said, oh, you should use these flags, too. That makes it even nicer. So. So it's fun to learn something old some as a new thing. And then somebody else told me, how about Git k. So get K is a is a graphical browser of your repository that just comes with most git installs that I didn't know was there. I'm like, do I need to install it? I'll just type it and see what happens. And it popped up this graphical interface. I'm like, This is great. This is exactly what I want. Oh, so get case. Pretty cool.

48:10 I didn't know about that one.

48:11 I've seen the command. I've never actually run it to see what happens. So I was not feeling quite brave enough.

48:20 Didn't mean get kill? Or was it something else?

48:25 That's just Mosca command scare me until I've run them the first 100 times or so? How about you? Yeah, I got a couple of extras. Okay, can I get my screen back up? There? It is, I was feeling a little bad about you know, being a bit self serving here. But then Michael just promoted his video series. So I don't feel too bad anymore. This is the python three dot 11 alpha five download page. And we have a new edition this time around, which is this Windows installer for ARM 64. So arm 64 is not a massive, massive platform for Windows yet, but it's growing. And we want to have Python support on it. So the builds have been running in the background for a while, but we've never actually released it. We're hoping to get it out with three dot 11. That is going to depend largely on do people use it? Do they love it? Do they hate it? My experience so far with it has been that it is noticeably faster on at least on the arm 64 devices I've had access to compared to the Intel devices, which is really, really cool. And there's like the test suite is kind of 30 to 50% Faster, which is a huge, huge really so so I think there's a lot of potential here. I may just have had awesome hardware. I'm not sure it was a virtual machine. So it's kind of hard to tell. Yeah. But this is fantastic. This is new. If you have an arm 64 device like Surface Pro X or that there's a couple out there from other manufacturers. I'm running

49:45 Windows 11 arm on my MacBook Pro and through parallels,

49:50 I then use please in download and install it and let me know how it goes. If you get it through the Windows Store, which is currently still not public, you need to get the The link from basically from one of my tweets to the Windows Store, you will automatically get the arm 64 version on ARM 64 as well. So this installer is the traditional one. Otherwise you get it through the store. The other thing, which I wasn't going to do and then I spent a bit of time working on this a couple couple years back at the other core dev sprints, forgetting I was chatting with I was chatting with one of the other core Devs about everyone typing from collections import deck and misspelling it and it's like you tell, you know that so a deck D qu E is Double Ended Queue, very useful datatype for certain purposes, but people would type it deck by deck isn't D ck, because it's phonetically what it sounds like. Exactly. And so as as a bit of a joke, I made a package that when you when you installed it, it would give you from collections import deck, and obviously the thing that that collection should be is a a deck a Double Ended Queue of 52 cards representing the cards in the normal deck. And overtime for various reasons. It's just kind of grown and and I recently you know added support for calculating poker hands to it. And so now if you can build a game with this, it does it uses a gnomes it's got shuffling, dealing, Joker's are optional, and you can calculate a poker hand. And yeah,

51:19 my even compare them poker hand one greater than poker hand two eyes, I

51:22 spent a lot of time most of my work on this over the last week was writing the tests that proved how incorrect that function was until I wrote tests for it. But now at this point, yeah, it's, you can look at the values it gives back, it's actually a tuple, with an enumeration saying what the hand is, and then a selection of the card values in a way that makes the tuples comparable. So you can actually look and see, you know, it's a pair of aces, it'll have the number 14 there for the ACE, and the next highest card was a 10. So if someone else has a pair of aces, and the next highest card was a nine, then you're still going to compare higher. So I'm pretty proud of that function. Yeah, that's clever. But yeah, this is and and it's code style

52:03 black. This. Oh, very nice. So

52:06 yeah, it's one short file. And it does still override Dec and the collections module for you. I love it. It doesn't. Alright, so that deck isn't there, right? It's like DK is untouched. But if you try and import D CK from collections, then you'll get it.

52:26 Nice. Hey, one other quick thing to shout out. We're hiring contractors to help develop features for pi pi.org. It says at the top of ipi.org. That's, do you know anything about this? I guess if people want to work on pi pi.org. That's pretty neat.

52:38 Yeah, no, they have funding. And there is a post that describes the surveys, I believe this is the organizational accounts project they're looking at, yeah, organization accounts. So if if, like me, you are kind of the one of the primary Python people at your company, then you will spend a lot of time helping people publish packages to pi pi, if that's the business you're in. Suddenly, As for us, there's a lot of packages from Microsoft up on pi pi. And the kind of corporate account for that is, by those exists, we have a user account with 483 projects. This is all manually curated right now. Because pipey, I just doesn't have the functionality to kind of hand out permissions to it safely. The teams and all that kind of stuff. Yeah. So So I believe the the idea of this is to add that functionality to API. And so I would love it if someone comes along and does this, I believe we've contributed some of the funding towards this. So

53:39 yeah, looks like

53:40 oh, so Steve, I've got a 311 question for you. So Okay, three elevens an alpha. So what what does that mean? Really? Does that mean? I can like start using 311? Or should I wait?

53:52 It means you can, it means we still may change stuff that will break you. And we won't apologize?

54:00 Okay, but if my code runs, can I trust it? Or

54:04 yeah, if you're, if all of your tests pass, then you should be you should be able to trust it fairly well, certainly existing code, there will be new features available in the alpha that have not been thoroughly tested yet, or may change again. But again, if you're running existing code, you won't be using those. So, so that won't matter. But yeah, it's totally viable to use, you can specify three dot 11. Dev on GitHub actions, I believe it compiles from source. When you do that, now, they don't have a build there. They should for beta. Beta is when we really want people to start doing stuff at this point. Alpha is so that people can test the new features kind of targeted testing on anything new that we've put in beta is when we really want people to start porting libraries, especially kind of the core libraries to be able to work with it. And and just test it because if existing code doesn't work on the beta, we want to hear about that so we can fix it in the runtime. and not forcing you to fix it in your code.

55:02 Okay, but if I'm like a package maintainer, I can start if it's got GitHub actions for it, I can start testing having my CI test against 311. Then to

55:09 absolutely. Okay, you will likely want to mock it as it's okay if it fails. Okay. Yeah. Awesome.

55:16 Okay, thanks. So we do a joke.

55:19 Let's do a joke. Let's, let's do a joke. Alright, so this one coming from the programming humor. One. And it's a, like you talked about the visualization stuff, Brian, and this one, it says, there's a search that says, how to get labels on MATLAB bar charts to be horizontal. Look what the results came back from Google was says you're not alone. Help is available. If you're experiencing difficult thoughts, please call 116 Dash 123 or via this is an emergency call 999. And the underlying vit here is it isn't that drastic Google, but I believe it might also work on Bing. I'm not sure. There's a physical out. I think there's a Bing equivalent down here. Yeah. Not just Google Bing things here and emerging as well. Yeah. That's awesome. Yeah. So it's not that that matters that much of an emergency. I'll go to Stack Overflow.

56:17 Nice to know that the big search engines are looking out for our mental health.

56:21 That's right. People become very upset after failing to get those bar charts. This is not the emergency but it's coming next and you realize what the answer something. Anyway, that's the job I found for us, guys.

56:36 Well, thanks, everybody. Thanks, Steve, for coming. And Thanks, Michael. Again.

56:40 Yeah. Thanks for having me. All right. Bye. Bye.

56:43 Thanks for listening to Python bytes. Follow the show on Twitter via at Python bytes. That's Python bytes as in by Ts. Get the full show notes over at Python byte set FM. If you have a news item we should cover just visit by them by setting them and click submit in the nav bar. We're always on the lookout for sharing something cool. If you want to join us for the live recording. Just visit the website and click Live stream to get notified of when our next episode goes live. That's usually happening at noon Pacific on Wednesdays over at YouTube. On behalf of myself and Brian Aachen, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page