Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book

« Return to show page

Transcript for Episode #215:
A Visual Introduction to NumPy

Recorded on Wednesday, Jan 6, 2021.

00:00 Hello, and welcome to Python bytes where we deliver Python news and headlines directly to your earbuds. This is Episode 215. Or to January six, my favorite dates. One. I'm Brian knockin.

00:11 I'm Michael Kennedy. And we have Jason. Oh, yeah. Hey, Jason, nice to have you here. Jason McDonald.

00:16 It's good to be here. Thank you for having me.

00:18 Yeah. Thanks for joining us. Oh, and Brian, I think he's gonna cover something we haven't really covered much on the show. Uh, gooeys.

00:24 Good.

00:26 Actually, to be honest, I know this is like a long standing joke in the show for longtime listeners. But we actually haven't covered gooeys that much recently. But there was a long stretch where we did.

00:34 Yeah, yeah. That was probably like, a year ago.

00:37 Yeah. Yeah. Like my programming projects on my brownies to be gooey.

00:42 And fudge, come on. fudge

00:44 is good. And you like bad jokes, so you'll fit in nicely? Oh, absolutely. No, if

00:48 anyone like puns, follow my Twitter, I posted original every Monday.

00:53 Nice. I heard that there's gonna be a lot of exciting news for space in 2021. So I kind of want to bring a little space in Python together.

01:00 That's good. Yeah. Yeah.

01:02 So the first topic that I want to talk about is this video done by a woman in the UK, who is a astrophysicist. She goes by the name, Dr. Becky, which is cool. She has a fantastic YouTube channel. She's also a Python developer. And she works in cosmology, which is pretty cool. And she did this video that I just like to highlight for people who maybe are coming into Python, not from the Hey, I'm going to create a micro service set of API is talking to Docker, but more from the Hey, I do some kind of science or data science or something like that. And the video is called the five ways that I use code as an astrophysicist. Cool. Yeah, yeah. So she basically lays out the idea of as a modern day scientists, you can barely do your job if you're not doing some sort of programming. And of course, one of the best languages technologies for programming these days is Python in the data science space, right, right. Yeah, no big surprise there since 2012, I would say, and so that covers five different things with examples of each. So I thought that was just a nice way for people who are either getting into Python from a science side, or maybe their teachers, and they want people ask, why should I not just use MATLAB or some other custom tool like, let me show you. So here's some really cool examples of real astronomy being done with Python. But it's also super accessible to even like middle schoolers, I would say. And number one is image processing of galaxies from telescopes. So you can do things like noise removal. So it turns out that when you're taking pictures of galaxies, even if there's no actual background, light, or disturbances, just the basic disturbance in the actual sensors themselves, will put little marks and imperfections in the images. So using Python to go through and clean those up, makes it much easier to get started. And the size of these pictures. And the amount of data coming in from some of these new telescopes is stunningly large. Yeah, for sure. Another one is data analysis. So if you're trying to find the brightness of some part of an image, say maybe you're looking for a transit of an exoplanet, right, you want to constantly monitor the brightness of a star, or in her case, what she's studying, it just blows my mind. She said in galaxies, like when you see pictures of stars and your zoom in you go, that's not a star, that's a galaxy. It's just, you know, like, I still can't really get my mind around that. But she talks about one of our data sets that has 600,000 rows of like brightness of galaxies, so 600,000 galaxies, they all have information about that they're comparing. That's pretty awesome, right? Yep. Model fitting. There's an example about theory that most galaxies have a supermassive black hole in the middle. There's also this idea that possibly the size of the black hole and the size of the galaxy, these things kind of grow and mash together. So she has all this data, she's like, well, let's do some statistical fits of black hole size and Galaxy size. Also, the color of galaxies can indicate the relative speed or rate of star formation and the issues and the age. Exactly, yeah. all tied together. And so she's using Python for that. Finally, data visualization, you know, pretty straightforward, but he's drawing graphs and pictures. And the last part that I was my favorite part is simulation. So there's two really cool examples. What happens if a star gets too close to a black hole, and gets she said, spaghettified. Cool. And the other one is examples of galaxies colliding, which is just, again, mind blowing, but really cool computational examples of all that. So I wanted to highlight this video because it's super accessible, but it's also really neat to show like concrete examples of real science being done with Python.

04:47 Yeah, I thought it was cool when she was talking to her colleague about you building the simulations of the of the universe. I know you have a simulation of universe where do you start on that? It's like you we think we have products Blocking, you know, it's like you start on a project. It's like, I'm just gonna build a tool. Where do I begin? It's like, I'm gonna build a simulation of the entire universe where?

05:07 Exactly. I'm gonna simulate gravity at scale. Let's just do that. Yeah, awesome. So if people are out there, and they're interested in this kind of stuff. Yeah,

05:19 this is all in one video.

05:20 Yeah, this is all in one video. Yeah, Robert says star galaxy. It's big. Yeah, they're they're both huge. But obviously, man, it's just like, I can't get my head around like galaxy size stuff. It is. So

05:30 so far isn't a star as a primitive type and in the universe, in the end that a galaxy is a collection. That's what I just immediately go to right there is

05:41 chocolate. Exactly. Yeah. So Brian, it's it's like a 15 minute video that half of it is the stuff that I talked about that half is what Jason touched on, she actually interviews one of her colleagues, who basically does the more the simulation side of programming.

05:55 That's pretty cool. Yeah, check that out.

05:57 Yeah, it's definitely worth it. Yeah, I

05:59 enjoyed it. I don't do very much data science actually, at all. And so is like, you know, understanding seeing data science stuff is always interesting, because I but most of my work is in like, you know, application development, I don't usually work in a lot of data. So it's the that side of it explained in this really cool relevant way. Instead of the wall, the statistics, the number of people you know, who, you know, by, you know, cheese every weekend at the supermarket is not interesting galaxies.

06:22 Exactly. Getting better click through rates on your ads is not not super compelling. But I think it's really valuable to see alternate perspectives, right, we all get into our own little world of like, this is what programming is, this is what Python is for. And then, you know, it's bigger.

06:36 I want to talk about NumPy a little bit. All right,

06:39 tell us about it. Well, I've

06:40 actually, I've used NumPy, often on a lot. And it's definitely a staple, scientific use of machine learning sorts of stuff. But, but I'm starting to use it more. And I realized, I realized that I had the wrong mental model. So I like to think of a raise kind of just like lists, but different. And so I came across this article, it's a couple years old, but it's a visual intro to NumPy. And data represent in To me, it really helps a lot like to help me understand what you can do with it, and just have a good mental picture of what what the arrays are in NumPy. So it talks about arrays, matrices, and then in India arrays, which are individually, but like, for instance, I'm even just creating an array, I know how to create an array, I mean, you just kind of initialize it with a list and you get an array, but it didn't know you could do like just say, I want a list of ones or a list of zeros or an array of ones. Just a random array prefilled with random numbers. That's pretty cool. And then he talks about, you know, arithmetic you can do with them and slicing.

07:45 O'Brian like when we talk about pythonic code all the time, like oh, you could write code in this way where you kind of hack a numerical for loop, but you should do this other way. And that would be more pythonic. I suspect there's also a no my way and a non pick way, right? Like filling up stuff like, Oh, you should just do ones on this one. And then then you don't you always like there's a lot of cool other ways of sort of conceptualizing things, right.

08:08 Yeah, well, and it's worth remembering. And I've said this quite a few times. Not here, obviously. But I really like to remind people abstractions are there to save us typing, never to save us thinking. It's like, it helps to have that mental model, as you put a Brian, you know, straight because if your mental model is wrong, it can really wind up well. you're prone to both cargo cult programming while I do it this way, because the way I was taught or trying to, you know, Ill fit a pattern that's familiar to you know, the wrong sort of problem. And you don't realize what it is you're really working with. So understanding what's happening under the hood, even if you don't know all the technical details of the application, still understanding how it's doing things important to you know, choosing the right. idiomatic patterns always. Yeah,

08:54 yeah. And you'll hear stuff like, Oh, well, Python is slow. It's like, well, because you're doing it wrong. Don't do it that way. Yeah, for example, use something like NumPy. Right.

09:01 And like, first, one of the things I really loved about this article was the explanation of dot product, because I've heard this before. I've never had to use a dot product, but it like somebody described it to me several times. And I'm like, Yeah, okay, weird. But then, like, the visual representation of it, like, like, just stared at it and read it for like, you know, 30 seconds. Oh, that's easy. Now I get it. And I'll have it forever now, because of that sunk in there. Pretty good. Um, one of the reasons why I went to it, I have this problem is that I get, like large arrays, but they're not like huge. They're like in the 1000s, say, of numbers. And I need to make sure that one array is like comparing to another. I know equal works carefree. But I wanted to compare item by item, if make sure every element is less than the other element in the other array. Less than or equal. They didn't know how to do that. And I'm like, I think NumPy would probably do that easy. Can you do one NumPy array, less than

09:59 The other.

10:00 Yeah. So if you say less than it compares it element by element, and it gives you a list of true or false. And then you can do all

10:07 Yeah, doing all on it and

10:09 just say all of these two arrays less than or equal to each other, and I get exactly what I want. And they're very expressive, simple line of code.

10:17 It's that kind of stuff I was thinking of when I was talking about like the num pick. bionic way.

10:24 Hi, thank

10:25 you. He's like, that's like one or two lines. And it's really fast. Whereas you could loop over each item individually. And it not only is more code, but it's also slower.

10:33 Yeah, well, and also I like, I also have to I like that there's the intermediate step of that there's gives me a list of true and false too, because I also on the debugging side, I need to be able to like wrap this in something and pick like, say, the first five elements that are not matching. I mean, I don't want if I if if it if it's false, the whole statements false. I don't want to like just say, you know, list all the 1000s that are wrong, but I I want to be able to like list a few to say at least these are not in the right. Yeah, yeah, it's good.

11:04 I want to try NumPy. Now, hey, I now have a reason that

11:08 exactly like why am I not using this in certain situations? Magnus? the live stream says two dimensions is okay. Three is hard. But in that, then my mind blows. Yeah, I actually did a bunch of math research and four dimensional stuff, two dimensional, but complex numbers. So four dimensional, sort of? Yeah, it's just, it's just hard.

11:26 Well, what one of my weird maxes are programmers, I actually can think in six dimensions. It's it's I mentioned before the podcast, I had a head injury few years ago. So I'm a minor traumatic savant, I can think in sixth dimension. And the best way I can explain it, if you're trying to do it without having a really bizarre brain like mine, is think of think of the fourth dimension as a timeline. And for each timeline, you have you have space represented as a cube. But then you have this row of cubes, which represents the timeline, it becomes a lot easier to think of four dimensional race when you think of it in that fashion.

11:57 Yeah. And the way that we did it, we actually had animations of that three dimension thing. And the animations were moving through that that bit, but still, it's, it's now Easy peasy.

12:07 Yeah, it's easier when you're an animator to wrap your head around for D than if you're just a, you know, normal run of the mill programmer. Like,

12:14 Brian, would you say that? That's a gooey type of solution? No. Maybe you could do something with cute. Yeah, okay. Oh, yeah. Jason

12:25 knows it's possible.

12:27 So that's our next topic to go grab Jason.

12:31 Yeah, well, okay. Well, I thought I was really excited to discover the cute six just released on December 8. So cute. Se Yeah, it is officially pronounced cute. Although it's much. It's very debatable. People know, it's cute. It's cute. You know, whatever you're gonna call it, it just released. And this includes the Python bindings, so pi side six shamokin six, which is the whole pi side two was Qt five as if that made sense. Pi side six is is Qt six cute sexy. Now I'm doing it anyway. So that just released. Um, and you also have the PI Qt six if you prefer riverbanks version, but in whatever case, you're going to wind up with, with all the cute six features. Um, I think the coolest thing here is if you're doing if you're doing like, you know, really fancy sort of graphics is that previously cute five and prior had this hard dependency on OpenGL. And they've actually put in a what they call the rendering hardware interface with an abstraction layer into into cute. So now, it can natively support whatever the 3d graphics driver is on that device, whether it's Direct X, Vulcan, metal, whatever, whatever you want it to work with the use of the native by default, you could you could tell it to use whatever, whatever you want.

13:45 That is so cool.

13:46 Yeah. And there's a bunch of other optimizations and fixes to have in here. I am really excited because I discovered and this was actually introduced in 515. But they now support snake case. For those of us who are like Pep eight addicts. Yeah, I really hate the fact that you kind of seem to force you to use the the camel case, you can use snake case there is a there is a setting for it. You can also use properties instead of getters and setters as of dx. So you can just rely on properties. And that is it makes it a lot easier to run eight, you know idiomatic Python code that is cute, which is fun.

14:25 Well, it just feels wrong to write, you know, get with. So with Yeah, other things. They also

14:29 have this cool thing called property binding where you can actually link those together now to is like, link the width and the height. So when you change the width height automatically changes.

14:37 Nice. Yeah, I really want to build some stuff with cute. I've got a few app ideas in mind. What I don't have is time. Sad. Like Can you help me that Jason can help me set more time? Well, I,

14:48 I I know I have a reputation as a Time Lord, but unfortunately, I can't control the stream of flow of time there. If I find my tardiness, I'll pick you up and drop you off, you know, 10 years ago. You can relive those 10 years additional things.

15:02 Okay. Nice. Nice. ob. Yep. Let's see, actually a couple of questions from the live stream Magnus asks any news about cute going mobile?

15:12 I actually am ashamed to admit,

15:15 I don't know. I don't know, either. But the I think the bigger more interesting question would could pi cute stuff like could would would you be able to write a Python cute application make it mobile? Right? I think that's where it gets really interesting. Because there's other if you pick another language like c++, or there's other options you might be able to choose. And then maybe you know, this one, you're gonna ask, are there any well known Python apps built with cute?

15:40 Oh, yeah, yeah, they're on the spot and trying to think of what mine it's not well known. But I built timecard. And cute if you look up the open lookup, timecard time tracking app that I that I built, but actually there's, there's there's quite a lot built, built with cute. Anyway, okay, in front of if you're, if you're into kte, the entire the entire kt stack is built on top of cute and there's actually quite a bit of a template. So names are escaping me off the top my head here, but yeah, there's any anything in the kte universes is cute. And so you're either gonna get c++ or Python, Python is certainly a lot faster, right? So

16:18 Oh file Zilla apparently is built, you know when that I know that's written in it, for sure that I it's like one of my favorite apps actually is Robo Mongo or Robo three t it got renamed to I believe it's just c++. It's not Python. Cute. But that one's a really nice one as well. Actually, there's a huge long list, I'll put it in the show notes. Over here, I have a bunch of apps written as well. So

16:43 it's definitely a lot easier to write write something. And I've used a lot of different UI toolkits, and keeps definitely one of the easiest.

16:50 Yeah, the thing that I like about it is it looks like it belongs because so many apps you build with these sort of cross platform things. And it's just like, Whoa, okay, well, that's not how the file dialog supposed to look. You use No, it's alien. But yeah, this looks, this looks like it belongs here.

17:04 Well, I packagings the other half of it, because like, I tried to build something with kivy. And I love kivy from from a development standpoint, it's really cool from a packaging standpoint is like beating yourself to death of a wet trout. So, um, and actually, if you're going to do cross platform, then, actually vtk is horrible, too, because it's really hard to get it to package on Windows, a lot of times, it just works. It just packages everywhere, which

17:28 Yeah, that's great. Nice. Nice. All right. Brian, I think this episode is brought to everyone by us. Wonderful. So we are we're doing a lot of work out there. And as everyone probably knows, if you're into testing, check out Brian's pi test book. If you're looking to take a Python course, we are just about to pass 200 hours of Python courses over at talk Python training. I'm working on a new course how to build web apps, not with API's, but web apps with fast API, super neat stuff. So that's that should be out in a week or two. So anyway,

18:00 I'm just gonna I wanted to bring up that there was kind of a spike in itis book sales in last last quarter of 2020. And open it like testing other teaching. So

18:13 yeah, that'd be super cool.

18:14 Yeah, it's nice to see more and more more more stuff about stuff other than unit test. I mean, unit test has its place. But I when I wrote the check, I've got a book coming out in May. And when I wrote the chapter on testing, and Amanda is like, thank you for not forcing me to edit yet one more unit test chapter.

18:30 Nice. What's your book on?

18:31 Oh, my books called dead simple Python. It just it it introduces the language of Python, the idiomatic practices of Python to people who are coming from another language. So if you don't want to have to sit through yet one more explanation of what a variable or a function is, or classes, you can have this up and it dives straight into the fine details of why idiomatic patterns are what they are in Python.

18:54 That's Yeah, that's a good idea. I the courses or books that say we're gonna pretend you know nothing about the world, and we're gonna force you to go through everything from scratch every time that drives me crazy. What else drives me crazy, Brian is when my Python GC is doing stuff when I know that it doesn't need to do stuff.

19:12 Yeah. Like to not have to think about the garbage collect. And you generally don't write

19:15 like one of the things that genuinely surprises me is the fact that we don't really talk about memory very much in Python.

19:22 It's like, Okay, I

19:23 think it cleans itself up. That's good. Now what? Let's go, let's go about stuff, right. But if you dig into it, it's pretty interesting. There's a lot of stuff around allocation we've covered before, but it's quite unique. But pythons also somewhat unique in the sense that it has like two modes. So it has reference counting, which I would say 98% of all like memory management, cleanup stuff is in the reference counting side. This is totally made up these numbers. But there's a little there's would say maybe even more like 99.5 unless you're building some kind of certain kind of app like with interesting algorithms. Most apps don't create cycles, and the only reason we have Have garbage collection, in addition to the reference counting is to catch those cycles, right? You know, I've got a customer object, I've got it out of a sequel alchemy database, it has a relationship over to the orders, I go to the orders, the orders have a link back to the customer, maybe like traversing that lazy loaded list has created a cycle. And now I need the GC to save me. So the rule for when the garbage collector runs is you can ask it you can say import the GC module and say GC dot get threshold or thresholds, I can't remember if it's singular or plural. on my screen, if I would switch to it singular get threshold, it returns three numbers. They're not the same units, which makes them really hard to understand. The first number is how many allocations of collection objects so classes, dictionaries, lists tuples, things that could contain other stuff. So things that could potentially be participants in a cycle, like numbers and strings are not even considered by the GC. But how many allocations of collection types are there that exceed the reference counting D allocation? So if I had a list and I put 1000 classes class objects in it by allocating and filling it up, then I would hold on to 1000. And none of them would have been come become garbage? So the first number that comes back is, well, how big is that number before we just run a GC no matter what the default is 700. So my example there by create a list of 1000 objects, that's a GC that's gonna run, it doesn't matter if they're cycles, there's no cycle that it does doesn't matter. Like I've made 1000 of them. That's over 700. So we're going to run a GC. And then the rest are like, how much do you run like a whole memory GC versus a local, a small, like recent object GC. And what it occurred to me is, you know, my website, there's a lot of pages that pull back 1000s of items in any website that uses the database, and an ORM that pulls stuff back and hangs on to it, and not just like streams over the items, but put some maybe in a list or something temporarily. Anytime you do that more with that 1000 you're gonna have the GC run, right?

21:51 They're just looking for anything to throw away, basically,

21:53 yeah, but you know, you're still in the process of building the list of them. I gotta get 10,000. Well, guess what? That means, you're gonna have 14 GCS. And you're just in the process of building the list. I'm like, that's kind of weird. That ad seems excessive to me. And then I went and looked at the sitemap on top Python training, where we're pulling back like 1000s of transcripts and all sorts of stuff to generate all the pages on there. 77 there 77, GCS to render the sitemap? There's no cycles. There's not one that's I'm like, That's not good. Let me think about that for a second. So what I ended up doing was I said, Well, what if I made the threshold 10,000? Actually, I ended up on 50,000. So only run the GC, if you get more than 50,000 allocations without deallocation. What was really interesting is doing that made my unit tests which were including many, many integration tests on top by line training, run 10 to 12%, faster, just setting that one line. And it basically does not use more memory, in my case,

22:46 is that crazy? What makes sense is most most issues of performance, just come down to memory and how memory allocation at the allocation date. I spend almost all my time and in c++ more time in samples plus than I do in Python, and we don't have a garbage collector over there. So you have to do all this manually. And yeah, and doing your

23:05 work is right. Yeah,

23:06 exactly. It's like doing it wrong is why why stuff slow. People like well, Python slower than c++ will not. It has the potential, it has the potential to be faster than Python. But it really depends on how you write that code. because well, written code is always going to run faster than poorly written code. It doesn't matter what the two languages

23:24 Yeah, yeah. But I realized that in in my world, in my type of application, I almost never create cycles. But I often get back more than a 700 class objects, which also have dictionaries, potentially, in the MCs editor like allocating the converting serializing into classes, like there's got to be a lot of places where that's happened. So I just set this number to say, you know what, let's waste a little bit of memory. And if there are cycles, we'll come back and get them later. And because there's almost no cycles, there's almost no memory growth, for example. So the server is running like eight worker processes, one of them, and I made this change. And I think over after running for a week without restarting any of the processes, it went from 1.89 gigs of memory usage to 1.91. So like 220 Meg's there was 20 Meg's more memory usage. And yet, like 10% speed up by just changing like one call at startup, it was insane.

23:24 Well, and think about what Dr. Becky's coded, like, you know, go back to the astrophysicist, thing here, you know, with with the sizes of data structures that she's doing, or any data scientist who's listening, you know, they're usually dealing with 10,000 100,000 million items, you know, you combine this with all the stuff that we talked about with NumPy. And with data processing, and, you know, we talked about how long it takes to do some of these data progressions. How much would this be?

23:24 Yeah, exactly. So if, if that data is being done in Python, and it's not just purely being pushed down into the C data science layer, then yeah, that's, that's really interesting, I think,

23:24 although I would, I would caution at the same time that that this is this, there's no such thing as a magic bullet. So you have to understand why this is gonna speed things up a notch. Well, I have to just copy and paste That line that my colleague has that he got from Michael Kennedy, because it'll make the code faster. No, you have to know why.

23:24 It's an easy test some cases that makes sense. People can check it out. I thought it was really it just so surprised me as I walk along with it. Like, wait a minute, that must mean something weird is going on. And then I put it on just on one of my pages. Like, why would I do 77 GCS on a single page load? That's crazy. And so I just started exploring this. And here we are.

23:24 So just did you hear the wherever you're linking to, to talk about how you can test how many garbage collections?

23:24 Ah, let me see. I'm linking to a Twitter thread and way deep down? No, but there's a way to do it. If you if you go to the GC, you can say I think it's set debug stats or something. I'll I'll look it up real quick. While we're talking. I'll throw it in the end here. But yeah, it's, there's a way to do it. Actually, I got it right here. Hold on, give me a sec, the way you do it, as you say, GC dot set, underscore debug, and then you pass an enumeration and the value is GC debug stats. So that thing was just lining up my you know, when I turn that on, it would just light up and just completely fill this the terminal with a debug, or, you know, GC, GC, GC, GC GC over and over and over when I hit the that one page and then changing it, guess what made it better?

23:24 Yeah. Now we should probably be PC about the GC and call the garbage collector, the the programmatic sanitation engineer, but

23:24 it doesn't have it doesn't it doesn't take offense. It's just there to help us out. Brian's probably pretty awesome library, honestly, the GC library,

23:24 probably, but it's built in. Yeah. So I'm, you know, of course, I click on a listicle. But the thing Come on, right, but we don't cover them very much. But I really liked it. So this this article is top 10 Python libraries of 2020. But their criteria was interesting. The criteria was it has to be a library that was launched or popular, it has to be well maintained as have maintenance changes since their launch date. And it has to be just outright cool. That you should check it out. So I'm gonna go through a handful of these they listed 10. I don't know if all of them since I'm, there's like four of them that are machine learning focused that I think cool is relative. But yeah, but the first one, the first one was typer. And I can't I'm like, I'm really a fan of typer. Now, was it really just 2020 and I went back and luckily, it was released like, yeah, in December of 2019. So Sebastian Ramirez is killing it, for sure. And then I looked into I'm like, well, fast API. When that came out. That was the previous December. So the end of 2018, released fast API and then type or a year later, he's just crushing it. Yeah. So yeah, nice. I'm both a huge fan of of those. A big fan of rich also, as a rich actually just showed up this in last year in 2020. enrich is a beautiful, beautiful formatting in the terminal. And yes, it's a beautiful, oh, it's really great.

23:24 That's glorious.

23:24 Um, I'm even using it even in applications where I just need these, the tables. So if you need to print out a table in the command line, the the tables, tables are kind of hard. And there were like weird other. There were other table, specialized table libraries. But this one is great that you can, it works. You don't have to specify the width, it like comes up with the width on its own. And then you if you shrink the terminal to really narrow or wide, it'll word wrap correctly and stuff. And that's kind of incredible. So even if even just for tables is awesome. The third one is do your PI gooey. I think we covered this. Maybe we could.

23:24 I don't remember. I mean, we did go on our gooey rant, so it feels like it should be.

23:24 Yeah, so it's a gooey project. Um, the nice pictures though.

23:24 Yeah,

23:24 I've been drooling over up. I've been drooling over here. I'm going for a while I haven't I haven't had an opportunity to use it yet. But I've been looking at it. So

23:24 yeah, so the last few I want to highlight pretty errors looks neat. I haven't tried that yet. But it's, it's a way to. Yeah, it's

23:24 glorious.

23:24 tracebacks. So,

23:24 I mean, ideally, you don't show airs to people, but if you're going to let's make them at least reasonable.

23:24 And let's train ourselves to you know, it's like, we're gonna have to read the Well, we're gonna spend at least half our life reading error message faces. So let's at least make it readable.

23:24 Yeah, another quarter crying about what we just couldn't figure out.

23:24 And then the last two that I want to highlight is diagrams and scaling. diagrams is a library. Look at that picture. It's a way to do it's intended for like, cloud architecture drawings. But the it's written in in Python, you write these diagrams in Python, in so because they're text you can check them in version control Haskell, which is nice. I'd like to see These sorts of diagrams would be great for not just, you know, network diagrams, other diagrams

23:24 flowcharts would be great. So flowchart?

23:24 Yeah. So the last one is scaling, which is a memory CPU and memory profiler in Python that handles multi threading well, and distinguishes between Python versus front. That's pretty cool. I definitely need to try this out. I also like that you don't have to modify your code using this. Yeah,

23:24 that's really cool. Yeah, absolutely. Yeah, those are cool. There's a bunch of great ideas there. And man, I really need to find a use for rich

23:24 solution, a source of a problem again,

23:24 but I write a lot of like, little terminal apps and stuff. And I'm just like, you know, maybe you'll put a little color in here or something. And just, you know, I just need to take the time and go, No, this is a UI that I should pay more attention to not just some random thing with text. Yeah, they

23:24 will find this cool stuff. It's like, I want to, I want to they feel the need to use this somewhere.

23:24 Well, actually, I had

23:24 a little, little application where it's just like I said, with the tables, and, and I'm like, I don't think it needs colors. I'm just showing a table. But the default for rich is to show colors. And you don't have to pick images. So the like the heading and the lines between were like different colors if you're on a terminal. And if you're not on a COVID terminal, it works anyway, it just figures that out for you. And lovely.

23:24 Yeah, that's awesome. It's awesome. It's very awesome. Awesome.

23:24 Speaking of awesome. So Pep 518 rolled out a while back, it was introducing this thing called pipe project to ml, I guess Tombow or whatever, I'll say that pipe project combo. So the idea behind this was that it was gonna be this configuration file, you know, one configuration file to rule them all. And of course, we're Python, we like things to be simple. Well, ironically, this turned into a really political thing, which I'm still trying to wrap my head around. So basically, the The nice thing about this repository is is is keeping track of all the projects that have adopted pi Project tommo, either, optionally are mandatory for configuration. So instead of having to have you know, a dozen configuration files in your project for all these different tools, you can just use this one. And so it's got this big list, what I find interesting is this part down here at the bottom, if you go down to Yeah, just scroll up just slightly here, just slightly, just a little bit up. That's gonna sound weird on the podcast anyway. If you're gonna, um, so these are projects that are quote, unquote, discussing the use of pipe project humble, but if you actually look at these, it's kind of odd. Um, you know, the big sticking points, because these are the projects that are like stopping people from really just going all in on on pipe project. tamo. And there's even some, you know, talk about circular, you know, dependencies are similar, like, well, I'll do it when they do and they're like, well, I will do it when they do it. Which makes you wonder if it's an excuse. So my pies the weirdest would have been Rossum himself said, well, it doesn't solve anything. You know, someone said, Can we just add this, please just add it. It's easy here. Here's the PR, somebody who did the PR, he's like, Nah, I doesn't solve anything. And he closed a close. It's like, Ah, it does solve something. That's one less fly. I have to deal with that is a solution. Um, flake eight. They have a couple of concrete objections. One is the fact we don't have the standard pommel parser in the Python standard library. So that could be you know, that could be a problem

23:24 as you're adding another dependency to just support having this format.

23:24 Exactly. Yeah, I'm Pip. But then again, it's a common dependency with a bunch of other you know, tools that are already in use, and it almost doesn't matter, Pip. Someone said, I don't understand this PIP to change his behavior. So the mere presence of the file doesn't change functionality. I can't wrap my head around what he's referring to there, maybe. But the stupid thing is someone already did flake nine, which is a which an exact fork of flake eight, that just adds I project pommel, so it's like it's done. They just have to merge it. But it's and actually the same thing happened with bandit, someone actually implemented in 2019. The PR has been sitting there untouched since 2019. So over the years gone by it there and bandit is not picking it up. They're just they're silent read the docs is saying it's too much work. Like it's a lot of work for us to have multiple pi oxidiser. Shockingly, hasn't even said anything since 2019. They're like the they're like the new trendy like the trendsetting packaging thing. And they haven't been saying anything about this, this. So I I'm trying to figure out why it is that this is so controversial, because it seems so obvious. You have one file to store all of the settings for all the different tools. And yet everybody seems to want to do their own thing with

23:24 Well, I know that, you know, hip M and poetry and flit and some of these other tools that suggest a workflow. I feel like I hear this file format being used along with those and you know, telling people, we're going to have a different way. For you to like work with your projects and manage dependencies and stuff, you know that i think that's part of the source of of this. And I don't know, if it's just necessarily all mixed together. And Brian, what do you think you know more about this idea?

23:24 Um, I think a lot of projects are on the side of, like, for instance, um, coverage was, was it I don't know, where they are on the list that they adopted? Did they? Okay, yeah, well, coverage had this thing in and other tools were talking about, you know, there's no tomo parser. And they, they didn't have any other dependencies. So they didn't want to add a third party dependency, just for this in if they're just using it for packaging, however, or settings or something. But the so I do think we will see a lot. I don't think it's a reasonable argument, because there's, there's reasons why, you know, the same reason why request is because there's making changes, but I do think that the like the format of tumble basic format enough to get a pipe project isn't going to change much. So they think enough of a project, tomo parser to handle pipe project. That's they think we need one of the something like that in the in built into Python.

23:24 Yeah. We have we have Pep 518. So like we have some we have some standard already.

23:24 Yeah. So I think we'll see a big, I would like to see at least even if it isn't the mainstream one if we have if the if most projects that are okay with a third party use something else, for a total parser. But there's some built in stripped down version in the standard library. I think that that's I think that's, yeah, I

23:24 I see you can solve that problem by just vendoring. It just here's the two files that make up the parser. We're just going to make it part of this package. So now we're good to go. I don't know. Sounds good. Well, I think that's it for all of our items. Brian, you got anything else you want to share with folks?

23:24 Yeah, I'm it's my birthday.

23:24 Happy birthday, man. I'm feeling good for what I was gonna say you didn't get for 28. Brother.

23:24 So I'm 51. And I heard today that that's just one. I'm just shy of a full deck.

23:24 I've never been accused of playing with a full deck myself. But I will say Don't let anyone tell you that you're old because it says in the first chapter or chapter of Genesis Tao. And then God said man's your shall be limited to 120 out of 120 is 62. What is biblical that 60 is middle aged. So you're not even middle aged. You've got a way to go. The Bible.

23:24 I keep telling everybody that I don't look a day over 73.

23:24 Good man, a couple happy birthdays. And also you're getting to ask if you're still a fan of flit.

23:24 Yeah, love it, especially since they adopted the source source directory.

23:24 Yeah, that's right. That's awesome.

23:24 That saved my life.

23:24 Jason, anything extra that you want to throw out there? I mean, maybe people have a place they could get notified about your upcoming book or something like that.

23:24 Yeah. You know, follow me on Twitter is probably the the best way to do that, um, code mass nine to one Twitter. And then actually follow no starch, press two. I mean, no starch press is awesome to begin with.

23:24 That's where you're doing the book. Yeah, exactly.

23:24 You're my publisher. No starch. I don't think they ever put out a bad book. I love that publisher. So I was I can you can actually, you can ask my mother. When I got when I got when my book contract got accepted. I actually screamed. Very high pitch.

23:24 That's awesome. Oh,

23:24 yeah. Follow up. Follow those starts press for updates on on that. And although Ross, they've got some other incredible books coming up next year.

23:24 So I'll go ahead and ask her. So what's your mom's Twitter in?

23:24 My my mom's Twitter handle? Oh, I'm sure the Twitter handle actually puts you in touch directly. I think that's

23:24 awesome. Well, cool. Thanks for being here again. So I have a couple of items to throw out here. Actually, this almost Brian's almost could have been an extra extra extra extra extra extra hear all about it. But the real short, so I didn't do that. Django 315 is released Django three. Didn't we just just go to Django two or something? That's I mean, that's good. That's really good to hear. So awesome. That Python 310 Alpha four is available for testing.

23:24 Now the new parser scheme new parser is going to be in that one. Oh, that's the peg parser

23:24 that good has been working on

23:24 Yeah, that's gonna be that's gonna, that's gonna revolutionize the language eventually.

23:24 Yeah, it'll definitely make it possible to do more. And then releases sy pi 160 was released. I learned about a cool project. So we talked about, like avoiding Excel for the Python data science stack, right? Like just stop doing Excel. There's all these weird errors, like the organization that defines or governs how you can name genes has come up with rules for names you can't use and the reason they can't be used is they'll be parsed incorrectly into other data types by Excel, for example. So there's a lot of issues you might run into with Excel. And, and that's all good, but there's this project called pi x L L. This is actually a paid product, they're not sponsoring the show, I just think it's kind of neat. So spreading the word. But anyway, if it's interesting for you, what you can do is it's a plugin for Excel that will embed Jupiter into Excel and allow you to write functions and macros in excel in Python. So basically at almost adds the programming Python the programming language to excel, which is good. Yeah, it's better than VBA. Ah, let's see, no, I

23:24 started in VBA. Tell me about

23:24 VBA. So someone on Twitter asked if pi charm works, okay, on my Apple Mac Mini m one, and they pi charm and JetBrains in general just released a whole bunch of their tooling with different installs for the apple silicon native versions. And so I've got a cool little video that I'm going to link to in the show notes. And it's just like a five second video here, I opened up pi charm. And you basically from the time you click on Open project till the project's open, if you've opened a project before, so that that caveat, but at that point, if you click on it, you cannot perceive click, like by the time you're letting up the mouse, the whole the project is loaded and ready to work on. It's like, it's insane.

23:24 I will I will consider picking up Python again, when they add live share. And they have they're working

23:24 on it. There's something called code with me. Yeah, yeah. So I have not tried it. I have no one to code with. I'm sorry. But

23:24 email me later will set for me

23:24 Yeah, exactly what it will look good together. So also, since I got my M one, like three, four weeks ago, whenever I've only used used this for all my Python work, and apparently, it's, it's still going strong. I even had to send in my MacBook Pro, because it had starting started shutting the battery was so bad, it would shut down at 75%. Like, you know, when it gets too low, it'll shut down. And as the battery gets bad, or maybe it shuts down at 10% instead of zero. If I'm doing video work, it'll actually shut down at 75% till I plug it back in. So so it's all in one until that comes back. No, I'm

23:24 I'm still on my system. 76 Linux, I can't speak out boy.

23:24 That that's cool. I just I think this whole, like new ARM architecture stuff that they're doing. It's gonna be interesting, you know, I think Microsoft following suit or trying in parallel with them. It just felt to me like Intel and AMD that's just the way it was gonna be forever. And it's not necessarily the case.

23:24 I I don't I don't have a problem with I don't have a problem with competition when I have a problem with the software companies making their own, you know, architecture that only works on their architecture that you move towards, and then you wind up with a totally fragmented industry. I think that's

23:24 Yeah, that's not gonna be great. Oh,

23:24 don't do it. Microsoft. It's not worth it.

23:24 Awesome. All right. Well, that's my extra extra extra extra extra extra Brian.

23:24 Nice. I want to get an M one. Like to get a mini

23:24 Yeah, the Mini is fantastic. I really, really like it. You don't even funny. It's not even. It's not even a joke. I'm being serious. But we do need a joke.

23:24 Yes. So I have a joke.

23:24 All right. Yeah. You got the joke this week.

23:24 I actually do have the joke this week. Yeah. And it was a while Why did the programmer always refused to check his code in the repository? Why he was afraid to commit

23:24 so yeah,

23:24 yeah. If you want to if you want to regular bills of my of my that is one of my original. So if you want to get a dose of my absolutely horrific puns, you can follow me on twitter at your own peril. I post every every Monday. I've got a new one. So awesome.

23:24 Nice. Yeah. Thanks for being on the show.

23:24 Yeah, it was fun. Yeah. Thanks. Thanks, everyone out there on the live stream man. Thanks, everyone who listened

Back to show page