Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book

« Return to show page

Transcript for Episode #311:
Catching Memory Leaks with ... pytest?

Recorded on Tuesday, Nov 22, 2022.

00:00 Hello, and welcome to python bytes where we deliver Python news and headlines directly to your earbuds. This is Episode 311, recorded November 22 2022. And I am Brian Aachen.

00:11 I'm Michael Kennedy. And I'm really Luke kuya. So welcome MaryLu.

00:15 So tell us a little bit about yourself before we jump into the topics.

00:18 TLDR is a machine learning engineer at a data inagh consultancy company called data nerds. I'm from Brazil, but actually live in Belgium. And I guess that's that's, that's it. Thanks for having me.

00:30 Thanks for showing up.

00:31 It's great to have you.

00:32 Well, Michael wants to kick us off with the first topic.

00:35 All right, let's kick it off. I've got I've got some fun stuff. I'll see what Marilla thinks about this. This is it's a little bit mathy. What I got going on here. That is not the right screen. How about that screen. So this comes in from a one of the big Friends of the show Brian skin, he sent me a tweet and she says what app Python bytes. And it's a, it's a quote, tweet from somebody. Holy Hell, a Taxify is the sexiest thing I've ever seen. And look at it. So when I studied a ton of math, and the symbols of mathematics are really important. And they communicate stuff, like really, really quickly, you can scan over and see the symbol for the real numbers. Or you can see the symbol for subsets or, you know, infinite, so many like, I know what that means. When you translate that into Python or into computer code, it usually becomes something kind of gnarly looking, right? So the example here on this tweet has a function called solve and it's solving the quadratic equation. I guess just for one variation of the root, not the plus minus, but that's fine. It is just as like negative b plus math dot square b star star to it's like symbol soup, right? Yeah. So this les Taxify thing, let Taxify law tech is the language of expressing those symbols, the way mathematicians would have written them in, you know, the 16th century or whatever, like the the fancy flowing, sort of, you know, some symbols and integral symbols and whatnot. And so what this does is you just put a decorator onto that Python function, you say latekka, phi dot with Latech. Allah tech, when you show that function, and a notebook, it shows the math, formal mathematics of it. Wow. Like, like, there's one that was doing like the quadratic equation, another one that says, if x is zero, return one else return Math, sine of x divided by x, and then the symbols is like this sort of like branching equation, you know, like, what you would write that in the Lytec? Conceptually, what do you think, wow, that's insane. This is

02:41 great.

02:42 But it just changes the record of the function, I guess, right? Like, if you call the function, it's all fine.

02:47 Yeah, exactly. It doesn't change the function at all, it changes the wrapper or the stir. So if you do this outside of a notebook, what it prints out, let me see if I can somehow communicate this back. So if you print it out what it it returns, I have it here. No, yes. There. No, that's not it. I don't have it. What it prints out is the Lytec escape codes. So it'll say like, backslash FRAC, have, you know, like, it's a weird, I don't know how to write Lytec. I did a little bit when I was studying math. And then I'd said, that's something I never need to remember. And it's, you know, shut it out of my brain ever again. Yeah, like that. Why do I need to know this? I don't need to know this. But yeah, so the repper is, is just the the low tech escape codes, and then the notebooks see that, and then they render it as, as low tech.

03:36 That's pretty cool. And then but one of the nice things about this, then as you can, you might have like the math that you're trying to convert to code, and then you can like check your answer. You can just see, did I get it right? In code?

03:50 So yeah, it's pretty cool. That's really nice. Yeah, you could you round trip it, right. Yeah, I'm

03:54 assuming people are doing this on their own code. So they're, you know, I guess

03:59 the question about the inverse. Yeah. Right. It's like a if I have the math symbols, could I turn this into a Python function? I mean, I don't see why I can't go both ways.

04:08 Sure. But I still think it would be easier to write the Python function than the LaTeX code.

04:14 Yeah, that's, that's true. I think it's a pretty nice use case. Well, you know, sure.

04:18 Sure. I'm sure someone's gonna find it. Use a cool use case for it too, right. Yeah.

04:23 This is pretty interesting. We've got a couple of live comments, though. Madison. Hey, Madison, on the audience. Madison has been on the show before I'm blown away by how libraries like this are able to make math approachable. I wonder how this can be used with auto generated documentation. Very cool. I agree. And Henry also says, I'm guessing it's working on the bytecode like number, but compiling it into a human language. Yeah, compile it into the law tech escape codes, which is not a simply which is the opposite of human readable, but it is text right. And, you know, related to it's just,

04:57 Oh, yeah. Henry senpai.

04:59 Okay. It's using inspect, get source and parsing the STL. Perfect. Another thing that's amazing that people check out like the sim pie stuff, it does some really, really interesting things like if you go say to like calculus, you take a limit here, it'll, it'll do similar outputs as well, right? So you could put in this, this, and it'll actually express it as symbolic math and it won't lose precision because it solves it symbolically, you can say like, you know, back to this equation. And so that's kind of related. But but this just says, given any arbitrary Python function, not written in the symbolic form, just turn it into law tech, which is pretty amazing. So anyway, thank you, Brian skin for pointing that out. That is pretty neat. One final comment, I could not get it to install on my apple silicone Mac, maybe that detail matters, but I couldn't get it to pip install I out of PPI had to pip install the get plus the GitHub URL, and then it would install. I don't know why. But if people want to play that, that might be necessary. Okay. Already, Brian?

06:02 All right. Well, a little while we're talking about math. I'm often working in the measurement world and where we care about like prefixes a lot, then you know, a lot of people do with big numbers or small numbers. And this was actually suggested by to us by Abraham. And I think he's the, I think he's either works on this or it's his project. It's a project called prefixed. And what this does is, it's just, it's a class and provides a class called float, capital F, that is a derives from the built in float. And it supports scientific, decimal, or scientific and yeah, IEC, which I'm not familiar with prefixes. So things like scientifically like k and s, and things like that feel to go look at all the metric prefixes you got, like, there's some new ones, but in K, mega, giga, things like that, and it just sort of adds these on to when you print them. So it acts just like a normal float, most of the time, you can, you know, use it in math equations and everything. But the interesting thing is, is if it is using used in math, math equation, the result will be a one of these prefix float types. But then the nice thing about it is when you convert it to a string, it includes the little little prefix prefix thing, or the suffix or whatever, the little micro or k or M or something like that. So I think this is actually super helpful. I'm going to use this right away, because I, you know, I use a lot of like big, big and small numbers and reporting out just the huge thing, or just the float is sometimes horrible to compare with. So this is, this is pretty cool.

07:50 It's very clever. I love how how simple the idea is. So you can just f string one of these floats and say, colon point to h, and that'll convert it to its, and the age tells it to be either, you know, kill, or micro or mega, or, you know, whatever suffix is needed at school.

08:07 And then there's a debate example where they they said, Well, I'm gonna, I'm gonna use the capital B for bytes, but that's after the the formatting of the number. And then the, the K comes in from the float things. So that's, that's pretty cool. One of the things that he passed along is there's some new prefixes. So this is, this is apparently new, new scientific prefixes over the last for new first ones, new new ones for the last 30 years, apparently. So we have 10 to the 21st, which is Zetta and 10 to 24th, which is Yatta. And then negative is upto. And Yocto. So these are fun.

08:49 Maybe why, why now why they decided to need to have more money now and you need to come up with new prefixes or,

08:58 exactly, I'm not sure why we need new prefixes, but

09:02 our microscopes can now see smaller things. We don't have words for this, that things are the small like,

09:08 but national debt maybe.

09:14 Yeah, very possible.

09:15 But the but also our rim notes that prefix does handle these new ones. So cool. Good

09:21 job. Cool. What the environment so you can put the underscore right to like, if you put underscore in the 1000s that also that's something that makes it easier, I think to read the numbers too. That's what

09:31 I was, like, a digit grouping. Yeah, yeah. Do you do that a lot?

09:35 Not a lot. But like some Whenever I can I do. I think it makes it easier to to distinguish how big the number is, I guess

09:42 I always forget to I just I know it's there, but I never use it.

09:47 I think usually like when I'm counting the zeros with my finger on the screen like that. Maybe just put a underscore there. It makes everyone's life easier.

09:54 Yeah, yeah, I've really started doing that a lot last couple years, but before then I did cool.

09:59 What is next row.

10:01 What do you got for us? That's me on the screen. Yeah. Deputy Have you ever she got her? You got to accept some cookies. Hold on. Oh. I'm just teasing. Things drive me crazy, man. I don't know. Yeah,

10:17 I think it's like, it's crazy how like, now they're just popping up everywhere. And then you see, like the data gathering all the time, and this and this. And he's like, Yeah, but maybe DBT. Have you ever heard of DBT? Is this something? Because in the data world in my field, it's super popular. But I don't know if it's a bubble as well.

10:33 I've never heard of it. Michael never heard. Yeah,

10:35 I think I think I've heard of it. But I couldn't tell you what it does. So basically, in the same spot, hey, I tell us,

10:42 no, it's a really cool tool. It's open sources, while they have their cloud option is right. So you can pay any hosted maybe a disclaimer as well that I never, I always see it. And I always want to use it. But I haven't found the use case. So I don't have first hand experience here. But basically, the way I would describe is that the ad best practices around SQL projects. So while my wife had mentioned this on Python bytes, it's built with Python, yay. And the other thing too, is that they actually mix ginger with SQL stuff, right? So you can actually do for loops, you can do stuff like this, you don't have to repeat every time and just change the variable. You they also have these like reference macros and stuff. So you can actually say, Okay, this comes from that table that is on that file, and this comes from this. So you can actually chain a lot of these dependencies, right? Like, there's a lot of projects that you have this ETL stuff, right, so you just have to basically transform in each step. And with DBT, they actually keep track of what depends on what and you can say, oh, I want the freshest data here, and you execute everything that needs to be executed there. Yeah, so it's super cool. They actually support a lot of like data platforms here, right? So you see, like BigQuery, data, bricks, snowflake, all these things as well. Another thing that they also more things they do they even have some data validation stuff, which in my field, it's a big thing to you know, like, maybe have an ID column that needs to be unique cannot be No, and you want to make sure that that always happens. And if it doesn't happen, you want to be flagged, right. So that's super cool. What else I also have some built in documentation. So if once you have the dependencies, you can say, oh, show me the DAG, you know, show me where the data comes from, and what depends on what so that's also super cool. And recently, actually, they actually started supporting so like an sql file kind of corresponds to a model right? Cookies again. And so they have SQL models. So that's the one but they also started supporting Python models, right? So this is very tied to data. So now you can actually mix and match right? You can say this step, this transformation is in SQL, but this one is actually in Python, right? So the way they don't run anything on the machine, they actually send it to the cloud. So snowflake has no Park, which is Python on its own snowflake. BigQuery has spark and data bricks as well. Right? So basically, you can mix and match this transformation, see additional information is there. But everything is like in a nice put in one place. And because it's on Git as well, we can have the ICD. I think also you mentioned I think it was you, Brian, I mentioned that SQL fluff and SQL fluff actually came from a DBT project as well. So it's all in Python. So super cool. Wow, that's really

13:08 neat. So what does the Python models look like? Are they straight Python classes? Are they Pydantic? Are I have

13:15 I? Maybe I'm a bit lazy because I just watched a video. And they were they were showing here. How it works. It is also doing a comparison, right? Maybe this is not this doesn't work? Does it? Does works? Yeah. Okay. This is what the quality is horrible. In a nutshell. Yeah, find a function. Yeah, you define a function that has a DBT in a session, and then you create a reference or reference, basically a table, right? And you can say, and then from that point on, you can see two bundles. And then you can just basically use the Plus API to transform that, right. So there's two some caveats, right? Because bundle is not super performance, depending on how much data you have and whatnot. So sometimes you probably still want to stick to the SQL stuff. But then there's a lot of possibilities there too, right? So even stuff like deploying machine learning models on the SQL infrastructure and everything. So

14:01 yeah, so it's kind of the same old, same old story, you know, even fewer kind of an ORM. Sometimes you don't want to bring all that data back to make some minor change. And then you would just do a sort of an update statement, instead of pull back 10,000 models change something in called Save 10,000 times, right? Like, it's probably that kind of trade off. But it's really cool that you can bring it back into Python this way. What are you using it for your work? Or like, what are you interested in using it for?

14:26 Well, I think we have a lot of these, like ETL pipeline stuff, right? A lot of these. We have some data here and we want to like basically clean it up and make sure it's all uniform and put in a dashboard, calculate some KPIs and whatnot, right? And so business people can see, are we doing better or making more money or not kinda. And like, a lot of the times, it's just SQL, right? It's also more accessible for a lot of people. So we stick to SQL. But there are also limitations, right? But before what I've seen is people just kind of go in the UI and just execute stuff ad hoc, right? So no, versioning nothing and I think this kind of puts everything in one place. You can even add safety. Because the CLI tool when everything just kind of makes sure that everything goes to that versioned method, let's say, I mean, again, yeah, if you need something more more fancy, right, and you can throw some Python stuff in there, but usually we try to avoid it. To be honest,

15:12 I can imagine. Let's see here, hold on the models, though, the way you express the code, it's like it's really nice looking for SQL, which is surprising, right? This this code you write, like, with customers as select these fields from this, this table. Yeah.

15:30 They also have like different macros. And like people can write different macros. So like the describe function in pandas, someone can just have written that and you can import that and like, it's really nice to share, like all these things as well. So super cool. Really, really eager to give it a try. To be honest, I've been just like trice crushed.

15:46 The next project that we get to use. All right, Brian, anything you want to add before we jump over to talk about a sponsor real quick?

15:56 Yeah, let's know. Let's talk about our sponsor. All right. So today's episode of Python bytes is brought to you by Microsoft for startups, founders of the Microsoft Microsoft for startups set out to understand what startups need to be successful and created a digital platform to help you overcome those challenges. And they came up with Microsoft for startups founders have the founders have provides all founders at any stage with free resources to help solve startup challenges. The platform provides access to expert guidance, skilled resources, mentorship and networking connections, technology benefits, and so much more founders hub is truly open to all you don't need to be investor backed, but you can be speed of development with free access to GitHub and the Microsoft Cloud. You can unlock credits over time. And there's also discounts and benefits from Innovative Companies partnering with founders hubs such as open AI, you'll have access to mentorship, their mentorship network, which includes hundreds of mentors across a range of disciplines need advice on marketing, fundraising, idea, validation. There's tons of topics including management and coaching, you'll be able to book one on one meetings with the with the mentors, many of whom many of whom are former founders themselves. It's no longer about who you know, get critical support you need from Microsoft for startup founders hub and make your ideas a reality today, join the program by visiting Python hub 2022. That link is also in your show notes. And

17:20 thanks, Microsoft for keeping us going strong. All right. What have I got next? This one is a chain of really cool things. So Roman brights have been a fame and other things tweeted about this project that Pablo Galindo Salgado has been working on. So Pablo was the release manager for Python 311. It was part of the live stream of releasing that was all fun. But he also I believe, works at Bloomberg, where they work on memory. I think we spoke about memory quite a while back Brian's memory profiling tool, maybe Marlowe, do you use profilers and that kind of stuff in your world?

17:58 No, I haven't used much haven't hadn't had the need to be honest now. yet. I feel like so far, there's no, try to keep it simple.

18:06 So a lot of times profilers are about performance, like how faster this code run, and if it's slower, should I look at this loop? Or that loop? Or, you know, where do you spend your time making it faster? Because it's really surprising when you look at code, like this part looks complicated. So that must be the slow part, like, no, that doesn't matter. Nothing you do to that will make any difference, you got to look over here, right? That kind of stuff. But memory, as the name would suggest, is more about memory profiling. And like talking about, you know, how many of these different things if you allocated, those kinds of things? What is coming up first, let me let me pull up a pipe s plugin, which is super cool. So with the PI test plugin, you can do two things. Now you can say pi test dash dash memory tests, and it'll tell you things like, you can actually set limits on how much memory can be allocated for a certain operation. And if it exceeds that, it'll say, oh, my gosh, there's something wrong, this thing is like way over it using the memory we expected. So that's an error. But it also gives you like, a cool, emoji filled summary, I guess, like total memory allocated, the number of allocations, a histogram of allocation sizes, so like Python memory has, like size classes, when we've talked about it's block arena, one other term, that forgetting, like that it uses to organize these data structures, and then you can actually get it overall, then for individual tests. And so it'll tell you like the different things that were allocated. And anyway, it's, it's pretty, pretty insane. Okay, so you can get that report. And then you can also the other one that he gets, was it and there's a place where you put a decorator and you just say on this test, if it exceeds this amount of allocation that should fail the unit test. It's just a PI test stop mark dot o coulomb rate limit or something like I don't think it's a limit or memory limit. I can't remember exactly what it's what it's called. You can say, if this test is seats one memory of one megabyte of memory allocation, then that's a failed test, which is pretty cool. Right? Right. That's

20:06 really great. So they gotta Yeah, they have a limit memory decorator and a check leaks decorator.

20:12 That's the one. So the check leaks is the new thing. And so what you can do now is you can say pi test dot marks dot check leaks as a decorator on your test. If there's a memory leak in the code that runs during that, it will let you know, I don't know if anyone else has tried to track down memory leaks, I would rather check track down like a multi threaded race condition than a memory leak. I don't want anything to do with memory leaks. This is no fun. And so if I can do with a decorator, let's do it

20:38 well, and also decorating your tests. So you're not having to modify your code at all to do this, that mean the code under tests, you're modifying your test code, if, if at all, or it looks like it gives you some benefits, even with no modification is pretty cool. Yeah,

20:52 maybe Pardon my ignorance here. But when would I worry about memory leaks in Python? I

20:57 think. So imagine you're writing. Imagine writing pandas, right. And you're, you've written a bunch of C code that's getting imported. And you know that, you know, there's a memory leak in there somewhere. And it's just like, Okay, well, I don't really know how to, but then

21:14 it's more likely that the C, the C part is the best that you can

21:18 also have memory leaks in the sense that you expected there to be no more things allocated after the function was called. But you could have assigned it to a global variable, or you could have you know, stored it held on to a reference in some way that you weren't expecting. So it's not a leak in the super traditional sense. But it could build up if you're doing something wrong in Python, but certainly outside of that. So I think this is pretty cool.

21:42 Really any long running service is going to have, you're going to be concerned about it. There's a lot of Python applications that are short running, and it just cleans up after itself when it's done. So there's there's a case as long running services, also things like that, maybe you care about things that are using large amounts of data and, and need all of the data that they can get ahold of without wasting any or that's, that's important as well. Makes sense.

22:10 I'm also wondering, you're right at the limit. Yeah, no, sorry. Go ahead. Yeah, if you're right at the limit of like, I'm using 15 and a half gigs. And I don't have more than that. So I need that. Or like I just checked, the top Python training site has been running for seven days in one hour. Like, if you've had a memory leak, you know, it's even if it's, you know, 100 kilobyte here and there, like it could turn out to be like a big hassle. So

22:31 I'm wondering if you could use this for edge device stuff, if you want to limit the memory, because we know the edge device can have that much. So

22:39 that's actually a really good point. Because if you're on one of these, like, circuit, Python, little boards, they've got like 256k of RAM. And that's, that's very different than 16 gigs, isn't it?

22:51 Yeah. Right. Yeah. So you could test your application on a larger computer and limit it limit how much memory you give it?

22:59 Right? Yeah, I think you would want to do that with the limit then, rather than the check leaks, but still,

23:03 it's the same. Yeah, cool. Yeah.

23:06 Awesome. All right. Let's see a couple of comments from the audience. Gareth out there. Haygarth says, I ended up writing Docker containers that swapped out every couple hours to solve it. I mean, that's actually what a lot of people do. Like, you know, what, if it runs more than 12 hours is the problem. So we just tell it to recycle itself. And then Madison says, it's so cool. I need memory profiling all the time with some of the data. I do work with regular so people, people are taking it cool. Yeah. Very cool. So thank you, Roman. I know you did that to us on purpose, but you shared it with us anyway. Thanks. Nice. Already, right.

23:35 Okay. Before we get on to the next topic, I want to point out that Henry Shriner. I'm going to paraphrase him by saying, Brian, you dork, you didn't even read the article. Yes, you're right. Anyway, sorry. So the new prefixes I was showing the previous new ones in 91, when they added Yocto and Zepto. These are not the new ones. The new ones are down here with Ron queda. ronto. And Coleco. Yes, yeah, the reason why those sounded familiar is because they've been around these new ones. They're the new ones. Okay, so thanks, Henry, for clarifying that. But on to the next topic. I is will McKagan says Please steal my source code. So he wrote an article Wilma coons wrote an article said stealing open source code from textual and he says, I would like to talk about a serious issue with free and open source software, steal the stealing code, you wouldn't steal a car, would you? And then actually, he has this funny video that embeds about like how, like digital piracy really is like stealing and it's a it's sort of a funny video. But the comment is, is real that like you need to you can steal code from open source projects, as long as you can. So please read the the MIT license or read the license to make sure that you can, and in a lot of cases you can. So like, I'm gonna give an example that I use a lot is, I'll think of something that I want to do like I'm interacting with a library. And I'm in maybe I don't quite get how to do that with the documentation. I could search GitHub for projects that use that library also, as an example. And so that's a way to look at other source code of how to how to interact with a project that maybe doesn't have the greatest documentation, you can see how it's done.

25:28 And I've honestly never thought to do that. That's a great idea. And I just got, I'll go look at tests and stuff. I'm like, these tests suck. There's not a single one that shows me like this use case that I'm looking for. This is brilliant.

25:41 Some of the Yeah, I use do that a lot with PI test plugins, because I look, I look at how other plugins are testing their stuff. And I'm like, Oh, how do they do that? So the but so the warning there is he's not advocating for piracy. Open source code gives you explicit permission to use it. And if you're, if you're actually just copying the whole thing in, you probably should reference it and use the same license, or if you're copying large chunks, but the MIT license, for example, says says it's a substantial copying, so a little bit of copying is fine. And, and willsez. Textual has some cool stuff in it that you might want to look at. So he's got a loop, he points out some things you might want to steal the loop first and last. So he's got a loop, iterator that he's got a couple of versions of it that will not only iterate through things, but it'll, it'll give you it'll note which ones the first and the last. So if you need to do something cool on something different on the first and the last one do that. He tweeted recently, or two did or whatever about the LRU cache as well. So the pythons got a built in LRU cache, but everything's global. So you can only click Clear, there's limits on how you can interact with it. So he has a more flexible LRU cache. He's got a color class, that looks pretty cool that you can convert to different color representations. That's pretty neat. And then, you know, he's been working on a ton of geometry stuff, 2d geometry. So he's like, you might want to use this for whatever 2d geometry you're using. So here's there. So kind of cool. A reminder, that open source, one of the benefits of open source is you can see the source and learn from people.

27:18 So I like it. I love your idea.

27:21 You've never done that. I'm like, it might dance. I just can't figure this out. Oh, how are other people using it? So

27:29 I just get frustrated and go on to a new library. This one sucks. I can't do this. I'm gonna find it. It's not good enough. Well, are you an open source thief? Do you do this kind of stuff?

27:37 I have to admit, yes. Yes, I am. StackOverflow. Thief. Open source is, especially in the early early days. Right. But I think Rich stuff too. It's very inviting for you to steal code, because even the on the rich package, right? Like if you do Python, that shame rich table or whatever, you always show some really nice stuff on the table on the on the terminal, right? And I was like, how did you honestly do that? And are like, I think for every component, he had a little demo that you can just run and it's very, very tempting. Even if you didn't want people to steal stuff from him. You have a hard time just keeping the thieves away, you know?

28:11 Yeah. Yeah. Very cool. And funny, too. I like it. Good job. Good job. Well, where are we on now? All right. Off off to Marlowe's final item. Yes, this one I've not heard of either. And it looks pretty interesting. Yeah.

28:25 I mean, it's, uh, I think kind of one of the things that I saw, I was like, Yeah, this makes so much sense. Why, how come I never I didn't think of this before. But this is shed, I'm a man. This is a podcast, right? So. It basically I think it's related to like bike shedding shedule, your legacy code, right? So it's like a superset of black, right? They call it black plus plus here. So they say you're a maximally opinionated auto formatting tool, right. So it's all about configuration over convention over configuration, which is also something that I can subscribe to. They have no configuration options. But basically, it's a bundling of a lot of tools, right? So they have black here, but they also have eisert. And with the profile black, so it doesn't clash. They also have pi upgrade, which I think you guys mentioned a couple of times, right? Yeah, we didn't like as well. Yeah. Out of like, I didn't know actually before. But basically, it's a it removes unused imports and unused variables from your Python code. So it's kind of like yeah, that's all I wanted. I was like, you

29:25 wouldn't have this last week.

29:28 There you go. Yeah. But yeah, it's the ones the ones that have shot and even do like blacking docs, right. So if you have docstrings, or markdown or everything, you will take that and black format that for you. So it was like yeah, this is this is what I wanted.

29:40 Okay, hold on black and Docs. And this is new to me too. All right. Yeah. Let's see. So this is on Python blocks, a sample code blocks. Yes.

29:50 So if you have restructured text, markdown, even doc strings, it will it will format that for you.

29:56 Like you like black and a readme for instance. Yes. Some Yes, yes. Oh, okay. Is this good? Yeah. So

30:04 I have some stuff to talk about at the very end just a little bit about blogging and writing and some platforms and stuff. And that's all in Markdown. Like, I could run this against all of my code samples on my blog to basically auto format. All code in the blog. I'll be cool. Yeah,

30:21 yes. Exciting. The next time I read a book, I'm totally gonna use that.

30:25 Yeah. Are you doing a book? Yeah, I mean, absolutely. Um, so

30:28 I literally just like yesterday, the day before I was cleaning up some code, I finally got, you know, I kind of I don't do it clean the whole time I get it to work. And then I like, you know, then I look at what I did stupid. And there's there might be some imports laying around that I thought I needed, because you add an import, and then you take that code out, you sometimes forget to take the import out. So I ran black on everything, of course. And then and then I ran flake eight, and I'm getting errors. I'm like, shoot, why didn't black just take those out? So now I've got shed and like goes out?

31:00 It doesn't all right. Like it's great now, because maybe it's the same, right? Like you're a flake eight is like, ah, yeah. And use variable. Okay, they have to go there one by one. You know, it's like, it feels like they should be a nicer way. Right?

31:10 Yeah. I mean, it's, you have to pay attention to that. Because your unused variable might be a typo. or something, you might think you're using

31:19 this trigger this Yeah.

31:20 Or it's like a global variable module is supposed to share with something else. And it's a library. But in general, I mean, you could probably put like a hash, no QA or something on it? Well, I mean,

31:29 yeah. And also your, your, your testing, so your test will catch it if you delete too much. So

31:37 let's say I have a really, really good one, take your code out to the shed and whip it into shape behind the shed. All right, I will, Brian, what else we got extras?

31:48 You got some extras? Who should go first?

31:50 You go first? Okay, well,

31:52 the thing that I've been working on is, is by test check, and I finally got, and I've been talking about this for like a month, because I've been slowly pulling this into shape. And so almost a complete, not really a rewrite, but I moved everything around. And the code is a lot easier to read. And so it makes me happy. And I also changed the API. So I wanted to mention to everybody that you can either use, so you can either use from pi test check import check, to get this check object, or you can stick the check object as a fixture in either way, you get access to everything in the library, that's the only thing you have to do. And that for people unfamiliar by text check is a is a library that allows you to have multiple failures per test. You know, normally, the recommendation is try to fail on one thing. But sometimes you need lots of data. And I just threw in a little example that uses both. So for like, it's using HTTP X to grab, grab the status code. And as long as the status goes 200, then I can check a whole bunch of stuff, I can check to make sure the redirect and encoding is right and check for some, some stuff inside the I mean, these could be multiple tests. But if it really is you're checking multiple parts of things. And for scientific work that I'm in measurement work that I do, I'm often checking like, tons of aspects of a waveform. And it's really just making sure the waveforms right and that rightness is multiple checks. So use that. Anyway, I didn't intend to break anybody. But I did break Brian skin. So Brian came up at the beginning of the article. But he tagged me in a GitHub issue on his project. And I looked at it and I'm like, Oh, I didn't intend to break that. So I fixed it this morning. So hopefully, if anybody gets broken by this, I was not intending to break anybody. Just let me know. And I'll try to fit. So looks great. How about you?

33:41 Marilla? I know you have some as well. It goes well. Sigh?

33:44 Yes. Maybe? Yeah, I feel like I should have opened the I didn't have the link up here. But talking about breaking stuff. Flake eight is not on GitHub anymore. And I actually didn't have issues with that. Because with pre commit, right, you have to specify the repo. I already was on GitHub, but I actually heard from people some people that they heard a lot of noise that flake eight is not on GitLab anymore. And then there was also this video from Anthony that is maintaining, right pre commit and flake eight, he was explaining a bit because why was the motivation from going from GitLab to GitHub, and yeah, like to what's relatable said like, sometimes you break people's code, but it's like he's not retention, right. But sometimes people can get very heated over these things. So yeah, just maybe public service announcement, you know, change your your Git repo to GitHub now for using flake eight as a pre commit.

34:33 You also had macedon.pi, right? Yes, yes. Yes,

34:38 I did. I just sorry, I flipped the order. Because I thought it was it was it was a segue there.

34:42 Yeah, yeah. Yeah. Just I wish I knew about this, like a week ago or so. That would have been

34:48 you covered a Brian I think, right? Yes,

34:51 we covered two That's

34:51 right. Yeah, yeah. So this is a this was a to be very honest. I wasn't the one that found this was my my boss. So shout out to BART. If you're listening right now. Uh hum. But this is basically just a wrapper around the Macedon API, right? So you don't have to do requests you can use, you have like a nice client library there to do all these things. So if you want to play around, create some bots, you know, whatever, then, yeah, there's a nice, convenient package now for you to do it.

35:15 This is really cool. And it has, you know, what documentation does it say what functions it has, I love it.

35:20 Documentation.

35:23 It doesn't have to be much like that, like that seven or eight lines of code that are in the readme like gives you a really good boost. But it lets you register your app, which is one of the things if you go to the website, it'll show you which apps are registered for your access keys on Mastodon, but it won't let you create one on the website. So you're just like a simple create app and just give it a name, your instance name and what file to save the access tokens over to and boom, you're good to go.

35:46 Yeah, have you guys already done stuff with a mastodon? Or?

35:49 Yeah, I, you know, I'm on the stream deck. The thing that controls the stream, I already wrote that thing where I might push the one button it it sends out the message automatically that this live stream is starting and yeah, uses that uses a little bit of toot and mostly just the straight API with HTTP X. But if I'd known about this, you know, I will use it. Now we know Yeah, no, thanks for sharing that. Anything else you want to share? Before we move on? Yes.

36:13 So there are a couple more things. But this one, this is the Brazilian me that couldn't resist the World Cup started. I don't know if you guys are soccer fans or not at all.

36:22 So we've fun soccer team. Here we go. See, I go see with the kids and stuff in town.

36:27 So the so I'm also in machine learning. So a lot of data and like this time of the year, you know, there's a lot of like, oh, yeah, AI models are predicting this, this and this one is one from Oxford. So I just wanted to give a quick shout out here. So they they have a video on YouTube as well, which is called Explain the math. And I'll go on a limb here and say the use Python because they've been mentioned matplotlib, and whatnot. But this is basically just a big excuse to say that they predict Brazil to win. So you know, if this doesn't happen, it's all rigged. This math suggests the math supports this. So Brazil must win this work up and anything that is going to be extremely disappointed.

37:02 That is really cool. People are always looking for like realistic examples to learn and explore libraries and tools. And this you know, if you're into soccer, and you care about the World Cup, this is great.

37:12 Yeah, I think if you Yeah, there's there's people people are very creative. I feel like there's a lot of uses.

37:18 Well, I'm sure this will happen, because there's absolutely no corruption in soccer. So yeah, for sure.

37:24 Yeah, not at all. Not at all. Cool. Should I just keep going? Or do you want to take over if

37:31 you got more items?

37:32 Yeah, can I have two more? Sorry, I know you said more than two. So

37:38 you can just that's what this whole section is about?

37:41 One. So for me as a data scientist or machine learning engineer, we use a lot of notebooks. Right. And I think they have their place in data science. But there are some tools that don't don't play very nicely with it. Right. And I think in good diffs or PRs, they don't they don't play so nicely. Right. So this is up, I think it's public preview, I want to say, but I haven't actually seen this. But now GitHub is going to start supporting notebook diffs. So if you have a pull request, they're going to have a nicer rendering of the notebook here. And you can actually see what the differences are. And I think before there was okay, called review and be that you could add to GitHub. But yeah, now they're just going to start supporting it. So I haven't seen how it looks. But I'm pretty excited about this to one less headache. For me,

38:21 that's excellent. Because before the diff would just be like, here's the diff of the JSON file, like I was like.

38:27 And also JSON is just JSON is just key value. So if you just changed the order of some keys, it's just the Yeah, you have a lot of changes, but it's not you don't care.

38:34 Yeah. Oh, this looks really useful. Yeah. And

38:37 maybe one last, if that's okay. Yeah, just pull this year. This is Lancer. So it's another CLI tool. I talked about linting before, right? So this is another kind of linting. And I say kinda so you know, some

38:51 definition of clean lands of linting or cleanup? Yeah,

38:55 so this is like black, almost like black, but it's the opposite. So instead of make your code look nice, it will just make it like a hideous but working mess right? So these are some of the features it turns all your comments to Pitbull lyrics or something, say for work that then if you want, it takes all your variable names and mix into like animal sounds and horribly looks similar looking character. So like bark, underscore, bark underscore 0000, it adds whitespace it adds completely irrelevant comments, and the code still runs after these improvements. So he was loving it. You have here some comments and everything. So before like nicely formatted and then afterwards, you see some comments like, Bada bing, bada boom, you know, there's nothing like Miami Heat, some alpha characters in your variable name. So pretty good stuff. Again, I must say I haven't used this, but this is a tool that I'm not as excited to

39:51 use. I mean, there's all there's always times that you need to send out your code to different places and you would rather share it less Spend more thinking of like, if you make a desktop app and you got to send out the code for that, or whatever, and you would want to obfuscate it, you wouldn't make it harder for people to just pick it up and like, do so you could hit it with this. They'd be like, yeah, no, now we're just not doing that.

40:15 So my favorite, my favorite ones on the screen is the adding obvious comments, like setting the value of some, like. That wasn't in the original. And it's just funny that I mean, that's actually not gibberish. It's just useless.

40:35 Yeah, it's, it's really good. The comments out in the live stream are really great as well, people are enjoying it. What am is is great for Twitter employees, you can maximize your lines of code for review, as that's coming up, then you just print it out. And you take it and sidebar, like if somebody says, print out my code, so we can review it. They're not, they're not equipped to review the code that you may have written, like if the word print involves in evaluating code like No, all right, just I don't think so. So leave that where that is. But you could you could put this on top of like, Yeah, I'm kind of funky. When I write code, it's a little different. Let's get used to. But yeah, I just, it's a farm. It's a code farm, like a coin coin.

41:16 It can have two sets of books, kinda, you got your real repo, and then you use this to put it into the actual one that you submit. And you're like, I understand it. I don't know what your problem is.

41:33 I kinda want to run this on a large codebase.

41:37 really complicated question all the commits, put, push, like,

41:42 like, releases texture along boink or something. Yes, I love it. Cool.

41:50 Well, this was this was a good, good find. Awesome. Thanks. I'll make one quick here. So a new YouTube video, I talked about how you can install the Macedon web app on your iPad as a native app as well as on your desktop. So if you're doing that kind of stuff, I'm not there. Basically, they just released Macedon for a couple days ago. And all the apps don't have features like edit and some of the other features that are there because they're like months behind. And so if you install the web app as an app, then guess what it looks like an app, it acts like an app, but it has like zero latency. So as soon as something is released on the website, you get it, which is pretty cool. So people can check that out. I saw Madison in the audience sent over a call for proposals are calling all Python Easter's if you will for PI cascades. So pi cascades is back in person this year in Vancouver, BC, it goes from Vancouver to Seattle to Portland and cycles through that there but so this year, it's going to be in Vancouver. So if you want to go up there and talk be part of the conference good conference, so call for proposals are open there.

42:55 Yeah, but they're not very much longer. So jump on that. The data is but

43:00 it it closes Wednesday. 30th. So what is that? Next Wednesday? Yeah, it is. And mass and film the audience. Thank you. It's in person this time we and we really value the first time speakers and a typical talks. So get out there and put yourself out there and get into public speaking. It's not a huge conference, but it's you know, it's big enough, a couple 100 people, three 400 people fun time. This is just really quick and fun. You know, if you're on a Mac, you are not as likely to get viruses sent your way that would actually be able to do something like 90% of viruses are written for Windows. But what's a really interesting fact I just if you do have a Mac, it turns out 50% of all Mac OS malware comes from one single app.

43:43 Can you believe that? What is it Safari?

43:46 No, it's Mac keeper. So if if you have Mac keeper it like organizes your files and it'll like clean up your your junky cache and stuff. But apparently it has to take over so much permissions and it is like it can get I guess plugin or I don't know what it does. But people can like plug into this and make it to all sorts of horrible stuff. So 50% of all malware is written for Mac keeper. So if you have Mac keeper, maybe on habit. I recently as Sunday launched a new website that I hope will bring me back to writing some more we'll see about how that goes. But here I'm I'm trying a new philosophy on on blogging, Brian, I don't know how you feel about it. But I have a blog been doing for a long time. But like I looked, the last article I wrote was like 2020 I'm like, Oh, that's not so good. And the reason is, I've always tried to write like 2000 word posts that are really not like, but I could post to Twitter and Macedon all day and it's like, I can just do that. That's no problem. I don't like fall behind on Twitter. I think that's because these really should be super short post. So I have I've got this new website that I wrote that are just super short, you know, fits on a page type of articles that people can Go and checkout. So

45:01 yeah, some of the people that some people were promoting like today I learned things. But short why not? I mean, if you think I if you think it's going to be a thread, right about a blog post.

45:12 Exactly. Yeah. Yeah. Cool. So all of these are written. And this is all based on Hugo, which is a just learned about it, but a ridiculously cool static site generator. If either of you played with Hugo,

45:23 I use it. I love it, Python. So Python was written on, Hugo.

45:28 It's ridiculous right. Now. Marla, you haven't? Sorry? Yeah. No, I

45:32 haven't used it. But I heard of it. Yeah, her nice things. Yeah. So

45:34 you basically just go to your directory, have markdown files and images, you just Hugo dash D server, whatever. And then as you write, you have your your web page open in your browser and automatically sees the Markdown file changes or the CSS changes, regenerates it and refreshes your browser just to your browser, it could be just over there. And it's just periodically, as you make changes, it instantly refreshes. So you don't even go and refresh the page and see how that is right. And the browser just watches and and reloads. It's cool.

46:02 Yeah. And so you got it so that you just you just push push your changes to GitHub, or your repo for and it just appears on your website.

46:11 Exactly. Yeah, exactly. So that was my next thing is that I set up a Netlify free account with CDN, SSL, custom domain name, push, it just has a prod branch that I connected it to. And when I push to prod, boom, it just goes there instantly. So anyway, people are looking at that. That is super cool.

46:26 Push to prod. Oh, that's kind of cool. I just added on prod. So

46:31 I just log in and it over SSH. Just enter the server. The server's the backup. Anyway, I have stuff on the screen, but then no more backups. That's just stuff I pulled up while we're talking. So no more extras. I mean, so Yeah, fun stuff. People check out the blog website and the video and apply for speaking at pi cascades.

46:51 Nice. Well, I feel like Lancer also was like already really funny. But do you have anything else funny for us? Or?

46:57 I do although I somehow forgot to pull them up on the screen. So give me just a second here. There's two, these are really good. Okay, these are these are pretty, pretty epic. So this one is called I think, Marlowe, you'll you'll really like this one because it has to do with like algorithms and data science. And it's called messing with the algorithm. And it shows this this dude here might don't mind the thing at the bottom. I have no idea what that's about. But see, there's this guy whose face is blurred out in the UK, I think I can't remember what it was an Oberlin. And he's got a wagon, like a little red wagon that you pull behind you full of 99 phones. And what he did is he caught them all running Google Maps and left them open and started walking down the street real slow. And notice his neighborhood has now red on the map and he got it so it thinks there's a traffic jam. It'll send cars around the neighborhood. Nice. I want to get one of these so bad. And whenever I take my dog for a walk, just walk with the wagon behind me to

47:55 no cars. Yeah.

47:57 So good. Is it? Nice. So ahead of our time. He's just like, Oh, he's

48:02 so really for his neighbors. Yeah, the next one. The next one here is going to take a little bit of a I gotta I gotta set the stage. Give me a second to SET set the history. Have you heard you've heard about these motivational posters you go into like a dentist's office. It'll be like an eagle soaring over like a sunset. Like if you don't spread your wings. You'll never soar as high as you could or something cheesy like that. Yeah. Well, there's this company called. Yes, exactly. There's a company called despair. And despair creates these but like in reverse. They're called the D motivators. Yeah. Nice. So have you have you seen these? No. Okay, so here's one like solutions. And what does it say? It has like a Rube Goldberg Ian type looking thing here. And it says solutions. This is what happens when the problem solver gets paid by the hour. Just out of control. Here's one what is this is a frog wearing with a snail on its head says collaborate. So the best of us have to carry the rest of us. It's just like, really? Alright, so that brings us to, I feel like this is Brian again show a little bit. This a tweet that he shared here. And it. It has a lot Taxify latekka fi thing but recursion. And for the recursion it has that demotivator is the picture that says recursion. Here we go again. And then embedded in that is recursion. Here we go again. It's like that. Yeah. Your screenshare you see her on screen? Yeah. So it's kind of like that poster. But for recursion. Yeah,

49:27 I kind of feel bad that people people that don't get the recursion joke because they can't even look it up because it just gets redirected.

49:35 It just keeps going. Like the definition is the definition. Right? Nice. Sigh Well, that's what I got for y'all.

49:40 Well, thanks, everybody. And thanks, Michael, of course, and thanks mirlo for coming on the show. Thanks for having me. It was great.

49:48 Yeah, you bet. Bye everyone.

Back to show page