Transcript #336: We found one of your batteries

00:00 Hello and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds.

00:05 This is episode 336 recorded May 16th, 2023. I'm Michael Kennedy.

00:12 And I'm Brian Okken.

00:13 And this episode is brought to you by InfluxDB from InfluxData. We'll tell you more about them later. Be sure to connect with us over on fostedon.org. I'm @mkennedy, Brian is @brianaukin, And the show is at Python Bytes.

00:29 The rights and status of the show are still undetermined, Brian, but I'm sure we'll figure that out someday.

00:34 See the last show to get the joke.

00:36 And join us over at pythonbytes.fm/live usually Tuesday at 11 a.m. Pacific time to be part of the show.

00:44 Or you can catch also the older episodes there.

00:47 Or of course on your podcast players.

00:49 And with that, Brian, let's dig into some batteries.

00:52 >> Okay. Well, as we know, Python is the language of the batteries included. But there's we also have lots of cool, cool extra packages on pipe. Yeah, I actually quite a few. And one of the things that I wanted to highlight was a few just a handful of utilities packages that are really kind of fun. And you probably knew about them, but maybe forgot. And we've covered some of these in the past. So I wanted to highlight this article from Martin Hines called Python's missing batteries, essential libraries you're missing out on.

01:25 And, the first project he talks about is bolt ons, which is actually an amazing, it's an amazing package, but it's so big, the comment here is he could, he could probably do an entire article just on bolt ons.

01:39 And I think that's wrong.

01:40 I think you could do an entire book on, on bolt ons and it would be a big book.

01:44 There's a lot in there.

01:46 but a few of the things that he highlighted were pretty, pretty cool that I kind of didn't know about.

01:52 Boltons has a JSON utils and a time utils and an iter utils that he's demoing.

01:58 So with JSON utils, you can just iterate with like a, for line and JSON utils, JSON iterator.

02:07 You can iterate through JSON elements.

02:09 That's pretty amazing.

02:10 That's pretty cool.

02:11 I like that.

02:12 The time utils example was, is a, using a date range, time utils date range and iterating through days, which is kind of neat. I didn't know you could do that.

02:24 Kind of a cool idea to maybe walk through days and get different date times.

02:28 But anyway, there's a different step size you can do.

02:31 You can walk through each week or whatever.

02:32 And then, iterutils has a, is highlighting a couple things in their utils.

02:38 One of them is get path, which isn't really like a file system path, but it's basically saying I've got a deeply nested structure and I want to access it without having to do all the access functions.

02:53 So it's a way to get access to deeply nested things.

02:56 And then a remap, which is neat.

02:58 Remap takes a deeply nested structure and just changes something inside of it, which is kind of cool.

03:05 I don't want to go through all of the details of this article, but a couple of quick highlights.

03:10 There's highlighting the SH package where you can do shell commands from Python in a fairly nice way.

03:20 Data validation actually, this is pretty neat.

03:24 There's pedantic, of course, but that's for like, which is awesome.

03:29 But there's also this validators library, which is neat.

03:32 And it can do things like validate, making sure that email, you like validating emails or Visa card numbers or an IP address is all format, just validate strings are formatted correctly and things.

03:44 It's pretty neat, cool.

03:46 And then the fuzz is a fuzzy matching string, fuzzy string matching library, which is kind of cool.

03:53 I wanted to jump down debugging.

03:56 There's a stack printer that has, it's basically a really nice stack trace that does the error messages, which is kind of cool.

04:05 What else? For testing, you can freeze time with the freeze gun library.

04:09 And then this, the last thing is kind of cool.

04:13 I write a lot of command line applications and there's a, I would not have thought to look for this package called TQDM.

04:22 I don't know what that stands for, but it does.

04:25 It does like, what are these things?

04:28 Progress bars, we think for command line utilities.

04:31 So TQDM comes from Taqwadum, which means progress in Arabic.

04:37 [LAUGHTER]

04:39 >> Of course.

04:40 >> I wouldn't have clued me in to go search for it.

04:42 I love that package. There's a lot of cool stuff here.

04:44 I use TQDM, it's my go-to for this stuff.

04:49 There's a lot of things like, "Oh, I need to go over millions of database records, and make some change, and do a test, and then maybe, I don't know, update some of them." That might take a while.

05:00 I just did something where I had to do a report on a bunch of stuff on the Talk Python courses and it took nine hours to go do a bunch of compute for a bunch of courses for an insane amount of stuff and I ran that and you could just see, I saw several things.

05:20 One, it shows you the progress.

05:22 You can see it doing progress, but it also tells you the per object per time.

05:28 So it'll say like processing 200 records per second, for example, as it goes through the list.

05:34 And it also estimates the time, which is why after five minutes, I'm like, oh, this is going to take nine hours.

05:39 I'm not going to wait for this.

05:40 It's really nice.

05:41 So can you use it if you don't really know how long something's going to take to begin with?

05:46 Yes.

05:48 And--

05:48 Or do you have to like kind of know--

05:51 do you have to give it like it's 10% done or it's 20% done?

05:54 No, it does it all automatically.

05:56 I don't really know how.

05:58 I think some things it can figure out.

06:00 >> Okay.

06:00 >> Others, I don't know how it can actually do that.

06:04 Because for example, on the example on the screen, it has a range from 0-100.

06:09 You can't ask the length of the range.

06:12 >> Okay.

06:13 >> But it somehow knows.

06:16 >> Well, I might play with that because right now I've got an application that command-line thing that reboots an instrument and then waits for it to finish.

06:24 and I just have dots and it'd be kind of nice to have like a, something like this.

06:30 - Yeah, my prior solution was, oh, let's put out a little dot every so often.

06:34 No, that's too many dots.

06:35 Let's mod it out a little bit higher to like maybe every 20 records we'll put a dot or something like that.

06:41 - Exactly.

06:42 - Yeah, so this is nice.

06:43 You can just wrap an iterator in a TQDM and then loop over it and magic happens.

06:48 - Cool, I'll try that.

06:49 - Yep.

06:50 All right, well, that's pretty awesome.

06:51 Wanna hear about more awesome things, Brian?

06:52 - Yeah, let's do awesome.

06:54 - Let's do some awesome, some pollers.

06:56 So pollers is as many things in Python are these days is the rustification in a good way, Python things.

07:04 So it's kind of like pandas, but redone and rust with more of a fluent API that allows it to be more database query engine like.

07:14 And so what I have for us today is the awesome pollers, a curated list of pollers, talks, tools, examples, and articles.

07:22 Now many of these awesome lists are extensions and there are a few things in here, like it talks about the Python library and you may not know there's actually a Rust library for pollers that you can directly use if you're integrating with Rust code, but also one for R, one for Node.

07:37 It's got some things like cheat sheets.

07:40 If people wanna go and check out the cheat sheet, it's got actually a really nice visualization to show you what reshaping data means with concat or appending columns side by side from two data frames in a different, with a horizontal concat flag, which I think the visualization of these things is really nice.

08:02 What do you think of this, Brian?

08:03 - I'm actually, the visualization is what I'm enjoying the most with this cheat sheet.

08:07 That's nice, so.

08:09 - Yeah, it's really, really nice.

08:11 And it has a bunch of tutorials and workshops.

08:13 So if you are trying to get into Polars, Come over here, there's maybe six or seven different examples and a bunch of blog posts, a whole bunch, how to integrate it with DuckDB or how it compares to DuckDB, and then a bunch of videos as well as people in the holders community, right?

08:34 Like Richie Vink, who created it, but also contributors, if you can follow them and ask them questions.

08:42 - That's kind of a nice addition.

08:43 It was like, on social media, who do you follow?

08:45 That's pretty cool.

08:47 Yeah, super nice.

08:48 So anyway, not a whole lot to go into it there, but yeah, really, really nice.

08:53 People are into Polars, put it here.

08:55 Also, I kind of wanted to give it a shout out because Polars is fairly new.

08:59 And if you've got something that integrates with Polars or builds on top of Polars in a way that itself is reusable, you know, come over here and do a PR.

09:06 I'm sure they're happy to accept it.

09:09 It says, "Contributions welcome!" So yeah, get in here and contribute.

09:14 They're so welcome.

09:15 (laughing)

09:17 You know what else is welcome?

09:19 Our sponsor this week.

09:20 So super happy to have a sponsor for the show.

09:24 As we mentioned at the top, InfluxDB.

09:27 So InfluxDB is all about the time series data.

09:31 So this episode is brought to you by Python.

09:34 This episode of Python Bytes is brought to you by InfluxData, the makers of InfluxDB.

09:39 InfluxDB is a database purpose built for handling time series data at a massive scale for real-time analytics.

09:46 So developers can ingest, store, and analyze all types of time series data, metrics, events, traces in a single platform.

09:53 Let me ask you a question.

09:55 How would boundless cardinality and lightning fast SQL queries impact the way you develop real-time applications?

10:01 Maybe make them real-time, huh?

10:03 InfluxDB processes large time series data sets and provides low latency SQL queries, making it a go-to choice for developers building real-time applications and seeking crucial insights.

10:14 For developer efficiency, InfluxDB helps you create IoT analytics and cloud applications using timestamp data rapidly and at scale.

10:23 It's designed to ingest billions of data points in real time with unlimited cardinality.

10:28 InfluxDB streamlines building once and deploying across various products and environments from the edge, on-premise, and to the cloud.

10:35 Try it for free at pythonbytes.fm/influxdb.

10:39 The link is in your podcast player show notes.

10:41 Thank you to Influx Data and InfluxDB for supporting the show.

10:46 All right, over to you, Brian, what's next?

10:49 - Well, this is a pretty quick one, but I wanted to, I know that a lot of people test with Selenium.

10:54 I know there's lots of other stuff you can do, like playwright and everything like that, but still, Selenium's heavily used, and I still have some tests in Selenium.

11:03 And, well, there has been a change, so I wanted to just make sure everybody's aware.

11:09 If you, there's an article called Running Headless Selenium in Python in 2023.

11:16 And the catch is basically if you're, well one, if you're not running headless already, why not?

11:25 The headless is awesome.

11:26 It can basically, you can run through a web browser but don't actually load, don't open it.

11:33 It just, you run it behind, there's no win, anyway.

11:35 It's faster, so use headless.

11:37 But if you are already using headless, there's been a change.

11:40 So the change is, let's go down, scrolling down, there's an example, which is great.

11:47 So Selenium 4.8.0 came out in January.

11:51 And the old way to do things was to do, you set up your web driver and you mark headless equals true.

12:00 And you can do this with both Chrome and Firefox had a little different setting, but it also had a headless equals true setup.

12:07 And then you can run headless and it was awesome.

12:10 They took away this dot headless.

12:12 So don't do that anymore if you're using Selenium 4.8 or above.

12:16 The new way is, so for Chrome, you add an argument of headless equals new, --headless equals new.

12:25 And it's really add argument.

12:26 If you're listening to this, there's a new options dot add argument.

12:29 And then the same sort of thing with Firefox, it isn't equals new, it's just --headless.

12:36 But this shows you an example.

12:38 Why did they do this?

12:39 Well, there's some description of why there was like an old way and a new way.

12:44 And then Chromium had a new headless option that you can add.

12:49 So we wanna be able to do the new way.

12:51 So they deprecated the old way to get people to use the new more powerful.

12:55 And we're also linking to an article from Selenium, which is kind of a funny title.

13:00 So they wanted to get everybody's attention So they knew, so they named the article, Headless is Going Away, yes, which is a funny name.

13:09 And then subtitled it with, now that we have your attention, headless is not actually going away, just the convenient method to set it in Selenium.

13:17 So I guess just a public service announcement, if you're using Selenium, you gotta change your code to use the new Selenium 4.8.

13:24 That's it.

13:26 Oh, you're on mute.

13:27 - So I am.

13:27 I do like it.

13:30 I wonder though, why you have to pass the command line argument directly and it doesn't just look like, oh, you said headless, that means in Chrome now pass --mode this, you know, because it's almost the same but not the same across the browser platforms.

13:44 Yeah, I think it's because there's different, I don't know, I haven't looked through the explanation but I think there's other options.

13:54 So it isn't necessarily just that they've changed the way you turn on headless, but there's more headless options.

14:01 So they're just building it in so that you can pass in new flags.

14:04 And I think Chromium might end up getting more versions in later or something, I don't know.

14:11 - Yeah, the browser space is a, it's an interesting time, isn't it?

14:15 - Yeah.

14:16 - We fought through the browser wars, we've beaten back Internet Explorer 6, only to come back and have Chromium even more dominant in certain ways.

14:26 - It's interesting because like as for a usability thing, I'm usually using Vivaldi now, but I use probably Vivaldi and Chrome for day to day use.

14:35 But for testing, yeah, it's still Chrome.

14:38 I use Chrome and Firefox.

14:41 That's what I use Firefox for, is still testing with Firefox.

14:44 - Yeah, absolutely.

14:45 Cool, you know, just a bit of follow up on the previous conversation about those different batteries that you talked about.

14:52 I love our audience, there's so much cool stuff going on over there.

14:55 So Blaise says, "I wonder if Rich does anything with TQDM?" And if you want a definitive answer, how about Will McGugan in the audience says, "TQDM has a Rich output option." Will obviously being the creator of Rich and many other awesome things there.

15:12 So nice follow-up.

15:14 - Awesome, we've turned into the water cooler of Python.

15:17 - We sure have.

15:18 All right, I have one more thing to share with you all.

15:22 Let's jump into it.

15:23 And that is Gracie.

15:25 So, Gracie's an interesting project.

15:28 It's a little bit like your first topic, Brian, in that it has a bunch of kind of utility features, and this one is around consuming APIs, so not creating APIs, but writing clients that talk to them, specifically around HTTPX, which is one of the absolute go-to ACP libraries for doing modern async style of APIs in Python, right?

15:54 So, Gracie, it says, "Gracely manage your API interactions." Gracie helps you handle failures, logging, retries, throttling, and tracking for all of your HTTP interactions.

16:08 And it uses HTTPX under the hood.

16:10 Let's you do the, like Gracie, do the boring stuff and you can focus on your app is the selling point here.

16:15 So this is pretty cool.

16:16 It's not super well known.

16:18 It's got like 180 stars and it's an interesting library that has a lot of cool functionality.

16:24 It feels like it could use a little bit more polish, but it's still quite neat.

16:29 So let me give you some ideas here.

16:30 So what you do is basically you model your API interactions through a class structure.

16:39 It's not quite a hierarchy, but you use classes to come up with it.

16:43 So you can come up with an endpoint here, and then you create something that derives from the API base class.

16:52 give it a URL and then you give it a bunch of settings.

16:54 The settings are where the useful stuff is.

16:57 For example, you can say, I would like to log the request as it's going out the door, but only in debug.

17:04 I'd like to log the response and that one a little more frequently at the info level, and then you can have a custom message that goes out there.

17:13 Then you also can have a parser that will parse the response as a set of functions.

17:20 The first example you see here just says, by default, just given any object, call.json on it, given the request, call.json on it, right?

17:30 So that's handy, but what you can do if you go down a little bit, custom validators, you can actually say, by default, just try to convert it to a JSON response.

17:44 But if the status code is not found, then do something else.

17:48 You can have a series of different status codes.

17:50 So if it by default use this parser, but if it's like a 400 bad requests, then we need to parse it as something else.

17:58 And that could even be like convert it from, you know, maybe in a success case, you get this particular say, Pydantic model back, but in an error case, you have a totally different structure and you might want to parse it differently into a different Pydantic model, something along the lines.

18:13 So you can do a lot of cool stuff like that there.

18:16 And yeah, and then you just give it, give it the functions that you call that basically invoke the API.

18:22 And of course, because it's based on HTTPX, you can await calling on this function. So, yeah, anyway, it looks it ends up with a pretty clean model for using it. What do you think?

18:33 Well, yeah, it it'll take some time to get your head around it because of the class-based thing.

18:39 But it's all stuff that you're going to have to develop anyway.

18:42 So having somebody else do the work, it's pretty good.

18:45 Yeah, there's some nice examples of like throttling and This might be interesting to you, Brian, is it has the ability, there's a bunch of different things.

18:54 It has ability to replay certain data.

18:57 You can also say, we're only allowing certain, by default, any 200 category status code is considered success.

19:06 You can say, no, for this one, it has to be a dot created like a HTTP status dot created, not 200 or something like that.

19:13 Or you can give it either okay or created.

19:16 You give it a set of options.

19:18 That's pretty cool. You can add custom validation.

19:21 You talked about validators at your beginning as well.

19:25 And if you're not using Pydantic or something that kind of does its own custom validation, you can still even add more stuff.

19:32 Like not only does this have to be a string, but it has to be, I don't know, an email of this type or whatever, right?

19:39 Like of this, say, the domain of our company, right?

19:42 Something like that.

19:43 So you can add these custom validators and It comes with a retry, built-in retry for how do you handle the retries, how many attempts, what do you do in terms of logging, you know, about retries and failures, what do you do if, you know, you can say, I want to retry three times and if it, none of them work, I don't care, just keep going, don't break my application, or please do.

20:06 Don't, you know, raise an exception.

20:08 You might say, well, why would you ever not want to break it?

20:10 Like, maybe you're trying to write to some sort of audit log to say this happened.

20:15 And if the server that just records what happened goes down, you don't want to start crashing your app, right?

20:19 There's like scenarios where you might not really care about that.

20:22 Also throttling, which is pretty neat.

20:24 You can say any time that the URL contains, the example is a Pokemon thing.

20:29 So it has a regular expression for Pokemon.

20:32 I want maximum 10 requests for every one and a half minutes.

20:36 And then you can actually, it has a cool output too.

20:39 If you print out just the rule it says, which is an object.

20:42 it says 10 requests per 90 seconds for URLs matching this regular expression, which is kind of nice.

20:47 - Oh, cool.

20:48 - And yeah, the final thing, some, I don't really know where it is in here, but yeah, you can also have it throw certain exceptions.

20:56 So how, you know how it has that parser type scenario for different HTTP status codes I told you about.

21:01 So you can say, if it's a bad request, please throw, you know, some exception class that you come up with, right?

21:07 So instead of just saying bad request, it could potentially have more details.

21:11 you might build a parse information into it and then raise that exception.

21:15 There's some pretty neat things.

21:17 And the final thing, by the way, rich integration right there, it requires you to install rich if you want fancy output on, it'll tell you sort of it's, it'll report on how it's interacted with the API endpoint.

21:32 So you've got to do like a bunch of processing.

21:34 You know, I told you about like, I'm going to transform a bunch of things.

21:36 I use TQDM.

21:38 But if you're going to do that, at the end you could ask, well, how did it go?

21:41 And it'll give you this like summary report of how much success and how much failure and what's the average latency and status codes and requests per seconds and all of these.

21:50 And it'll do that in text form or in rich style.

21:53 Final thing, it will record and replay API interactions for testing purposes.

22:01 So if you want, you know, if it's really tricky to mock out some complex interaction, you'd say, well, I want it to be as exactly close to real as possible, you could just one time do those API calls and then replay them back, put it either record mode or replay mode and the backend that stores that could be a SQL light database or a MongoDB database that's automatically integrated and you just give it that and say, when I talk to the server, remember what you did and store it over here.

22:29 And then you can play that back for testing.

22:31 Oh, wow.

22:31 Cool.

22:32 So yeah.

22:33 Anyway, people can check this out and see what they think, but.

22:36 I think it almost looks like it was a system pretty much designed, well, one of the obvious use cases is to build a custom thing to test your application because there's a bunch of utilities there to really interrogate something.

22:54 Absolutely, yeah, you get that report and you get the replay, record replayability, the logging, yeah, a lot of that stuff is there.

23:02 It's pretty neat.

23:03 Yeah, cool.

23:04 Nice.

23:05 - Nice. - Well, that's it for that one.

23:06 Yeah, I guess that's all of our items, isn't it?

23:09 - It is.

23:10 And for extras, I don't have any extras.

23:14 Do you have any extras?

23:15 - I do, I just have one.

23:17 And then we'll get to our joke.

23:18 So for the extras, do you know what, Brian?

23:20 Look at this, look at, here it is.

23:22 - You got in the App Store, yay.

23:24 - I got in the iOS App Store too.

23:26 So finally, finally, finally, the Talk Python mobile apps are out on all of the App Stores.

23:32 So go get them.

23:34 Just talkpython.fm/apps, I believe will take you there.

23:38 Redirect over to the training site.

23:39 But yeah, they're available on iPhone, Android tablets, iPad, Android tablets as well.

23:48 Maybe more coming.

23:49 We might have even desktop apps coming pretty soon, depending on how successful we are with all this.

23:55 But yeah, so this is out.

23:57 People can check it out.

23:58 And as a way to celebrate finally getting this done, after four months of work.

24:04 First of all, wrote a blog post, maybe I'll add it, throw the link in the notes, yep, I'll throw it in there for people.

24:09 I talked about some of the design choices about how and why we chose things like Flutter and so on as the mobile app framework.

24:19 But the one thing for people to know out there, and this is a bit timely, is if you download and install the mobile app before, what day, today is Tuesday, May 16th, If you do that before May 22nd, so download the app before May 22nd, inside the app only, the Up and Running with Git course, which is normally $39, is completely free.

24:42 So we'll sort of celebrate the launch of the app.

24:45 So you go in there, go find the courses, go to the free section, join the Git course, and you'll have it forever, not just for a little time.

24:51 But the only way to get it is to download and install the app, which is free, and then go put the Git course into your account.

24:58 >> I just downloaded it.

24:59 I'm opening it right now.

25:01 >> Awesome.

25:02 >> One of the things I'm excited about this is because when I'm doing a course, not giving a course, but learning from one, I do like to have it on my computer screen, but there's oftentimes where I've got time to kill.

25:16 I'd like to listen to some of the conversation and listen to it.

25:19 Yeah, I'm going to look at some of the stuff on my phone, but a lot of it is following along, but I'm listening and then I'll go through and watch the same stuff later on the computer and walk through it.

25:31 So I really like this addition of having a mobile app.

25:34 This is pretty cool.

25:36 - Thanks, thanks so much, Brian.

25:37 Yeah, there's a couple things why you might need it.

25:39 People are like, well, why are you just watching the web?

25:40 Like, especially on iPhone, you can't get rid of that navigation section around the browser.

25:47 So you end up watching like a postage stamp size thing, which is not ideal.

25:51 It won't auto-advance because ad companies are evil and iOS blocked them from playing ads all the time, which gobbles up everyone else as well, unfortunately.

26:02 >> Okay. On your app, it'll just jump to the next thing then?

26:07 >> Yeah. It just keeps playing smoothly as you would imagine.

26:10 Then the other thing that's important is you can download content offline like if you're going on a trip, or on the train, or some people even use it if they work at government institutions that have high levels of security, and they want to like research labs and stuff.

26:27 If they want to be able to take the course at their work, but their work is super restrictive about what they can interact with, you could download a whole course onto your tablet, set it next to you and watch it at your work.

26:39 Yeah, so those are the reasons why it exists.

26:43 But anyway, long time coming, super happy about it.

26:45 That's my extra.

26:46 - Cool.

26:47 - Download it, get the Git course.

26:49 - All right, well, how about a joke?

26:51 - Ah, this is a good one.

26:53 So you may wonder, you may have friends who are like, "Brian, you do Python, you do C++, "you wrote a book on pytest, "like how did you get so good at this?" So this kind of riffs on that theme.

27:06 There's two developers here.

27:10 First one, she says, "How do you code so well?" The expert developer, she says, "Practice." And the first person didn't really hear, like, "It must be an innate gift, a gift from God.

27:22 It's practice.

27:23 I'll never understand how some people are so talented.

27:26 A mystery, practice.

27:28 - Yeah.

27:30 - Right?

27:30 - Yeah.

27:31 - What do you think?

27:32 - Well, this is great.

27:33 And it applies to so many things, of course.

27:35 But one of my daughters is dealing with this right now.

27:39 She's been doing for about a year, doing aerial silks, aerial arts, and she's working on it and exercising and stuff every day.

27:49 And it was really hard at first, and now she's pretty good.

27:53 And so many people have said, oh, you're just naturally talented at that.

27:57 She's like, it makes her mad because it's not natural.

28:00 I've had to work at it.

28:02 Coding as well, so obviously.

28:05 - Obviously, yeah.

28:06 This is, it's not just coding, but coding certainly.

28:09 - Yeah, podcasting, writing blog posts, everything around what we do.

28:15 Practice.

28:16 - Yeah, absolutely, practice.

28:17 - Nice.

28:18 - That's the way they end it, so good job.

28:19 - Yeah, absolutely, very, very uplifting.

28:22 We'll end it on a growth mindset today, Brian.

28:24 Thanks for being here.

28:25 - Thank you.

28:26 - And thanks everyone for coming.