Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to
your earbuds. This is episode 336, recorded May 16th, 2023. I'm Michael Kennedy.
And I'm Brian Okken.
And this episode is brought to you by InfluxDB from Influx Data. We'll tell you more about them
later. Be sure to connect with us over on Fostadon.org. I'm at M. Kennedy. Brian is
at Brian Okken. And the show is at Python Bytes. The rights and status of the show are still
undetermined, Brian, but I'm sure we'll figure that out someday. See the last show to get the joke.
And join us over at pythonbytes.fm/live, usually Tuesday at 11 a.m. Pacific time to be
part of the show. Or you can catch also the older episodes there. Or, of course, on your podcast
players. And with that, Brian, let's dig into some batteries.
Okay. Well, as we know, Python is the language of the batteries included. But there's, we also have
lots of cool, cool extra packages on PyPI. Actually, quite a few. And one of the things that I wanted
to highlight was a few, just a handful of utilities packages that are really kind of fun. And you
probably knew about them, but maybe forgot. And we've covered some of these in the past. So I wanted
to highlight this article from Martin Hines called Python's Missing Batteries, Essential
Libraries You're Missing Out On. And the first project he talks about is Bolt-ons, which is actually
an amazing, it's an amazing package, but it's so big. The comment here is he could, he could probably do
an entire article just on Bolt-ons. And I think that's wrong. I think you could do an entire book
on Bolt-ons and it would be a big book. I agree. There's a lot in there. But a few of the things
that he highlighted were pretty, pretty cool that I kind of didn't know about. Bolt-ons has a JSON
utils and a time utils and an iter utils that he's demoing. So with JSON utils, you can just iterate
with like a, for line and JSON utils, JSON iterator, you can iterate through JSON elements. That's pretty
amazing. That's pretty cool. I like that. The time utils example was, is a using a date range,
time utils date range and iterating through days, which is kind of neat. I didn't know you could do
that. Kind of a cool idea to let me walk through days and get different daytimes. But anyway, there's
a different step, a step size you can do. You can walk through each week or whatever.
And then iter utils has a, he's highlighting a couple of things in their utils. One of them is
get path, which isn't really like a file system path, but it's basically saying I've got a deeply
nested structure and I want to access it without having to do all the access functions. So it's a way
to get access to deeply nested things. And then a remap, which is neat. Remap takes a deeply nested
structure and just changes something inside of it, which is kind of cool. I don't want to go through
all of the details of this article, but a couple of quick highlights. There's highlighting the sh package
where you can do shell commands from Python in a fairly nice way. Data validation actually
is pretty neat. There's pedantic of course, but if that's for like, which is awesome. But there's also
this validators library, which is neat. And it can do things like validate, making sure that email,
like validating emails or visa card numbers or an IP address is all format, just validate strings are
formatted correctly and things. It's pretty neat. Cool. And then the fuzz is a fuzzy matching string,
fuzzy string matching library, which is kind of cool. I wanted to jump down debugging. There's a stack
printer that has a, it's basically a really nice stack trace that has, does the error messages, which is kind of
cool. What else for testing and you can freeze time with the freeze gun library. And then this, the last thing
is kind of cool. I write a lot of command line applications and there's a, I would not have thought to look for
this package called TQDM. I don't know what that stands for, but it does, it does like a, what are
these things? Progress bars would think for command line utilities.
So TQDM comes from tech quantum, which means progress in Arabic.
Of course.
That wouldn't have, I wouldn't have clued me in a good search for it. I love that package. There's a lot of cool
stuff here. I use like TQDM is just, it's my go-to for this stuff. You know,
there's a lot of things like, Oh, I need to go over, you know, millions of database records and make
some change and do a test. And then maybe, I don't know, update some of them. And that might take a
while. I just did something where I had to do like a report on a bunch of stuff on the talk Python
courses. And it took nine hours to like go do a bunch of computes for a bunch of courses for each,
you know, like an insane amount of stuff. And I ran that and you could just see, I saw
several things. One, it shows you the progress. You can see it doing progress, but it also tells
you the per object per time. So it'll say like processing, you know, 200 records per second.
For example, as it goes through the list and it also estimates the time, which is why after five
minutes, I'm like, Oh, this is going to take nine hours. I'm not going to wait for this. It's really
nice.
So can you use it if you don't really know how long something's going to take to begin with?
Yes. And do you have to like, I don't know, like, do you have to give it like it's 10% done or it's 20% done?
No, it does it all automatically. And I don't really know how I think some things it can figure out.
Okay.
And others. Yeah. I don't know how it can actually do that because for example, on the example, on the
screen, it has a range from zero to a hundred, right? And you can't ask the length of the range.
Okay. Right.
But it, it somehow knows.
Well, I might play with it because right now I've got a, I've got a application that command line thing
that reboots an instrument and then waits for it to finish. And I just have dots and, and it'd be
kind of nice to have like a, like a, something like this.
Yeah. My prior solution was, Oh, let's put out a little dot every so often. Oh, that's too many dots.
Let's, let's, let's mod it out a little bit higher to like it. Maybe every 20 records,
we'll put a dot or something like that. Exactly. Yeah. So this is nice. You can just
wrap an iterator in a TQDM and then loop over it and magic happens. Cool. We'll try it out.
Yep. All right. Well, that's pretty awesome. Want to hear about more awesome things, Brian?
Yeah. Let's do awesome. Let's do some awesome, some polars. So polars is as many things in Python
are these days is the rustification in a good way of Python things. So it's kind of
like pandas, but redone and rust with more of a fluent API that allows it to be more database
query engine like. And so what I have for us today is the awesome polars, a curated list of polars,
talks, tools, examples, and articles. Now, many of these awesome lists are extensions,
and there are a few things in here, like it talks about the Python library. And you may not know,
there's actually a rust library for polars that you can directly use if you're integrated with rust code,
but also one for R, one for node. It's got some things like cheat sheets. If people want to go and check out the cheat sheet,
it's got actually a really nice visualization to show you what reshaping data means with concat
concat or appending columns side by side from two data frames in a different, with a horizontal concat flag, which I think the visualization of these things is really nice. What do you think of this, Ryan?
I'm actually, the visualization is what I'm enjoying the most with this cheat sheet. That's nice.
Yeah, it's really, really nice. And it has a bunch of tutorials and workshops. So if you are trying to get into polars,
come over here. There's maybe six or seven different examples, a bunch of blog posts,
a whole bunch, how to integrate it with DuckDB, or how it compares to DuckDB, and then a bunch of videos,
as well as people in the polars community, right? Like Richie Vink, who created it, but also contributors,
if you can follow them and ask them questions.
That's kind of a nice addition of like, on social media, who do you follow? That's pretty cool.
Yeah. Yeah, it's super nice. So anyway, not a whole lot to go into it there, but yeah, really, really nice. People are into polars, put it here. Also, I kind of wanted to give it a shout out because polars is fairly new.
And if you've got something that integrates with polars or builds on top of polars in a way that itself is reasonable, you know, come over here and do a PR. I'm sure they're happy to accept it. It says contributions, welcome, exclamation point. So yeah, get in here and contribute.
They're so welcome.
You know what else is welcome? Our sponsor this week. So super happy to have a sponsor for the show. As we mentioned at the top, InfluxDB. So InfluxDB is all about the time series data. So this episode is brought to you by Python. This episode of Python Bytes is brought to you by InfluxData, the makers of InfluxDB. InfluxDB is a database purpose built for handling time series data at a massive scale
for real-time analytics. So developers can ingest, store, and analyze all types of time series data, metrics, events, traces in a single platform.
Let me ask you a question. How would boundless cardinality and lightning-fast SQL queries impact the way you develop real-time applications?
Maybe make them real-time, huh?
InfluxDB processes large time series data sets and provides low latency SQL queries, making it a go-to choice for developers building real-time applications and seeking crucial insights.
For developer efficiency, InfluxDB helps you create IoT analytics and cloud applications using timestamp data rapidly and at scale.
It's designed to ingest billions of data points in real-time with unlimited cardinality.
InfluxDB streamlines building once and deploying across various products and environments from the edge, on-premise, and to the cloud.
Try it for free at pythonbytes.fm/InfluxDB.
The link is in your podcast player show notes.
Thank you to InfluxData and InfluxDB for supporting the show.
All right, over to you, Brian. What's next?
Well, this is a pretty quick one, but I wanted to, I know that a lot of people test with Selenium.
It's a, I know there's lots of other stuff you can do, like Playwright and everything like that, but still Selenium's heavily used.
And I still have some tests in Selenium.
And there's a, there has been a change.
So I want to just make sure everybody is aware.
If you, there's a, there's, here's an article called running headless Selenium in Python in 2023.
And the catch is basically if you're, well, one, if you're not running headless already, why not?
The headless is awesome.
It can basically, you can run, run through a web browser, but don't actually load, don't open it.
It just, you run it behind, there's no wind anyway.
It's faster.
So use headless.
But if you are already using headless, there's been a change.
So the change is, let's go down, scrolling down.
There's an example, which is great.
So Selenium 4.8.0 came out in January.
And the old way to do things was to, to do, you set up your web driver and you mark headless equals true.
And you can do this with both Chrome and Firefox had a little different setting, but it also had a headless equals true setup.
And then you can run headless.
And it was awesome.
They took away this dot headless.
So don't do that anymore.
If you're using Selenium 4.8 or and above the new way is so for Chrome, you add an argument of headless equals new, dash, dash, headless equals new.
And it's really add argument.
If you're listening to this, there's a new options dot add argument.
And then the same sort of thing with Firefox.
You just, it isn't a equals new.
It's just dash, dash, headless.
But this, this shows you an example.
Why did they do this?
Well, it was, there's some description of why there was like an old way and a new way.
And then Chrome, Chromium had a new headless option that you can add.
So we want to be able to do the new way.
So they deprecated the old way to get people to use the new, more powerful.
And we're also linking to an article from Selenium, which is kind of a funny title.
So they wanted to get everybody's attention.
So they knew.
So they named the article headless is going away.
Yes.
Which is funny name.
And then subtitled it with, now that we have your attention, headless is not actually
going away.
Just the convenience method to set it in Selenium.
So I guess just a public service announcement.
If you're using Selenium, you got to change your code to use the new Selenium 4.8.
So that's it.
Oh, you're on mute.
So I am.
I do like it.
I wonder, though, why you have to pass the command line argument directly.
And it doesn't just look like, oh, you said headless.
That means in Chrome now pass --mode this verse.
Yeah.
Because it's almost the same, but not the same across the browser platforms.
Yeah.
I think it's because there's different.
I don't know.
I haven't looked through the explanation, but I think there's other options.
So it isn't necessarily just that they've changed the way you turn on headless, but there's
more headless options.
So they're just building it in so that you can pass in new flags.
And I think Chromium might end up getting more versions in later or something.
I don't know.
Yeah.
The browser space is an interesting time, isn't it?
Yeah.
We fought through the browser wars.
We've beaten back Internet Explorer 6, only to come back and have Chromium even more dominant
in certain ways.
It's interesting because for a usability thing, I'm usually using Vivaldi now, but I use probably
Vivaldi and Chrome for day-to-day use.
But for testing, yeah, it's still Chrome.
I use Chrome and Firefox.
That's what I use Firefox for is still testing with Firefox.
Yeah, absolutely.
Cool.
You know, just a bit of follow-up on the previous conversation about those different batteries
that you talked about.
I love our audience.
There's so much cool stuff going on over there.
So Blaze says, I wonder if Rich does anything with TQDM.
And if you want a definitive answer, how about Will McGugan in the audience says,
TQDM has a Rich output option.
Will obviously be in the creator of Rich and many other awesome, awesome things there.
So nice follow-up.
Awesome.
We've turned into the water cooler of Python.
We sure have.
All right.
I have one more thing to share with you all.
Let's jump into it.
And that is Gracie.
So Gracie's an interesting project.
It's a little bit like your first topic, Brian, in that it has a bunch of kind of utility
features.
And this one is around consuming APIs.
So not creating APIs, but writing clients that talk to them specifically around HTTPX,
which is one of the absolute go-to ACP libraries for doing sort of modern async style of APIs
in Python.
Right?
Yeah.
So Gracie, it says, Gracie, graciously manage your API interactions.
Gracie helps you handle failures, logging, retries, throttling, and tracking for all of your HTTP
interactions.
And it uses HTTPX under the hood.
And it lets you do the, like Gracie, do the boring stuff and you focus on your app is the
selling point here.
So this is pretty cool.
It's not super well-known.
It's got like 180 stars.
And it's an interesting library that has a lot of cool functionality.
It feels like it could use a little bit more polish, but it's still quite neat.
So let me give you some ideas here.
So what you do is it's basically you model your API interactions through a class structure.
It's not quite a hierarchy, but kind of use classes to come up with it.
So you can come up with an endpoint here.
And then you create something that derives from the API base class, right?
Give it a URL.
And then you give it a bunch of settings.
And the settings are where kind of the useful stuff is.
So for example, you can say, I would like to log the request as it's going out the door,
but only in debug.
I'd like to log the response and that one a little more frequently at the info level.
And then you can have a custom message that goes out there.
And you also can have a parser that will parse the response as a set of functions.
The first example you see here just says, by default, just given any object called .json on it.
All right.
Given the request called .json on it, right?
So that's kind of handy.
But what you can do if you go down a little bit, custom validators, is you can actually say, by default,
just try to convert it to a JSON response.
But if the status code is not found, then do something else.
And you can have a series of different status codes.
So if it, by default, use this parser.
But if it's like a 400 bad request, then we need to parse it as something else.
And that could even be like convert it from, you know, maybe in a success case, you get this particular, say,
pydantic model back.
But in an error case, you have a totally different structure.
And you might want to parse it differently into a different pydantic model, something along the line.
So you can do a lot of cool stuff like that there.
And yeah.
And then you just give it the functions that you call that basically invoke the API.
And of course, because it's based on HTTPX, you can await calling on those functions.
So yeah.
Anyway, it looks, it ends up with a pretty clean model for using it.
What do you think?
Well, yeah, it'll take some time to get your head around it because of the class thing.
But it's all stuff that you're going to have to develop anyway.
So having somebody else do the work.
So that's pretty good.
Yeah.
There's some nice examples of like throttling.
And this might be interesting to you, Brian, is it has the ability.
There's a bunch of different things.
It has the ability to replay certain data, right?
So you can also say we're only allowing certain, you know, by default, any 200 category status code is considered success.
You can say, no, for this one, it has to be a dot created, like HTTP status dot created, not 200 or something like that.
Or you can give it either okay or created, right?
You give it a set of options.
That's pretty cool.
You can add custom validation.
You talked about validators at your beginning as well.
And if you're not using Pydantic or something that kind of does its own custom validation, you can still even add more stuff.
Like not only does this have to be a string, but it has to be, I don't know, an email of this type or whatever, right?
Like of this, say the domain of our company, right?
Something like that.
So you can add these custom validators.
And it comes with a retry, built in retry for how do you handle the retries?
How many attempts?
What do you do in terms of logging?
You know, about retries and failures.
What do you do if, you know, you can say, I want to retry three times.
And if none of them work, I don't care.
Just keep going.
Don't break my application.
Or please do.
Don't, you know, raise an exception.
You might say, well, why would you ever not want to break it?
Like maybe you're trying to write to some sort of audit log to say this happened.
And if the server that just records what happened goes down, you don't want to start crashing your app, right?
There's like scenarios where you might not really care about that.
Also throttling, which is pretty neat.
You can say any time that the URL contains examples of Pokemon things.
So it has, you know, a regular expression for Pokemon.
I want maximum 10 requests for every one and a half minutes.
And then you could actually, it has a cool output too.
You know, if you print out just the rule it says, which is an object, it says 10 requests per 90 seconds for URLs matching this regular expression, which is kind of nice.
Oh, cool.
And yeah, the final thing.
Some, I don't really know where it is in here, but yeah, you can also have it throw certain exceptions.
So how, you know, how does that parser type scenario for different HTTP status codes I told you about?
So you can say, if it's a bad request, please throw, you know, some exception class that you come up with.
Right.
So instead of just saying bad request, it could potentially have more details.
You might build a parse information into it and then raise that exception.
There's, there's some pretty neat things.
And the final thing, by the way, rich integration right there, it requires you to install rich.
If you want fancy output on, it'll tell you sort of it's, it'll report on how it's interacted with the API endpoint.
So you've got to do like a bunch of processing, you know, I told you about like, I'm going to transform a bunch of things.
I use TQDM.
But if you're going to do that at the end, you could ask, well, how'd it go?
And it'll give you this like summary report of how much success and how much failure and what's the average latency and status codes and requests per seconds and all of these.
And it'll do that in text form or in rich style.
Final thing.
It will record and replay API interactions for testing purposes.
So if you want, you know, if it's really tricky to mock out some complex interaction, you'd say, well, I want it to be as exactly close to real as possible.
You could just one time do those API calls and then replay them back, put it either record mode or replay mode.
And the backend that stores that could be a SQLite database or a MongoDB database that's automatically integrated.
And you just give it that and say, when I talk to the server, remember what you did and store it over here.
And then you can play that back for testing.
Oh, wow.
Cool.
So, yeah.
Anyway, people can check this out and see what they think.
But I think it almost looks like it was a system pretty much designed.
Well, one of the obvious use cases is to build a custom thing to test your application.
Because there's a bunch like all the utilities there to really interrogate something.
Absolutely.
Yeah.
You get that report and you get the replay, record replayability, the logging.
Yeah.
A lot of that stuff is there.
It's pretty neat.
Yeah.
Cool.
Nice.
Well, that's it for that one.
Yeah.
I guess that's all of our items, isn't it?
It is.
And for extras, I don't have any extras.
Do you have any extras?
I do.
I just have one.
And then we'll get to our joke.
So, for the extras, do you know what, Brian?
Look at this.
Look at, here it is.
You got in the App Store.
Yay.
I got in the iOS App Store, too.
So, finally, finally, finally, the Talk Python mobile apps are out on all of the app stores.
So, go get them.
Just talkpython.fm/apps, I believe, will take you there.
Redirect over to the training site.
But, yeah.
They're available on iPhone, Android, tablets, iPad, Android tablets as well.
Maybe more coming.
We might have even desktop apps coming pretty soon, depending on how successful we are with all this.
But, yeah.
So, this is out.
People can check it out.
And as a way to celebrate finally getting this done after four months of work, first of all, wrote a blog post.
Maybe I'll add it, throw the link in the notes.
Yep.
I'll throw it in for people.
I talked about some of the design choices about how and why we chose things like Flutter and so on as the mobile app framework.
But the one thing for people to know out there, and this is a bit timely, is if you download and install the mobile app before, what day?
Today is Tuesday, May 16th.
If you do that before May 22nd, so download the app before May 22nd, inside the app only, the Up and Running with Git course, which is normally $39, is completely free to sort of celebrate the launch of the app.
So, you go in there, go find the courses, go to the free section, join the Git course, and you'll have it forever, not just for a little time.
But the only way to get it is to download and install the app, which is free, and then go put the Git course into your account.
I just downloaded it.
I'm opening it right now.
Awesome.
So, one of the things I'm excited about this is because, I mean, when I'm doing a course, like not giving a course, but learning from one, I do like to have it on my computer screen.
But there's often times where I've got like time to kill.
So, I'd like to sort of listen to some of the conversation and listen to it.
And yeah, I'm going to look at some of the stuff on my phone, but a lot of it is kind of following along, but I'm listening, and then I'll go through and watch the same stuff later on the computer and walk through it.
So, I really like this addition of having a mobile app.
This is pretty cool.
Thanks so much, Brian.
Yeah, there's a couple things why you might need it.
People are like, well, why don't you just watch it on the web?
Like, especially on iPhone, you can't get rid of that navigation section around the browser.
So, you end up watching like a postage stamp size thing, which is not ideal.
It won't auto-advance because ad companies are evil, and iOS blocked them from playing ads all the time, which, you know, gobbles up everyone else as well, unfortunately.
Okay, so on your app, it'll just jump to the next thing then?
Yeah, it just keeps playing smoothly, as you would imagine.
And then the other thing that's important is you can download content offline.
Like, if you're going on a trip or on the train, or some people even use it if they work at, you know, like government institutions that have high levels of security, and they want to, like, research labs and stuff.
If they want to be able to take the course at their work, but their work is super restrictive about what they can interact with, you could, you know, install, download a whole course onto your tablet, set it next to you, and watch it at your work.
Yeah, so those are the reasons why it exists.
But anyway, long time coming, super happy about it.
That's my extra.
Cool.
Download it, get the Git course.
All right.
Well, how about a joke?
Ah, this is a good one.
So, you may wonder, you may have friends who are like, Brian, you do Python, you do C++, you wrote a book on pytest.
Like, how did you get so good at this?
So, this kind of riffs on that theme.
There's two developers here.
First one, she says, how do you code so well?
The expert developer, she says, practice.
And the first person didn't really hear, like, it must be an innate gift, a gift from God.
It's practice.
I'll never understand how some people are so talented.
A mystery.
Practice.
Yeah.
Right?
Yeah.
What do you think?
Well, this is great.
And it applies to so many things, of course.
But one of my daughters is dealing with this right now.
She's been doing, for about a year, doing aerial silks, aerial arts.
And she's working on it and exercising and stuff every day.
And it was really hard at first.
And now she's pretty good.
And so many people have said, oh, you're just naturally talented at that.
It makes her mad because it's not natural.
I've had to work at it.
Coding as well.
So, obviously.
Obviously.
Yeah.
It's not just coding, but coding certainly.
Yeah.
Podcasting, writing blog posts, everything around what we do.
Practice.
Yeah.
Absolutely.
Practice.
Nice.
Nice way to end it.
So, good show.
Yeah.
Absolutely.
Very, very uplifting.
We'll end it on a growth mindset today, Brian.
Thanks for being here.
Thank you.
And thanks, everyone, for coming.