Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to
your earbuds. This is episode 173, recorded March 12th, 2020. I'm Michael Kennedy.
And I am Brian Okken.
And we have one of our favorite sponsors, Datadog, sponsoring the show. Check them out
at pythonbytes.fm.datadog. Get a cool t-shirt, get some cool software. Tell you more about that
later. You know, I think people who, when they're getting started, Brian, they write basic code,
but they realize that Python is so easy to use because you can just do something like
request.get. And hey, you have a website already. Like you've already downloaded like something from
an API or you downloaded something, some HTML or a file. And then it's easy to forget that
maybe there's more to it. Yeah, I actually totally forgot there was more to it. And
we know of requests as being both powerful and really easy to use. And there's just about a
million tutorials on how to put something together quickly with requests. So I was really thrilled to
come across this article by, I think it's Danny Hadovic, titled Advanced Usage of Python Requests.
He covers timeouts, retries, hooks, and more. And kind of that comment is it's easy to be immediately
productive with requests. But there's some really cool things that you can do that I had no idea you
could do this stuff. So some of the cool tricks that he runs down, actually all the tricks he runs
down, there's a hooks thing, you can attach a hooks to a session. And so you can use hooks to call
raise for status on every call. So raise for status is a way you can say, well, if when I request for
something, if a certain status comes back, then call this other function. But like a 404, there's certain
wildcard ones that you might want to like all the error ones you might want to always call
something. So there's a way to hook that up. That's pretty cool.
Oh, yeah. Nice.
Base URLs. I had no idea you could do this. This is so neat. Instead of doing like the full path to a URL
for every if you're doing a whole bunch of them, you can use a set up a base URL that essentially gets
prepended to everything else later.
Yeah. And this comes from request tool belt, which we covered before. So it like wraps and adds on to
requests, which is pretty awesome.
Oh, does it? Okay.
Yeah. Yeah. Yeah.
That's cool.
Yeah. And this is something I do all the time when I'm writing code that talks to consumes APIs or talks
to APIs or whatever is it's like, here's the base URL we set. And there's usually an if statement,
like, are you in development mode? Do this. Are you in production? Use this other URL as the base.
And then, you know, just do a slash, whatever it gets. But for me, it's always been.
Oh, that's a great use of base.
Yeah. For me, it's I've always been like, well, okay, get base URL plus whatever that I'm like.
Oh, yeah. Does it have the slash on the end or does it not have the slash on the end? Do I need
to put the slash on that? You know what I mean? And it's super clear here how that works. So I really
like this. Like I could totally have made use of this and I haven't been so I should.
And then he covers things like retries on failure, timeouts, timeouts with retries, a little bit of
testing and mocking of requests. But there's, you know, there's a lot about that.
Yeah. I'm looking at the section here on timeouts. I feel like, do you think people could get frustrated
about this or something? What's up with this guy in this card game, man?
I'm not sure.
There's like this, like under setting default timeouts. There's like a giphy of a guy just
beating a card table to death, right? Because he lost. It's playing Settlers of Catan and I'm
guessing he lost. Anyway, it's a great, great giphy to go along with it.
I'm not sure why that is relevant to here, but it's funny. It's a funny little video.
Why is it relevant? This is irrelevant to the podcast, but how is this relevant?
Well, it says it's great that stuff will just wait, but it can really frustrate you when someone's
forgot to set a timeout and it halts a program in production. So I'm guessing this
is an expression of being frustrated because production's lagged. Yeah. Yeah. Yeah. Anyway,
the one that I also thought was interesting, the base URL is cool, is the retry on failure is pretty
neat actually.
Yeah. And then you can even customize it to say, well, not, don't retry everything, but certain
things you want to make sure you retry.
Yeah, for sure. And how many times and whatnot. That's a great feature.
Yeah. And then this is kind of clever. The last one that I thought was clever was the,
if you're, you want to mimic being different browsers by adjusting the user agent header
request information, because, you know, sometimes you want to make sure that you're,
whether or not you're testing your stuff, you want to make sure that you can, deal with
different user agents. Or if you want to try to, if some people have, have a security stuff
on where they, they only allow certain user agents cause they don't want bots coming in. Well,
you can write your bot to be a different user agent then.
Yeah, absolutely. You could lie to them. Tell your, tell them your internet explorer six,
see what they think about that.
Yeah.
So this next item that I want to cover, I ran across last week and I thought it was really
cool. And I'm like, I've been looking for something like this for a while. So have you
heard of this library for testing? It's called pytest. Have you worked with it?
Yeah, a little bit. Yeah.
Awesome. So pytest is super clean and great. And you know, the way you express the assertions and
whatnot is you would, you want to assert something's not none. You would assert the thing is not none.
And like literally you write that code. And that's really cool for simple stuff like is not none.
But if you have to test multiple aspects of a thing, like it's not none, it is a number and it's
greater than zero, or it's between some range that can be a little bit tricky and you've kind of got to
write some code and it's not, not as descriptive or as expressive, right? That is one of the things
that I was kind of looking for is like a library that has more complex tests built in. Like, is this
thing a subtype of that? Or is this string parsable as an integer? I'd like to know that beforehand,
you know, like something like that. And I don't want to have to do the little, like the complicated,
small, but still somewhat complicated and non-obvious code in these assertions, right?
I'm also a big fan of fluent APIs. And what do you mean by fluent? I would love to see more fluent
APIs in Python, the standard library. So what I mean is a function or property that you can call
that returns the same object. Oh, okay. Sort of functional. Yeah, a little bit. So if I had say a
list, I could say list.sort.someotherlistoperation. If sort would return a list, the list that it was
called on, right? And maybe it had an order by, or I guess that's sort. If it had a filter and then you
could say dot, right? Transform or something like that, right? You could sort of chain these together
without doing multiple lines. Yeah. Right? So that's the fluent API. And so this thing I want to talk
about is something called fluent assertions. And I ran across this from Dean Egan and he was talking
about it on Twitter. And basically the idea is it has all these checks, like tons and tons of checks
built into it. I'm not sure how many, I would guess 50 different checks. Is that number complex? Is it not
complex? Is a value between these two things? Is this set? Is it a set? Is it non-empty? Does it contain
spaces? Does this string contain spaces? Is it shorter than this? Right? So you could say things
like, given an object, is it a string that contains no spaces? That is all lowercase. You could do that
as like one line, just like really clearly in this fluent API. And it throws assertion errors,
which are the same thing that you would get if you failed in assert directly. So it integrates really
well into things like pytest. That's pretty cool, right? Cool. Yeah. Yeah. So the example,
there's an example in the show notes here, and I took this from their docs. It basically says,
if you wanted to test something like given two parameters, I want to check that the value is not
none. It's an, that it's a float, that it's between zero and one, and you're given an object and it's not
none. And it's a type of something, right? That's kind of complicated. Or you can go to their API and say,
check n dot is not none dot is float dot is between. And I thought this idea is really cool,
but I don't like it so much. There's like too many words, right? So I've been working on a PR and it's
basically accepted. The guys running the project said, sure, this looks great. They've reviewed it.
And it's a simpler one and it uses properties a lot of the time. So it's a little more English,
less function he called. So you can say, is n dot not none dot float dot between parentheses zero and
one close parentheses. So the things that assert like, I guess, properties about it, like whether
it's a float or not none are just properties. And the stuff that takes arguments would be still
function calls. So I think it's a really clean way to write these test assertions in nice,
simple ways, especially if you're testing like a couple aspects, like it's not none and it's a float.
Yeah, I'm on the fence. I know that that a lot of people like this sort of a style. I like your
updates to it. I hope that goes in because that's a lot cleaner. It's a lot easier to read. So good
luck with that. I'm perfectly a fan. That sounded harsh. But no, I'm a fan of just the straight
asserts myself there. I think they're easy to read. But I think having a couple ways to do it,
that sounds neat.
What I like about it is it packages up some of the more complicated types of tests,
like the character only has, like the character, the string has some characters,
which are spaces or things like, like it's not as obvious if you've write the actual code that does
the test some of the times, right? I don't know. Anyway, that's what I like about these kinds of
APIs is the sort of English telling me what you're looking for more than the code of checking for it.
Yeah. And also you're fitting a whole bunch of stuff on one line. It's kind of nice too.
Yeah. Yeah. This is a way that you can have multiple assertions on one test without
people like, say, hey, you're doing more than one test. Well, sort of. Anyway, that's,
people can check that out. It's a pretty cool library. If that sounds useful to you,
it's already working for the check API and probably will for the is later. Cool. Cool. Now,
before we move on to the next, I want to tell you about Datadog because they're sponsoring this episode,
as they have been great supporters of the show. Let me ask you a question,
dear listener. Do you have an app in production that is slower than you like,
or its performance is all over the place? Sometimes it's fast, but sometimes it's slow
and you don't really know why. That's the most important thing is why is your app behaving this
way? Do you know what's causing it to be slow or to be kind of all over the place? And if you have
Datadog, you'll know. You can troubleshoot your app's performance with Datadog's end-to-end tracing,
use their detailed flame graphs to identify bottlenecks and latency in that finicky app of yours.
So be the hero that got your app back on track at the company and get started with a free trial at
pythonbytes.fm/Datadog. Very nice. Yeah. Thank you, Datadog. Good stuff. Also good stuff,
GitHub. Yeah, GitHub and GitHub Actions. I don't know how long they've been out of beta,
but GitHub Actions, I think, are available to everyone now. And these are different than their
webhooks, right? These are more automation workflow inside GitHub. Yeah. And GitHub's sort of by part of
the Azure Pipelines sort of stuff. I mean, GitHub's associated with Microsoft now. So Azure Pipelines
are one way to do actions on a, when you commit something or actions at an event time within GitHub.
But GitHub Actions are a way also. They're more of a lightweight pipeline thing. But for a lot of Python
projects, I think they're a very good, clean way to go because a lot of our Python projects are not.
You know, if you're building a package, they're kind of perfect. So there's an article called Python and
GitHub Actions. And it's by Hynik. And it's sort of really cool. He says he's currently recommending that
people use GitHub Actions for Python stuff. It's simple, easy integration. So how you do that?
And that's what this article is about. And he goes through running your tests through Tox,
using coverage, testing against multiple Python versions, and shows you the YAML that you have
to set up to configure GitHub Actions to do that right. And you also have to put a little bit of a,
make a change to your Tox.ini file to make sure this all works. And then I think it's good.
He was reminding people that if you've got an open source project, it's kind of nice to clean up
your old stuff. So if you've, if you are switching from some other CI system to GitHub Actions,
make sure to clean up the old stuff. And then he even goes through and tells some other things like
changes you want to make to make sure that you're hooked up to the CodeGov system and some other stuff.
And then making sure it builds on multiple operating systems, the sort of stuff you'd want to do with
continuous integration. Most of it is available through Azure or not Azure through GitHub Actions.
So that's kind of cool.
I love that it's all part of GitHub now. That's great. You know, it's, if you're already there,
it can be a pain to integrate these other systems. You can just push a few buttons or
few files and just make it run there.
Yeah.
Kind of the secret sauce is that they're really the same thing. I think the Azure Pipelines and the
GitHub Actions are all running on the same system, but the GitHub Actions is a little simpler interface
for people that aren't. I mean, Azure Pipelines is, and the Azure Workflow is powerful, but it is,
it is quite overwhelming when you get into it.
Yeah.
That whole system is pretty much all of Azure's overwhelming to me. I go there, I'm like,
why are there 400 things here? There's so many things. Like, I don't care about most of these.
What is this place?
Yeah.
Yeah. Yeah. Going back to digitalization.
Okay, cool. This next one I want to talk about touches on a sort of a similar topic as the request
one that you covered. As also the assertion one that I did. So one of the challenges of testing
your code can be when you're talking to external systems, right? I want to call the Stripe API and I got
to provide it all this information. And if you call it more than once with the same token,
it'll say, sorry, you can't do that. That token's used. So you got to go get a new token. And there's
just a lot of stuff. And if you necessarily want to be calling real live APIs inside your test,
that's going to make it quite slow potentially. And so on, right?
Yeah.
You can get charged. Like we use a geolocation service on the training site to figure out which
server to serve the video from. And that one, it's not super expensive, but we have to pay
per use. So I don't want to hammer it in continuous integration and pay more. So there's this cool
project called VCR.py. Have you heard of this?
Yeah.
Yeah. VCR.py is really cool. I heard about this from Tim Head, who was on Talk Python recently
for the binder project. And that's going to be out shortly. But the idea is that this simplifies
testing things that make HTTP requests as well as speeds it up. So all you got to do is the first
time you run it, you decorate the function. And what it's going to do is going to basically
instrument and record all the HTTP interaction, what gets sent out and then what comes back.
And it'll save that into a YAML file hanging around, which is called a cassette. Do you like
that?
Yeah.
Plug the cassette to the VCR, right?
And the next, the second and third and fourth time that you run the test, if it finds that
cassette file and the same inputs, it's like, well, you asked for this and here's your answer. And it
just replays it back to you.
Yeah. It's pretty clever.
Yeah. It's super clever. You don't have to worry about if the system is maybe slow or you've got to
set things up just right to call it because once it has that little cassette file, it's good to go.
So it lets you test these external service, test your integration with these external services
in a semi-realistic way because you're really playing back at least snapshot in time, real data
that you got from it without any effort on your part. Let's you work offline. Your tests are
completely deterministic. If it passes once, it's always going to pass because you always get the
same data back. You know, definitely speed up the execution speed because it's just throwing back
JSON that it has in a file rather than hitting an external service and all that. And if you decide,
you know what, this request is stale and out of date, all you got to do is delete that cassette
file, run your test once, it'll hit the real system and then it'll go back to playing the
new cassette.
Yeah. Nice.
Yeah. Yeah. Pretty cool. And then for people who do like pytest, there's a cool little plugin
called pytest-VCR. And then all you got to do is for your tests that might use something
like URL open or request or something, you just say pytest.mark.vcr for the test. And then that's
it. Magically, it works.
Yeah. Okay. Nice.
Yeah.
I don't know if it's really magical. I think it's very useful when it can be useful.
Yeah.
I personally don't have the experience, but I'm going to lean on somebody that does. On
episode 102 of Testing Code, I talked with Harry Percival. And part of that, we're talking
about how to set up testing an application that has external dependencies through, you know,
APIs and stuff. He does talk about both good experience with things like bcr.py and some
difficulties. Like if there needs to be timestamps or different, if you call an API twice and expect
to get something different back.
Right.
Well, this isn't going to help you. But so I recommend if you're running down this route,
then also listen to the half hour of episode 102 with Harry Percival. It'll help a lot.
Yeah. Super. We should put a link in the show notes for that.
Yeah.
To me, one of the things that I really like about this is I could go pip install Stripe or
pip install MailChimp or some other thing that I'm integrating with that who knows what kind of
complicated badness it's doing on the inside to make all of its stuff work. You know, and I don't have
to think about how am I going to mock out their internals? And if their internals change, how's that
going to affect my tests? And I can just say, I'm going to grab this higher level API that somewhere
deep in its guts does network traffic, throw this at it, and then it'll be reproducible. And to me,
that's the big appeal.
Yeah. And also speed. So like, even if you're using a test server and not incurring the, all the overhead
costs of, of the actual server, even with the test server, it's time and it's latency and you can speed
things up by caching the, the, the return values. So it's a cool idea.
Well, glad you like it. I know it's probably not something you do as
much in the hardware world, but yeah, it's a cool one. What else is cool? What else you got?
I got eight cool things. I'm a sucker for listicles. If there is actually good information.
And so Jeremy Grifsky wrote the eight coolest Python programming languages features. And I was just
smiling the whole time I was reading this. It's a quick article, but it talks about a whole bunch
of features. I was reading it thinking, man, this is why I love Python. And I really miss all this stuff
when I'm writing C++ code. So there's code examples of course, in, in this, in this article, but
we've got list comprehensions. And that's something that when you first learn about comprehensions,
it's like, Oh my gosh, this is so cool. We also have, dictionary comprehensions and
other comprehensions. Now you can have, there's all sorts of stuff. Generator expressions are nice
and really helpful. Slice assignment. I sometimes forget that we can do this. So you can take like
part of a list and assign, like if you have a three values or something, you want to stick in the middle
of a list, you can assign those with slice assignment. It's pretty, pretty powerful.
Iterable unpacking. So if you've got a tuple and you want to unpack that and pass that to a function,
but as separate values, you can do that. Negative indexing. I mean, you want to grab the last thing
off of an array. You can say minus one or minus two. I love negative indexing so much. That's a
really clever feat. It's so simple. And yet it's really nice to just go, I want the thing minus one
bracket minus one. Give me that. That's the one I want. I sort of, I want minus one. I've tried to do
this in C++. It's a bad idea. So negative indexing school chaining comparisons. So one is less than X is
less than five to make sure X is between one and five. That's not something you can do in most
languages. And it's just, it's how we talk. It's how we do math and Python has it. It's nice.
He finalizes it out with f-strings, which we love f-strings. And then a whole bunch of a big list of
honorable mentions. And I was thinking as I was reading this, like this entire thing, plus his
honorable mentions at the, at the other stuff of like things I could have also talked about,
that would be a really great, just like a, an introduction to Python course of just like,
here's a half an hour of why I think Python's awesome. So it's good.
I agree. I think that would be awesome. Let's see. As I look through this, f-strings definitely stand
out as something that's awesome. Negative indexing already riffed on that. That's cool.
Feeding comparisons, list comprehensions, and I'll throw all the expressions and other comprehensions
in there. I love them, but I wish they did a little bit more. Like why can't I sort a list expression,
list comprehension, or something like that? There's just a few things where it's like, oh man,
if I could just, I find myself a lot of times, here's the comprehension. And then here's the
little bit of things that I wish it could still support. I got to do afterwards. Anyway,
it would be nice.
Interesting.
Like for example, paging, right? I would like to be able to skip. So if I'm on page five of groups
of 20, I would like to be able to skip four times 20, take 20 with this clause sorted by that and have
the sorting happen before the paging. Like that would be so nice. And it's just, you know, it's on the
cusp. So half the time I'm like, God, I love these, but why can't I, you know, whatever.
Or the little thing that I wish I could extend it a little more database and memory type of
behavior, but still great, great stuff. I would totally miss them if they weren't there. That
is for sure. Yeah. Yeah. Cool.
Actually. So slice assignments, I would, I didn't never even occurred to me that you could do that
with a language. So it's neat that you can do that, but I probably wouldn't miss them if they
weren't there because I wouldn't have expected them to be there.
You shouldn't be able to do that. Yes, exactly. Exactly. I wouldn't miss light assignment at all.
Although it is neat. You know what I definitely would not miss is bugs in my web app.
Yeah. Me either. Do you have bugs in your web apps?
Well, I always think no. And then I learned yes, but not as often, not that often.
Right.
So there's this cool project called Bento that I just learned about. Have you heard of Bento before?
Just the lunch style?
Yeah, exactly. I do love a good Bento box, but this has nothing to do with that. Maybe the name is
inspired, but other than that, no. So Bento at bento.dev, this is basically an analysis system that will
look at your Flask and coming soon, other languages, Django, SQLAlchemy, whatnot. Look at your Flask app
or your request calls and look for known bugs, especially in the security realm. So you don't
end up with little Bobby tables. This is your school calling. Did you really name your table,
your son, dash, dash, semicolon, drop, whatever? Yeah. So what the idea is you can basically
pip install this thing. You call Bento init. It's going to create a Docker container with the
analysis tools. And then it's going to run those against your Flask app at the moment. Like I said,
Django, SQLAlchemy and stuff is coming along. And it'll find things like if you have a missing JWT
token or you're missing a no opener or the content set, or if you're using requests, if you're sending
username and passwords over an HTTP, not an HTTPS request, it'll automatically detect that and tell
you. Oh, very cool.
Yeah. It's pretty nice, right?
Yeah. It's very nice. It even does like Jinja template checking and stuff.
Right. Exactly. So this is open source on GitHub and yeah, you could check it out. It looks pretty
nice. There's a little Giphy. Thank you guys. Well done on your project. There's a Giphy showing how it
works right on the page. You can just go to the GitHub repo, scroll down a tad and just watch them
find a bug, fix a bug and so on. Yeah. Very good. Cool. Cool. Cool. Anyway. So yeah, if people have
for the moment Flask apps in the future, it looks like they're coming with other things, but yeah,
you can just run this and it'll check it out. There's also a list of all the checks that they
have, which I put into the show notes. So a bunch of stuff like, you know, some of the obvious ones are
like, did you ship Flask into bug mode and stuff, but other things that are not so obvious.
Yeah. That's it for our main topics, Brian. Do you have anything you want to share with folks
where we get to the laughter, the hilarity? No, we could use a good laugh, but do you have
anything extra? We haven't talked about it yet, but I feel like it's all the tech conferences are
either canceled or they're on coronavirus watch before being cut, right? Like we've had E3,
the big game one canceled. We have the game developer conference. We had South by Southwest.
canceled some other ones. Some, I think some Facebook's F8, I think was canceled. Possibly
Google IO. I'm not sure about the one. I don't remember exactly what they said, but
PyCon's still on for the moment at the time of this recording. There's going to be an announcement
tomorrow, which may change things, but just what, I just want to remark, like what a crazy time,
both for the world, but also for tech.
Yeah, definitely crazy. I'm curious what it's going to do for, now this is totally self-centered.
I do want everybody to be healthy, but I also wonder with less people commuting,
if less people are listening to podcasts, that would be terrible.
Oh no, they've got a lot of housework. You all folks out there, you definitely got to keep
listening. Got it. No, that's, that's actually a legitimate question. Whether or not that makes
sense. Yeah.
Maybe it does. Here's some of the effects I think are going to happen. I think a lot of companies,
especially larger companies that believe you must have a meeting every Wednesday and it has to be
like two hours with this group and an hour of that group. And you must commute every day into the
office, even though you just work and through GitHub and Slack and email anyway, they're going to realize,
you know what? We don't actually have to have these big offices and we don't have to have our people
always come in. We were at least as well off and what's going to be like the work life change that
comes from that realization.
Yeah. We made sure, I mean, a year ago today, it was rare for anybody to be, to be working from
home in our company, even though the work is software for the most part. But now we've made sure that
everybody, I think we have out of like 80 people, something like that, we've got like only a handful
of people that are not set up yet to be able to work remotely. I think that's a good change actually.
I do too. It's going to be really interesting to see the knock-on effects. I think there's going to
be stuff like that, like, oh wait, we actually could work in this way or we could hire people
from other places or whatever. It's going to be interesting outside the just, you know,
the potential chaos of people getting sick and whatnot.
This has been like talking about the elephant in the room, but it's talking about the virus in the room?
Yeah. Well, I hope it's not in either of our rooms. Let's just put it like that. Like right now.
Yeah, definitely not. There's nobody else in my room. I think I'm clean.
So speaking of mysteries, how about I tell you a joke?
Oh, please.
Okay. So let me give you a definition straight out of the dictionary. Debugging. Pronounced
debugging. It's a verb. Primary definition. Being the detective in a crime movie where you are also the
murderer. Yeah. That's a good one, right? Yes. You probably get that less since you're
often coding solo, but there's times where I'm like just hot under the collar, mad about
some bug in the system and trying to figure out who did this.
Dude, it's time for some version control blame, some get blame, some version blame.
Yeah.
No.
It was me.
Yeah. I've been there. I've been there like, oh my gosh. Now, now what do I do?
Definitely.
Yeah.
Love that.
It's good. It's good for your humility level though, right?
Yeah. And then you just remember that next time you're mad at somebody else.
Exactly.
Or doing something boneheaded.
Awesome. Well, thanks for being here. Great to chat with you as always.
Thank you. Bye.
Bye.
Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's
Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news
item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout
for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy.
Thank you for listening and sharing this podcast with your friends and colleagues.