Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to
your earbuds. This is episode 185, recorded June 4th, 2020. I'm Michael Kennedy. And I am Brian
Arkin. And this episode is brought to you by Datadog. More on that later. Check them out at
pythonbytes.fm/Datadog. Brian, I feel like we're all working from home. Everyone's life is
scrambled. Even like my sleep schedules are scrambled. Like some crazy stuff happened and
I slept from like 6 to 9.30 and I was up for like four hours and I slept in. Like it's just,
it's weird. Don't we need more structure in our life? Nice, nice intro. Yes, more structure.
Yeah. I'm a fan of Markdown also. Believe it, trust me, it's not a tangent. Though we have a,
just a repo that we want to point people to called Myst. It's got to be called Myst, don't you think?
Oh yeah, definitely. M-Y-S-T, which is markedly structured text. And what this is, is a fully
functional Markdown parser for Sphinx. It's Markdown plus a whole bunch of stuff from restructured test,
restructured to text. So Myst allows you to write Sphinx documentation entirely in Markdown.
And things that you could do in restructured text, but could not do in Markdown have been put in a,
there's a new flavor of Markdown. So you can do all of your directives and all sorts of cool things,
like anything you could do in restructured text with Sphinx you can now do in Markdown.
It's based on CommonMark and some other tools. So they're standing on other tools that are already
doing things really well and just extending them a bit. But this is pretty powerful. One of the things
I like about this is I particularly don't use a lot of Sphinx, but this also includes a standalone
parser so you can see how somebody's extended Markdown for these extra directives and even use some of
them in your own code if you want. Yeah, this looks really, really nice. Like restructured text is good
and all, but I don't know. If I'm going to write something like restructured text, my heart just wants
to write Markdown. I got to tell you. Yeah, me too. And I think one of the things that was holding a lot of
people back is some of the extra directives, the information boxes and other things like that,
that you can't necessarily do in Markdown off the shelf, but some extensions are nice. I played with
it a little bit doing some just, I didn't pull it down with Sphinx. I just pulled it down so that I
could run some Markdown through it and some of the extra directives to see what it has. So for instance,
some of the directives, like I tried like an information box, you can have structure around
putting an information box somewhere. And what you end up with is a div that has a class to it.
Oh, nice. If you're not using Sphinx, then you'll have to use your own CSS, I guess, to style it,
but it puts in enough hooks for you to be able to do that. That's really nice. I do wish you could
sort of indicate CSS styles and Markdown because, wow, that would just, that would be the end of what
you need HTML for, for many, many things. That would be nice. So last week you brought up
dir-inf. We were talking about how do you store your secrets? How do you activate and configure
different environments? I think I even said something about like specifying where Python was running. I
don't remember what the context was exactly, but you're like dir-inf. And actually I've been meaning
to cover this. Dunderdan, LinkedIn on Twitter, don't know what his last name is. Thanks, Dan.
sent this over to us as a recommendation. And I'm like, yeah, like you brought it up. It seems
definitely cool. So let me tell you about dir-inf. D-I-R-E-N-V. So it's an extension that goes into
your shell. And normally what you do is you open your shell and it runs your bash RC, ZHRC, whatever,
and sets up some stuff. Or if you're over on Windows, it works a little bit different, but I
think dir-inf is only for the POSIX type systems. Anyway, it'll set up some values that you put in
there like environment variables and whatnot. And that's just global, right? You can also set up when
you activate a virtual environment to export other values. That's pretty cool. But what it doesn't really
do is allow you to have like a hierarchy of values. So if I'm in this subdirectory over here,
I want this version of Python active or this version of where the Flask app lives. And then if I change
to another directory, I want it to automatically go, well, that means different values. And dir-env
basically does that.
Yeah. So as you go into different parts of your folder system, it'll look for certain files,
.envrc. And if it finds that, it'll automatically grab all the, basically all the exports and then
jam them into whatever your shell is. And it's also cool because it's not a shell, right? It's not like,
well, here's a shell that has this cool feature. It works with bash, ZShell, TCShell,
fish, and others, right? So it's basically a hook that gets installed for, like I use,
oh my, ZShell because, oh my gosh, it's awesome. And then I would just plug this into it. And as I
do stuff with ZShell, it will just apply its magic.
Yeah. And so one of the things that, one of the things you can do with this is to automatically
set a virtual environment. If you go into special directories, that's not the only thing it can do,
but that's one of the reasons why a lot of people use it.
Right. You basically, well, I guess you can't do aliases. You can't change what Python means,
but you can say where the Python path is. Yeah.
Yeah. And that's one of the things that's a limitation of this that people should be aware
of is it doesn't, the way to think of it is not as a sub RC, right? It's not a sub bash RC where
like it runs aliases and all sorts of stuff. The way it works is it runs a bash shell,
like a little tiny hidden bash shell. It imports that as the bash RC and it captures what the
exported variables are, throws away that shell, and then jams that into whatever active shell you
have, like ZShell or bash or fish or whatever.
Yeah. I would probably use this all the time if I wasn't somebody that used both Windows and
Mac and Linux frequently.
You know, probably, I bet somebody could come up with this thing for Windows as well. It's just
got to be like totally from scratch, different type of thing, right?
People have already pointed me to Windows versions of it, but it's one of those things of like,
you got to jump through hoops to make it work. And it's just not, for me, it's not solving a big
enough problem that I have that I need to jump through the hoops. But I agree. I agree. It is
cool, but it doesn't, it's not like life changing in that regard. I guess one more thing to point out is
it's, you don't have to like go to the directory where the environment RC file is. It looks up the
parent directories until it finds one. So you have this like hierarchy, like I'm down here in the,
you know, like views part of my website and the top level of that git repo, I have one of these
EMV RCs. It would find that and like activate that for you. So that's pretty cool that it has,
it's kind of like node JS where the node modules live in that regard. That's pretty cool.
Yeah. That's a really nice feature. Yeah, for sure. Also nice data dog. So before we get to the next
thing, let me talk about them real quick. They're supporting the show. So thank you. They've been
sponsors for a long time. Please check them out and see what they're offering. It's good software and
it helps support the show. So if you're having trouble visualizing bottlenecks and latency in your app,
and you're not sure where the issues are coming from or how to solve it, you can use data dogs,
end to end monitoring platform with their customizable built-in dashboards to collect
metrics and visualize app performance in real time. They automatically correlate logs and traces
at the individual level of requests, allowing you to troubleshoot your apps and track requests across
tiers. Plus their service map automatically plots the flow of these requests across your application
architecture. So you can understand dependencies and proactively monitor performance of your apps.
So be the hero that got that app at your company back on track.
Get started with a free trial at pythonbytes.fm/data dog. You can get a cool shirt.
All right, Brian, what's next? Yep. Thanks data dog.
I had a problem. So my problem was a little application that had a database. It was a,
I was using tiny DB just for development. You could use Mongo similar. It's a document database,
thrown some data into it, no problems. But I, that was one of the values that I decided to change to
use Python enums because I thought enums are cool. I don't use them very often. I'll give these a shot
because they seem like perfect. And then everything blew up because I can't, couldn't save it to the
database because enums are not serializable by default. So I'm like, there's got to be an easy
workaround for this. And, and I first ran around, ran into questions about, or topics about creating your
own serializer. That just didn't seem like something I wanted to do.
You could do it, but it's not so fun, right?
Yeah. Well, so I ran across an article, a little short article written by Alexander Holtner
called convert a Python enum to JSON. And I didn't need it converted to JSON, but I did need it
serializable. And the trick is to just, if you're, you're doing your, when you use enums, you,
you do from enum import the capital enum type, and then you have a class that derives from that.
And then you have your values. Well, if you also derive from not just enum, but another solid,
a concrete type, like, like int or string. And in my case, I was using, I used string so that my
string values would be stored. Now it is serializable and it works just the same as it always did before.
It's just, it uses the serializer from the other type and it just works incredible. So for instance,
I'm, I'm going to put a little example in the show notes about using a color, which is red and blue.
And if you just, you derive from enum, you can't convert it to JSON because it's not serializable.
You can either do an int enum, which is a built-in one or a combine a stir and enum. Now it serializes
just to the string red and blue, if that's the values. And then that's what's stored in your,
like your database too. So when I'm using, it's really handy for debugging to be able to have
these, these readable values as well.
Yeah, this is really cool. It's a little bit like abstract based classes versus concrete classes or
something like that, right? You've like the sort of general enum, but if you do the int enum,
then it has this other capability, which is cool. Or yeah. Multiple inheritance,
stir comma enum is the one you went for, right?
Yeah. So the multiple inheritance is the thing that Alexander recommended in his post. That's what
I'm using. It works just fine. But I was interested to find out that in the Python documentation for
int enum, int enum is almost just there as an example to say, we realize that it might not be
integers that you want. You might want something else, but there's an example right in the,
in the Python documentation on, on using multiple inheritance to create your own type. It doesn't
talk about serializability there, but that's one of the benefits.
Yeah. It seems like it works anyway. Awesome. How much time did it take you to figure that out?
Was it a long time?
No, I don't know. 10 minutes of Googling.
Yeah, that's pretty cool. Well, you could compute it with Python, of course, but you know,
the daytimes in Python and time spans, they're, they're pretty good actually, but they're a little
bit lacking. There's certain types of things you might want to do with them. And so there's a
couple of replacement libraries and one that Tucker Beck sent over. It's called pendulum.
That's pretty cool. Have you played with pendulum?
I haven't, but I like the name.
Yeah, I do too. It's, it's really good. I've played with arrow. So this is a little bit like
arrow, but it doesn't seem like it tries to solve exactly the same problem. It's just like,
let's make Python date times and time deltas better, which is kind of the goal of both of them.
So it's more or less a drop in replacement for standard date time. So you can create like time
deltas, which are pretty cool. Like I could say pendulum dot duration days equals 15. I have this
duration and it has more properties than the standard date time or the time delta. You know,
you get like total seconds or something like that, but that's, you know, that's not that helpful.
So this one has like duration dot weeks, duration dot hours, and so on, which is pretty cool. You can ask
for the duration in hours, like the total number of hours, not just the number of hour, you know,
like three hours and two days or whatever. But you also have this cool, like human friendly
version. So I can say duration in words and give it a locale and say like locale is U S English.
And it'll say that's two weeks in one day. Nice. You can also like, let's suppose I'm trying to do
some work with like calendars or some kind of difference. I say the time from here to there,
I want to do something for every weekday that appears. Right. So skip Saturday and Sunday.
But if it's like from Thursday to Wednesday, I need to go Thursday, Friday, Monday, Tuesday, Wednesday.
Yeah. So I could say pendulum dot now, and then I could go from that and subtract three days. So that
would be a period of three days. And that gives you what they call a period, which is a little bit
different. And then I can go to it and say, convert yourself to in weekdays. Okay. Right. Not interesting.
Then you can loop over it. You can say for each day or each time period in this period and go,
it would go, you know, over the weekdays that are involved in that time span.
That's pretty cool.
Yeah. Cause that would not be so much fun to do yourself. Right. There's a bunch of stuff that
it does. And I don't want to go like read all the capabilities and whatever, but that gives you a
sense. Like if these are the kinds of problems you're trying to work through and you're like,
man, this is a challenge to do with, with a built-in one. Check out pendulum. Also check out arrow.
I think we've covered arrow a long time ago. If we haven't, we'll, I'll cover it at some point.
It's a good one. Yeah. And I think actually, I don't think that's a matter of which one's the
best either. It's a, it's whatever seems to speak to you and, and, and has an API that
thinks like you do. Yeah. It's good that lots of people have solved things like this.
Yep. Absolutely. All right. Well, what's this next one? I'm trying to be like a private detective
or what's going on with this? Yeah. Private detective looking into and spying on your code.
So this was sent off by a Twitter account called PyLang and this is PySnooper. The claim is never
use print for debugging again. And I have to admit, I am one to lean on the print statement every once
in a while, especially if I'm just, sometimes I don't really want to do a use breakpoint because I,
I've got some code that's getting hit a lot and I really do want to see what it looks like over time.
So one of the things that people often do is throw a print statement somewhere in a line just to say,
Hey, I'm here. The other thing they do is like print out a variable name right after an assignment so
that they can see when it changes, but that's exactly.
It was this and now it's that.
Yeah. So this is exactly kind of what it does. So by default, it's just a, you can throw a decorator
onto a function and that's the easiest way to apply it for PySnooper to create a function.
And now every time that function gets run, you get a play by play log of your function.
And what it logs is it logs the parameters that gets past your function. It logs all the,
the output of your function, but also every line of the code of the function that gets run.
And every time a variable is changed, changes its value. And then even at the end, it tells you the
elapsed time for the function. So that's quite a bit. If that's great for you, great. But if it's
too much information, you can also isolate it with a width block and just take a section of your
function under test and just log a subset. And then if a local value, local variables are not enough
and you're changing some global variable, you can tell it to watch that as well. Anyway, it's a pretty
simple API and there's actually quite a few times. I think I'll probably reach for this.
When I first saw this, I'm like, ah, yeah, it's kind of cool. There's a lot of these replacements
where I think like, you know what, you've got PyCharm or you've got VS Code, you're better off just
setting a break point. And the tooling is so much better than like, say, PDB or something
like that, right?
Yeah.
This though, this solves a problem that always frustrates me when I'm doing debugging, which
is you're going around, you've got to keep a track in your mind. Okay, this value was that,
now it's this, and then it became that. And like sort of the flow of data, like at any frozen
point, you can see really well with the visual debuggers, right? Like PyCharm or whatnot,
what the state is, you can see even what's changed, but like this number of way, this list
was empty, empty, then this was added, then this was added. And here's how it evolved over time.
People should check out the read me for this because that view of it is like, there's a loop
where it shows going through the loop four times. And as like all the values and variables like build
up, so you can just like review it and see how it flows. I think it's pretty sweet actually.
Yeah. One of the other things that I forgot to mention is, is if you're like debugging a process on a
server, maybe it's a, you've got a, yeah, small service that's running and instead of standard out,
you can pipe these logs to a file and, you know, review them later.
Yeah. For definitely for a server as well, it would be nice to flip that on. And I guess with the,
with the conditional, but you could probably even in code say, do you feel like you're running into
trouble? Turn on the PySnooper for a minute and then turn it out. You know, like there's,
there's probably options there, but yeah, you definitely wouldn't want to attach a real debugger to like
production. Dude, why wasn't the site work? Oh, somebody's got to go back to their desk and hit F,
you know, F5 or continue or whatever. Yeah. That's not going to go well. So I have something that's a
pretty similar to follow this up with that's, you know, this is about debugging and seeing how your code
is running. Like per usual, we talk about one tool and people are like, oh yeah, but did you know about,
so we've talked about Austin and we've talked about some of the other cool debugger profilers.
And so over on PyCoders, they talked about Phil, F-I-L, which is a new memory profiler for data
scientists and well, general scientists. And you might wonder like, why did data scientists,
right? You know, biologists, why can't they just use our memory profile? Like why is Austin not their
thing? Right. And it may or may not be like, it may answer some great questions for them. Like
obviously they do a lot of computational stuff, making that go much faster, faster to let them
ask more questions. Right. So maybe profilers in general are like things they should pay attention
to. But you know, when they talk about this, they say, look, there's a really big difference between
servers and like data pipeline or sort of imperative, just top to bottom code. We're just going to run
scripts sort of right. And that's what scientists and data scientists do a lot. So like, I just need to do this
computation and get the answer. So with servers, if you're worried about memory, remember, this is a memory
profiler. What you're worried mostly about is, you know, this has been running for three hours. Now the server's out of
memory. That's a problem, right? Like it's, it's probably an issue of a memory leak somewhere. Something is hanging on to a
reference that it shouldn't. And it like builds up over time, like cruft and it just eventually wears it down.
And it's just like bloated, you know, with too much memory. Right. So that's the server problem. And I think
that's what a lot of the tooling is built for, but data pipelines, they go and they just run top to bottom
and they don't, for the most part, don't really care about memory leaks because they're only going to run for 10
seconds. But what they need to know is if I'm using too much memory, what line of code allocated that memory?
Like I need to know what line where I'm using too much memory and how can I like maybe use a generator
instead of a function in a list or something like that. Right. So that's what the focus of this tool
is, is it's like, it's going to show you exactly what your peak memory usage is and what line of code
is responsible for it. This is actually pretty cool. It is right. At first I thought, what is this? Like,
why do they need their own thing? But as I'm looking through, I'm like, yeah, this is actually pretty cool. And
if you go to the site, you can actually see they give you this graph, like a nice visualization of
like, here are the lines of code. And then it's like more red or less red, depending on how much
memory it's allocated. Oh wow. Yeah. And then the total amount and you can like dive into like,
okay, well I need to see like this loop or the sub function that I'm calling. How much is it? So you
can like navigate through this visual, like red, pink, gray of like memory badness, I guess. I don't know.
Memory usage. Yeah. It's not bad, right? No. Yeah. And when you're staring at code,
it's not obvious where the huge array might get generated or used. Yeah. And the example they
have here, it's like, okay, well they have a function called make big array. Okay. So like
probably you might look there and there's also things like, like using numpy, like, okay, here we're
creating a bunch of stuff with numpy and you might say, well, here's the numpy thing that we're doing
that makes too much, but you could be doing like a whole bunch of, you know, numpy and pandas work.
And like one line is actually responsible, but you're probably pretty sure it has to do with pandas,
but you're not sure where exactly. Right. So you could, you know, dig into it and see, I think it's
cool. Yeah. We thought we were using arrays and suddenly we have this huge matrix that accidentally
exactly. Why is all this stuff still in here? Yeah. Yeah. Cool. Well, anyway, if you're doing data
science and you care about memory pressure, this thing seems super easy. It even has like a try it on
your own code on the website, which I don't know what that means. So that's crazy.
Not uploading my code there, but it's fine. All right. Well, Brian, that's it for our main items.
You got anything? I don't. I've just been trying to get through the day lately. Yeah. I hear you.
Well, I have one really quick announcement and then an unannouncement in a sense. So I sent out a message
to a ton of people. So unannouncement is for them. So what I'm trying to do is I'm trying to create some
communities for students going through the courses to go through them together. And I'm calling these
cohorts. Right. So I set up like a beginner Python cohort in a web Python cohort and put like 20 or
30 people. I had 20 or 30 slots, let's say for people to go through over like three or four,
three months or so, where they each work a little, like they all work on the same part of the course
at the same time. And they're there to help each other. There's like private Slack channels and
other stuff around it. So that's really fun. But it turns out that after one day of having that open,
I got many hundreds of applicants for like 20 spots. So I had to stop taking applications. So
if people got those messages and like, Oh, I want to apply, but it looks like the form is down.
It's because there's like an insane number of applicants per spot. So those will come back
and people can sign up to get notified. There's a link in the show notes, but I just want to say like,
that's what I was doing, which is fun. But for those of you who didn't get a chance to apply,
cause it got closed right away. That's why.
And that's for training at talkpython.fm.
Yes, exactly. So there's like certain courses. And if you got one of the courses and you want to go
through it with a group of students all on the same schedule, this was like a free thing that I was
doing to try that out.
Yeah.
Right.
I think it's a neat idea.
Yeah. Thanks. Yeah. People seem to like it.
Yeah. Too many.
But yeah, I've got to give it a try, get it dialed in, then we can open up some more groups.
Yeah.
All right. Well, I've got it. I've got a joke. I kind of like for you here.
I love this one.
Are you ready for it?
Yeah.
You want to be, why don't I be the junior dev? You can be the senior dev. So the junior dev and
senior dev are having a chat. And I feel like that you may be a little skeptical of what I've done
here. Let's just do this. All right. Why don't you hit me with a question?
Okay. So where did you get the code that does this? Where did you get the code from?
Oh, I got it from Stack Overflow.
Was it from the question part or the answer part?
Isn't that so good? It's like people say copy from Stack Overflow is bad. I think this is the
real question.
You definitely don't want to copy from the question part.
Yeah. But actually, I've never heard anybody like, you know, spell that out. You know,
you can look up stuff on Stack Overflow, but at the top with the question, don't copy that. That's
the code that somebody's saying this doesn't work. Yeah.
Exactly. Exactly.
That's funny.
Yeah. This is a good one.
It's too funny.
It's too funny. All right. Well, thanks as always. Great to chat with you and
share these things with everyone.
Thank you.
Yeah. Bye-bye.
Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S.
And get the full show notes at pythonbytes.fm. If you have a news item you want featured,
just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something
cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and
sharing this podcast with your friends and colleagues.
Thanks.