Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to
your earbuds. This is episode 220, recorded February 10th, 2021. I'm Michael Kennedy.
I'm Brian Okken.
And we have a special guest, Hannah. Welcome.
Hello.
Hannah Stepnick, welcome to the show. It is so great to have you here.
Thank you. I'm happy to be here.
Yeah, it's good to have you. It's so cool. The internet is a global place. We can have
people from all over. So we've decided to make it an all Portland show this time.
We could do this in person, actually. Well, not really, because we can't go anywhere. But
theoretically, geographically, anyway. Yeah, so all three of us are from Portland, Oregon. Very nice.
Before we jump into the main topics, a few quick things. One, this episode is brought to you by
Datadog. Check them out at Pythonbytes.fm.datadog. And Hannah, do you just want to give people a quick
background on yourself?
Yeah, so I'm Hannah. I have written a book, which is weird to say, about pandas. But I also just go
around, like, give talks at various conferences, like on Python. So yeah, like I gave re-architecting
legacy code base recently.
That sounds interesting and challenging.
Yeah.
What was the legacy language? Was it Python or something?
It was Python. It was like a Flask web application. And then also the front end of it was Vue, like
Vue.js.
Oh, yeah.
So yeah, that's been a fun project. That was through work as developers. Like, you're pretty much always
working with some form of legacy code. Just depends on how legacy it really is.
Well, what could be cutting edge in one person's viewpoint might be super legacy in another, right?
Like, it's Python 3.5. You wouldn't believe it.
Right.
Yeah. Very cool. Well, it's great to have you here. I think maybe we'll start off with our
first topic, which is sort of along the lines of the data science world, some tie-ins to your book.
And of course, whenever you go to JetBrains, you've got to run your CLI to accept the cookies,
which is fantastic. And so this topic, this first topic I want to cover is from JetBrains. And it's
entitled, we downloaded 10 million Jupyter notebooks. I almost said 10,000. 10 million Jupyter notebooks
from GitHub. Here's what we learned. So this is an article or analysis done by Elena, who's a Harina.
And yeah, pretty neat. So they went through and downloaded a whole bunch of these notebooks and
just analyzed them. And there's many, many of them are publicly accessible. And a couple of years ago,
there were 1.2 million Jupyter notebooks that were public. As of last October, it was eight times as
many. 9.7 million notebooks available on GitHub. That's crazy, right?
Wow.
Yeah. So this is a bunch of really nice pictures and interactive graphs and stuff. So I encourage
people to go check out the webpage. So for example, one of the questions was, well, what language do you
think is the most popular for data science, just by judging on the main language of the notebook?
Anna, you want to take a guess?
Oh yeah. Python, for sure. Without a doubt.
That's for sure. The second one, I'm pretty sure no one who's not seen this, there's no way they're
going to guess. It's Nan. We have no idea. Like we look, we can't tell what language this is in there.
But then the other contenders are R and Julia. And often people say, oh yeah, well, Julia,
maybe I should go to Julia from Python. Well, maybe, but that's not where the trends are. Like
there's 60,000 versus 9 million, you know, as the ratio, I don't know what that number is,
but it's a percent of a percent type of thing. Wow.
They also talk about the Python two versus three growth or different. So in 2008, it was about 50%
was Python two. And in 2020, it's a Python two is down to 11%. And I was thinking about this 11%.
Like, why do you guys think people, there's still 11% there hanging around?
I mean, I would guess, speaking of legacy applications, probably it's just hasn't been
touched, but yeah. Yeah. Those are very likely the ones that were like the original 2016, 17 ones that
were not quite there. They're still public, right? GitHub doesn't get rid of them. The other one is I
was thinking, you know, a lot of people do work on Mac or maybe even on some Linux machines that just
came at the time with Python two. So they're just like, well, I'm not going to change anything. It just,
I just need to view this thing. I don't have Python problem solved, right? They didn't know
that there's more, more than one Python. There's a good breakdown of the different versions. Another
thing that's interesting is looking at the different languages, not language, different libraries used
during this. So like NumPy is by far the most likely used. And then a tie is pandas and matplotlib,
and then scikit-learn, and then OS actually for traversing stuff. And then there's a huge long tail.
And they also talk about combinations like pandas and NumPy are common, and then pandas,
and then like seaborn, scikit-learn, pandas, NumPy, matplotlib, and so on as a combo. And so
that's really interesting, like what sets of tools data scientists are using. Yeah. And then another
one is they looked at deep learning libraries and PyTorch seems to be crushing it in terms of growth,
but not necessarily in terms of popularity. So it grew 1.3 times or 130%, whereas TensorFlow is more
popular, but only grew 30% and so on. So there's a lot of these types of statistics in there. I think
people will find interesting if they want to dive more into this ecosystem. You know, it's one thing
to have survey and go fill out the survey, like ask people, what do you use? You know, what platform do
you run on? Vue.js or Linux? Like, okay, well, that's not really a reasonable question, but I guess
Vue.js, you know, like, but if you just go and look at what they're actually doing on places like
GitHub, I think you can get a lot of insight. Yeah, for sure. Yeah, I know I use, like I'll go to GitHub
pretty frequently, like at work when I'm, you know, just like browsing, like, I wonder how you do this
thing or like, what's the most common way to do this? Or yeah, absolutely. There's just look up,
like, what's the most popular. So it's a pretty good sign if a lot of people are using it.
It is. One thing I should probably make more better use of is I know they started adding dependencies,
like, oh, if you go to Flask, it'll show you Flask is used in these other GitHub repos and stuff.
Like you could find interesting little connections. I think, oh, this other project uses this cool
library. I know nothing about, but if they're using it, it's probably good. Yeah, for sure.
Yeah. I love the dependency feature of looking who's using it. It's neat. Yeah, absolutely. So,
Brian, you going to cover something on testing this time? Yeah. Will we make you?
I wanted to bring up something we brought up before. So there's a project called pytest Python Path,
and it's just a little tiny plugin for pytest. And we did cover it briefly in way back in episode 62,
two. But at the time I brought it up as, so, okay. So the, I brought it up as a way to, to,
to just shim, like be able to have your test code, see your source code, but as just like a shortcut,
like a stop gap until you actually put together like proper packaging for your source code. But the
more I talked to real life people were testing all sorts of software and hardware, even there's,
there that that's a simplistic view of the world. So thinking of everybody is working on,
on packages is, is not real. There's applications for instance, that, that they're never going to set
up, pull their code together as a package. And that's, that's, that's legitimate. So if you have an
application and your, your source code is in your source directory and your test code is in your test
directory, it's just, your tests are just not going to be able to see your source code right off
the bat. So what's more, tricky is depending on how you run it, they will, or they won't.
Yeah. Right. Right. If you say run it with PyCharm and you open up the whole thing and it can like put
together the past, you're all good. But if you then just go into the directory and type pytest, well,
maybe not.
It doesn't work. And it just confuses a lot of people. And so more and more, I'm recommending people
use this, this little plugin and really, the, the, the big benefit is it gives you there's,
there's, there's a, it does a few things, but the big biggie is just, you can add a Python path,
uh, setting within your pytest, any file, and you stick your, any file at the top of your project.
And then you just give it a relative path to where your source code is like source or SRC
or something else. And then pytest from then on, we'll be able to see your source code.
It's a really simple solution. It's just, I, I, that's way better than what I do.
I don't think it's a stop gap. I think it's awesome. So yeah, I totally agree. What I do a lot of times
is certain parts of my code. I'm like, this is going to get imported. For me, the real tricky thing is
a limbic, the database, database migration tool and the tests and the web app. And usually I can get the
tests and the web app to work just fine running them directly. But for some reason, a limbic always
seems to get weird, like working directories that don't line up in the same way. So it can't import
stuff. So a lot of times I'll put at the top of some file, you know, go to the Python path and add,
you know, get the directory name from dunder file and go to the parent, add that to the Python path.
And now it's going to work from then on basically. And, this seems like a nicer one, although it doesn't
help me with the limbic, but still, but it, it might, you might be able to add the limbic path right to it.
So yeah, yeah, for sure. Very cool. So it says, yeah, go ahead, Hannah.
Oh, I was just going to say, yeah, like this is something I like pretty much every time I set up a new project.
Like I always have to screw with the Python path. I always like run it initially. And then it's like, Oh,
can't find blah, blah, blah. And I'm like, Oh, here we go again.
But I usually always run my projects from Docker though. So I just, you know, hard code that stuff,
like just once you get it set up. That's cool. Nice. I dream of days when I can use Docker again,
have an M one Mac and it's in super early, early beta stages. Yeah. It's okay. I don't,
I don't mind too much because I don't use it that much, but still cool. Brian,
it says something about dot PTH. I'm guessing path files. What do you know anything about this?
I have no idea what those are. Oh, dot PTH files. So there's yeah, there's there.
There are a way to, I don't know a lot. I don't know the detail, the real big details, but it's,
it's a way to have a you can have a list of different paths when it, within that file. And if you import it
or don't import it, if you include it in your path, then Python, I think includes all of the
contents into anyway, I'm actually, I'm blowing smoke. I don't know the details. Okay. Sorry.
Yeah. But apparently you can have a little more control with TH files, whatever those are.
Yeah. I don't know much about that either. Yeah. Unfortunately. I mean, I've been using
OS dot path. So what do I know? All right. Speaking of what do I know? I could definitely learn more
about pandas and that's one of your items here, huh? Hannah? Yeah. So I thought maybe I just give
like a little snippet of kind of like some of the stuff I talk about in the book. Fantastic. So yeah,
here we go. So if we're looking at pandas in terms of like the dependency hierarchy, well,
and I guess I should start at the beginning. So what is pandas if you're not familiar with it?
It's a data analysis library for Python. So it's used for doing big data operations. And so like,
if we look at the dependency hierarchy of pandas, it kind of goes like pandas, which is dependent on
numpy, which deep down is dependent on this thing called BLOS, which is basic linear algebra subprograms.
Right. And wasn't there something with BLOS and a Windows and a Windows update in a certain version,
I think recently? I can't remember. I feel like there was some update that like made that thing
that wasn't working. Yeah. Usually a big challenge around numpy and versioning and stuff to make it
work in the short term. Yeah. Usually the BLOS library is built into your OS already. And it just
points at that. But if you're using something like Anaconda, I think by default, like it installs
Intel MLK and uses that. But yeah, if you're using like Linux or just like out of the box,
whatever's on Windows, which is what it is, if you like pip install it, then yeah, there could
certainly be issues with like dependencies mismatches. Yeah. So, and I've like greatly simplified this,
but in terms of kind of like the languages and walking down that dependency hierarchy,
you start out in Python with pandas and then numpy is partially Python and partially C and then BLOS is
pretty much always written in assembly. And if you don't know what assembly is, it's basically like a
very, very, very, like probably the lowest level language you can program in. And it's essentially
like CPU instructions for your processor. And so I've taken this just like basic example here and I'm
going to kind of like roll with it. So if we're doing just like a basic addition in pandas, say like
we have column A and we want to add that with column B and like store it back into column C.
Like a traditional linear algebra vector addition type thing.
Traditional vector math. So pandas, like if you, if you look at these operations, each,
each of these like additions on a per row basis is independent, meaning like you could conceivably run
like each of those additions for each row, like in parallel. Like there's no reason why you have to go
like row by row. and that's essentially like what kind of like big data analysis libraries are
like at their core is they, they like understand this conceptually and try to parallelize things as
much as possible. and so that's kind of like the first like fundamental understanding that you have
to have, like when working with pandas is like, you should be doing things in parallel as much as you
can. which means understanding the API and understanding like which functions in the API
will let you do things in parallel. so like if we're just not using pandas at all, say like
we're just inventing our own sort of like technique for this, like you might think, well, like each of
these rows could be broken up like into a thread, right? So like we could say like thread one is going to
run like the first row addition. And then like thread two is going to run the second row, et cetera.
but you might find that we'll run into issues with this, in terms of the GIL. So like the gill
is now otherwise known as the global interpreter lock in Python, prevents us from really like
running a multi-threaded app, operation, like in parallel. basically Python can run the rule
is it can run one Python op code at a time and that's it. All right. It doesn't matter if you've
got, you know, 16 cores, it's one at a time. Yeah. Yeah. And this like is really terrible for,
yeah. For, for like trying to do things in parallel. Right. So like that, that kind of
use cases out, like pandas and numpy and, and all that stuff is, is not going to be able to use
multi-threading. and so, and like, I just want to point out like Python, like at its core has
this like fundamental problem, which is why they went with the GIL. So like Python manages memory for you.
Um, and it, how it does that is it keeps track of references to know when to, free up memory.
Uh, so like when memory can be like completely destroyed and somebody else can use it essentially.
Um, and like that's something you've got to do stuff like Brian sometimes probably has to do with C and
like free and all those things. Right. Yeah, exactly. Yeah. Yeah. So like C, you have to do this with
yourself with like Malik and free and all that stuff. But, with Python, it does it for you,
but that comes at a cost, which means like every single object in Python has this little like counter,
which is like a reference counter. and so basically like way back in the day, like when
threading first became a thing, like in order to kind of like avoid this threading problem,
um, they came up with the gill, which basically says you can only run one third at a time or like
one opcode at a time as, as you said.
And attempts have been made to remove it. Like Larry Hastings has been working on something
called the galectomy, the removal of the GIL for a while. And the main problem is, if you take
it away, the way it works now is you have to do lock on all memory access, all variable access,
which actually has a bigger hit than a lot of the benefits you would get, at least in the single
threaded case. And I know Peter said like, if we really don't want to make changes to this,
if it's going to mean slower, single threaded Python, they'll probably not for a while.
Yeah. Yeah. Yeah. And that, that is a big problem. So like, I mean, if generally what people use,
like instead of threads in Python is they use like multi-process and they spin up multiple Python
processes. Right. And like that truly kind of like achieves the parallelism. but anyways,
I digress. so, so we can't use the gill, but what's interesting to note is when you're,
uh, running NumPy at its very low level in C, like when you enter and look at the C files,
it actually is not subject to the GIL anymore because you're in C. and so you can potentially
run, you know, multi-threaded things in C, and call it from Python. so, but beyond that,
if we look at BLOS, BLOS has, built in like parallelization for like, hardware parallelization.
Um, and how it does that is through vector registers. so if you're not familiar with like the
architecture of CPUs and stuff, like at its core, you basically, only have like, only can
have a certain small set, maybe like three or four values in your CPU at any one time that you're running
like ads and multiplies on. and like how that works is you load those values like into the CPU from
memory. And that load can be quite time consuming. It's really just based on like how far away your memory is from
from your CPU at the end of the day, like physically on your board. Right. Right. Is it in the cache?
Is it in the RAM? Yes. Yeah. And that's why we have caches. So like caches are like memory that's closer
to your CPU. consequently it's also smaller. but that's, that's how you can kind of, you might hear
like people say like, oh, like so-and-so wrote this really performant program and it like utilizes like the
size of the cache or whatever. So like basically like if you can load all of that data, like into your cache and
run the operations on it without ever like having to go back out to memory, like you can make a really
fast program. Yeah. Yeah. It could be like a hundred times faster than regular memory. Yeah. Yeah. And so
essentially like that's what, BLOS is trying to do like underneath and, and to NumPy is they're trying
to take this giant set of data and break it into chunks and load those chunks into your cache and
operate on those chunks. and then dump them back out to memory and load the next chunk.
yeah, very cool. Yeah. Thanks for pointing that out. Like I didn't realize that BLOS leveraged some of
the OS native stuff, nor that it had like special CPU instruction type optimizations. That's pretty cool.
Yeah. Yeah. so like it has, like on top of the registers, it also has these things called
like vector registers, which actually can hold like multiple values at a time in your CPU. so like,
we could take this like simple example of, like the addition and we could actually, well, we can't
run those like row per row calculations, in parallel with threads. We can with vector registers.
Okay. and the limitation there is that the memory has to be, sequential when you load it in.
Um, this is definitely at a level lower than I'm used to working at. How about you?
But yeah, so, anyways, this is just like kind of the stuff that I talk about in my book.
Um, it's not necessarily about like how to use pandas. but it's, it's about like kind of like
what's going on underneath pandas. And then like, once you kind of like build that foundation of
understanding, like you can understand like better how pandas is working and like how to use it correctly
and what all the various functions are doing. Fantastic. Yeah. So people can check out your book.
Got a link to it in the show notes. So, very nice. It's offering me the European,
uh, the Euro price, which is fine. I don't mind. So yeah. So like, I mean, it's on Amazon too.
It's on a lot of different platforms, but I figured I'd just point directly to the publishers.
Yeah, no, that's perfect. Perfect. quick comment. Roy Larson says,
NumPy and Intel MKL cause issues. Sometimes you could learn on windows. If something else in the system
uses Intel MKL. Okay. Yeah. Interesting. I have no experience with that, but I can believe it. Intel
has a lot of interesting stuff. They even have a special iPhone, compiled version,
I think for Intel if you use potentially, I'm not sure they have some high performance version.
Yeah. Yeah. Yeah, they do. Yeah.
Nice. Also in Portland, you can keep it in Portland. There we go.
Now, before we move on to the next item, let me tell you about our sponsor today.
Thank you to data dog. So they're sponsoring data dog. And if you're having trouble visualizing latency,
CPU, memory bottlenecks, things like that in your app, and you don't know why you don't know where it's
coming from or how to solve it, you can use data dog to correlate logs and traces at the level of
individual requests, allowing you to quickly troubleshoot your Python app. Plus they have
a continuous profiler that allows you to find the most resource consuming parts of your production code
all the time at any scale with minimal overhead. So you just point out your production
server, run it, which is not normally something you want to do with diagnostic tools, but you can with
their continuous profiler, which is pretty awesome. You'll be the hero that got that app back on track at
your company, get started with a free trial at pythonbytes.fm/datadog, or just click the link in
your podcast player show notes. Now, I'm sure you all have heard that working with pickle has all sorts
of issues, right? The pickle is a way to say, take my Python thing, make a binary version of bits that
looks like that Python thing so I can go do stuff with it, right? That's generally got issues, not the
least of which actually are around the security stuff. So like you unpickle something to deserialize it,
sit back is actually potentially running arbitrary code. So people could send you a pickle virus.
I don't know what that is like a bad, a rotten pickle or whatever. That wouldn't be good.
So there's a library I came across that solves a lot of the pickle problems.
It's supposed to be faster than pickle and it was cleverly named quickle.
Either of you heard of this thing?
No.
Yeah, it's cool, right? So here's the deal. It's a fast serialization format for a subset of Python types.
So you can't pickle everything, but you can pickle like way more say than JSON. And the
reasons they give to use it are it's fast. If you check out the benchmarks, I'll pull those up in a
second. It's one of the fastest ways to serialize things in Python. It's safe, which is important.
And unlike pickle deserializing a user provided message does not allow arbitrary code execution.
That seems like the minimum bar. Like, oh, I got stuff off the internet. Let's try to execute that.
What's that going to do? Oh, look, it's reading all my files. That's nice.
All right.
It also, it's a flexible because it supports more types. And we'll also learn about a bunch of other
libraries while we're at it here, which is kind of cool. A bunch of things I heard of like
MSG pack or well, Jason, you may have heard of that. And the other main problem you get with some
of these binary formats is you can end up where in a situation where you can't read something.
If you make a change your code, like, so imagine I've, I've got a user object and I've pickled them
and put them into a Redis cache. We upgrade our web app, which adds a new field to the user object.
That stuff is still in cache. After we restart, we try to read it. Oh, that stuff isn't there anymore.
You can't, you know, user cache anymore. Everything's broken, et cetera, et cetera. So it has a concept of
schema evolution, having different versions of like history. So there's ways that older messages can be
read without errors, which is pretty cool. Yeah. That's nice. Yeah. Neat, huh? Yeah. I'll pull up the benchmarks.
There's actually a pretty cool little site here. It shows you some examples on how to use it. I mean,
it's incredibly simple. It's like, dump this as a string, read this, you know, deserialize this.
It's real simple. So, but there's quite interesting analysis, live analysis where you can click around
and you can actually look at like load speed versus reads like serialized versus deserialized speed,
how much memory is used and things like that. And it compares against pickle tuples,
protobuf, pickle itself, ORJSON, MSGPAC, quickle, and quicklestrux.
There's a lot of things. I mean, I knew about two of those, I think. That's cool.
But these are all different ways. And you can see, like in all these pictures, generally,
at least the top one where it's time shorter is better. Right? So you can see if you go with
there, like quicklestrux, it's quick rule of thumb, maybe four or five times faster than pickle,
which I presume is a way faster than JSON, for example.
You know, you'll also see the memory size, which actually varies by about 50% across the
different things. Also speed of loading up a whole bunch of different objects and so on. So yeah,
you can come check out these analysis here. Let's see all the different libraries that we had. Yeah,
I guess we read them all off basically there, but yeah, there's a bunch of different ways which are,
you know, not pickle itself to do this kind of binary serialization, which is pretty interesting.
I think it does. Protobuf, that's pretty cool. Actually, I want to try this out. It looks neat.
Yeah. Yeah, it looks really right. And one of the things I was just looking at the source code,
I love that they use pytest to test this. Of course, you should use pytest. But the, I can't believe
I'm saying this, but this would be the perfect package to test with a Gherkin syntax. Don't you think?
Because it's a pickle thing. Oh my gosh. You've got to use the Gherkin syntax.
So yeah, you definitely should. And Roy threw out another one like UQ foundation,
Dill package deals with many of the same issues, but because it's binary and has all the same
sort of versioning challenges you might run into. Well, Dill, the Dill package. That's funny.
Yeah, pretty good. Pretty good. All right. So anyway, like, you know, I'm,
I'm kind of a fan of JSON these days. I've had enough XML with custom namespaces in my life that
I really don't want to go down that path and XSLT and all that. But, you know, I've really shied away
from these binary formats for a lot of these reasons here. But, you know, this might make me interested.
If I was going to say throw something into a cache, the whole point is put it in the cache,
get it back, read it fast. This might be decent. Yeah. Yeah. It definitely seems to address a lot of the
concerns I have with pickle for sure. Yeah. And I don't, did I talk about the types
somewhere in here? We have time. Yeah. Here's, there's quite a list of types. You know, one's
really nice. Date time. I can't do that with JSON. Why is, why in the world doesn't JSON support
some sort of time information? Oh, well, but you've got most of the fundamental types that you might run
into. All right. So, quick, give it a quick look. All right, Brian, what you got here?
Well, I was actually reading a different article. But the, it came up, we, I think we've talked about
friendly traceback. It's a package that just sort of tries to make your tracebacks nicer. But,
but when I didn't realize it had a console built in. So I was pretty blown away by this. So there's a,
it's, you know, it's not trivial to get set up. It's not that terrible, but you,
you have to start your own console, start the REPL, import friendly traceback, and then do friendly
traceback start console. But at that point, you have just like the normal console, but you have better
tracebacks. And then also you have all these different cool functions you can call like,
what, what, what, where, why, and explain and more. And basically if something goes
wrong while you're playing with Python, you can interrogate it and ask like for more information.
And that's just pretty cool. The, the why is really great. So if you have the, one of the examples I saw
before, and I'm, I think I might start using this when teaching people is, we often have like
exceptions, like you assigned to none or you assigned to something that can't be assigned,
or you, you, you didn't match up the bracket and the parentheses or something like that correctly.
Um, and you'll get like just syntax error and it'll point to the syntax error, but you might
not know more. So you can just type why a W H Y with parentheses. Cause it's a, or yeah,
because it's a function and it'll tell you why, why it's like a, the great storytelling,
right. The five Y's of a bug. Yeah. so then you get W's of a bug. Yep. You can, you can say
what, like to, to repeat what the error was, why we'll tell you why that was an error. And then
specifically what you did wrong. And then where it will show you if you've, if you've been asking
all sorts of questions and you lost where the actual trace back was, you can say where, and it'll point
to directly to it. And, I think this is going to be cool. I think I'll use this when trying to teach,
especially kids, but really just people new to Python. Tracebacks can be very helpful for them.
Yeah. Like even, I know, like I sometimes have to look up like certain error messages that I'm like,
not familiar with. So yeah, that would be super helpful. I could just do it right in the console.
Yeah. I totally agree. You're going to have to help me find a W that goes with this,
but I want the, what would be effectively Google open closed privacy?
You know, because so often you get this huge trace back and you've got these errors. And if
you go through and you select it, like for example, the area you see on the screen,
unbound local error, local variable greetings in quotes, reference before assignments. Well,
the quotes means oftentimes in search, like it must have the word greeting. And that's the one thing that
is not relevant to the, the, the Googling of it. Right? So if I'm a beginner and I even try to Google
that I might get a really wrong message. Right? So if you could say, Google this in a way that is
most likely going to find the error, but without carrying through like variable details, file
name details, but just the essence of the error, that would be fantastic. Now, how do we say that with W?
You could just say, Whoa, or, or maybe www or WTF. I mean, come on, there's a lot of WTF.
But wouldn't that be great. And so that's also part of this package that you see,
at their main site where you've got these really cool, like visualized stuff, right? Where it's
sort of more tries to tell you the problem of the error with the help text and whatnot.
Yeah. Yeah. This is cool. Also uses rich, which is a cool library we talked about as well.
I love rich. I include rich in everything now, even just, just to print out simple,
better tables. It's great. Yeah, for sure. Hannah, do you see yourself using this or is it,
are you more, more in a notebooks? Oh no. I, I mean, I usually use like the PDB debugger. So yeah,
I mean, I'm not sure if really this as it is would be, like a problem. It would depend on how
much information it has about like obscure errors from dependent libraries which is usually what I
end up looking at these days but yeah I mean conceivably like yeah that could be helpful
yeah if we get that WTF feature added then yeah oh yeah for sure gosh
speaking of errors let's uh cover your last item last item of the show uh yeah so um I uh at work
uh work in um the security org and I write uh like automation tools for them which means uh
sometimes the repos that we work on get to be like test subjects um for for new like requirements and
such um and so recently uh our org was exploring uh like static code analysis looking for like
security vulnerabilities in the code um and so I ran across bandit and I integrated bandit
into our we don't have time to uh go through these old legacy code and fix these problems oh wait this
is what it means oh sorry yes we can do that right now that's the kind of report you gave you got from
bandit yeah exactly um so yeah we integrated bandit into our legacy code base and we actually it's funny
you say that because I the bug that I found using bandit was actually like a from the legacy code um
that does not surprise me yeah uh so it was it was a pretty stupid like error um like it was pretty
obvious like if you were doing code review but because it was legacy code and it was like already there
um I just like never noticed um but it was basically like issuing like a request with like no verify
uh so it was like an unverified like http request um and bandit was like yeah this broken ssl
certificate keeps breaking it I just told it to ignore it oh yeah yeah well and I honestly like I think that
might have been why it was there in the first place because I I know like the oh like several years ago
like had some certificate issues so yeah that might be and it was it was like an internal talking to
internal so it was like maybe even a self-signed certificate that nothing trusted but they get
technically there yeah yeah it was like we'll just we'll just do that um but yeah so um bandit is
basically like a linter but it looks for security issues um so you could just like pip install it um
and then just like run it on your code and it will find a bunch of different potential security issues
like just by like statically analyzing your code um and I've uh pretty much like come to the opinion
that like why haven't I done this on all of my other projects like I should be doing this on every single
project um like because you know like as as like a developer I always run like lint and black and stuff
like that um so I figure you know I should probably be running bandit too yeah cool yeah well very nice uh
it's a good recommendation for people as well and it's got a lot of cool you can go and actually see
the list of the things that it tests for and even has test plugins as well which is pretty cool yeah
yeah so you can like make your make your own if you want um and it has like all the common linter sort of
like functionality like ignore these files or like ignore these rules or even you know like ignore this
rule on this particular line stuff like that yeah absolutely which is pretty sweet I love that things like
bandit are around because um uh thankfully uh developing web stuff is becoming easier and easier
but it's then now the barrier to to entry is lower you still have to have all those security concerns
that you had before that normal I mean usually people were just had more experience but they would make
mistakes anyway but now I think this is one of the reasons why I love this is because people new to it
might be terrified about the security part but having uh bandit on there looking over their shoulder is
great yeah yeah like don't publish with the debug setting on and blast or jango or anything like that
simple obvious stuff and like honestly like having worked in the security org for about a year now like
I've come to the understanding that a lot of security issues stem from just like basic like duh sort of
misconfigurations so like something like this is perfect and I really really like that you added um
you you wrote in the show notes um some pre-commit uh how to how to hook this up with pre-commit because
I think having it in pre-commit or in a ci pipeline is important because um like you guys were joking
about often security problems come in because somebody's just trying to fix something that broke yeah but
they don't really realize how many other things it affects so yeah yeah yeah besides down just we got
to make it work quick just just turn on the debug thing we'll just look real quick and then you forget
to turn it off or whatever yeah yeah for sure yeah yeah just stupid human errors nice all right I want to go
back real quick Brian because uh your uh mentioned a friendly trace back got a lot of stuff so let me just do a
quick uh audience reaction Robert says it is cool Brian John Sheehan says I was just thinking it's something
the same would be cool it's a great teaching concept Anthony says super useful um John says I've been doing
more demo code in the console rather than the idea this looks like it would help w how to fix it w
wow how w i love it Robert very good Zach says uh what is this magic this looks amazing and so on all
right well thanks everyone uh I'm glad you all like that uh so that's it for our main items you know um
Brian you got any extras you want to throw out there you were uh doing something with climate change or what
are you doing this week um yeah I'm sharing the room with some people just a sec uh the uh I did do
two meetups uh with uh with uh Noah and uh then with the Aberdeen python meetup wait wait I got
I got to interrupt you really quick did all the talk that Hannah did about bandit and viruses get you
it's all right I'm sorry sorry about that carry on well I missed all this talk with Hannah that Hannah
had about viruses and in hacking and stuff with bandit did it make you nervous and you had to put on your
your mask no I just I'm in a group meeting in their group room and somebody came in but it's okay I'm
just teasing carry on um the that's funny I also wanted to look like a bandit yeah exactly but I was
thrilled that uh Noah uh asked me to to speak to them that was neat and then the python Aberdeen people
um and also like but they mentioned that Ian from the python Aberdeen group said that he had an arrangement
with you that when you Michael that when the pandemic is over you're gonna go over and they're
gonna you're gonna do like a whiskey tour or something like that so I'm I don't know the
details but it sounds good to me already anyway if that happens I want to go along yeah it's a python
bites outing let's do it and then we have uh uh there are pdx west meetup tomorrow you're gonna speak
that's kind of exciting so yeah it's gonna be fun and people as virtual so people can attend however
um I'm also I've got feedback from both uh you and um and Matt Harrison gave me some feedback so I'm
updating my training page on testing code so um because I really like working with teams so I'd and
anybody else wants to give me feedback on my training page maybe I could I'd love to hear it so yeah or maybe
they even want to have some high test training for their team yeah I mean testing is something that uh
I think teaching a team at a time is a great thing because people can uh can really um I don't know that
we can talk about their their particular problems not general problems it's good so yeah for sure well
you also need more of a team buy-in on testing right because like if one person writes code and
won't write the test another person is like really concerned about making the test fast it's super
frustrating yeah the person who doesn't want to run the test keeps it breaking the build but like you know
anyway it's a team sort of sport in that regard yep yeah all right awesome so I got a couple quick
things PEP 634 structural pattern matching in python has been accepted for python 310 that's like
imagine a switch case that has about a hundred different options that's what it is yeah with
like reg x not quite but sort of like style like you can have like these patterns and stuff that
happen in the cases I don't know how to feel about this like if uh let me put a perspective like if the
walrus operated was controversial like this is like this is like a way bigger change to the language so
I don't know it it's both awesome and terrifying yes exactly yeah I was gonna say I'm kind of surprised
yeah yeah so am I Hannah that like this got accepted it seemed to be sort of counter to the simplicity of
python like I I did not at all against having a simple switch statement that does certain things but
this seems like a lot I may come to love it one thing that maybe would help me come to a better
understanding and acceptance was if the PEP page had at least one example of it in use like the
whole page that talks about all the details says I don't believe there's a single code sample ever
well there's a tutorial page as well so oh is there there's the tutorial page okay maybe that's where
I should be going to check it out yeah but it still sort of feels like a five barrel foot gun yeah
it does well but the page that I'm looking like the pip thing that I'm listening to the official PEP I
don't think it has uh does it have the tutorial yeah no you're right it does it does um somewhere down
yeah PEP 636 yeah it's a different PEP that is the tutorial for the PEP interesting I didn't realize
that it's kind of meta honestly anyway I to me I'm a little surprised it's accepted fine um I know people
worked really hard on it and congratulations a lot of people really want it comes from Haskell right so
Haskell had this like pattern matching like alternate struct thing I don't know I just feel
like Haskell and Python are far away from each other so that's my first impression I will
probably come to love it at some point uh PyCon registration is open so if you want to go to PyCon
you want to attend and be more part of it than just like watching the live stream on YouTube be part of
that I think I'm going to try to make a conscious effort to attend the virtual conference not just
catch some videos so you can do that yeah PyCon is awesome like just I my first conference was PyCon
and then I went to other conferences and I was like what are wrong with these conferences like
yeah I know I feel the same way I know it's uh it's really really special I'm sure the virtual one
will be good I can't wait for the in-person stuff to come back because it really for sure yeah it's a
whole another experience in person I consider it basically my um geek holiday where I get get away
and like just get a hang out with my geek friends I happen to learn stuff on there totally
and then Python web comp is coming up and that's uh registration is open for that as well um and I
suppose probably PyCascades which Brian and I are on a panel at there as well oh nice I put I put a link
into an hour of code for Minecraft which has to do with programming Minecraft with Python if people are
looking to teach kids stuff uh that looks pretty neat so um my daughter's super into Minecraft I don't
do anything with it but if if you are and you want to make it part of your curriculum uh that's pretty
cool Hannah anything you want to throw out there before uh we break out the joke nope I'm good
awesome do it do it all right all right so this one we have something a little more interactive for
everyone we've got a um a song about PEP 8 about writing clean code this is written and and uh produced
sung by Leon Sandoy uh goes by Lemon and him and his team over at Python Discord he runs Python Discord and
apparently it was a team effort creating this and the reason I'm covered is a bunch of people sent it
over so Michael Rogers Valet uh sent it over so you should cover this Dan Bader said check this out
Alan McElroy said hey check out this thing so all right I actually uh spoke to Lemon and said hey do
you mind if we play this he said no that'd be awesome give us a shout out of course so we're
gonna actually play the song as part of this in the live stream you get the video on the audio you get
well audio so I'm gonna kick this off and we'll come back and I'd love to hear Brian and Hannah's thoughts
here we go
you don't need any curly braces just for spaces just for spaces
wildcard imports should be avoided in most cases in most cases
try to make sure there's no trailing white space it's confusing it's confusing
trailing commas go behind list items get blamed titans get blamed titans
and comments are important as long as they're maintained when comments are misleading it will drive people insane
just try to be empathic just try to be empathic just try to be a friend
it's really not that hard just adhere to
papade. Papade.
constants should be named, all capital letters, and live forever, live forever.
And camel case is not for python, never ever, never ever.
And never use a bare exception, be specific, be specific.
No one likes the horizontal scroll bar, keep it succinct, keep it succinct.
And comments are important, as long as they're maintained.
When comments are misleading, it will drive people insane.
Just try to be empathic, just try to be a friend.
It's really not that hard, just adhere to.
Papade.
That was amazing.
I can sympathize with so much of what he's saying.
I'm just having flashbacks to a discussion I had with my teammate about comments.
And being like, "No, this comment doesn't actually describe what the comment is doing."
It's worse than having no comment. It really is.
It really is, yeah.
I love it.
Or if it describes literally what the code is doing and not high-level.
Exactly.
Why or background or anything other than...
The why.
The why is important.
Yeah.
I love it.
So, two things.
Lemon and team, well done on the song.
And man, you've got a great voice.
That's actually...
It was beautiful and funny.
Yeah.
It was amazing.
All right.
Well, Brian, we probably should wrap it up.
Yeah.
All right.
Well, Hannah, thanks so much for being here.
It's good to have you on the show.
And Brian, thanks as always.
Everyone, thanks for listening.
Thanks for having me.
Bye.
Bye, all.
Thank you for listening to Python Bytes.
Follow the show on Twitter via @pythonbytes.
That's Python Bytes as in B-Y-T-E-S.
And get the full show notes at pythonbytes.fm.
If you have a news item you want featured, just visit pythonbytes.fm and send it our way.
We're always on the lookout for sharing something cool.
On behalf of myself and Brian Okken, this is Michael Kennedy.
Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.