Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to
your earbuds. This is episode 160, recorded December 4th, 2019. I'm Michael Kennedy.
And I'm Brian Okken.
And this episode is brought to you by DigitalOcean. Check them out at pythonbytes.fm
slash DigitalOcean. $50 credit for new users. Let's start with a topic that we've covered
quite a bit, Brian. Type hints. Or do you call them type annotations?
I usually call them type hints.
Yeah, I go back and forth. And the article you're going to mention actually says you can call
them both. But yeah, it's kind of funky that they don't have like a definitive name. Maybe they do.
And I just don't have certainty around it.
Yeah, one of them's wrong. And whatever the one I'm using is probably the wrong one.
So we have covered it a lot. And there's a lot of tools that we've talked about, like mypy and
others. So hopefully we've convinced everybody that they should at least do a little bit of type
hinting. If you're in that boat, then this article is perfect for you. Al Swigert wrote an article
called Type Hints for Busy Python Programmers. And that's everybody, right? I mean...
We're all busy.
We're all busy. I really like the article. It's a kind of a... When I first went to it,
I was a little shocked because there's a lot of words there. But it reads like a conversation.
It's a conversational thing as if it's a programmer talking to an expert. So that sort of tone of,
I think I should do this, but what do I do? And so he just sort of walks through a whole bunch
of questions with it. He covers using mypy and typing as the two modules you need and how to use
those. And there are other tools, but really most people are using mypy. And that's what... That's
Al's saying that. I don't know if most people are using mypy or some of the other tools.
We don't... I don't think there's any way to gain statistics on that. But I think mypy is a good way
to start. And a reminder that using types doesn't affect runtime. You have to run a tool like mypy
to get it to work. And so I don't think he covers this, but I think that people should hook running
mypy into their talks system or into their continuous integration pipeline to make sure that these are
met.
Yeah. Everything's hanging together. Yeah.
Yeah. There isn't any compiler to catch you. So one of the things I liked about it is it gets you
started and it covers some of the basics. But there's... As I've started running it, started using
type hints, the exact things that I've run into, Al covers. For instance, you've got a function that
takes an integer, but it also could be none or the default is none or it's optional or something.
How do you deal with that? There is stuff built in the type int system for unions. So you can say
it's either an int or none and there's optional parameters. And then a shout out to stuff that
if you kind of get into it and you might... Like if you're passing callables or sequences around,
yes, there are ways to type that. And so I think this is actually just a really great quick read and
it'll get you started pretty fast.
Yeah. I really like it. I love the conversational style. It seems super approachable and it's kind
of from the skeptics perspective. It seems a little bit like, oh, I want to do this, but what about...
I don't know if I'm sure about this. Why can I do this? And yeah, it seems like a lot of fun. I'm
definitely a fan of type hints these days, not for the mypy reasons. I understand that makes sense for
very large projects. But for me, like the editors are just so much better if you can just give them
these little hints here and there. I'll give you an example. Like for the database for Python Bytes,
we are talking to MongoDB with Mongo Engine. And the way you specify it is much like SQLAlchemy
or even maybe more so like Django ORM where you say this column is a Mongo Engine column of type such
and such. Well, the type editor, the editors think the type is like a column, but it's not a column.
It's probably a date time or it's a string or it's a list or it's anything, but what do they think
it is? And so you can use type hints to say this column is an integer and it's equal to one of these
column things. Oh, okay. That'll tell all the editors when I hit dot on this, don't tell me it's
one of these column things. I don't want to know that. I want to know what its value is because
that's what I program against when I run it. So there's just all these different angles that make
it super valuable. I'll have to take a look at how you're using that. That's pretty cool.
Yeah, for sure. Speaking of cool, another thing that we have beaten to death, but is very tough
and leathery and has not died is the ability to create an application out of Python code,
right?
Yes.
Well, I stumbled across this thing. I don't believe it is new, but it actually is interesting
on layers. There are layers of interest here called auto-py-2-exe. Auto-py-exe.
Okay.
Okay. So this is a GUI and it's built with eel. First of all, it's a GUI Python thing built
with eel. Remember eel?
I know. Remind me.
Yeah. It lives in the ocean. Some of them are electrical. You shouldn't touch them. No
wait, hold on. It's a library for making electron-like HTML and JavaScript GUI apps with Python.
Oh, okay. It says electric snake. Got it.
Yeah. Electron. Yeah, exactly. So it's very much like that. I think it's for kind of simplistic
ones, but it certainly works for this app, which is pretty cool. So here's a Python app built in eel.
That's cool. And it's kind of meta in the sense that it is, its job is to make other Python
applications out of native Python code.
Okay. Interesting.
Okay. Yeah. So what it does is basically is this is automation over top py installer. And you might
think, okay, well, I could just, you know, pass the command line arguments and just do that.
But there's something really nice about having a GUI say, here's the icon. I just put the icon
into the app. And here's the, the name I want for the title and such and such. And, you know,
here are the additional binary files or additional resource files I want bundled and shipped with my
application. We're using this JSON file and we're using this, I don't know, this image that we're
going to embed somewhere. Just include those. And like, so there's all sorts of settings in this
auto py.dxe where you can just pick all that stuff out and you don't really have to be an expert at
all with py installer, but you can build pretty nice stuff. Like there's a checkbox. Do you want just a loose
set of files or do you want it bundled into a single .exe that I can just give and it'll contain
Python and the dependencies and all that? Okay. This does sound neat. Yeah. It's pretty neat, right?
Yeah. And so really people want to, if this sounds interesting to you and you want to find out,
Hey, is this cool? Is this interesting? There's a three minute YouTube video. You can just check that
out. Okay, cool. So I put the link to the three minute video at the end. People can click on that,
watch it. And then within three minutes, you'll decide whether or not this is useful for you. But
some of the conversation I saw people saying, yeah, I know it just automates PyInstaller and
that's cool. But there were things I couldn't get to work with PyInstaller, but doing it here,
it worked fine. So I feel like there's a lot of value in this. Yeah. Nice. Yeah. Speaking of value,
DigitalOcean, they are adding a lot of value to all the hosted things of the world, including our stuff
that's hosted there across all the different sites and whatever. I believe we're doing like 15 terabytes
of traffic a month through DigitalOcean. That is not messing around. That's a lot. So been rock solid.
The other thing that they have that they're just launching is the Kubernetes, the container, their own
container registry for things like their Kubernetes service. So we've got like Docker hub and places
that you can put your containers, but that's on the public internet and things like that. It'd be nice
to have a private little one for your private code that you want to make sure doesn't somehow leak with,
you know, making it public or private on accident or whatever. So you can use their DigitalOcean
container registry and have nice private container definitions, which is pretty cool, among other things.
So check them out at pythonbytes.fm/DigitalOcean. Like I said, $50 credit for new users to get started
and play around with this. Speaking of playing around, I've never played with Sphinx. Oh, you never have?
I haven't. I've had no reason to do it, but it seems something like something that's useful. Maybe you should learn about it.
Yeah. So one of the things like, so I have, what I've wanted to use Sphinx for,
so Sphinx can be used for building up documentation and makes like a documentation website based on
either REST or Markdown or not the REST in that. Restructured text. Restructured text.
Yeah. Thanks a lot, programming world, for having two things that could be abbreviated REST.
In the C++ land, there was some auto-documentation tools to where you can take, and I know other
programming languages have this too, where you can take information about the structure of a program
and automatically generate some web documentation about how to use it and what the parameters are
for different functions and stuff like that. So you can do that with Sphinx, and that's the thing
I've been wanting to do, and I've tried several times and failed. And it's usually because,
probably on my part, but the tutorials get me part of the way, and then, you know, they reference a
bunch of documentation and stuff, and I don't want to become a Sphinx expert. I want, that's kind of
the point of this. I want to be able to just point it at my code and go, go do it. And like, for instance,
pulling out, so what does a function do? What do different tests do? I'd like that to be embedded
in the doc string so that when I'm coding, I can make sure that that's correct, and then that's just
automatically generated. So what I'm highlighting today is an article called How to Document Python
Code with Sphinx. It's a short tutorial by Mosh Zadka. That's a cool name. And so, walks through,
it has a little example of a little module with the doc string, and then the small example of how to set
up your index.rst. And I like Markdown better, but this is so short, I'd probably just use the
restructured text version. Points out that you need a few extensions. You need Autodoc, Napoleon,
and ViewCode. And then it gives you, just gives you the conf.py file you need to make this work.
And then how do you generate this stuff? Well, there's this long, weird command. It's not that
weird, but I don't want to memorize it. So this tutorial also shows you how to set up a tox.any
file with this Sphinx built in. So you can just say tox and your docs get rebuilt. So I like this
tutorial, and I'm going to give it a shot. Yeah, it's a really cool thing, and automatically
generating your documentation. Not terrible at all. Of course, it doesn't cover, like,
okay, well, now that you've generated it, it's a bunch of web pages and stuff. Where do you put
that? But for internal documentation within a team, you got your own way to do it, whether you're
pushing it into a, like a small web server, or just leaving it with your code, or whatever. So
leave that to the user. Yeah, or put it on DigitalOcean, or not DigitalOcean,
GitHub Pages, right? Yeah. That's one thing you do. You put it on Netlify. A lot of options for just
hosting HTML. Yeah, or throw it into a GitHub cluster,
or a Kubernetes cluster. Maybe not. That's right. So, you know, something that's always
amused me is the association of Python ecosystem and language with snakes. Because that's not the
origin, right? That's not what Python is named after. It's named after Monty Python. All these
apps and stuff should have, like, medieval knight type of names, but they don't. So I'm going to tell
you about one that is called Snake. Snake? Snake? S-N-E-K. Snake? Don't quite know the right
accent to put on it to pull it off, but Snake? It's pretty awesome. So it's a PowerShell module
for integrating standard CPython, which is Python code, and various packages with PowerShell. So if
you're doing anything on Windows, any sort of automation or advanced shell stuff, you're probably
using PowerShell. Although the regular terminal is getting better as well. And, you know, the story of
Python on Windows has always had a little caveat, or maybe a little star, like, yeah, you always do this,
except for on Windows, you type this other thing. And it's slowly getting better, right? Steve Dower's
done a lot of work to improve that. Yeah. With, say, the Python installer that comes off the Windows 10
store has Python 3 as a command, not just Python and so on. But here's one more thing that you can do to
make working on Windows with Python better. If you do anything with PowerShell, you can write scripts
now that have sections of those scripts or libraries for PowerShell written in Python. And it uses this
library called Python for .NET, which is a new modern way to share the entire last library of .NET and all
the objects you might create there with Python code. And similarly, say, this Python thing returns some
funky .NET type that can be converted so, like, it looks normal to the rest of the PowerShell land,
which is generally .NET stuff. So if you're doing stuff with PowerShell, check this out. It's pretty
cool. The syntax is funky, but, you know, it's PowerShell syntax. So there you have it.
Yeah. Well, okay. So I think that people can't be blamed for doing the snake thing because our logo
is two snakes. I know. I think that that is absolutely true. I think it should be like maybe
some cheese. Yeah, exactly. Or maybe a shield. Yeah, I don't know. Whatever. I don't mind having it
being snake. The snake is cool. And there's the, you know, the Python snake. It's just funny.
That's not actually the name. No. Killer Rabbit would be nice. Yes. The Killer Rabbit would be
fantastic. If you just got an exception and that appeared as an emoji, that would be beautiful.
Yeah. Or maybe a failed test in pytest, right? You could write a plugin. It unleashes the killer,
the killer rabbit and all that. All right, we're going down a bit of a hole. So maybe get us out and
talk, get us back to the databases. Okay. So databases. Since in Python world, one of the things we like to do
is use Pandas because Pandas makes dealing with data like so much easier with data frames and stuff.
And also one of the things with databases that we can use is SQLAlchemy if it's a SQL based database.
And most of them are. So you can just take Pandas and put it together with SQLAlchemy and put it
together with your database and just slurp your entire data tables into Pandas data frames and then
manipulate them there. So you're good to go, right? Right. Unless you have like five gigs of data,
then you're good to swap and be very slow. Yeah. Or if terabytes of data, that'd be bad. Yeah.
Very bad. So all hope is not lost because I'm going to guess Arena Trong wrote this article called
How to Use Pandas to Access Databases. And it's really great. One of the recommendations she uses
is don't be afraid of the command line for SQL because you can play around with queries and look
to see how big the shape of your data is to start with, which is a good thing. But then goes on to
tips on how to put this together correctly. And I'm just going to read out the highlights, but
there's examples of all these. So I think it's great to have a look. Don't grab a whole table limit
to just the fields that you care about that you're going to actually use. Tables often have a lot of
columns that you don't need. So don't grab everything. Limit the number of records. You can either use the
limit field to say, I only want to grab like a thousand elements or something. You can also combine
that with just the number of fields, but also you can also pick out, say, I know I'm going to already
throw away all the rows that don't have a particular value in this one field. So you can
limit that in your query. One of the things you'll often run into with SQL databases, of course, is to
need the need to use joins. And the quick recommendation there is let the database do that.
Put it in your query. Don't do the join within Pandas because you can do it with Pandas and SQL
coming together, but don't. It's going to be slower.
Right. Load it on a memory and have the data table merge it. That seems probably not like the
best way to do it.
Right.
Databases are pretty good at joining stuff.
Yeah. And I mean, if you find, of course, it didn't talk about this, but if you find you're doing
that a lot, you can kind of restructure some of your queries within. I think you can build that in,
like make a join be faster. I don't know. I'm not a database person, so.
You could put indexes. You could write a store procedure and call that. There's a couple options.
Okay. Ask Michael how to do it. Anyway, maybe not. So a lot of this is we're caring about memory. So
how do you figure it out? Like how much memory you're going to use? And I didn't know you do this,
but there's, she has a tip of using query called memory, memory underscore usage, and then summing it.
This is, I don't know what the field comes back, but you can do a query and grab the memory, but you do
like a small query, like just a hundred, a hundred rows or something, and then take that. And then you
can use that to estimate the, the total size. If when you're going to read out like a thousand or
2000 or something. And then the last one, I'm actually didn't quite follow, but I think I kind
of followed. The tip is to talk that the types get converted, like integer types get converted to like
64 bit types. I think, and you don't really need that for a lot of stuff. So there's a,
I'm calling this out because it's definitely a, for the user read this if you need it, but
there's a little bit on using chunking to, and then combining that with specifying what types,
pandas types to use so that you get smaller data footprints.
Yeah. Yeah. So for example, maybe one of the columns in the data frame that just the way it
came back from the database is a 64 bit type integer and you know, everything can actually fit into an
untyped and unsigned 16 bit integer. Well, that'll save you. It'll be a quarter of the size of the data
if you'd make that conversion. Right. So got a million of them, a quarter of the size of a million,
it's possibly meaningful.
Yeah. I mean, a lot of stuff that like, for instance, a lot of stuff I'm, one of the tables I deal with has,
has a numbers for the different specifying, which kind of a result we're looking at, for
instance. And it's, they each have like a number code. There's only like six. So the number can be
one through six. I don't need 64 bits to store that. So yeah, for sure. So it's good. Do you use a
pandas with SQLAlchemy and any of our doing it?
Not really. I, there's a few times I've been thinking I should probably try to do some BI,
like graphing type stuff, maybe about popularity of episodes or things like that, you know, figure
out what day I should release our podcast on, but I haven't done anything. So no.
Okay. But you do use a SQLAlchemy though, right?
Some, but not for the main site because that's MongoDB.
Oh, right.
I'm using something equivalent, like we're using Mongo Engine, which is super equivalent.
Okay.
Yeah.
Nice.
So yeah, I suspect there's probably some interesting MongoDB integration with pandas
that I just don't know about, but I don't know any equivalent of there. Well, this next
one I'm going to cover is very similar to what you talked about. It talks about how do I basically
efficiently work with data from this one source databases? And how do I not just select star
from all the tables into pandas and work on it? Another problem you might have is you might
have JSON data. JSON is sweet, but one thing JSON is not very obviously good at is streaming
data, right? Because how do you know where the closing curly braces? Maybe you've got some
key and then it's got an array of stuff. And the closing bit of that array is not until, you
know, maybe 500 megabytes down the file. That's a problem potentially, right? So if you load
it with, if you just load up a, you know, import JSON and do a JSON load, give it a file pointer,
it's just going to hoover that all up and jam into it in one giant dictionary and spit it out.
But if it's huge, obviously what you want to do is iterate over it and process it item by
item. Yeah. I mean, that's what you would do for a regular file if you could take it line
by line, but because it's this formatted thing, it's much harder. And something I ran
across last time, time before we were talking about some other project and it said, oh,
this project uses iJSON. I'm like, wait a minute, what is iJSON? That's interesting.
Yeah. Yeah. So iJSON is an iterative JSON parser with the standard Python iterator interface. So
what you can do is basically point it at a file stream, some like search expression kind of in
the JSON document, and then it will stream those back to you generator style one at a time.
That's so cool.
Yeah. So there's a little example I put in the show notes. It says, imagine we've got some object
called earth in our JSON document. It has a bunch of continents and within each continent, it lists
out say cities and states and so on. If that was huge, probably it's not, but if it were, you would
want to iterate over it. So you just go to the iJSON and go, give me the items of this from this file
for earth.europe.item. And item is just a way to say, stop there. And then what you do is just say
for thing in, you know, whatever came back from that load that you did and you just iterate over
it in a for in loop.
That's really neat.
Yeah.
And it's doing, like I was looking at the website, there's an example with grabbing it from like a
URL because that's, you might be getting it from an API endpoint.
Yeah.
So I'm guessing that it's not grabbing everything from the endpoint at first.
It probably depends on how the endpoint is set up or even maybe how the object that's
querying it back. So I think they're using URL lib or something like that there.
Okay.
If you would set that up to do streaming, then you pass the stream over to iJSON, then it would
probably stream it from the server and never keep it all in memory at once anyway. That'd be cool.
Yeah, that'd be cool.
Yeah. But there's probably several layers you got to set that up as.
I'm using generators and iterators more and more because this lazy, doing lazy work is
awesome. It speeds things up quite a bit sometimes.
It's so awesome. And you basically no longer have a memory issue, no matter how much data
you have. You might still have a time issue, but you're not going to run out of memory, right?
You're just going to blast through it one at a time instead of blasting through them all
and then working on them.
Oh, nice. I got to check this out. Cool.
Yeah. So people, if you have large JSON, bring out the iJSON.
Bring out the iJSON.
Yep.
For sure. All right. Well, that's it for all of our main items, Brian. I have, for some reason,
it's like a crazy news week and I actually decided to not throw a bunch of stuff in here,
but to save it till next time because I just don't want to overwhelm everyone. But yeah,
I've got a bunch of stuff to talk about. How about you?
I am just kind of in a lull after Thanksgiving.
Still in the turkey coma?
Turkey coma. Yeah. My head's spinning. There's a lot of stuff coming up in the next couple of
months and I'm pretty excited about it.
That's awesome. Yeah. I'm trying to juggle between getting over that holiday break,
doing all the Black Friday stuff at Talk Python training, and then getting ready for vacation.
It's a lot. But in that little gap where I'm really still paying attention, there's a bunch
of stuff that I ran across this week. So first of all, let's start with this. Django 3 was released.
Oh, sweet. Yeah. So that's pretty cool. And Django 3, there's a couple of things that are
important about it. One is it has official support for MariaDB, which is pretty cool. The other one is
that it begins the journey to the async world.
Dun, dun, dun.
Dun, dun. We'll cross over to the ASGI world.
Yeah. So things like async and await are now making their way to Django.
I saw some messages from some of the developers there saying that this is not like the full story.
This is the first steps. So we've got a long ways to go, but it's first steps, right? It's great to
see them moving that way so that you get super high scalability if you need it. Most people actually
don't need it. But if you do, it's cool to say the answer is not, well, now you move to go or node.
It's you put the async keyword in front of your view method and you're good again.
Yeah. I'm excited to watch this. And actually the Django 3 release, I had dipped my toes in Django
before and I wanted to sort of wait until I knew the 3 was coming. So I was sort of waiting for this
to be official before I started up again. So now I got no excuse. I can get started again.
Yeah. That's awesome. Speaking of excuses running out, the death clock does toll, doesn't it?
Yeah.
Pretty soon the death clock is going to toll for Python 2.7 or 2 in general and in particular
2.7. And as part of that, Python 2 will be removed from Homebrew.
Wow. That's big.
Yeah. That's pretty. It's just another sign that, you know, you probably shouldn't be using
this. It used to be installable. Homebrew install Python at 2 or 2 or whatever the keyword was.
Yeah. That doesn't work. Not yet. Very soon that's going to stop working. So that's cool.
Big news, although I'm not really sure what spin to put on it, so I'll just give you the
news, is Guido is stepping down from the steering council. And this came to us from Vincent Polio.
Great. It's a thing. I don't really know how to spin it other than that.
Well, yeah. I mean, basically, from what I got out of doing a little bit of reading is
there were new elections to re-elect people to the steering council. And Guido didn't put his
name in the hat. And people said, he didn't put his name in the hat. Here, let me put it in a
forum. And then there was a conversation that followed. Like, you know, a lot of people are
enjoying being part of the steering council. I've been on that part of Python for a long
time. And it's not, it's just not fun for me right now. And I've earned some fun in the
programming world. So I want to work on projects and not politics and all that kind of stuff.
So I'm going to let someone else take it. And he also said, the reason I'm comfortable doing
this is it's in good hands. The people already, the other council members there are great.
So y'all don't need me. I'm still going to be around and working on stuff, but I just
don't want to be on the steering council. So anyway, thanks for sending that over. That's
big news, but that's about all I got to say. There wasn't a whole lot written about it really.
GitHub is freezing open source for hundreds of thousands of years to preserve it for humanity.
That's pretty awesome.
Yeah. So the GitHub archive program, I watched this video.
It's well done, right? It's like a polished little video.
Yeah. So I don't know what to make of it other than it's a thing. They're going to put a whole
bunch, all the, as a, like they've already put a bunch of stuff in there, but I guess next
spring they're going to, or when next summer or something, they're going to go up and take
all open source code on GitHub and digitize it into these film, these weird.
Microfish code.
Yeah.
Digital microfish or something that'll last for very long.
Yeah. It's supposed to last a thousand years or something like that. And they're going to put
it in like some cave in, on an Island somewhere.
Greenland, Iceland, some, one of those two, I can't remember.
Right. So why?
To preserve it. I don't know. I'm pretty excited. I have over a hundred public repos that are going
to be up there though. That's going to be awesome. I'm excited for them to get cold and.
Yeah. If, all of, so if civilization collapses, we're counting on future civilizations
to be able to find this cave and figure out some technology to read these microfiche things.
Wow. Those ancient creatures wrote in a very weird symbol form.
Yeah. Okay. Well, good luck future selves.
Yeah. Yeah. Good luck future selves. Hopefully they got, some, some hints, but anyway,
that's pretty cool. And the last thing I had talked about this Python for decision makers course,
that I'm working on. And basically it's a high level of why Python, when keys it, what it's good
for, what it's not good for. I'm also going to do a live webcast that people can check out after the
winter break. So mid January, I'm doing just a free webcast. People can sign up and come ask questions
and we can just share our thoughts on, you know, whether Python makes sense for whatever they're
doing. So people can check that out, click the link and register for free. Cool. That's a lot of stuff,
man. It is. I think we should wrap it up with a joke though. Yeah, let's do that.
This is our tradition now. All right, you go first.
Okay. So, okay. Web developer walks into a restaurant. He immediately leaves in disgust
as the restaurant is laid out in tables.
Oh, must've been a no SQL web developer. Does it want that relational stuff in there?
Yeah, that's pretty funny. That comes from a joke API that Sinjitsu sent over. And so appreciate
that. And this next one as well is from there, but I kind of like it. So I'm going to cover it.
So last time when we were talking about pint, you talked about using prefixes and suffixes
for variable names. Like if I'm going to have something in meters, I'd use an underscore M
potentially or centimeters underscore CM, whatever convention you use. So what is the best prefix
for global variables?
I'm not sure. What? Maybe G?
That would be that. G. G underscore something like that. If it's global. Or hash.
Like you shouldn't have global variables. So let's just comment those babies.
Comment those out.
Pretty sweet. Yeah. Thanks, Sinjitsu, for sending those in. Those are funny.
Yep.
All right, Brian. Well, thanks as always.
Thank you. Bye.
Yep. Bye.
Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's
Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news
item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout
for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy.
Thank you for listening and sharing this podcast with your friends and colleagues.