#160: Your JSON shall be streamed
Sponsored by DigitalOcean: pythonbytes.fm/digitalocean
Brian #1: Type Hints for Busy Python Programmers
- Al Sweigart, @AlSweigart
- We’ve (Michael and myself, of course) convinced you that type hints might be a good thing to help reduce bugs or confusion or whatever. Now what?
- Al’s got your back with this no nonsense guide to get you going very quickly.
- Written as a conversation between a programmer and an type hint expert. Super short. Super helpful.
typing
andmypy
are the modules you need.- There are other tools, but let’s start there.
- Doesn’t affect run time, so you gotta run the tool.
- Gradually add, don’t have to do everything in one go.
- Covers the basics
- And then the “just after basics” stuff you’ll run into right away when you start, like:
- Allowing a type and None:
Union[int, NoneType]
Optional
parameters- Shout out to
Callable
,Sequence
,Mapping
,Iterable
, available in the documentation when you are ready for them later
- Allowing a type and None:
- Just really a great get started today guide.
Michael #2: auto-py-to-exe
- A
.py
to.exe
converter using a simple graphical interface built using Eel and PyInstaller in Python. - Using the Application
- Select your script location (paste in or use a file explorer)
- Outline will become blue when file exists
- Select other options and add things like an icon or other files
- Click the big blue button at the bottom to convert
- Find your converted files in /output when complete
- Select your script location (paste in or use a file explorer)
- Short 3 min video.
Brian #3: How to document Python code with Sphinx
- Moshe Zadka, @moshezadka
- I’m intimidated by sphinx. Not sure why. But what I’ve really just wanted to do is to use it for this use of generating documentation of code based on the code and the docstrings. Many of the tutorials I’ve looked at before got me stuck somewhere along the way and I’ve given up. But this looks promising.
- Example module with docstring shown.
- Simple
docs/index.rst
, no previous knowledge of restructured text necessary. - Specifically what extensions do I need: autodoc, napolean, and viewcode
- example
docs/conf.py
that’s really short - setting up
tox
to generate the docs and the magic command like incantation necessary:sphinx-build -W -b html -d {envtmpdir}/doctrees . {envtmpdir}/html
- That’s it. (well, you may want to host the output somewhere, but I can figure that out. )
- Super simple. Awesome
Michael #4: Snek is a cross-platform PowerShell module for integrating with Python
- via Chad Miars
- Snek is a cross-platform PowerShell module for integrating with Python.
- It uses the Python for .NET library to load the Python runtime directly into PowerShell.
- Using the dynamic language runtime, it can then invoke Python scripts and modules and return the result directly to PowerShell as managed .NET objects.
- Kind of funky syntax, but that’s PowerShell for you ;)
- Even allows for external packages installed via pip
Brian #5:How to use Pandas to access databases
- Irina Truong, @irinatruong
- You can use pandas and sqlalchemy to easily slurp tables right out of your db into memory.
- But don’t. pandas isn’t lazy and reads everything, even the stuff you don’t need.
- This article has tips on how to do it right.
- Recommendation to use the CLI for exploring, then shift to pandas and sqlalchemy.
- Tips (with examples, not shown here):
- limit the fields to just those you care about
- limit the number of records with
limit
or by selecting only rows where a particular field is a specific value, or something. - Let the database do joins, even though you can do it in pandas
- Estimate memory usage with small queries and
.memory_usage().sum()
. - Tips on reading chunks and converting small int types into small pandas types instead of 64 bit types.
Michael #6: ijson — Iterative JSON parser with a standard Python iterator interface
- Iterative JSON parser with a standard Python iterator interface
- Most common usage is having ijson yield native Python objects out of a JSON stream located under a prefix. Here’s how to process all European cities:
// from: { "earth": { "europe": [ ... ] } }
stream each entry in europe as item:
objects = ijson.items(f, 'earth.europe.item')
cities = (o for o in objects if o['type'] == 'city')
for city in cities:
do_something_with(city)
Extras:
Michael:
- Python decision makers webcast on January 14th, 9:30am US Pacific
- Guido steps down from Steering Council via Vincent POULAILLEAU
- GitHub Archive Program, Preserving open source software for future generations, video
- Python 2.7 will be removed from Homebrew, via Allan Hansen
- Django 3.0 released
Joke:
- Question: "What is the best prefix for global variables?"
- Answer:
#
via shinjitsu
A web developer walks into a restaurant. He immediately leaves in disgust as the restaurant was laid out in tables.
- via shinjitsu
Episode Transcript
Collapse transcript
00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to
00:04 your earbuds. This is episode 160, recorded December 4th, 2019. I'm Michael Kennedy.
00:10 And I'm Brian Okken.
00:11 And this episode is brought to you by DigitalOcean. Check them out at pythonbytes.fm
00:15 slash DigitalOcean. $50 credit for new users. Let's start with a topic that we've covered
00:21 quite a bit, Brian. Type hints. Or do you call them type annotations?
00:24 I usually call them type hints.
00:27 Yeah, I go back and forth. And the article you're going to mention actually says you can call
00:30 them both. But yeah, it's kind of funky that they don't have like a definitive name. Maybe they do.
00:34 And I just don't have certainty around it.
00:37 Yeah, one of them's wrong. And whatever the one I'm using is probably the wrong one.
00:41 So we have covered it a lot. And there's a lot of tools that we've talked about, like mypy and
00:45 others. So hopefully we've convinced everybody that they should at least do a little bit of type
00:50 hinting. If you're in that boat, then this article is perfect for you. Al Swigert wrote an article
00:56 called Type Hints for Busy Python Programmers. And that's everybody, right? I mean...
01:01 We're all busy.
01:02 We're all busy. I really like the article. It's a kind of a... When I first went to it,
01:07 I was a little shocked because there's a lot of words there. But it reads like a conversation.
01:11 It's a conversational thing as if it's a programmer talking to an expert. So that sort of tone of,
01:17 I think I should do this, but what do I do? And so he just sort of walks through a whole bunch
01:22 of questions with it. He covers using mypy and typing as the two modules you need and how to use
01:29 those. And there are other tools, but really most people are using mypy. And that's what... That's
01:35 Al's saying that. I don't know if most people are using mypy or some of the other tools.
01:40 We don't... I don't think there's any way to gain statistics on that. But I think mypy is a good way
01:45 to start. And a reminder that using types doesn't affect runtime. You have to run a tool like mypy
01:52 to get it to work. And so I don't think he covers this, but I think that people should hook running
02:00 mypy into their talks system or into their continuous integration pipeline to make sure that these are
02:08 met.
02:08 Yeah. Everything's hanging together. Yeah.
02:10 Yeah. There isn't any compiler to catch you. So one of the things I liked about it is it gets you
02:16 started and it covers some of the basics. But there's... As I've started running it, started using
02:21 type hints, the exact things that I've run into, Al covers. For instance, you've got a function that
02:28 takes an integer, but it also could be none or the default is none or it's optional or something.
02:34 How do you deal with that? There is stuff built in the type int system for unions. So you can say
02:39 it's either an int or none and there's optional parameters. And then a shout out to stuff that
02:46 if you kind of get into it and you might... Like if you're passing callables or sequences around,
02:50 yes, there are ways to type that. And so I think this is actually just a really great quick read and
02:57 it'll get you started pretty fast.
02:59 Yeah. I really like it. I love the conversational style. It seems super approachable and it's kind
03:03 of from the skeptics perspective. It seems a little bit like, oh, I want to do this, but what about...
03:08 I don't know if I'm sure about this. Why can I do this? And yeah, it seems like a lot of fun. I'm
03:13 definitely a fan of type hints these days, not for the mypy reasons. I understand that makes sense for
03:19 very large projects. But for me, like the editors are just so much better if you can just give them
03:25 these little hints here and there. I'll give you an example. Like for the database for Python Bytes,
03:32 we are talking to MongoDB with Mongo Engine. And the way you specify it is much like SQLAlchemy
03:37 or even maybe more so like Django ORM where you say this column is a Mongo Engine column of type such
03:45 and such. Well, the type editor, the editors think the type is like a column, but it's not a column.
03:50 It's probably a date time or it's a string or it's a list or it's anything, but what do they think
03:54 it is? And so you can use type hints to say this column is an integer and it's equal to one of these
04:00 column things. Oh, okay. That'll tell all the editors when I hit dot on this, don't tell me it's
04:04 one of these column things. I don't want to know that. I want to know what its value is because
04:07 that's what I program against when I run it. So there's just all these different angles that make
04:11 it super valuable. I'll have to take a look at how you're using that. That's pretty cool.
04:14 Yeah, for sure. Speaking of cool, another thing that we have beaten to death, but is very tough
04:19 and leathery and has not died is the ability to create an application out of Python code,
04:25 right?
04:25 Yes.
04:26 Well, I stumbled across this thing. I don't believe it is new, but it actually is interesting
04:32 on layers. There are layers of interest here called auto-py-2-exe. Auto-py-exe.
04:40 Okay.
04:40 Okay. So this is a GUI and it's built with eel. First of all, it's a GUI Python thing built
04:47 with eel. Remember eel?
04:48 I know. Remind me.
04:49 Yeah. It lives in the ocean. Some of them are electrical. You shouldn't touch them. No
04:53 wait, hold on. It's a library for making electron-like HTML and JavaScript GUI apps with Python.
04:58 Oh, okay. It says electric snake. Got it.
05:02 Yeah. Electron. Yeah, exactly. So it's very much like that. I think it's for kind of simplistic
05:07 ones, but it certainly works for this app, which is pretty cool. So here's a Python app built in eel.
05:12 That's cool. And it's kind of meta in the sense that it is, its job is to make other Python
05:18 applications out of native Python code.
05:20 Okay. Interesting.
05:21 Okay. Yeah. So what it does is basically is this is automation over top py installer. And you might
05:26 think, okay, well, I could just, you know, pass the command line arguments and just do that.
05:30 But there's something really nice about having a GUI say, here's the icon. I just put the icon
05:36 into the app. And here's the, the name I want for the title and such and such. And, you know,
05:41 here are the additional binary files or additional resource files I want bundled and shipped with my
05:47 application. We're using this JSON file and we're using this, I don't know, this image that we're
05:51 going to embed somewhere. Just include those. And like, so there's all sorts of settings in this
05:56 auto py.dxe where you can just pick all that stuff out and you don't really have to be an expert at
06:00 all with py installer, but you can build pretty nice stuff. Like there's a checkbox. Do you want just a loose
06:05 set of files or do you want it bundled into a single .exe that I can just give and it'll contain
06:10 Python and the dependencies and all that? Okay. This does sound neat. Yeah. It's pretty neat, right?
06:14 Yeah. And so really people want to, if this sounds interesting to you and you want to find out,
06:19 Hey, is this cool? Is this interesting? There's a three minute YouTube video. You can just check that
06:23 out. Okay, cool. So I put the link to the three minute video at the end. People can click on that,
06:27 watch it. And then within three minutes, you'll decide whether or not this is useful for you. But
06:31 some of the conversation I saw people saying, yeah, I know it just automates PyInstaller and
06:36 that's cool. But there were things I couldn't get to work with PyInstaller, but doing it here,
06:39 it worked fine. So I feel like there's a lot of value in this. Yeah. Nice. Yeah. Speaking of value,
06:44 DigitalOcean, they are adding a lot of value to all the hosted things of the world, including our stuff
06:49 that's hosted there across all the different sites and whatever. I believe we're doing like 15 terabytes
06:54 of traffic a month through DigitalOcean. That is not messing around. That's a lot. So been rock solid.
07:00 The other thing that they have that they're just launching is the Kubernetes, the container, their own
07:05 container registry for things like their Kubernetes service. So we've got like Docker hub and places
07:11 that you can put your containers, but that's on the public internet and things like that. It'd be nice
07:16 to have a private little one for your private code that you want to make sure doesn't somehow leak with,
07:21 you know, making it public or private on accident or whatever. So you can use their DigitalOcean
07:26 container registry and have nice private container definitions, which is pretty cool, among other things.
07:31 So check them out at pythonbytes.fm/DigitalOcean. Like I said, $50 credit for new users to get started
07:37 and play around with this. Speaking of playing around, I've never played with Sphinx. Oh, you never have?
07:41 I haven't. I've had no reason to do it, but it seems something like something that's useful. Maybe you should learn about it.
07:46 Yeah. So one of the things like, so I have, what I've wanted to use Sphinx for,
07:51 so Sphinx can be used for building up documentation and makes like a documentation website based on
07:57 either REST or Markdown or not the REST in that. Restructured text. Restructured text.
08:01 Yeah. Thanks a lot, programming world, for having two things that could be abbreviated REST.
08:06 In the C++ land, there was some auto-documentation tools to where you can take, and I know other
08:13 programming languages have this too, where you can take information about the structure of a program
08:18 and automatically generate some web documentation about how to use it and what the parameters are
08:25 for different functions and stuff like that. So you can do that with Sphinx, and that's the thing
08:30 I've been wanting to do, and I've tried several times and failed. And it's usually because,
08:35 probably on my part, but the tutorials get me part of the way, and then, you know, they reference a
08:40 bunch of documentation and stuff, and I don't want to become a Sphinx expert. I want, that's kind of
08:45 the point of this. I want to be able to just point it at my code and go, go do it. And like, for instance,
08:51 pulling out, so what does a function do? What do different tests do? I'd like that to be embedded
08:57 in the doc string so that when I'm coding, I can make sure that that's correct, and then that's just
09:02 automatically generated. So what I'm highlighting today is an article called How to Document Python
09:08 Code with Sphinx. It's a short tutorial by Mosh Zadka. That's a cool name. And so, walks through,
09:16 it has a little example of a little module with the doc string, and then the small example of how to set
09:23 up your index.rst. And I like Markdown better, but this is so short, I'd probably just use the
09:29 restructured text version. Points out that you need a few extensions. You need Autodoc, Napoleon,
09:34 and ViewCode. And then it gives you, just gives you the conf.py file you need to make this work.
09:40 And then how do you generate this stuff? Well, there's this long, weird command. It's not that
09:45 weird, but I don't want to memorize it. So this tutorial also shows you how to set up a tox.any
09:50 file with this Sphinx built in. So you can just say tox and your docs get rebuilt. So I like this
09:57 tutorial, and I'm going to give it a shot. Yeah, it's a really cool thing, and automatically
10:01 generating your documentation. Not terrible at all. Of course, it doesn't cover, like,
10:05 okay, well, now that you've generated it, it's a bunch of web pages and stuff. Where do you put
10:09 that? But for internal documentation within a team, you got your own way to do it, whether you're
10:15 pushing it into a, like a small web server, or just leaving it with your code, or whatever. So
10:21 leave that to the user. Yeah, or put it on DigitalOcean, or not DigitalOcean,
10:24 GitHub Pages, right? Yeah. That's one thing you do. You put it on Netlify. A lot of options for just
10:30 hosting HTML. Yeah, or throw it into a GitHub cluster, or a Kubernetes cluster. Maybe not. That's right. So, you know, something that's always
10:39 amused me is the association of Python ecosystem and language with snakes. Because that's not the
10:48 origin, right? That's not what Python is named after. It's named after Monty Python. All these
10:52 apps and stuff should have, like, medieval knight type of names, but they don't. So I'm going to tell
10:57 you about one that is called Snake. Snake? Snake? S-N-E-K. Snake? Don't quite know the right
11:05 accent to put on it to pull it off, but Snake? It's pretty awesome. So it's a PowerShell module
11:10 for integrating standard CPython, which is Python code, and various packages with PowerShell. So if
11:18 you're doing anything on Windows, any sort of automation or advanced shell stuff, you're probably
11:23 using PowerShell. Although the regular terminal is getting better as well. And, you know, the story of
11:29 Python on Windows has always had a little caveat, or maybe a little star, like, yeah, you always do this,
11:35 except for on Windows, you type this other thing. And it's slowly getting better, right? Steve Dower's
11:39 done a lot of work to improve that. Yeah. With, say, the Python installer that comes off the Windows 10
11:44 store has Python 3 as a command, not just Python and so on. But here's one more thing that you can do to
11:51 make working on Windows with Python better. If you do anything with PowerShell, you can write scripts
11:57 now that have sections of those scripts or libraries for PowerShell written in Python. And it uses this
12:04 library called Python for .NET, which is a new modern way to share the entire last library of .NET and all
12:14 the objects you might create there with Python code. And similarly, say, this Python thing returns some
12:20 funky .NET type that can be converted so, like, it looks normal to the rest of the PowerShell land,
12:25 which is generally .NET stuff. So if you're doing stuff with PowerShell, check this out. It's pretty
12:31 cool. The syntax is funky, but, you know, it's PowerShell syntax. So there you have it.
12:35 Yeah. Well, okay. So I think that people can't be blamed for doing the snake thing because our logo
12:41 is two snakes. I know. I think that that is absolutely true. I think it should be like maybe
12:47 some cheese. Yeah, exactly. Or maybe a shield. Yeah, I don't know. Whatever. I don't mind having it
12:52 being snake. The snake is cool. And there's the, you know, the Python snake. It's just funny.
12:55 That's not actually the name. No. Killer Rabbit would be nice. Yes. The Killer Rabbit would be
13:00 fantastic. If you just got an exception and that appeared as an emoji, that would be beautiful.
13:05 Yeah. Or maybe a failed test in pytest, right? You could write a plugin. It unleashes the killer,
13:10 the killer rabbit and all that. All right, we're going down a bit of a hole. So maybe get us out and
13:16 talk, get us back to the databases. Okay. So databases. Since in Python world, one of the things we like to do
13:22 is use Pandas because Pandas makes dealing with data like so much easier with data frames and stuff.
13:29 And also one of the things with databases that we can use is SQLAlchemy if it's a SQL based database.
13:35 And most of them are. So you can just take Pandas and put it together with SQLAlchemy and put it
13:42 together with your database and just slurp your entire data tables into Pandas data frames and then
13:48 manipulate them there. So you're good to go, right? Right. Unless you have like five gigs of data,
13:52 then you're good to swap and be very slow. Yeah. Or if terabytes of data, that'd be bad. Yeah.
13:58 Very bad. So all hope is not lost because I'm going to guess Arena Trong wrote this article called
14:05 How to Use Pandas to Access Databases. And it's really great. One of the recommendations she uses
14:11 is don't be afraid of the command line for SQL because you can play around with queries and look
14:16 to see how big the shape of your data is to start with, which is a good thing. But then goes on to
14:23 tips on how to put this together correctly. And I'm just going to read out the highlights, but
14:30 there's examples of all these. So I think it's great to have a look. Don't grab a whole table limit
14:35 to just the fields that you care about that you're going to actually use. Tables often have a lot of
14:40 columns that you don't need. So don't grab everything. Limit the number of records. You can either use the
14:45 limit field to say, I only want to grab like a thousand elements or something. You can also combine
14:50 that with just the number of fields, but also you can also pick out, say, I know I'm going to already
14:56 throw away all the rows that don't have a particular value in this one field. So you can
15:02 limit that in your query. One of the things you'll often run into with SQL databases, of course, is to
15:07 need the need to use joins. And the quick recommendation there is let the database do that.
15:13 Put it in your query. Don't do the join within Pandas because you can do it with Pandas and SQL
15:19 coming together, but don't. It's going to be slower.
15:23 Right. Load it on a memory and have the data table merge it. That seems probably not like the
15:28 best way to do it.
15:29 Right.
15:29 Databases are pretty good at joining stuff.
15:32 Yeah. And I mean, if you find, of course, it didn't talk about this, but if you find you're doing
15:36 that a lot, you can kind of restructure some of your queries within. I think you can build that in,
15:41 like make a join be faster. I don't know. I'm not a database person, so.
15:46 You could put indexes. You could write a store procedure and call that. There's a couple options.
15:50 Okay. Ask Michael how to do it. Anyway, maybe not. So a lot of this is we're caring about memory. So
15:57 how do you figure it out? Like how much memory you're going to use? And I didn't know you do this,
16:01 but there's, she has a tip of using query called memory, memory underscore usage, and then summing it.
16:07 This is, I don't know what the field comes back, but you can do a query and grab the memory, but you do
16:13 like a small query, like just a hundred, a hundred rows or something, and then take that. And then you
16:18 can use that to estimate the, the total size. If when you're going to read out like a thousand or
16:24 2000 or something. And then the last one, I'm actually didn't quite follow, but I think I kind
16:30 of followed. The tip is to talk that the types get converted, like integer types get converted to like
16:37 64 bit types. I think, and you don't really need that for a lot of stuff. So there's a,
16:43 I'm calling this out because it's definitely a, for the user read this if you need it, but
16:48 there's a little bit on using chunking to, and then combining that with specifying what types,
16:55 pandas types to use so that you get smaller data footprints.
17:01 Yeah. Yeah. So for example, maybe one of the columns in the data frame that just the way it
17:05 came back from the database is a 64 bit type integer and you know, everything can actually fit into an
17:11 untyped and unsigned 16 bit integer. Well, that'll save you. It'll be a quarter of the size of the data
17:18 if you'd make that conversion. Right. So got a million of them, a quarter of the size of a million,
17:22 it's possibly meaningful.
17:24 Yeah. I mean, a lot of stuff that like, for instance, a lot of stuff I'm, one of the tables I deal with has,
17:28 has a numbers for the different specifying, which kind of a result we're looking at, for
17:37 instance. And it's, they each have like a number code. There's only like six. So the number can be
17:43 one through six. I don't need 64 bits to store that. So yeah, for sure. So it's good. Do you use a
17:48 pandas with SQLAlchemy and any of our doing it?
17:51 Not really. I, there's a few times I've been thinking I should probably try to do some BI,
17:57 like graphing type stuff, maybe about popularity of episodes or things like that, you know, figure
18:03 out what day I should release our podcast on, but I haven't done anything. So no.
18:06 Okay. But you do use a SQLAlchemy though, right?
18:08 Some, but not for the main site because that's MongoDB.
18:12 Oh, right.
18:13 I'm using something equivalent, like we're using Mongo Engine, which is super equivalent.
18:16 Okay.
18:16 Yeah.
18:17 Nice.
18:17 So yeah, I suspect there's probably some interesting MongoDB integration with pandas
18:22 that I just don't know about, but I don't know any equivalent of there. Well, this next
18:26 one I'm going to cover is very similar to what you talked about. It talks about how do I basically
18:31 efficiently work with data from this one source databases? And how do I not just select star
18:37 from all the tables into pandas and work on it? Another problem you might have is you might
18:43 have JSON data. JSON is sweet, but one thing JSON is not very obviously good at is streaming
18:50 data, right? Because how do you know where the closing curly braces? Maybe you've got some
18:56 key and then it's got an array of stuff. And the closing bit of that array is not until, you
19:01 know, maybe 500 megabytes down the file. That's a problem potentially, right? So if you load
19:07 it with, if you just load up a, you know, import JSON and do a JSON load, give it a file pointer,
19:13 it's just going to hoover that all up and jam into it in one giant dictionary and spit it out.
19:18 But if it's huge, obviously what you want to do is iterate over it and process it item by
19:23 item. Yeah. I mean, that's what you would do for a regular file if you could take it line
19:27 by line, but because it's this formatted thing, it's much harder. And something I ran
19:31 across last time, time before we were talking about some other project and it said, oh,
19:37 this project uses iJSON. I'm like, wait a minute, what is iJSON? That's interesting.
19:42 Yeah. Yeah. So iJSON is an iterative JSON parser with the standard Python iterator interface. So
19:49 what you can do is basically point it at a file stream, some like search expression kind of in
19:57 the JSON document, and then it will stream those back to you generator style one at a time.
20:02 That's so cool.
20:03 Yeah. So there's a little example I put in the show notes. It says, imagine we've got some object
20:09 called earth in our JSON document. It has a bunch of continents and within each continent, it lists
20:15 out say cities and states and so on. If that was huge, probably it's not, but if it were, you would
20:20 want to iterate over it. So you just go to the iJSON and go, give me the items of this from this file
20:25 for earth.europe.item. And item is just a way to say, stop there. And then what you do is just say
20:32 for thing in, you know, whatever came back from that load that you did and you just iterate over
20:38 it in a for in loop.
20:38 That's really neat.
20:39 Yeah.
20:40 And it's doing, like I was looking at the website, there's an example with grabbing it from like a
20:46 URL because that's, you might be getting it from an API endpoint.
20:50 Yeah.
20:51 So I'm guessing that it's not grabbing everything from the endpoint at first.
20:55 It probably depends on how the endpoint is set up or even maybe how the object that's
21:01 querying it back. So I think they're using URL lib or something like that there.
21:05 Okay.
21:05 If you would set that up to do streaming, then you pass the stream over to iJSON, then it would
21:10 probably stream it from the server and never keep it all in memory at once anyway. That'd be cool.
21:14 Yeah, that'd be cool.
21:15 Yeah. But there's probably several layers you got to set that up as.
21:17 I'm using generators and iterators more and more because this lazy, doing lazy work is
21:23 awesome. It speeds things up quite a bit sometimes.
21:26 It's so awesome. And you basically no longer have a memory issue, no matter how much data
21:30 you have. You might still have a time issue, but you're not going to run out of memory, right?
21:34 You're just going to blast through it one at a time instead of blasting through them all
21:37 and then working on them.
21:37 Oh, nice. I got to check this out. Cool.
21:39 Yeah. So people, if you have large JSON, bring out the iJSON.
21:43 Bring out the iJSON.
21:44 Yep.
21:45 For sure. All right. Well, that's it for all of our main items, Brian. I have, for some reason,
21:50 it's like a crazy news week and I actually decided to not throw a bunch of stuff in here,
21:55 but to save it till next time because I just don't want to overwhelm everyone. But yeah,
21:59 I've got a bunch of stuff to talk about. How about you?
22:01 I am just kind of in a lull after Thanksgiving.
22:06 Still in the turkey coma?
22:08 Turkey coma. Yeah. My head's spinning. There's a lot of stuff coming up in the next couple of
22:12 months and I'm pretty excited about it.
22:15 That's awesome. Yeah. I'm trying to juggle between getting over that holiday break,
22:19 doing all the Black Friday stuff at Talk Python training, and then getting ready for vacation.
22:23 It's a lot. But in that little gap where I'm really still paying attention, there's a bunch
22:27 of stuff that I ran across this week. So first of all, let's start with this. Django 3 was released.
22:34 Oh, sweet. Yeah. So that's pretty cool. And Django 3, there's a couple of things that are
22:38 important about it. One is it has official support for MariaDB, which is pretty cool. The other one is
22:45 that it begins the journey to the async world.
22:48 Dun, dun, dun.
22:49 Dun, dun. We'll cross over to the ASGI world.
22:52 Yeah. So things like async and await are now making their way to Django.
22:58 I saw some messages from some of the developers there saying that this is not like the full story.
23:02 This is the first steps. So we've got a long ways to go, but it's first steps, right? It's great to
23:07 see them moving that way so that you get super high scalability if you need it. Most people actually
23:13 don't need it. But if you do, it's cool to say the answer is not, well, now you move to go or node.
23:19 It's you put the async keyword in front of your view method and you're good again.
23:22 Yeah. I'm excited to watch this. And actually the Django 3 release, I had dipped my toes in Django
23:28 before and I wanted to sort of wait until I knew the 3 was coming. So I was sort of waiting for this
23:34 to be official before I started up again. So now I got no excuse. I can get started again.
23:39 Yeah. That's awesome. Speaking of excuses running out, the death clock does toll, doesn't it?
23:44 Yeah.
23:45 Pretty soon the death clock is going to toll for Python 2.7 or 2 in general and in particular
23:51 2.7. And as part of that, Python 2 will be removed from Homebrew.
23:56 Wow. That's big.
23:57 Yeah. That's pretty. It's just another sign that, you know, you probably shouldn't be using
24:00 this. It used to be installable. Homebrew install Python at 2 or 2 or whatever the keyword was.
24:05 Yeah. That doesn't work. Not yet. Very soon that's going to stop working. So that's cool.
24:12 Big news, although I'm not really sure what spin to put on it, so I'll just give you the
24:16 news, is Guido is stepping down from the steering council. And this came to us from Vincent Polio.
24:21 Great. It's a thing. I don't really know how to spin it other than that.
24:25 Well, yeah. I mean, basically, from what I got out of doing a little bit of reading is
24:29 there were new elections to re-elect people to the steering council. And Guido didn't put his
24:35 name in the hat. And people said, he didn't put his name in the hat. Here, let me put it in a
24:39 forum. And then there was a conversation that followed. Like, you know, a lot of people are
24:42 enjoying being part of the steering council. I've been on that part of Python for a long
24:47 time. And it's not, it's just not fun for me right now. And I've earned some fun in the
24:52 programming world. So I want to work on projects and not politics and all that kind of stuff.
24:56 So I'm going to let someone else take it. And he also said, the reason I'm comfortable doing
25:00 this is it's in good hands. The people already, the other council members there are great.
25:06 So y'all don't need me. I'm still going to be around and working on stuff, but I just
25:09 don't want to be on the steering council. So anyway, thanks for sending that over. That's
25:12 big news, but that's about all I got to say. There wasn't a whole lot written about it really.
25:16 GitHub is freezing open source for hundreds of thousands of years to preserve it for humanity.
25:25 That's pretty awesome.
25:25 Yeah. So the GitHub archive program, I watched this video.
25:29 It's well done, right? It's like a polished little video.
25:31 Yeah. So I don't know what to make of it other than it's a thing. They're going to put a whole
25:37 bunch, all the, as a, like they've already put a bunch of stuff in there, but I guess next
25:43 spring they're going to, or when next summer or something, they're going to go up and take
25:47 all open source code on GitHub and digitize it into these film, these weird.
25:53 Microfish code.
25:55 Yeah.
25:55 Digital microfish or something that'll last for very long.
25:58 Yeah. It's supposed to last a thousand years or something like that. And they're going to put
26:01 it in like some cave in, on an Island somewhere.
26:05 Greenland, Iceland, some, one of those two, I can't remember.
26:07 Right. So why?
26:09 To preserve it. I don't know. I'm pretty excited. I have over a hundred public repos that are going
26:13 to be up there though. That's going to be awesome. I'm excited for them to get cold and.
26:16 Yeah. If, all of, so if civilization collapses, we're counting on future civilizations
26:23 to be able to find this cave and figure out some technology to read these microfiche things.
26:29 Wow. Those ancient creatures wrote in a very weird symbol form.
26:33 Yeah. Okay. Well, good luck future selves.
26:37 Yeah. Yeah. Good luck future selves. Hopefully they got, some, some hints, but anyway,
26:40 that's pretty cool. And the last thing I had talked about this Python for decision makers course,
26:46 that I'm working on. And basically it's a high level of why Python, when keys it, what it's good
26:53 for, what it's not good for. I'm also going to do a live webcast that people can check out after the
26:57 winter break. So mid January, I'm doing just a free webcast. People can sign up and come ask questions
27:03 and we can just share our thoughts on, you know, whether Python makes sense for whatever they're
27:07 doing. So people can check that out, click the link and register for free. Cool. That's a lot of stuff,
27:11 man. It is. I think we should wrap it up with a joke though. Yeah, let's do that.
27:15 This is our tradition now. All right, you go first.
27:18 Okay. So, okay. Web developer walks into a restaurant. He immediately leaves in disgust
27:24 as the restaurant is laid out in tables.
27:27 Oh, must've been a no SQL web developer. Does it want that relational stuff in there?
27:33 Yeah, that's pretty funny. That comes from a joke API that Sinjitsu sent over. And so appreciate
27:38 that. And this next one as well is from there, but I kind of like it. So I'm going to cover it.
27:42 So last time when we were talking about pint, you talked about using prefixes and suffixes
27:46 for variable names. Like if I'm going to have something in meters, I'd use an underscore M
27:51 potentially or centimeters underscore CM, whatever convention you use. So what is the best prefix
27:57 for global variables?
27:59 I'm not sure. What? Maybe G?
28:00 That would be that. G. G underscore something like that. If it's global. Or hash.
28:06 Like you shouldn't have global variables. So let's just comment those babies.
28:11 Comment those out.
28:12 Pretty sweet. Yeah. Thanks, Sinjitsu, for sending those in. Those are funny.
28:15 Yep.
28:15 All right, Brian. Well, thanks as always.
28:17 Thank you. Bye.
28:18 Yep. Bye.
28:18 Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's
28:23 Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news
28:30 item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout
28:34 for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy.
28:39 Thank you for listening and sharing this podcast with your friends and colleagues.