« Return to show page
Transcript for Episode #163:
Meditations on the Zen of Python
00:00 KENNEDY: Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is Episode 163, recorded live on location here at the Portland West Python meetup. Hello, everyone. And it was recorded January 7th, 2020. Brian, it's 2020.
00:20 OKKEN: Yeah, was that difficult to remember?
00:21 KENNEDY: It's, you know, I'm really not used to it. I just got used to 2019. So we're having problems, but we'll get there. Yeah, well, it's really great to be here with everyone. This is the first time we've done a live recording in a while. We do this at PyCon a lot, but PyCon's not that frequent. So, here we are in Portland. It's great.
00:36 OKKEN: Yeah, thanks, and thanks for everybody for coming out. This is wonderful.
00:38 KENNEDY: Yeah, absolutely.
00:39 OKKEN: I've got the first one, right?
00:40 KENNEDY: You've got the first item, so some zen. I think it's a new year, it's a new decade. Let's kick it off with a little zen.
00:47 OKKEN: Yeah, a little zen. So we going to take 20 minutes and just meditate. No. The first one I want to talk about, there's, probably going to get this name wrong. Why did I pick this? Moshe Zadka. It's a pretty cool name, though. Wrote an article called Meditations on the Zen of Python. And if you don't know about The Zen of Python, hopefully you do. We going to put a link in the show notes, but there's also, in any REPL you can say import this and it will show you a whole bunch of little koans, little snippets of The Zen of Python.
01:16 KENNEDY: One of the cooler things you can do in Python, second only to import antigravity.
01:19 OKKEN: Yeah, that's a good. So, Moshe says that this is, the Zen of Python is not the rules of Python or the guidelines of Python. It is full of contradictions and illusions. It is not intended to be followed. It is intended to be meditated upon. So you can read these and think about your code, and actually, I kind of like it. I like that idea, because when I've read through them, there are some that contradict each other, and I want to bring out a few of them that he points out. The first is beautiful is better than ugly. And one of his comments is that consistency helps. So things like Black and Flake8 and Pylint are very useful in making consistent looking code.
01:57 KENNEDY: Right, but those are only table stakes, right? It's just not ugly, it's not beautiful because of that.
02:01 OKKEN: Right, so even more important, only humans can judge what humans find beautiful. So code reviews and collaboration and a collaborative approach to writing code are the only realistic way to build a beautiful code. So listening to other people is an important skill in software development, and also just looking at code and seeing if you think it looks nice. I think it's good. So, a couple more. Complex is better than complicated. And that one always threw me. I wasn't sure why that was in there. Moshe says, "When solving a hard problem, it's often a case that no simple solution will do. In that case, the most Pythonic strategy is to go from the bottom up. Build simple tools and combine them to solve the problem." That's kind of a Unix style. Is that how you view the complex is better than complicated, or?
02:49 KENNEDY: I don't know if I've ever thought of in these terms, and I do like this article 'cause it did make me think like that, but I certainly think about software that way. I feel like so many people get caught trying to overthink it and they're like, oh, I can't get started. I'm trying to get the right, the exact right thing before I can write my first line of code. And it's like, no, you're never going to know until you work on it for two days, then you're like, "Oh, I should've done this." But, well, you didn't know what you knew now. So what are you going to do? Just get started. Right? It's my philosophy.
03:14 OKKEN: And the last one I want to talk about is readability counts. So in the face of immense pressure to throw readability to the side and just solve a problem, the Zen of Python reminds us that readability counts. Writing code so it can be read is a form of compassion for yourself and others. And one of the reasons why I highlighted this is we're going to talk about optimization and speed later on. And speed is great, but making sure that your program is readable and maintainable is very important.
03:39 KENNEDY: I really like this article. Well done. Because we've all heard about The Zen of Python. We've probably all typed import this. But it's a little vague. And this is not here's what it means. This is here's me trying to think about it sort of philosophically, And I've never seen anyone write that way about it before, and I thought it was really cool.
03:57 OKKEN: I'd love other people to come up with their ideas about it. So that'd be cool.
04:01 KENNEDY: Yeah, absolutely. So Python in the web doesn't usually have a James Bond sort of places getting raided by the police, Secret Service, international angle to it. But this next item does. So you know what the most popular web server is in the world? Used to be Apache, now it's Nginx. Our stuff runs on Nginx, for example. However, there was some news a few weeks ago. The Nginx offices in Russia, were raided by the Russian police and some of the core developers were detained. And the reason is, this is not as interesting as I made it out to be, I don't really think. But the reason was, the guy who created it, I have his name here, Sysoev. He created it, and at the time he was working for Rambler.ru. And Rambler.ru is like a Google, Yandex type of company, search engine in Russia, and he worked on this in his spare time, and he open sourced Nginx, and then later turned it into a company. Well, Rambler came along and said, "Hey, you know what? You worked on that while we were employing you. Nginx is ours. We're taking it over." Meanwhile, Nginx has been bought by F5, an American company, and they own it. And so there's all this intrigue around it. And yeah, so that happened.
05:20 OKKEN: So a bunch of my friends from Spokane that when Agilent closed down there went to work for F5, and I'd never heard of 'em before, and then they show up in this news article. That's pretty interesting.
05:30 KENNEDY: Yeah, they're all about the networks, but apparently servers. So I received an update, an email from Nginx a few days later, and I'll just read it so I get it right. They said promptly following this event that I just described, "We took measures to ensure the security of our master software builds for Nginx, Nginx Plus, Nginx WAF, and Nginx Unit, all of which are stored on servers outside of Russia. No other products are developed in Russia. F5 remains committed to innovating with Nginx." Nginx Plus, et cetera, et cetera, all the products, And we continue to provide the best in class support." You can expect that to come. So who knows where this is going to go? That's pretty interesting.
06:10 OKKEN: If you use Nginx, should you worry about this, or?
06:13 KENNEDY: That was why I brought it up, because if it's the most popular web server in the world, some folks, their ear's going to perk up and say, "Wait a minute, what?" I mean, I don't think this is like Russia trying to impose its will on the world. I think this is just a dispute between a Russian company and some Russian folks who created Nginx, but it could have knock-on effects quite far. So yeah, it's pretty interesting. I think just keep an eye on it, really. This episode is brought to you by well, us.
06:41 OKKEN: Really?
06:41 KENNEDY: Yeah, normally is brought to you by other companies, but this time, you know, we both have interesting things to talk about and we have a gap, so I just want to tell you about some of our stuff. So if you want to check out the courses that I'm creating or set up the business stuff, just visit pythonbytes.fm/biz. And here there's something about testing over at pythonbytes.fm/pytest and people can check that out there, as well.
07:04 OKKEN: Cool, did you set up a URL redirect from there?
07:07 KENNEDY: I didn't want to say the whole name of, the whole domain name and URL, so yeah, that's yours.
07:11 OKKEN: It points to Python Testing with Pytest, which was published in 2017, but I still think is the very best way to get up to speed very quickly on Pytest.
07:20 KENNEDY: Yeah, absolutely. And also, we also have a Patreon that you set up at patreon.com/pythonbytes. So I have some thoughts on this next one, but why don't you kick it off? This one's from the creator of Flask, but not the current maintainer of Flask.
07:33 OKKEN: Oh, that's true? Right, so I brought this up because I was curious what your thoughts were. So the next one is from Armin...
07:40 KENNEDY: Armin Ronacher.
07:42 OKKEN: Armin Ronacher, thank you. He wrote an article called I'm Not Feeling the Async Pressure. The idea is like async is all the rage. We've actually talked about async quite a bit on the podcast. And I think a lot of people are concerned about it, and it's one of the reasons why it's going in places because I think there's some pressure of people to leave Python to go to Go or other things.
08:02 KENNEDY: Node.js was an early example of that, yeah.
08:04 OKKEN: Yeah, but before you go towards async, Armin is warning people to make sure you understand flow control and back pressure. And I had never heard of back pressure, but back pressure is the resistance that opposes the flow of data through a system. Back pressure sounds quite negative, but is here to save your day. If parts of your system are async, you have to make sure the entire flow through the system doesn't have overflow points. And then Armin goes through an example with a reader and writer, and it seemed like simple code, but it really got hairy really kind of fast. In the example, yeah, it got hairier than I expected, and he says, "Async brings you new footguns. Async and await are great ways to encourage writing stuff that will behave catastrophically when overloaded. For you developers of async libraries, here's a New Year's resolution for you. Give back pressure and flow control the importance they deserve in documentation and API." And there's just some hidden things within buffers and things like that, so.
09:08 KENNEDY: Yeah, well, this is a pretty nuanced article, and it's pretty interesting, and it comes from someone who knows a thing or two about the network programming, Armin being the original creator of Flask. That said, my reaction to reading it, my reaction was, there were a lot of examples of, and here's if you overpressure the system when you write an async system, it will fall down, right? Imagine you only allow 50 database connections and suddenly you get 10,000. My sort of reaction to this was, well, what is the response of the system going to be when it's single threaded and you give it 10,000 connection requests? They're all going to time out except for, it's just going to crash. So there's a crash in an async world, there's a crash in a non async world with enough traffic, yes, but at the same time, if you can write basically the same code, put async and await in front of a few bits and you can get 10 or 100 times the amount of performance out of a given piece of hardware before it crashes, that seems better to me. So I mean, I'm sympathetic to the problem, but at the same time, it's always like, well, if we give it way too much pressure, it's going to crash. Well, if it wasn't parallel, it would crash before then.
10:16 OKKEN: My thoughts were like, can you, are there ways to throttle? I don't know enough about all the way to do network stuff. So if I'm setting up a web service, for instance, can I throttle the input to say, to not allow 10,000 connections and just allow 5,000 or something?
10:32 KENNEDY: Right, right potentially. So maybe with something like Uvicorn or something, you could set that up. I honestly don't know. It seems to me the danger that he's addressing is when the system itself is generating the input. Like, we had this example of a guy who sent us a message and said, "Hey, I had this web scraping thing. It was really slow. We turned async loose on it and it crashed the server 'cause it ran out of memory 'cause it processed it too quickly," right? Like, there you need to find a way to slow it down. But when you're running a web server, you don't control how many people want to get to it. There's just people want to get to it, and they either can or they can't. And with async, more of them would be able to then otherwise. That's my thoughts.
11:09 OKKEN: Okay, and I guess that the, I guess my thoughts would be, if you're going to throw async at a problem, make sure that you do capacity testing on it, as well.
11:18 KENNEDY: Yeah, well, it's going to fail somewhere else, right? And so you're going to, maybe your database isn't set up for it. Maybe your microservices can't handle all the traffic. Like, there's going to be something, right? It's just going to show up somewhere else, but in general, I think you're going to get better scalability with it than without it. So if you're not generating that pressure, if you're not generating the traffic, then I don't know what choice you have. That's my thought.
11:38 OKKEN: It was an interesting read.
11:39 KENNEDY: Yeah, it definitely was, it definitely was. Let's go for something a lot simpler than a deep thing on streaming and buffers and async. How about a new way to time your code?
11:49 OKKEN: Yeah, that sounds good.
11:50 KENNEDY: Yeah, so this one comes to us from Doug Farrell, who works on the Real Python team, and as part of their work, they've got to time all sorts of things for their articles. You know, Dan Bader and crew over there. And so they came up with this thing, either they came out of Real Python, I think, or possibly they were just using it a lot. But it's this thing called codetiming. So if you've got some code and you want to know how long it takes to run, you could say, create a time, capture the date time, do some work, capture dot now again and do a delta, or you could even use performance counters and other things, but you can use this library, you just, there's a bunch of different things you can do. You create a timer class, you can call start, do some work, stop, and out comes the time. Or you can put it in a context manager, like a with block. As soon as it goes through the with block when it's done, that bit is timed, or you can use it as a decorator. And you can also wire up a logger, which is kind of cool. So you can see it'll just output standard Python logging with time information of when it's doing bits of its thing. Give it a name and it'll tell you how long it took.
12:53 OKKEN: Cool, well, they add statistics, too, so I can get min, max, and average and standard deviation.
12:58 KENNEDY: Yeah, that would actually be cool. Well, it's open source. I bet they accept PRs. Actually, there's a bunch of features I want to add to it, and I started messing around with it, and I'm like, put it down. I have other things to do. I'm already too busy. I don't need this. I'm going to leave it alone. But yeah, it's a pretty cool little timer class, and I'm going to probably use that. I like it.
13:15 OKKEN: Yeah. I thought this, a nice follow on for this, the timer would be an article called Making Python Programs Blazingly Fast.
13:24 KENNEDY: That sounds good. We all want that.
13:26 OKKEN: You need to time stuff. You should never, I mean, hopefully we've all learned that premature optimization is one of the most horrible things you can do as a programmer.
13:34 KENNEDY: What I've learned is it's incredibly hard to guess where something is slow. Even if you know this takes a second, you look at the code like, "Oh, I bet it's there." like, no, no, no, that's, like, 5%. It's over here.
13:44 OKKEN: Yeah, because...
13:44 KENNEDY: It's super hard to know.
13:45 OKKEN: You're first, you're throwing the first version of the rough draft or your code down. And you write something down you go like, "I know I can do this better but I'm going to just make it work here." And you know you're going to have to optimize that, and it turns out to not be the slow part.
13:59 KENNEDY: Right, exactly.
14:00 OKKEN: Yeah, so you need to find out where the slow part is. So this article called Making Python Programs Blazingly Fast by Martin Heinz goes through are a few things. He doesn't cover this timer, but there's a few other ways you can time it. There's a, you can use the command line time function to just time how long something runs.
14:17 KENNEDY: That might work. You might just go, I made a change. It was five seconds, is it more or less?
14:22 OKKEN: Yeah, exactly.
14:22 KENNEDY: That'll tell you.
14:23 OKKEN: Python -m cProfile can tell you a whole bunch about your program, about some...
14:28 KENNEDY: Do you use cProfile much?
14:29 OKKEN: I don't really.
14:30 KENNEDY: Yeah, I've used it some. It's pretty awesome, actually.
14:33 OKKEN: Yeah, and then he goes through an example of writing a wrapper function for a timer, which is similar to what this last article was.
14:40 KENNEDY: It's one facet of Pro Timer.
14:42 OKKEN: One of the things that he doesn't cover is the timeit function within, that's built into Python, which allows you to just run a single function a whole bunch of times and then it tells you how long that takes.
14:53 KENNEDY: Yeah, then you get your average and your deviation and all that.
14:56 OKKEN: But then the article goes through how to make things faster. So once you've found the slow parts, how do you make it faster? And these are some good suggestions. Use built-in types over custom types. Use caching and memoization through LRU cache, which is a built-in thing into Python. Local variables and local aliases when looping. This is something I didn't expect. This is something like, if you've got a multiple dot, dot dot dot something, even to a function call, creating a local copy of that makes things run faster.
15:30 KENNEDY: Every traversal of that dot is expensive in Python, whereas like C++, not so much.
15:35 OKKEN: Yeah, especially if it's in a loop. So, use functions. I don't understand why this was there, kind of duh, but, you know.
15:43 KENNEDY: Well, apparently the variable lookup in a function scope is faster than a global variable lookup or something like that, that he was talking about.
15:52 OKKEN: Oh, okay.
15:52 KENNEDY: And so by forcing all the variables into the function scope, they actually come out faster. So there's all these little weird edge cases.
15:58 OKKEN: Yeah, I don't have any code that's not in a function, so. Don't repeatedly access attributes in loops. Okay, this is a similar sort of thing. One of the things I didn't realize is that f-strings have been tuned to be very fast. So if you're doing string formatting, use f-strings over other things.
16:12 KENNEDY: Yeah, yeah. How many of you out there are using f-strings these days? Right on, like, that's five, 10 times faster? I don't know. There was a thing by Raymond Hettinger that's mentioned in the article? Yeah. So it's way, way faster than other ways, which is awesome.
16:25 OKKEN: And then use generators, because, I added at least experiment with generators, because generators are really about saving memory. But if you're really dealing with some large data, memory saving could result in speed up. So I would say throw those in and see if it's faster.
16:40 KENNEDY: As soon as you start hitting that page file, you're done.
16:43 OKKEN: I love generators. I throw 'em everywhere, yeah.
16:45 KENNEDY: I do, too.
16:46 OKKEN: Anyway, I think this was an interesting article on speeding up Python, and I warn people against premature optimization though. So, but it's fun.
16:54 KENNEDY: Perfect, yeah, this is a really good one. I like it and it's a good follow-on to the other ones we have. Brian is, yeah, you're here. So you spoke about CDK, the Cloud Development Kit from AWS. One of the big peeves I have with working with the cloud is I work from home. I want to go work in a coffee shop. Maybe I'm traveling, I want to work from the hotel and the internet's bad. I still want to be able to work on my code and know the internet is not available. Whoops, I guess my app won't run anymore, right? Well, that is a problem which I mostly solved by avoiding the cloud directly. But there's another cool project called LocalStack. I talked about Moto before, which is a way to mock out AWS. And this actually built on Moto, but it actually does quite a bit more. This comes to us from Graham Williamson and Jan Gazda, so thank you both for sending this in. And basically what it is, is it's like a local AWS. Not just a little bit, a lot of local AWS. It has S3, it has SES for simple email, it has EC2, it has Dynamo DB, it has Lambda, it has Elastic Search, all of that stuff, locally.
18:03 OKKEN: He showed us tons of a huge list, though. Does it have all that stuff?
18:06 KENNEDY: It has a bunch. I don't know how many it has, but I would say it's somewhere on the order of, I don't know, 20, 25 different services.
18:14 OKKEN: Probably the most common ones.
18:15 KENNEDY: Yeah, yeah, probably the most common ones. And then apparently there's also some kind of of pro thing I've not used, but then you get more services if you buy the pro version. But the lesser one, I guess, is open source, which is pretty cool.
18:25 OKKEN: That's neat, that's great. Like if you're on an airplane or something.
18:28 KENNEDY: Yeah, or you just want to have a little local dev environment.
18:31 OKKEN: You'd have to pay for that, even though it's less than pennies, but yeah.
18:34 KENNEDY: It depends what you're doing, I guess, yeah. So yeah, it basically, it brings together some of these tools. It brings together Moto. It brings together this thing called Dynalite. And puts sort of a unifying layer on top of it. So it's pretty cool. A lot of it runs in Docker, so it helps to kind of get a repeatable experience there.
18:50 OKKEN: That sounds neat.
18:51 KENNEDY: Yeah, absolutely. Alright, well, that's it for our main items, everyone. Got any extras you want to share with folks?
18:56 OKKEN: Well, I just, I don't know if we've covered this before. I saw an advert for the Python job board on python.org?
19:05 KENNEDY: Yeah, yeah, I saw that. I hadn't seen it before, but yeah, apparently, there's the, there's now. Yeah, we're joking around. We're laughing because the internet's not quite cooperating. And that's fine, we don't need it. Who needs the internet? What did I say about the cloud? What did I just say, Brian? You'd better hope you're not trying to test something right now. Anyway, so, carry on. Yeah, so the job board is cool, right? I hadn't noticed it either. But it's on python.org.
19:28 OKKEN: I don't know if you have to pay for stuff, but it just list jobs, so that's cool.
19:31 KENNEDY: Yeah, and Python's in demand. People want to have jobs writing code in Python, right? Like, you know.
19:37 OKKEN: Do you have any extras for us?
19:38 KENNEDY: I do have a couple. So let me pull these up here for the audience as well. So I have pictures 'cause some of these are very, very graphical. So there's this really cool one. You know Guido van Rossum, probably heard of the guy who created Python. So he was interviewed. He's Dutch. He was interviewed by this Dutch newspaper about programming, and the title. My Dutch is a little off, but it's like Python is Half My Life, right? So I worked on Python for half my life or something like this. And they said, we're going to, this is like a developer thing. Let's put some code and show some Python. What code do we see there?
20:10 KENNEDY: Document.getElementByID? Yeah, not so much. Not so much. That was a pretty interesting little thing that actually happened. The next one that's pretty interesting. I don't have a picture for it. I would just be a bar that's rusting or something. But no, it's pretty cool. So Microsoft, you know, they're all about C and C++, right? Like Windows is based on C and C++. They are actually been doing experimentations with Rust and they're coming out with a Rust-based programming language for rewriting things like Windows in Rust. That's a pretty big job for Rust.
20:42 OKKEN: Wow.
20:43 KENNEDY: And the reason is Rust is especially good at memory management and memory ownership. So things like buffer overflow vulnerabilities and stuff just go away in Rust. Which is like, you know, every first Tuesday, here's the seven buffer overflows. They're going to lose all your data if you don't patch by the next two days. That's what you get, and they're trying to avoid that, I'm guessing. So that sounds interesting.
21:05 OKKEN: You know Rust yet?
21:06 KENNEDY: I've looked at Rust. It looked like C. I said, "I'm going to go back to Python." I mean, not exactly, but it looked like C-ish.
21:12 OKKEN: Maybe I should take a look at it.
21:14 KENNEDY: Yeah, it's pretty interesting. Two more quick things. So I'm doing a webcast of Python for the .NET developer kind of interactive, one-hour thing at Crowdcast, on the Crowdcast platform. I think that'll be fun. So, link's in there. It's free to sign up people and check that out. And then Reuven Lerner, I was talking to him today, and he has a new free course that he released called Ace Python Interviews. So people out there looking for a Python job, here's, like, 50 little exercises and questions answered live and, like, live coding responses to 50 interview questions that are explained.
21:47 OKKEN: Reuven's a really cool guy, so I think that'd be cool to look at.
21:50 KENNEDY: Yeah, yeah, absolutely. It looks really good. And that's also free, so no harm, no foul there, if people want to check that out.
21:54 OKKEN: And I've got a job opening. So if anybody's looking, I'm mostly, last time I interviewed was for a Python person, so I'm probably just going to take some of these things and convert 'em to C++. And so, you know.
22:08 KENNEDY: So if you want to pass Brian's interview, it may be a good idea to take this course.
22:11 OKKEN: Don't tell my boss.
22:11 KENNEDY: Yeah, yeah. Alright, so are we ready for a joke? We always end our podcasts with a joke. This one's very visual, so I'm going to put this up on the screen for you, as well. And this is really like a sort of infographic. I'm a fan of infographics. And this one helps you understand the different types of jobs in software development, which can be very tricky, right? Like, what is the difference between a lead developer and a full stack developer and a coder? Well, here we go. So it says, there's a person, and pretty much it's the same looking person for every job description. And it says there's a coder, and there's a little caption that says he writes, software engineer, he writes code. Lead developer, he writes code. Devops, well, he writes code. Infrastructure as code, right? Data engineer, actually, what does he do?
22:56 OKKEN: He writes code.
22:56 KENNEDY: He writes code. Full stack developer, he writes code alone. Computer programmer, he writes code, too. Sysadmin, he writes, this is actually a guy eating doughnuts with a big beard and looking very disheveled. It says he writes, in fact, we don't really know what he does. Well, that's the joke for today, and I guess that's the podcast for today, as well.
23:18 OKKEN: So thanks a lot.
23:18 KENNEDY: Yeah, thanks a bunch. And thank you, everyone. Bye. Thank you for listening to Python Bytes. Follow the show on Twitter via @PythonBytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.