Transcript #163: Meditations on the Zen of Python

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 163, recorded live on location here at the Portland West Python Meetup. Hello, everyone. And it was recorded January 7th, 2020. Brian, it's 2020.

00:19 Yeah. Was that difficult to remember?

00:21 It's, you know, I'm really not used to it. I just got used to 2019, so we're having problems, but we'll get there. Yeah. Well, it's really great to be here with everyone. This is the the first time we've done a live recording in a while. We do this at PyCon a lot, but PyCon's not that frequent. So here we are in Portland. It's great. - Yeah, thanks. And thanks for everybody for coming out. This is wonderful. - Yeah, absolutely. - I've got the first one, right? - You've got the first item. So some zen. I think it's a new year. It's a new decade.

00:46 Let's kick it off with a little zen. - Yeah, a little zen. So we're gonna take 20 minutes and just meditate. Now, the first one I wanna talk about, there's probably gonna get this name wrong.

00:54 Why did I pick this? Mosh Zadka. It's a pretty cool name though. I wrote an article called Meditations on the Zen of Python. And if you don't know about the Zen of Python, hopefully you do, we're going to put a link in the show notes, but there's also in any REPL, you can say import this and it will show you a whole bunch of little cones, little snippets of the Zen of Python.

01:15 - One of the cooler things you can do in Python, second only to import antigravity.

01:19 Yeah, that's good. So Moshe says that this is a, the Zen of Python is not the rules of Python or the guidelines of Python. It is full of contradictions and illusions. It is not intended to be followed. It is intended to be meditated upon. So you can read these and think about your code. And actually I kind of like it, like that idea because when I've read through them, there are some that contradict each other. I want to bring out a few of them that he points out. The first is beautiful is better than ugly. And one of his comments is that consistency helps. So things like black and flake aid and pylint are very useful in making consistent looking code. Right. But those are only table stakes, right? It's just not ugly.

01:59 It's not beautiful because of that. Right. So even more important, only humans can judge what humans find beautiful. So code reviews and collaboration and a collaborative approach to writing code are the only realistic way to build beautiful code. So listening to other people is an important skill in software development and also just looking at code and seeing if you think it looks nice. I think it's good. So a couple more. Complex is better than complicated and that one always like threw me. I wasn't sure why that was in there. Mosh says when solving a hard problem it's often a case that no simple solution will do. In that case the most Pythonic strategy is to go from the bottom up, build simple tools and combine them to solve the problem.

02:43 That's kind of a Unix style. So is that how you view the complex is better than complicated?

02:49 I don't know if I've ever thought of it in these terms. And I do like this article because it did make me think like that. But I certainly think about software that way. I feel like so many people get caught trying to overthink it. And they're like, "Oh, I can't get started. I'm trying to like get the right, the exact right thing before I can write my first line of code." And And it's like, no, you're never going to know until you work on it for two days.

03:07 Then you're like, oh, I should have done this.

03:08 But well, you didn't know what you knew now.

03:10 So what are you going to do?

03:11 Just get started.

03:12 Yeah.

03:12 Right.

03:12 It's my philosophy.

03:13 And the last one I want to talk about is readability counts.

03:16 So in the face of immense pressure to throw readability to the side and just solve a problem, the Zen to Python reminds us that readability counts writing code.

03:24 So it can be read is a form of compassion for yourself and others.

03:28 And one of the reasons why I highlighted this is we're going to talk about optimization and speed later on.

03:33 And speed is great, but making sure that your program is readable and maintainable is very important.

03:39 - I really like this article, well done, because we've all heard about the Zen and Python, we've probably all typed import this, but it's a little vague.

03:47 And this is not, here's what it means, this is here's me trying to think about it sort of philosophically, and I've never seen anyone write that way about it before, and I thought it was really cool.

03:57 - I'd love to have other people to come up with their ideas about it, so that'd be cool.

04:01 - Yeah, absolutely.

04:02 So Python and the web doesn't usually have like a James Bond sort of places getting raided by the police, secret service, international angle to it, but this next item does.

04:17 - Really?

04:18 - Yeah, it does.

04:19 So you know what the most popular web server is in the world?

04:22 Used to be Apache, now it's Nginx.

04:24 Our stuff runs on Nginx, for example.

04:26 However, there was some news a few weeks ago.

04:28 The Nginx offices in Russia were raided by the Russian police and some of the core developers were detained.

04:39 And the reason is, this is not as interesting as I made it out to be, I don't really think, but the reason was the guy who created it, I have his name here, Sisyov, he created it and at the time he was working for Rambler.ru.

04:53 And Rambler.ru is like a Google Yandex type of company, search engine in Russia, and he worked on this in his spare time, and he open sourced Nginx, and then later turned it into a company.

05:01 Well, Rambler came along and said, "Hey, you know what, you worked on that "while we were employing you.

05:08 "Nginx is ours, we're taking it over." Meanwhile, Nginx has been bought by F5, an American company, and they own it, and so there's all this intrigue around it, and yeah, so that happened.

05:20 - So a bunch of my friends from Spokane, when Agilent closed down there, went to work for F5, and I'd never even heard of them before, and then they show up in this news article.

05:29 - Yeah. - It's pretty interesting.

05:30 - Yeah, they're all about the networks, but apparently servers.

05:32 So I received an update in email from Nginx a few days later, and I'll just read it so I get it right.

05:39 They said, "Promptly following this event "that I just described, we took measures "to ensure the security of our master software builds "for Nginx, Nginx Plus, Nginx WAF, and Nginx Unit, all of which are stored on servers outside of Russia.

05:54 No other products are developed in Russia.

05:56 F5 remains committed to innovating with Nginx, Nginx Plus, etc.

06:00 etc. all the products.

06:02 And we continue to provide the best in class support.

06:05 You can expect that to come.

06:07 So, who knows where this is going to go.

06:09 That's pretty interesting.

06:09 If you use Nginx, should you worry about this?

06:13 That was why I brought it up.

06:14 Because if it's the most popular web server in the world, some folks, their ears are going to perk up and say, wait a minute, what?

06:21 I mean, I don't think this is like a Russia trying to impose its will on the world.

06:26 I think this is just a dispute between a Russian company and some Russian folks who created Nginx, but it could have knock-on effects quite far.

06:34 So yeah, it's pretty interesting.

06:36 I think just keep an eye on it, really.

06:38 This episode is brought to you by, well, us.

06:41 - Really?

06:41 - Yeah, normally it's brought to you by other companies, but this time, you know, we both have interesting things to talk about and we have a gap, so I just want to tell you about some of our stuff.

06:49 So if you want to check out the courses that I'm creating or, you know, set up the business stuff, just visit pythonbytes.fm/biz.

06:57 And here, there's something about testing over at pythonbytes.fm/pytest.

07:02 And people can check that out there as well.

07:04 Cool.

07:04 Did you set up a URL redirect from there?

07:07 I don't want to say the whole name of the whole domain name and URL.

07:10 So yeah, that's yours.

07:11 It points to Python testing with pytest, which I was published in 2017, but I still think is the very best way to get up to speed.

07:19 very quickly on Python.

07:20 Yeah, absolutely.

07:21 And we also have a Patreon that you set up at patreon.com/pythonbytes.

07:25 So, I have some thoughts on this next one, but why don't you kick it off?

07:29 This one's from TheCreatorOfFlask, but not TheCurrentMaintainerOfFlask.

07:33 Oh, that's true. Right.

07:35 I brought this up because I was curious what your thoughts were.

07:38 So, the next one is from Armin Roneker.

07:42 Armin Roneker, thank you.

07:43 He wrote an article called, "I'm Not Feeling the Async Pressure." The idea is like async is all the rage.

07:49 We've actually talked about async quite a bit on the podcast.

07:52 And I think a lot of people are concerned about it.

07:55 And it's one of the reasons why it's going in places because I think there's some pressure of people to leave Python to go to Go or other things.

08:02 Node.js was an early example of that.

08:04 Yeah, but before you go towards async, Armin is warning people to make sure you understand flow control and back pressure.

08:12 And I had never heard of back pressure, but back pressure is the resistance that opposes the flow of data through a system.

08:20 Back pressure sounds quite negative, but is here to save your day.

08:25 If parts of your system are async, you have to make sure the entire flow through the system doesn't have overflow points.

08:32 And then Armin goes through an example with a reader and writer, and it seemed like simple code, but it really got hairy really kind of fast.

08:40 And the example, yeah, got hairier than I expected.

08:44 And he says, "Async brings you new foot guns.

08:47 "Async and await are great ways to encourage "writing stuff that will behave catastrophically "when overloaded.

08:53 "For you developers of Async libraries, "here's a New Year's resolution for you.

08:57 "Give back pressure and flow control "the importance they deserve in documentation and API." And there's just some hidden things within buffers and things like that.

09:08 - Yeah, well, this is a pretty nuanced article, and it's pretty interesting.

09:12 It comes from someone who knows a thing or two about the network programming, Armin being the original creator of Flask.

09:18 That said, my reaction to reading it, my reaction was there were a lot of examples of, and here's if you overpressure the system when you write an async system, it will fall down, right?

09:31 Imagine you only allow 50 database connections and suddenly you get 10,000.

09:35 My sort of reaction to this was, well, what is the response of the system gonna be when it's single threaded and you give it 10,000 connections?

09:43 requests, they're all going to time out except for a call.

09:45 It's just going to crash.

09:46 So is it crashing in an async world, or is it crashing in a non-async world?

09:51 With enough traffic, yes.

09:53 But at the same time, if you can write basically the same code, put async and a wait in front of a few bits, and you can get 10 or 100 times the amount of performance out of a given piece of hardware before it crashes, that seems better to me.

10:06 So I mean, I'm sympathetic to the problem, but at the same time, it's always like, well, if we give it way too much pressure, it's gonna crash.

10:14 Well, if it wasn't parallel, it would crash before then.

10:16 - My thoughts were like, can you, are there ways to throttle?

10:20 I don't know enough about all the way to do network stuff.

10:23 So if I'm setting up a web service, for instance, can I throttle the input to say, to not allow 10,000 connections and just allow 5,000 or something?

10:32 - Right, right, potentially.

10:33 So maybe with something like Uviacorn or something, you could set that up.

10:37 I honestly don't know.

10:38 It seems to me the danger that he's addressing is when the system itself is generating the input.

10:44 Like we had this example of a guy who sent us a message and said, "Hey, I had this web scraping thing. It was really slow.

10:50 We turned async loose on it and it crashed the server because it ran out of memory because it processed it too quickly." Right? Like there you need to find a way to slow it down.

10:58 But when you're running a web server, you don't control how many people want to get to it.

11:02 There's just people want to get to it and they either can or they can't.

11:05 And with async, more of them would be able to than otherwise.

11:08 That's my thoughts.

11:09 - Okay, and I guess that my thoughts would be if you're gonna throw async at a problem, make sure that you do capacity testing on it as well.

11:18 - Yeah, well, it's gonna fail somewhere else, right?

11:20 And so you're gonna, maybe your database isn't set up for it.

11:24 Maybe your microservices can't handle all the traffic.

11:25 Like, there's gonna be something, right?

11:27 It's just gonna show up somewhere else.

11:28 But in general, I think you're gonna get better scalability with it than without it.

11:32 So if you're not generating that pressure, if you're not generating the traffic, then I don't know what choice you have.

11:38 - That's my thought.

11:38 - It was an interesting read.

11:39 - Yeah, it definitely was, it definitely was.

11:41 Let's go for something a lot simpler than like a deep thing on streaming and buffers and async.

11:47 How about a new way to time your code?

11:49 - Yeah, that sounds good.

11:50 - Yeah, so this one comes to us from Doug Farrell, who works on the RealPython team, and as part of their work, they've gotta time all sorts of things for their articles.

11:59 You know, Dan Bader and crew over there.

12:01 And so they came up with this thing, either they came out of RealPython, I think, or possibly they were just using it a lot.

12:09 But this thing called code timing.

12:12 So if you've got some code and you wanna know how long it takes to run, you could say, you know, create a time, like capture the date time, do some work, capture dot now again and do a delta, or you could even use performance counters and other things, but you can use this library.

12:27 You just, there's a bunch of different things you can do.

12:29 You create a timer class, you can call start, do some work, stop, and out comes the time.

12:34 Or you can put it in a context manager, like a width block.

12:38 Soon as it goes through the width block, when it's done, that bit is timed.

12:40 Or you can use it as a decorator.

12:42 And you can also wire up a logger, which is kind of cool.

12:45 So you can see, it'll just output standard Python logging with time information of when it's doing bits of its thing.

12:52 Give it a name and it'll tell you how long it took.

12:53 >> Cool, well, they should add statistics too, so I can get min, max, and average, and standard deviation.

12:58 >> Yeah, that would actually be cool.

12:59 Well, it's open source, I bet they accept PRs.

13:02 >> Yeah. >> Actually, there's a bunch of features I want to add to it.

13:04 I started messing around with and I'm like, put it down.

13:08 I have other things to do.

13:10 I'm already too busy.

13:11 I don't need this.

13:12 I'm going to leave it alone.

13:13 But yeah, it's a pretty cool little timer class.

13:14 I'm going to probably use that.

13:15 I like it.

13:16 Yeah.

13:17 I thought this nice follow on for this, the timer would be an article called making Python programs blazingly fast.

13:24 That sounds good.

13:25 We all want that.

13:26 You need to time stuff.

13:27 You should never, I mean, hopefully we've all learned that pre premature optimization is one of the most horrible things you can do as a programmer.

13:34 >> What I've learned is it's incredibly hard to guess where something is slow.

13:38 Even if you know this takes a second, you look at the code, I bet it's there.

13:41 Like no, no, no, that's like five percent, it's over here.

13:44 >> Yeah, because you're throwing the first version of the rough draft of your code down, and you write something down, you go like, I know I can do this better, but I'm going to just make it work here.

13:56 You know you're going to have to optimize that, and it turns out to not be the slow part.

13:59 >> Right, exactly.

14:00 - Yeah, so you need to find out where the slow part is.

14:02 So this article called "Making Python Programs Blazingly Fast" by Martin Hines goes through a few things.

14:09 He doesn't cover this timer, but there's a few other ways you can time it.

14:12 There's a, you can use the command line time function to just time how long something runs.

14:17 - That might work.

14:18 You might just go, I made a change.

14:20 It was five seconds.

14:21 Is it more or less?

14:22 - Yeah, exactly.

14:23 - That'll tell you.

14:23 - Python-MC profile can tell you a whole bunch about your program, about some-

14:28 - Do you see profile much?

14:29 - I don't really.

14:30 - Yeah, I've used it some.

14:31 It's pretty awesome actually.

14:33 - Yeah, and then he goes through an example of writing a wrapper function for a timer, which is similar to what this last article was.

14:40 - It's one facet of GoTimer.

14:42 - One of the things that he doesn't cover is the time it function within, that's built into Python, which allows you to just run a single function a whole bunch of times, and then tells you how long that takes.

14:54 - Yeah, then you get your average and your deviation and all that.

14:56 - But then the article goes through how to make things faster.

15:00 So once you've found the slow parts, how do you make it faster?

15:03 And these are some good suggestions.

15:06 Use built-in types over custom types.

15:08 Use caching and memoization through LRU cache, which is a built-in thing into Python.

15:15 Local variables and local aliases when looping.

15:19 This is something I didn't expect.

15:20 This is something like if you've got multiple dot, dot, dot, dot, something, even to a function call, creating a local copy of that makes things run faster.

15:30 - Every traversal of that dot is expensive in Python, whereas like C++, not so much.

15:35 - Yeah, especially if it's in a loop.

15:37 So, I use functions, I don't understand why this was there, kinda duh, but you know.

15:43 - Well, apparently the variable lookup in a function scope is faster than a global variable lookup, or something like that that he was talking about.

15:52 So by forcing all the variables into the function scope, they actually come out faster.

15:56 So there's all these little weird edge cases.

15:58 >> Yeah, I don't have any code that's not in a function.

16:01 Don't repeatedly access attributes in loops.

16:04 Okay, there's a similar sort of thing.

16:06 One of the things I didn't realize is that f-strings have been tuned to be very fast.

16:09 So if you're doing string formatting, use f-strings over other things.

16:12 >> How many of you out there are using f-strings these days?

16:15 Right on, like that's five, 10 times faster.

16:18 I don't know, there was a thing by Raymond Henninger that's mentioned in that article.

16:22 >> Yeah. >> Yeah, so it's way, way faster than the other ways that it was awesome.

16:25 And then use generators because I added at least experiment with generators because generators are really about saving memory.

16:32 But if you're really dealing with some large data, memory saving could result in speedup.

16:37 So I would say throw those in and see if it's faster.

16:40 - As soon as you start hitting that page file, you're done.

16:42 - I love generators. I throw them everywhere.

16:45 - I do too.

16:46 - Anyway, I think this was an interesting article on speeding up Python.

16:50 And I warn people against premature optimization though, so, but it's fun.

16:54 - Perfect, yeah, this is a really good one, I like it.

16:56 And it's a good follow on to the other ones we have.

16:58 Brian, yeah, you're here.

16:59 So you spoke about CDK, the Cloud Development Kit from AWS.

17:06 One of the big gives I have with working with the cloud is I work from home, I wanna go work in a coffee shop, maybe I'm traveling, I wanna work from the hotel and the internet's bad, I still wanna be able to work on my code and know the internet is not available.

17:20 Whoops, I guess my app won't run anymore, right?

17:23 Well, that is a problem, which I mostly solve by avoiding the cloud directly.

17:30 But there's another cool project called LocalStack.

17:33 Talked about Modo before, which is a way to mock out AWS.

17:39 This is actually built on Modo, but it actually does quite a bit more.

17:43 This comes to us from Graham Williamson and Jan Gazda.

17:45 So thank you both for sending this in.

17:49 And basically what it is, it's like a local AWS.

17:49 Not just a little bit, like a lot of local AWS.

17:52 It has S3, it has SES for simple email, it has EC2, it has DynamoDB, it has Lambda, it has Elasticsearch, all of that stuff locally.

18:03 - He showed us like tons of huge list though.

18:06 Does it have all that stuff?

18:06 - It has a bunch, I don't know how many it has, but I would say it's somewhere on the order of, I don't know, 2025 different services.

18:14 - Probably the most common ones.

18:15 - Yeah, probably the most common ones.

18:16 And then apparently there's also like some kind of pro I've not used, but then you get more services if you buy the pro version, but the lesser one I guess is open source, which is pretty cool. - That's neat, that's great.

18:27 Like if you're on an airplane or something.

18:29 - Yeah, or you just want to have a little local dev environment.

18:31 - You don't have to pay for that, even though it's less than pennies. - It depends what you're doing I guess.

18:37 Basically it brings together some of these tools, it brings together Modo, it brings together this thing called DynaLite, and puts sort of a unifying layer on top of it.

18:45 It's pretty cool, a lot of it runs in Docker, or it helps to kind of get a repeatable experience there.

18:50 - That sounds neat. - Yeah, absolutely.

18:52 All right, well, that's it for our main items, everyone.

18:55 Got any extras you want to share with folks?

18:56 - Well, I just, I don't know if we've covered this before.

18:58 I saw an advert for the Python job board on python.org.

19:03 - Yeah, yeah, I saw that.

19:06 I hadn't seen it before, but yeah, apparently there's the, there's now, yeah, we're joking around, we're laughing because the internet's not quite cooperating.

19:15 That's fine, we don't need it.

19:16 Who needs the internet?

19:17 What did I say about the cloud?

19:18 What did I just say, Ryan?

19:21 You better hope you're not trying to test something right now.

19:23 Anyway, carry on.

19:24 Yeah, so the job board is cool, right?

19:25 I hadn't noticed it either, but it's on python.org.

19:28 - I don't know if you have to pay for stuff, but you can just list jobs, so that's cool.

19:31 - Yeah, and Python's in demand.

19:33 People want to have jobs writing code in Python, right?

19:36 Like, you know.

19:37 - Do you have any extras for us?

19:38 - I do have a couple.

19:40 Let me pull these up here for the audience as well.

19:42 So I have pictures, 'cause some of these are very, very graphical.

19:45 So there's this really cool one.

19:46 You'd have asked him, probably heard of the guy who created Python.

19:49 So he was interviewed, he's Dutch.

19:50 He was interviewed by this Dutch newspaper about programming and the title, my Dutch is a little off, but it's like Python is half my life, right?

19:59 So I worked on Python for half my life or something like this.

20:02 And they said, this is like a developer thing.

20:04 Let's put some code and show some Python.

20:07 - Yeah.

20:08 - What code do we see there?

20:09 - I don't know, is it JavaScript?

20:10 - Document.getElementById.

20:12 Yeah, not so much.

20:14 - Not so much.

20:15 That was a pretty interesting little thing that actually happened.

20:18 The next one that's pretty interesting, I don't have a picture for it, it would just be like a bar that's rusting or something.

20:24 But no, it's pretty cool.

20:25 So Microsoft, they're all about C and C++, right?

20:28 Like Windows is based on C and C++.

20:31 They are actually been doing experimentations with Rust, and they're coming out with a Rust-based programming language for rewriting things like Windows in Rust.

20:40 That's a pretty big jump for Rust.

20:43 And the reason is Rust is especially good at memory management and memory ownership.

20:49 So things like buffer overflow, vulnerabilities and stuff just go away in Rust.

20:53 And which is like, you know, every first Tuesday, here's the seven buffer overflows.

20:59 They're going to like lose all your data if you don't patch by the next two days.

21:02 That you get in, like, they're trying to avoid that, I'm guessing.

21:04 So that sounds interesting.

21:05 You know, Rust yet?

21:06 I've looked at Rust.

21:07 It looked like C.

21:08 I said, I'm going to go back to Python.

21:09 I mean, not exactly, but it looked like C-ish.

21:12 Maybe I should take a look at it.

21:14 - Yeah, it's pretty interesting.

21:15 Two more quick things.

21:16 So I'm doing a webcast of Python for the .NET developer, kind of interactive one hour thing at Crowdcast on the Crowdcast platform.

21:23 I think that'll be fun.

21:24 So links in there.

21:25 It's free to sign up, people can check that out.

21:26 And then Reuben Lerner was talking to him today and he has a new free course that he released called Ace Python Interviews.

21:34 So people out there looking for a Python job, here's like 50 little exercises and questions answered live and like live coding responses to 50 interview questions that are explained.

21:47 Rubin's a really cool guy. So I think that'd be cool to look at.

21:50 Yeah, yeah, absolutely. It looks really good. And that was also free. So no harm, no foul there. People want to check that out.

21:54 And I've got a job opening. So if anybody's looking, I'm mostly last time I interviewed was for Python person. So I'm probably just going to take some of these things and convert them to C++. And so, you know, So if you want to pass Brian's interview, it may be a good idea to take this course.

22:11 Don't tell my boss.

22:12 Yeah.

22:13 All right.

22:13 So are we ready for a joke?

22:14 We always see in our podcasts with a joke.

22:16 Yeah, this one's very visual.

22:17 So I'm going to put this up on the screen for you as well.

22:19 And this is really like a sort of infographic.

22:21 I'm a fan of infographics and this one helps you understand like the different types of jobs in software development, which can be very tricky, right?

22:29 Like what is a difference between a lead developer and a full stack developer and a coder?

22:33 Well, here we go.

22:34 So it says, there's this person, and pretty much it's the same looking person for every job description.

22:40 And it says there's a coder, and there's a little caption that says, "He writes, software engineer, he writes code.

22:46 "Lead developer, he writes code.

22:49 "DevOps, well, he writes code.

22:51 "Infrastructure is code, right?

22:53 "Data engineer, actually, what does he do?" - He writes code. - He writes code.

22:57 "Full stack developer, he writes code alone.

22:59 (laughter)

23:01 computer programmer, he writes code too, sysadmin, he writes, this is actually a guy eating donuts with a big beard and looking very disheveled.

23:10 He says he writes, in fact, we don't really know what he does.

23:13 (laughing)

23:14 All right, well, that's the joke for today.

23:16 I guess that's the podcast for today as well.

23:18 - So thanks a lot.

23:19 - Yeah, thanks a bunch and thank you everyone.

23:20 (audience applauding)

23:23 Bye.

23:25 Thank you for listening to Python Bytes.

23:27 Follow the show on Twitter via @pythonbytes.

23:29 And that's Python Bytes as in B-Y-T-E-S.

23:32 And get the full show notes at pythonbytes.fm.

23:35 If you have a news item you want featured, just visit pythonbytes.fm and send it our way.

23:40 We're always on the lookout for sharing something cool.

23:42 On behalf of myself and Brian Okken, this is Michael Kennedy.

23:46 Thank you for listening and sharing this podcast with your friends and colleagues.