Transcript #12: Expanding your Python mental model and serving millions of requests per second with Python

00:00 This is Python Bytes, Python headlines and news delivered directly to your earbuds.

00:06 It's episode 12 recorded February 6, 2016.

00:10 This is Michael Kennedy, one of your hosts here with my co-host Brian Okken.

00:15 Hey, Brian, how's it going, man?

00:16 It's going really good.

00:17 Yeah, I'm super excited to talk about what we got to cover today.

00:21 But I also want to say thank you.

00:22 I want to say thank you to Rollbar because they're sponsoring this episode and I'll tell you more about their cool stuff that they do later.

00:28 Let's start out with the way that you think about programming, because learning to program is somewhat learning the syntax, but it's also just learning the mental models of what programming means, right?

00:39 Yeah, and I did realize that when you think about Python, the problems in problem solving is different in Python than in other language, just because of the idiosyncrasies of the language and how people tend to do things.

00:53 couple of good articles that I stumbled across and I think actually I think one of them is a anyway so there's there's Python functions aren't what you think and this is on a site called Powerful Python and it's a short little article just talking about that functions are really just objects that that happen to have a name that's the name of the function that you gave it and they're really that much different than giving a variable the value of five.

01:24 You just give a variable a value of a function.

01:27 It does change the way you, if you can think about functions that way, you can change the way you do some things.

01:34 You know, not surprisingly, I think it really enables functional programming type of metaphors.

01:39 Just the whole concept of treating functions as first class objects, right?

01:43 Functions are like variables or like classes or like whatever.

01:47 And I think that's really interesting.

01:48 It's really super important for things like lambdas and closures that we can treat functions this way.

01:54 But even normal functions are basically the same thing, right?

01:58 Yeah, and that you can just throw one in a data structure and use it later.

02:03 It is cool.

02:05 What I thought was weird was you can change the dunder name of a function.

02:08 That was news to me.

02:09 Oh, yeah.

02:10 Yeah, I've never tried to do that.

02:13 Neither have I.

02:14 It seems a little evil, but.

02:15 (laughing)

02:17 - A little bit.

02:18 - And then another article called the Tao of Python, which I think is just a cute name, but it's actually a Jupyter notebook that talks about the relationship between objects, classes, and meta-classes.

02:31 And again, it's a similar thing of not just functions or objects, but even classes themselves are objects.

02:40 - Right, right.

02:41 And their classes are meta-classes, right?

02:43 which is, it's very crazy.

02:46 It definitely is a mental shift, especially the meta programming, and that's something I could learn more about personally as well.

02:53 - Yeah, me too, and I haven't really made use of it a lot, but it's kind of one of those things of if you can think about the whole thing and how it fits together in just a little mental overview, I think it helps.

03:08 - Yeah, excellent.

03:09 All right, number two, the one that I chose is an article by a friend of the show, Dan Bader, And his article he wrote, which I helped him a little bit with, you'll see at the very end, it's called "Why Learn Python?

03:20 Here are 8 Data-Driven Reasons." And people send me messages and say, "Hey, Michael, I'm thinking about changing careers or aspects of my career.

03:29 Maybe I'm a Java developer and I'm considering learning Python.

03:33 Is it worth it?" And you'll see from this article, the answer is yes, yes, and absolutely yes.

03:39 Python is the second most popular language in the world according to the popularity of programming language index.

03:46 It's one of the hottest job skills to have according to Dice.

03:49 And Dan lays out eight really well researched reasons why this is the case, basically to support these trends.

03:58 And so, you know, it says you can learn pretty much anything with Python.

04:01 It's widely used by data scientists.

04:04 Python pays well.

04:05 In fact, it's number two only to Ruby developers, according to Guru, but it has many more job openings than Ruby does.

04:15 And it's on average above all the other developer salaries if you're looking at the US.

04:20 So for example, they say, Indeed says the average Python developer in the US across all the regions, they're thinking that something between 103 and 116,000 a year, which is amazing really.

04:33 Yeah.

04:34 That's a good place to be, right?

04:36 Yeah, I remember when C++ was up near one of the tops and it's now down at number seven now.

04:43 Yeah, it's a Python world.

04:46 One of the things that he didn't list, which I'm confused by, was that you can listen to this podcast.

04:54 It's totally bizarre he forgot that.

04:56 I mean, there's like a couple of good Python podcasts around this at least, right?

04:59 Yeah, that's one of the good reasons to learn Python.

05:03 - We'll add one in there, that's for sure.

05:05 It definitely helps you.

05:06 And so there's demand is growing for Python developers.

05:09 It saves you time.

05:10 It's beginner friendly, but you don't top out really quickly, right?

05:13 Like it can be an advanced language as well.

05:16 All the big companies are using it.

05:17 So if you want to get a job somewhere cool, like YouTube or IBM or Mozilla or Quora or Instagram, all these companies are making major use of Python.

05:27 So this could be your ticket there.

05:28 And finally, Python has a great ecosystem.

05:31 - Yeah, and one of the great things about articles like this are, we're preaching to the choir, of course, but a lot of people get those questions of like, well, why should I learn Python?

05:42 And all I have usually have is it's fun and cool, and why not?

05:47 So these are better reasons.

05:49 - Yeah, this is really, it's a really great article, and of course we're linking to it.

05:52 I think it's not, I mean, it's great to be preaching to the choir, but it's also interesting to provide this as fuel for people having these debates.

06:01 Maybe they're working on a team, and they're like, "Well, what technology should we choose "for our next topic, our next project or something?" And having all that information there is like, "Hey, look, choosing Python is a really good bet.

06:12 "You'll have lots of people who know it.

06:13 "There's a lot of, you know, just, it's a good bet, "and I think this will help people make that case." - And I'm also a firm believer in just resume building.

06:21 You don't know what the next job's gonna be like, and it probably won't use Python somewhere, so.

06:27 - Yeah, absolutely.

06:28 - Anyway.

06:28 - All right, I bet people do some testing with Python.

06:31 - Yeah, they do.

06:32 And I was thrilled to have somebody write a pytest article 'cause it's one of my favorites.

06:37 So there's, and cool last name too.

06:40 I think I'm gonna try this, Kevin Ndungu, I think.

06:44 Ndungu?

06:45 Anyway, testing Python applications with pytest.

06:49 And it's a good introduction article to try to not scare people off too bad about some of the advanced features.

06:58 But after jumping into it a little bit, He shows how easy it is, of course, to write tests, but then also takes a look at one of the common things that happens is you grow a whole bunch of tests, and then you want to refactor them.

07:13 And there's a little bit about refactoring your tests by using fixtures and pulling out some of the common starting state.

07:22 And then he jumps into a couple of the nice--

07:25 one of the nice features of pytest, of course, is the parameterization of easily being able to throw multiple data sets at individual test function.

07:35 And that was demonstrated pretty cleanly, I like that.

07:38 - Nice.

07:39 - Good job, Kevin, and I like the article.

07:41 - Yeah, I really like the let's treat test code as production code, 'cause I feel like a lot of people will write tests and they're like, these tests are hard to maintain.

07:50 And then you look at their test code and it's, you're like, why did you duplicate this bit of code 20 times?

07:55 Like, what is this?

07:56 Would you ever do this in a real app?

07:58 That's insane.

07:59 it's not a function or some other thing that you can use throughout your test, right?

08:03 Yeah, definitely.

08:04 Speaking of production, you don't want errors in your production, do you?

08:08 No, of course not.

08:09 No, not not too much.

08:11 One of the first steps in solving errors in production is knowing about them.

08:15 And that's where rollbar comes from.

08:17 So rollbar supporting the show this week.

08:19 Thank you, rollbar.

08:20 And I've used rollbar for a long time on the website that delivers Python bytes and my training courses and all those things.

08:26 And I would say at least five times Rollbar has saved me from some seriously bad choices that I've made.

08:34 I've tested something locally where there was data in the database and I put it in a production I was about to start putting data but it was empty at the time and the code couldn't deal with empty sets and all sorts of stuff going on where something crashed on the site and I didn't realize it.

08:49 And of course, Rollbar sent me a message popped up on my phone immediately and said, error in production, here's the exact details of what you need to fix.

08:57 And I was like, ah, crap, you know, five minutes later, though, it was fixed.

08:59 So thanks, roll bar for helping me out then.

09:01 And thanks for sponsoring the show.

09:03 Awesome.

09:04 It was special offer for listeners.

09:05 And I'll put that link in the show notes right at the top.

09:08 Speaking of production, we would all like our code to run a little bit faster.

09:12 And I feel like we're in this cool place with Python, especially Python versus legacy Python, Python 3.5, 3.6, and 3.7, the one that's coming, have really been focused on performance.

09:25 And so there's a, you know, we talked about a web framework called Sanic that was based on AIO HTTP, which is an async pipeline for web processing.

09:36 Well, here comes another one.

09:38 And this one, it turns out, if the benchmarks are to be believed, is dramatically faster than even Sanic and lots of other things out there like Go.

09:50 So it's called Jepronto, I'm not sure which, how you pronounce it, but I'll link to it.

09:55 It describes itself as a screaming fast Python 3.5 and above web micro framework integrating with pipelining HTTP servers based on UV loop and Pico HTTP parser.

10:08 So it came out just a few months ago, it has like 2,600 stars on GitHub.

10:15 And the title of the article introducing it was handling a million requests per second with Python.

10:21 Yeah, I think that's on on one server.

10:25 So that's crazy.

10:26 It is crazy fast.

10:27 It's pretty exciting.

10:29 If you read this article that we're linking to the guy who built it, he's doing a crazy amount of careful thought and low, low, low level OS type of optimization.

10:41 Just to give you a sense, most of this is written in C, although he's thinking of rewriting it in Rust.

10:47 He talks a bit about that and says, look, I'm going to call it Gepronto, going with that one.

10:53 Gepronto tries to delay the creation of Python counterparts of its internal structures until explicitly asked.

11:00 For example, normally when a request come in, the request object would be populated with headers.

11:07 This framework won't actually create the headers dictionary unless you try to access it.

11:13 It gives you only what you need at many, many levels and has some really cool low-level optimizations around pipelining.

11:20 It's quite cool.

11:21 Yeah, the pipelining description, I thought that was pretty interesting.

11:24 Yeah, I hadn't really thought too much about that.

11:27 Super, super cool.

11:29 Related to this, I think that somebody on Twitter was commenting that, "Why are we worried about speed? Why isn't it better to be correct?" I guess I like to have both, and I think I'm intrigued by other people trying to get performance up. And I think it's, to me, it's important because I would hate to have somebody say, "Oh, eventually you're going to scale out of Python and need to switch over to go.

11:58 Exactly.

11:59 That that's the primary primary thing I think.

12:02 And even if you're not going to scale out, if you start to hear lots of major companies saying we're all abandoning Python because they won't run our workloads.

12:09 Even if that's not the situation you're in, it starts to like cast a shadow over the general ecosystem.

12:16 Right.

12:17 Where, whereas if you say, no, no, look at these amazing things that are being done with Python on the web.

12:23 your stuff is going to be way lower traffic and load than that.

12:27 Surely this will work for you as well, right?

12:29 So I think it's really interesting.

12:31 I mean, to be honest, Pyramid plus MicroWZ plus Nginx just straight up is way fast enough for what I'm doing, even with a couple million HTTP requests a month.

12:41 But it's super cool to see that it's sort of expanding the upper bound.

12:46 And I think that's going to have some great stories down the road.

12:48 Yeah, definitely.

12:50 So the next topic we've got is a good news thing that came out, was just announced today.

12:57 The Rethink, and I'm not sure if you had RethinkDB on--

13:01 - I did, I had them on my show.

13:02 I can't remember the number, I think it's in the 60s on Talk Python TV, we talked about RethinkDB and the company behind it, yeah.

13:09 - Yeah, and when I, I think I've heard about it because of Talk Python, but the, it was really intriguing to me, and I like the idea of somebody just from the ground up rethinking how databases work with live data and real-time data.

13:25 But in September of last year the company shut down and I was curious, like everybody else was, is what's going to happen to RethinkDB. But I guess somebody called Cloud Native Computing Foundation, they made an announcement that they purchased the rights to the source code and contributed it to the Linux Foundation and it's changing the licensing as well to the Apache license version 2 and everybody's can continue using it now and we'll be able to have this project live on. That's really cool so RethinkDB is supposed to be like a real-time it's like a document database that is a real-time database for your server so it's definitely got some some interesting components and I believe all the team that worked there has made their way over to Stripe. I'm pretty sure that's what's happening. So everybody who used to work at Rethink has kind of moved over to Stripe and that's really cool that they had a soft landing somewhere. And it's great to see the Linux Foundation taking up this project and keeping it going. And I guess there were some companies that wanted to, that are relying on it and wanted to somehow donate to try to keep it going and there wasn't a way to do that at first and now there is and they've got a special under the Linux Foundation and Stripe has put together 25,000 in matching funds to to match other donations.

14:51 So I wish them all well and hope hope to rethink DB lives for a long time.

14:56 Yeah, they're definitely occupy cool space in the database world.

14:59 So happy to see them rolling.

15:00 All right, let's round out this conversation with my sixth item here, which is the Python top 10 articles of the past year. And this comes from this interesting like work reading app or like technical reading app called MyBridge at my my bridge dot co. And so this is kind of like Zite or Flipboard but it's specifically focused on technology and like reading for work and for your career. Okay so it's an interesting like iPad app and whatnot but what they did is they said how many articles they had some crazy number of articles that for the past year we've ranked nearly 10,000 Python articles and we picked the top 10 stories with a point 1% chance that will help you advance your career in Python.

15:48 Oh, nice criteria.

15:49 Yeah, it's really nice, right?

15:51 So the first one is from Kenneth hitchhikers guide to Python best practices guidebook for writing for humans.

15:58 That's really cool.

15:59 Then SciPy lecture notes, 30 essential Python tips and tricks, computational and inferential thinking and data science.

16:09 Welcome to Python cheat sheet, data mining in Python, Python FAQ, why should I use Python 3 by Eve.

16:17 Remember the whole Zed Shah, Python 2 is going to live forever, no Python 3.

16:23 This guy wrote the beautiful rebuttal to that article, so this article is also by him.

16:28 Introduction to Stock Market Analysis, NumPy Tutorial, and Build Your First Python and Django App.

16:33 So, I like this pick for a couple of reasons.

16:36 One is it's just cool to have some really nice reading and this is kind of a cool app, this MyBridge thing.

16:42 I've just started playing with it.

16:44 But I think it gives you a pretty interesting view into the Python space.

16:48 People ask, "What can you do with Python?

16:50 What should I learn with Python?" Well, this is like 60% data science, 25% web and 25% language by my super quick estimation.

17:00 So I think that's a kind of an interesting map to draw.

17:03 Actually, I hadn't run across a lot of these articles.

17:06 So I was pretty glad to hear, see some of these.

17:09 So even though that's our six items, actually, I have one more for you.

17:12 Last week, we talked about Red Hat distribution, Red Hat Enterprise Linux, and how people are having trouble using Python 3 because they keep sending this message going I can't use Python 3. Well, friend of the show Chip Warden sent, actually added a comment. If you want to add comments in the show there's a whole discuss section at the bottom of each show page and said hey what do you mean people can't do this they should just check out Red Hat software collections. So I checked it out yeah absolutely people should check this out it's the latest stable updates of development technologies for Red Hat Enterprise Linux. Wow. That's cool so if If you want to work with Node, Perl, PHP, Python, or Ruby, it is basically an approved Red Hat Enterprise Linux way of getting Python 3, for example, on your environment.

17:57 Perfect.

17:58 Yeah, so that's cool.

17:59 So that's definitely going to be in the show notes.

18:01 All right.

18:02 Well, that's one more reason to switch to Python 3, right?

18:06 Definitely.

18:07 And I can't believe we're done already.

18:08 Gone so fast.

18:09 I know.

18:10 Well, I think we're going to have to scour the internet for more cool Python stuff for next week.

18:15 What do you say?

18:16 want to encourage anybody to help us out by sending us what you think we ought to cover next week.

18:21 Yeah, we don't always cover everything people send but we if it matches up, we definitely try to cover it.

18:27 So please help us out if you know something like chip did here that we didn't cover, you know, send it our way.

18:33 All right.

18:34 Well, thank you everyone for listening.

18:35 Brian.

18:36 Thank you.

18:37 Great to catch up with you like always, man.

18:39 Thank you for listening to Python bites.

18:41 Follow the show on Twitter via at Python bites.

18:44 pythonbytes as in b-y-t-e-s. And get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy.

19:00 Thank you for listening and sharing this podcast with your friends and colleagues.