Transcript #12: Expanding your Python mental model and serving millions of requests per second with Python

00:00 This is Python Bytes. Python headlines and news delivered directly to your earbuds.

00:05 It's episode 12 recorded February 6th, 2016.

00:09 This is Michael Kennedy, one of your hosts, here with my co-host Brian Okken.

00:15 Hey Brian, how's it going, man?

00:16 It's going really good.

00:17 Yeah, I'm super excited to talk about what we got to cover today.

00:21 But I also want to say thank you.

00:22 I want to say thank you to Rollbar because they're sponsoring this episode.

00:25 And I'll tell you more about their cool stuff that they do later.

00:28 Let's start out about with the way that you think about programming.

00:31 Because learning to program is somewhat learning the syntax, but it's also just learning the mental models of what programming means, right?

00:38 Yeah, and I did realize that when you think about Python, the problems and problem solving is different in Python than in other languages.

00:47 Just because of the idiosyncrasies of the language and how people tend to do things.

00:52 And a couple of good articles that I stumbled across.

00:56 Actually, I think one of them is a...

00:58 Anyway, so there's Python functions aren't what you think.

01:01 And this is on a site called Powerful Python.

01:05 And it's a short little article just talking about that functions are really just objects that happen to have a name.

01:15 That's the name of the function that you gave it.

01:17 And they're really that much different than giving a variable the value of five.

01:24 You just give a variable a value of a function.

01:27 It does change the way you...

01:29 If you can think about functions that way, you can change the way you do some things.

01:33 You know, not surprisingly, I think it really enables functional programming type of metaphors.

01:39 Just the whole concept of treating functions as first class objects, right?

01:43 Functions are like variables or like classes or like whatever.

01:46 And I think that's really interesting.

01:48 It's really super important for things like lambdas and closures that we can treat functions this way.

01:54 But even normal functions are basically the same thing, right?

01:58 Yeah, and that you can just throw one in a data structure and use it later.

02:03 It is cool.

02:04 What I thought was weird was you can change the Dunder name of a function.

02:08 That was news to me.

02:09 Oh, yeah.

02:10 Yeah, I've never tried to do that.

02:13 Neither have I.

02:14 Seems a little evil.

02:15 A little bit.

02:17 And then another article called the Tau of Python, which I think is just a cute name.

02:23 But it's actually a Jupyter notebook that talks about the relationship between objects, classes, and meta classes.

02:30 And again, it's a similar thing of not just functions are objects, but even classes themselves are objects.

02:39 Right.

02:40 Right.

02:41 And their classes are meta classes, right?

02:43 Which is, it's very crazy.

02:45 Yeah, it definitely is a mental shift, especially the meta programming.

02:48 And that's something I could learn more about personally as well.

02:53 Yeah, me too.

02:53 And I haven't really made use of it a lot, but it's kind of one of those things of if you can think about the whole thing and how it fits together in just kind of a little mental overview, I think it helps.

03:07 Yeah, excellent.

03:08 All right, number two, the one that I chose is an article by a friend of the show, Dan Bader.

03:13 And his article he wrote, which I helped him a little bit with, you'll see at the very end, is called Why Learn Python?

03:20 Here are eight data-driven reasons.

03:23 And people send me messages and say, hey, Michael, I'm thinking about changing careers or aspects of my career.

03:29 Maybe I'm a Java developer and I'm considering learning Python.

03:33 Is it worth it?

03:34 And you'll see from this article, the answer is yes, yes, and absolutely yes.

03:39 Python is the second most popular language in the world, according to the popularity of Programming Language Index.

03:45 It's one of the hottest job skills to have, according to Dice.

03:48 And Dan lays out eight really well-researched reasons why this is the case, basically to support these trends.

03:58 So, you know, it says you can learn pretty much anything with Python.

04:00 It's widely used by data scientists.

04:03 Python pays well.

04:05 In fact, it's number two only to Ruby developers, according to Guru.

04:10 But it has many more job openings than Ruby does.

04:15 And it's, on average, above all the other developer salaries if you're looking at the U.S.

04:20 So, for example, they say Indeed says the average Python developer in the U.S. across all the regions, they're thinking that's something between $103,000 and $116,000 a year, which is amazing, really.

04:32 Yeah.

04:33 Yeah, that's a good place to be, right?

04:36 Yeah, and I remember when C++ was up near one of the tops, and it's now down at number seven now.

04:43 Yeah, it's a Python world.

04:46 All right, so.

04:47 One of the things that he didn't list, which I'm confused by, was that you can listen to this podcast.

04:53 It's totally bizarre.

04:55 I forgot that.

04:56 I mean, there's like a couple of good Python podcasts around this, at least, right?

04:59 Yeah.

05:00 You know, that's one of the good reasons to learn Python.

05:02 Yeah, so we'll add one in there, that's for sure.

05:04 It definitely helps you.

05:05 And so there's demand is growing for Python developers.

05:09 It saves you time.

05:10 It's beginner-friendly, but you don't top out really quickly, right?

05:13 Like, it can be an advanced language as well.

05:15 All the big companies are using it.

05:17 So if you want to get a job somewhere cool like YouTube or IBM or Mozilla or Quora or Instagram,

05:23 all these companies are making major use of Python.

05:26 So this could be your ticket there.

05:28 And finally, Python has a great ecosystem.

05:31 Yeah, and one of the great things about articles like this are we're preaching to the choir, of course,

05:36 but a lot of people get those questions of like, well, why should I learn Python?

05:41 And all I usually have is it's fun and cool and why not?

05:47 So these are better reasons.

05:49 Yeah, this is really, it's a really great article.

05:50 And of course, we're linking to it.

05:52 I think it's not, I mean, it's great to be preaching to the choir, but it's also interesting to provide this as fuel for people having these debates.

06:01 Maybe they're working on a team and they're like, well, what technology should we choose for our next topic, our next project or something?

06:08 And having all that information there is like, hey, look, choosing Python is a really good bet.

06:12 You'll have lots of people who know it.

06:13 There's a lot of, you know, just it's a good bet.

06:16 And I think this will help people make that case.

06:17 And I'm also a firm believer in just resume building.

06:21 You don't know what the next job is going to be like, and it probably will use Python somewhere.

06:26 Yeah, absolutely.

06:27 Anyway.

06:28 All right.

06:28 I bet people do some testing with Python.

06:30 Yeah, they do.

06:31 And I was thrilled to have somebody write a pytest article because it's one of my favorites.

06:37 So there's, and cool last name too.

06:40 I think I'm going to try this.

06:41 Kevin Ndungu, I think.

06:44 Ndungu?

06:45 Anyway.

06:45 Testing Python Applications with pytest.

06:49 And it's a good introduction article to try to not scare people off too bad about some of the advanced features.

06:58 But after getting, jumping into it a little bit, he shows how easy it is, of course, to write tests.

07:05 But then also takes a look at one of the common things that happens is you grow a whole bunch of tests.

07:11 And then you want to refactor them.

07:13 And there's a little bit about refactoring your tests by using fixtures and pulling out some of the common starting state.

07:21 And then he jumps into a couple of the nice, one of the nice features of pytest, of course, is the parameters.

07:28 And then you're going to have a lot of the standardization of easily being able to throw multiple data sets at an individual test function.

07:35 And that was demonstrated pretty cleanly.

07:38 I like that.

07:38 Nice.

07:39 Good job, Kevin.

07:40 And I like the article.

07:41 Yeah, I really like the let's treat test code as production code.

07:45 Because I feel like a lot of people will write tests and they're like, these tests are hard to maintain.

07:49 And then you look at their test code and it's, you're like, why did you duplicate this bit of code 20 times?

07:55 Like, what is this?

07:56 Would you ever do this in a real app?

07:58 That's insane.

07:59 Like, why is this not a function or some other thing that you can use throughout your test, right?

08:03 Yeah.

08:03 Yeah, definitely.

08:04 Speaking of production, you don't want errors in your production, do you?

08:08 No, of course not.

08:09 No, not too much.

08:10 One of the first steps in solving errors in production is knowing about them.

08:15 And that's where Rollbar comes from.

08:17 So Rollbar is supporting the show this week.

08:18 Thank you, Rollbar.

08:19 And I've used Rollbar for a long time on the website that delivers Python bytes and my training courses and all those things.

08:26 And I would say at least five times Rollbar has saved me from some seriously bad choices that I've made.

08:33 I've tested something locally where there was data in the database and I put it in a production.

08:39 I was about to start putting data, but it was empty at the time.

08:42 And the code couldn't deal with empty sets and all sorts of stuff going on where something crashed on the site and I didn't realize it.

08:49 And of course, Rollbar sent me a message, popped up on my phone immediately and said, error in production.

08:54 Here's the exact details of what you need to fix.

08:57 And I was like, ah, crap.

08:58 You know, five minutes later, though, it was fixed.

08:59 So thanks, Rollbar, for helping me out then.

09:01 And thanks for sponsoring the show.

09:02 Awesome.

09:03 It was a special offer for listeners.

09:05 And I'll put that link in the show notes right at the top.

09:08 Speaking of production, we would all like our code to run a little bit faster.

09:12 And I feel like we're in this cool place with Python, especially Python versus legacy Python.

09:18 Like Python 5, 3.6, and 3.7, the one that's coming, have really been focused on performance.

09:24 And so there's a, you know, we talked about a web framework called SANIC that was based on AIo HTTP,

09:31 which is an async pipeline for web processing.

09:36 Well, here comes another one.

09:38 And this one, it turns out, if the benchmarks are to be believed, is dramatically faster than even SANIC

09:47 and lots of other things out there like Go.

09:49 So it's called Jepronto.

09:51 Jepronto?

09:51 I'm not sure which, how you pronounce it, but we'll link to it.

09:55 It describes itself as a screaming fast Python 3.5 and above web micro framework integrating with pipelining HTTP servers based on UV loop and Pico HTTP parser.

10:07 So it came out just a few months ago.

10:10 It has like 2,600 stars on GitHub.

10:15 And the title of the article introducing it was Handling a Million Requests Per Second with Python.

10:21 Yeah.

10:22 I think that's on one server.

10:23 So that's crazy.

10:26 It is crazy fast.

10:27 It's pretty exciting.

10:28 If you read this article that we're linking to, the guy who built it, he's doing a crazy amount of careful thought and low, low, low level OS type of optimization.

10:41 So just to give you a sense, most of this is written in C, although he's thinking of rewriting it in Rust.

10:47 He talks a bit about that and says, look, I'm going to call it Jepronto.

10:52 I'm going with that one.

10:53 Jepronto tries to delay the creation of Python counterparts of its internal structures until explicitly asked.

11:00 So, for example, like normally when a request come in, like the request object would be populated with like headers.

11:06 Well, this framework won't actually create the headers dictionary unless you try to access it.

11:13 It gives you only what you need at many, many levels and has some really cool low level optimizations around pipelining.

11:19 It's quite cool.

11:20 Yeah.

11:21 The pipelining actually description was very, I thought that was pretty interesting.

11:24 Yeah.

11:25 I hadn't really thought too much about that.

11:27 So super, super cool.

11:28 Yeah.

11:28 And related to this, I think that somebody on Twitter was commenting that,

11:34 why are we worried about speed?

11:37 Why isn't it better to be correct?

11:41 I guess I like to have both.

11:42 And I think it would be, I think I'm intrigued by other people trying to get performance up.

11:49 And I think it's, to me, it's important because I would hate to have somebody say,

11:54 oh, eventually you're going to scale out of Python and need to switch over to go.

11:59 Exactly.

11:59 That's the primary thing.

12:02 I think, and even if you're not going to scale out, if you start to hear lots of major companies saying,

12:06 we're all abandoning Python because they won't run our workloads.

12:09 Even if that's not the situation you're in, it starts to like cast a shadow over the general ecosystem.

12:15 Right.

12:16 Whereas if you say, no, no, look at these amazing things that are being done with Python on the web,

12:23 your stuff is going to be way lower traffic and load than that.

12:27 Surely this will work for you as well, right?

12:29 Yeah.

12:29 So I think it's really interesting.

12:31 I mean, to be honest, Pyramid plus MicroWisgi plus Engine X just straight up is way fast enough for what I'm doing,

12:37 even with a couple million ACP requests a month.

12:41 But it's super cool to see that it's sort of expanding the upper bound.

12:46 And I think that's going to have some great stories down the road.

12:48 Yeah, definitely.

12:49 So the next topic we've got is a good news thing that came out, was just announced today.

12:56 The Rethink, and I'm not sure if you had RethinkDB on.

13:01 I did.

13:01 I had him on my show.

13:02 I can't remember the number.

13:04 I think it's in the 60s on Talk Python TV.

13:06 We talked about RethinkDB and the company behind it.

13:09 Yeah, and I think I've heard about it because of Talk Python.

13:13 But it was really intriguing to me.

13:16 And I like the idea of somebody just from the ground up rethinking how databases work with live data and real-time data.

13:25 But in September of last year, the company shut down.

13:31 And I was curious, like everybody else was, is what's going to happen to RethinkDB?

13:35 But I guess somebody called Cloud Native Computing Foundation, they made an announcement that they purchased the rights to the source code and contributed to the Linux Foundation.

13:46 And it's changing the licensing as well to the Apache license version 2.

13:52 And everybody can continue using it now and will be able to have this project live on.

14:00 That's really cool.

14:01 So RethinkDB is supposed to be like a real-time, it's like a document database that is a real-time database for your server.

14:09 So it's definitely got some interesting components.

14:13 And I believe all the team that worked there has made their way over to Stripe.

14:17 I'm pretty sure that's what's happening.

14:20 So everybody who used to work at Rethink has kind of moved over to Stripe.

14:23 And that's really cool that they had a soft landing somewhere.

14:27 And it's great to see the Linux Foundation taking up this project and keeping it going.

14:32 And I guess there were some companies that are relying on it and wanted to somehow donate to try to keep it going.

14:39 And there wasn't a way to do that at first.

14:40 And now there is, and they've got, especially under the Linux Foundation, and Stripe has put together $25,000 in matching funds to match other donations.

14:51 So I wish them all well and hope that RethinkDB lives for a long time.

14:56 Yeah, they definitely occupy cool space in the database world.

14:59 So happy to see them rolling.

15:01 All right, let's round out this conversation with my sixth item here, which is the Python Top 10 Articles of the Past Year.

15:11 And this comes from this interesting work reading app or technical reading app called MyBridge at mybridge.co.

15:21 And so this is kind of like Zite or Flipboard, but it's specifically focused on technology and reading for work and for your career.

15:30 Okay, so it's an interesting like iPad app and whatnot.

15:33 But what they did is they said, how many articles?

15:37 They had some crazy number of articles that for the past year, we've ranked nearly 10,000 Python articles.

15:42 And we picked the top 10 stories with a 0.1% chance that will help you advance your career in Python.

15:48 Oh, nice criteria.

15:49 Yeah, it's really nice, right?

15:51 So the first one is from Kenneth Hitchhiker's Guide to Python Best Practices Guidebook for Writing for Humans.

15:58 That's really cool.

15:59 And then SciPy Lecture Notes.

16:01 30 essential Python tips and tricks.

16:04 Computational and inferential thinking and data science.

16:08 Welcome to Python Cheat Sheet.

16:10 Data mining in Python.

16:12 Python FAQ.

16:13 Why should I use Python 3?

16:16 By Eve.

16:17 Remember the whole Zed Shaw.

16:19 Python 2 is going to live forever.

16:22 No Python 3.

16:23 This guy wrote the beautiful rebuttal to that article.

16:25 So this article is also by him.

16:27 Introduction to Stock Market Analysis, NumPy Tutorial, and Build Your First Python and Django app.

16:33 So I like this pick for a couple of reasons.

16:36 One is it's just cool to have some really nice reading.

16:40 And this is kind of a cool app, this MyBridge thing.

16:42 I've just started playing with it.

16:43 But I think it gives you a pretty interesting view into the Python space.

16:48 People ask, what can you do with Python?

16:49 What should I learn with Python?

16:51 Well, this is like 60% data science, 25% web, and 25% language by my super quick estimation.

17:00 So I think that's kind of an interesting map to draw.

17:03 Actually, I hadn't run across a lot of these articles.

17:06 So I was pretty glad to hear, see some of these.

17:09 So even though that's our six items, I actually have one more for you.

17:11 Last week, we talked about Red Hat distribution, Red Hat Enterprise Linux,

17:16 and how people are having trouble using Python 3 because they keep sending this message going,

17:21 I can't use Python 3.

17:22 Well, a friend of the show, Chip Warden, actually added a comment.

17:27 If you want to add comments in the show, there's a whole discuss section at the bottom of each show page.

17:31 And said, hey, what do you mean people can't do this?

17:33 They should just check out Red Hat software collections.

17:36 So I checked it out.

17:38 Yeah, absolutely.

17:39 People should check this out.

17:40 It's the latest stable updates of development technologies for Red Hat Enterprise Linux.

17:45 Wow.

17:46 It's cool.

17:46 So if you want to work with Node, Perl, PHP, Python, or Ruby, it is basically an approved Red Hat Enterprise Linux way of getting Python 3, for example, on your environment.

17:57 Perfect.

17:57 Yeah.

17:59 So that's cool.

17:59 So that's definitely going to be in the show notes.

18:01 All right.

18:02 Well, that's one more reason to switch to Python 3, right?

18:05 Definitely.

18:06 And I can't believe we're done already.

18:08 Going so fast.

18:09 I know.

18:10 Well, I think we're going to have to scour the internet for more cool Python stuff for next week.

18:15 What do you say?

18:15 Yes.

18:16 And I want to encourage anybody to help us out by sending us what you think we ought to cover next week.

18:21 Yeah.

18:21 We don't always cover everything people send, but if it matches up, we definitely try to cover it.

18:27 So please help us out.

18:28 If you know something like Chip did here that we didn't cover, send it our way.

18:32 All right.

18:34 Well, thank you, everyone for listening.

18:35 Brian.

18:35 Thank you.

18:36 Great to catch up with you.

18:37 Like always, man.

18:38 Thank you for listening to Python Bytes.

18:41 Follow the show on Twitter via at Python Bytes.

18:43 That's Python Bytes as in B-Y-T-E-S.

18:46 And get the full show notes at pythonbytes.fm.

18:50 If you have a news item you want featured, just visit pythonbytes.fm and send it our way.

18:54 We're always on the lookout for sharing something cool.

18:57 On behalf of myself and Brian Okken, this is Michael Kennedy.

19:00 Thank you for listening and sharing this podcast with your friends and colleagues.