Transcript #71: We can migrate to Python 3, careful please

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.

00:04 This is episode 71, recorded March 22nd, 2018.

00:09 I'm Michael Kennedy, and normally I would introduce Brian, but Brian is not here this week.

00:13 So I'm here with Trey Hunter. Trey, hello, and welcome to the show.

00:16 Hi, thanks, Michael. How are you doing?

00:18 I'm doing great, doing great.

00:19 Thanks for dropping in.

00:21 There's a bit of a shuffle with the whole Spring Break travel chaos going on, and so it's really great that you're here to record with us, and Brian will be back with me next week.

00:31 So, you know, probably a lot of people out there in the audience know you, but maybe a really quick introduction just so people know who you are.

00:37 Yeah, I am super happy to be here because I'm a big listener of the show.

00:40 I do on-site Python training for teams, and I help folks learn Python on the internet through weekly things, and I tweet a whole bunch.

00:46 So folks have probably seen me on the internet, if anywhere.

00:49 You work on the internet. That's awesome.

00:51 Yes.

00:52 And do things on that internet. All right, awesome.

00:53 So before we get to the first item that you picked, I just want to say thank you to Digital Ocean.

00:58 They're sponsoring this episode like they do many.

01:00 Check them out at do.co/python.

01:03 Trey, what do you got for us this week?

01:04 So my first pick is kind of related to a talk I'm doing at PyCon.

01:08 I've been researching this stuff a whole bunch.

01:10 It's the conservative Python 3 porting guide.

01:12 This was written by a whole bunch of folks at Red Hat.

01:15 And it's pretty much the best porting guide I've seen for how to take your code from Python 2 and switch it to Python 3.

01:22 It's not perfect, but none of the guides out there really are.

01:25 Yeah, this is something that there's really not as much material on as I kind of would expect there to be. I mean, there are tools, we have six, we have two to three, we have Anthony Shaw's Pluralsight course that he just launched, which is really a good course.

01:39 But, you know, that's brand new.

01:41 And Guido just recently made a statement, which I thought was great, that really solidified the date. Because for a long time it's like, well, Python 2 is going to go out of out of support, out of security patches and all that stuff in 2020. And like, that's a whole year, right?

01:58 We need a more time frame than a year, right?

02:01 So when is it actually going out?

02:02 It's January 1st, 2020. So it is less than two years away now.

02:06 So people should probably start paying attention to this if they're ever going to.

02:10 Yeah, this is a really big deal.

02:12 A lot of my clients use Python 2 and switching to Python 3 is going to be a really big headache.

02:18 I do wish this guy'd used the Future library, which I know Anthony uses in his courses at Pluralsight or his one course there on switching from Python 2 to 3, instead of the modernized library.

02:28 But I mean, none of these guides is perfect.

02:30 Some of them are a little bit older than others.

02:31 This is just one of the best ones I found out there.

02:33 Yeah, that's really awesome.

02:34 I really like it.

02:35 It's called the conservative Python 3 porting guide.

02:38 Why conservative?

02:39 Conservative because they don't try to get you using Python, new fancy Python 3 features.

02:45 They're just trying to get your code working on Python 3, even if in a kind of begrudging way.

02:49 If you don't want to switch, you have to out of necessity, this is a conservative guide.

02:53 I see. So it's not like, oh, and while you're here, wouldn't it be awesome to throw a little async and await in there?

02:59 Exactly.

02:59 And maybe some stars to force some keyword arguments and off you go.

03:04 No, this is just like, you need to get it to your older code, your legacy code to run on modern Python.

03:11 And that's it. No changes.

03:13 Absolutely. Yeah, this is you just need to get running.

03:15 Yeah, I think that's a really good way to do it because On one hand, I'm sure it's super tempting, you're in there, you want to make all these changes, but on the other, you don't want to conflate the making it run on Python with adding new features, re-architecting your code, and so on, right? Maybe that's a second project.

03:31 Absolutely.

03:32 So I have a question for you.

03:34 Okay.

03:35 Which companies use Python?

03:36 Well, there are a ton of them, actually.

03:39 Some of the best companies on the web that you know of definitely use it.

03:43 So there's a really cool article on realpython.com by Jason Reynolds, And I think a lot of people come to Python.

03:52 I don't know, you give me your opinion on this.

03:54 I feel like they come and they're like, "Oh, I write a few scripts and I kind of like tinker with Python, but I don't write apps in Python." Like, that's a different thing.

04:02 Do you get this experience?

04:03 Absolutely. This is a big thing with a lot of my clients is we use Python, we don't use it for making deployable software our clients use.

04:09 It's just for scripts.

04:10 Right, exactly.

04:11 And so, obviously, that's possible with Python, but there's so much more that you can do.

04:15 So I just wanted to pull this article up as some examples for how people might use Python.

04:22 And if you're thinking of using Python at your company and they're like, "No, no, we use C#." That compiles. That's something we can use in large projects.

04:29 We can't use a toy language, right?

04:30 Like, think about what these companies are doing.

04:32 So the first one, Industrial Light & Magic, they do all the special effects from any of the sci-fi movies and so on.

04:39 And they use Python to track and audit pipeline functionality, maintain databases and basically move items and assets through like a pipeline.

04:49 That's pretty awesome.

04:50 We have Google.

04:51 So in the beginning, of course, right, of course, Guido worked there.

04:55 Many of the luminaries in the Python space have or do.

04:59 And they said, the founder said, we're going to use Python where we can and C++ where we must.

05:05 That's pretty cool, right?

05:06 That is really cool.

05:07 I love that phrase.

05:08 And of course, Python powers YouTube still to this day, handling millions of requests, per second, that's a pretty insane number, right?

05:16 - That's a ton of stuff to be putting through Python there.

05:19 - I know, that's quite a bit.

05:21 It's really impressive how much is going on there.

05:23 And actually, there's some really interesting comparisons.

05:27 Like there used to be Google Video, which was a competitor to YouTube when YouTube was a startup.

05:32 And they were doing all their work in C++ at Google Video.

05:36 And the people, many fewer, like 20 versus hundreds of engineers were writing Python and just like running circles around Google Video.

05:43 So Google solved it by buying YouTube.

05:46 It's a pretty interesting history.

05:47 - I never knew that.

05:48 I mean, I knew they acquired YouTube while having video.

05:50 I didn't know that Python was a deciding factor there.

05:53 - Yeah, it was one of the deciding factors that let like the small team outperform Google 'cause Google is using C++ for their video service.

06:00 All right, so speaking of large services, Facebook.

06:03 They use this mostly on the infrastructure side, but Facebook is really interesting in that they have did a really interesting thing moving from Python 2 to Python 3 as sort of like changing the engineering culture.

06:15 Jason Freed did this really awesome talk on that and the whole steps in which they took to make this happen are pretty amazing.

06:21 Did you happen to see that talk?

06:22 - I did, you know, I think he's given a similar talk at PyCon actually this year.

06:26 - Oh, that'd be awesome, I'm definitely going.

06:27 Definitely going.

06:28 Closely related is Instagram because kind of like YouTube, Instagram was its own thing and now it is like engulfed into Facebook.

06:36 But they run the largest deployment of Django web framework in the world.

06:41 and they have like 800 million active users.

06:45 I think maybe the most notable thing was Instagram's keynote at the 2017 PyCon with Lisa Guo and Hui Ding, who gave like the most inspiring talk, how they migrated from old version Django to new version Django and Python 2 to Python 3 without branching and continuing to ship features.

07:04 Yeah, that was amazing. I really liked that talk because in the beginning they started with, here's how we did it at a high level and towards the end, they dove right into the code that they use and the difficulties they encountered.

07:13 And, you know, if you are afraid of encountering difficulties, look at the ones that Instagram encounter because they've got a big code base.

07:20 Yeah, they have like millions and millions of lines or something. It's insane.

07:24 But the way they did it was truly inspiring.

07:26 And I felt like that was actually like a really clear playbook of, hey, we could do the same thing.

07:30 Absolutely.

07:31 We also have Spotify, they're doing a bunch of stuff with Python, primarily as back-end services and data analysis.

07:37 Quora, I think Quora is the best Q&A place on the internet.

07:40 It always seems to have such like super thoughtful questions and answers.

07:43 You know, I don't use Quora much, mostly because whenever I go, it asks me to sign up or sign in, so I mostly go to the other ones for that reason.

07:50 But I have actually found a lot of useful stuff in Quora before.

07:53 Yeah, I'm just surprised how much time and energy people are willing to put into answers there.

07:57 Absolutely.

07:58 It's interesting because that runs on Python, of course, and they were thinking about maybe C#, maybe Java, maybe Python, maybe C++ and so on.

08:06 And in the end, they ended up kind of doing what Google did, is Python where they can, C++ where they must.

08:12 Netflix, obviously Netflix is insane.

08:14 They do, like, in the evenings, they account for one third of the bandwidth in the United States.

08:19 - Wow.

08:20 - Yeah, and they run all their infrastructure and management and operations stuff over Python.

08:24 I actually talked to those guys over at Talk Python on episode 16 way back in the day.

08:28 Really interesting stuff they're doing.

08:31 Dropbox, when I think of the center of the universe of Python, I think Dropbox.

08:35 What do you think?

08:36 - Well, Guido works there.

08:38 - That's a big reason.

08:39 - It seems to be one of the companies that like, is one of these major, major sort of technology companies that is all in on Python, right?

08:47 Like even their little widget thing that ships as a client app that is in your tray or your menu bar, that's even Python.

08:55 - Yeah, they definitely got a ton of deployed Python code on desktop computers that they don't have control over, which is pretty amazing.

09:02 I think they also have some open source packages up on GitHub, their Python.

09:05 - Yeah, I think they do have a lot of really interesting stuff.

09:07 And they've had some stuff that they put out that's kind of like left to go as experiments.

09:11 Very, very interesting.

09:12 So last one I'll leave you with, Reddit.

09:14 So they have half a billion visitors a month.

09:17 They had, let's see, it was the fourth most visited site in the United States and seventh in the world.

09:24 And I guess in 2015, they said they had 82 billion page views, which is kind of insane.

09:31 And of course, that's all driven on Python, SQLAlchemy, all sorts of goodness.

09:35 - That's pretty incredible.

09:36 - Yeah, awesome.

09:37 Those are the companies that I kind of wanted to highlight out of that article and just thought, like, these are really great examples.

09:42 And if people are saying, "Oh, but we have like 100,000 users.

09:45 "We can't use Python.

09:46 "It's not gonna scale." Like, well, yeah, probably not the real problem.

09:51 There's probably some kind of architecture or infrastructure thing that is actually gonna make this work easily.

09:55 - For sure.

09:56 - Yeah, nice.

09:57 So do you think we should stop writing classes, Trey?

10:00 - Maybe.

10:01 I don't know if we should stop writing classes entirely, But my favorite, well, one of my favorite PyCon talks, I tend to tell people it's my favorite because I recommend it so much, is "Stop Writing Classes" by Jack Diedrich.

10:14 This is from PyCon 2012, so it's kind of an old talk, but it is definitely the days of Python 2, but all the code in it, all the advice in it is still really applicable today.

10:25 - Yeah, that's really awesome.

10:27 So give us the main takeaway from it.

10:29 - One of the biggest bits of advice in it that I tend to tell people is if you have a class that only has two methods, one of those is your dunder and knit method, and the other one is some other method, especially if it's called call, you probably need a function instead of a class.

10:44 So this is something I recommend a lot to folks moving to Python from Java and C++ because I teach a lot of folks who are moving programming languages and you know, Python is just different from the other languages.

10:54 Especially if what you're going to have is effectively a static class, right? Like, there's just one copy of the variables.

11:00 it's very likely that you could get away, and you're not doing any inheritance, you're not doing any of that kind of stuff, you probably could just have a set of functions and module level variables, and you get effectively the same output.

11:11 Yeah, absolutely, and that's something I see a lot in code.

11:13 On the flip side of things, though, one of the great things about this talk is it doesn't just show you when not to write classes, it does show you a couple examples of when classes are really handy in Python.

11:21 Yeah, that's cool. I'm still a big fan of object-oriented programming.

11:25 I think there's still, you know, people, it fades in and out of popularity. I still think it's a really great way to model stuff.

11:30 But when you're creating more than one of the things typically, right?

11:33 Absolutely. Yeah, it's when you need to make more than one of them, when you need to use it more than once, that's the time where classes need.

11:40 They really can be abused.

11:41 Yeah, absolutely.

11:42 So what's up with PyPI, Michael?

11:44 I'm going to tell you about PyPI. I'm super excited. But before I do, I want to tell you about DigitalOcean. So there's this thing coming up where I might need a whole lot more power for my web server, 'cause there's some big promotion and it could kind of end up overwhelming my servers, which so far they've never had an issue.

12:00 Like you log in, it's like three, 4% CPU.

12:02 But one of the things that's really cool about DigitalOcean is if you have one of their servers, you can go over there and you can flip a switch and within just a couple minutes, it will take it offline for a second, resize, re-whatever, however you change the infrastructure.

12:16 Say I want four CPUs and eight gigs of RAM, change it to that.

12:20 And you could do that for like one day and then go, no, no, back to the one gig, one CPU the next day, right?

12:25 Spend five bucks and get like some super huge server to handle some spike that you know is coming.

12:30 So pretty cool.

12:32 A lot of the stuff that I do on the web is running through DigitalOcean, so definitely recommend them.

12:37 Check them out at do.co/python.

12:40 They help make the show possible.

12:42 So another thing that I think is doing really awesome stuff on the web, finally, is the new PyPI.

12:49 Do you remember back when there's pypi.python.org/pypi because it's better if it ends up on both ends of the URL.

12:57 That's the original gray that's been around forever.

13:01 And then for a while there was a pypi.io, which was like the new one.

13:07 Then there's a pypi.org, which they redirected to.

13:11 I think that's the same site.

13:12 - It's getting confusing here.

13:13 - I know.

13:14 And so there's been a really, almost a big problem with the old PyPI in that it was based on just super custom hand-rolled socket processing.

13:25 And like the entire PyPI was running out of like a handful, two, three Python files, thousands of lines long, purely custom socket code.

13:35 You know, no, it's not based on Flask or Pyramid or any of those things.

13:38 It's just like, yeah, we're just going to start like at the lowest level.

13:41 I think it was more or less a prototype and it just became what PyPI was for many years.

13:46 And that was actually a big problem because people will come on, "Hey, I got this cool idea, I want to add a feature, like, can I just plug that in as an extension?" Or, "Oh my God, what is that? I'm not going to contribute to this project, I'm out of here." And so finally, finally, Donald Stuft and crew have released PyPI.org as an official thing.

14:06 You still have that red bar up there, like, "Warning, this is like a testing area." It's no longer a red bar, like, you go there and it's properly a website.

14:13 I'm going there right now to make sure.

14:15 Oh, wait, it's back again. It was gone earlier.

14:18 There is it. So there is a red bar. I think you can close it.

14:21 However, this I love that this is officially it's coming up here, which is super exciting.

14:27 Yeah. So the big news is not the website itself, although that's sort of the portal to it. The big news is when you pip install the thing, it now goes through pypi.org.

14:36 Oh, that's really exciting. So this actually is not only the future, it is the present of using pip with Python.

14:42 - Yes, that is where we are.

14:44 So there's a really cool tweet.

14:47 Sorry, I forget the guy's name who sent it out.

14:48 It has like graphs of the analytics of sort of the number of requests per second and errors per second and memory usage and latency, all that for the new pypi.org handling the pip install requests.

15:01 - That's exciting.

15:02 - Yeah, so I'm really glad to see that.

15:03 I guess I did just close that X and it stayed closed.

15:05 You're right, so I opened it in like an incognito window.

15:07 It's still there, but it's almost gone, almost gone.

15:10 - All right, it's good that at least cookie you.

15:12 That's all right, it is good.

15:14 So one of the biggest challenges, you opened this whole section with getting started and converting from Python 2 to Python 3, and I would say the biggest thing that people complain about, saying this is really hard to get this transition over, is the old strings, byte strings, Unicode bytes array madness around that, right?

15:36 Yeah, absolutely. This is one of the biggest challenges, is figuring out what is text and what is data and separating those because Python 2 didn't make a distinction, Python 3 forces you to make that distinction.

15:47 So there's a nice talk by Ned Batchelder, right, that you pulled up for us?

15:51 Yeah, absolutely. It's called "Pragmatic Unicode." I think it's "Pragmatic Unicode" or "How Do I Stop the Pain?" is the full talk title.

15:57 It's counterintuitively, this is from PyCon 2012, and it is mostly Python 2 code.

16:02 And so if you're switching to Python 3, this might seem odd to watch this talk.

16:05 But the reason this is still relevant today is Unicode hasn't changed.

16:09 The default way we see Unicode in Python 2 versus 3 has changed.

16:14 And there's a big analogy in this talk that I really like, a metaphor that Ned uses, the Unicode sandwich, which I'll let you go watch the talk to figure out what that is.

16:22 But this really helps shape your mental model of how text works versus how bytes work in the Python interpreter.

16:30 And I think this is really important, especially if you're not familiar with Unicode, and you're going to switch to Python 3 or to start using Unicode, you've got to understand a little bit about how it works.

16:38 Yeah, this is really, really interesting.

16:40 And Unicode is definitely one of those tricky things.

16:42 And I think growing up in the United States definitely at least puts, I felt like put me at a disadvantage of appreciating Unicode.

16:50 You know, because it's like, well, ANSI, what's wrong with ANSI?

16:54 I don't know, it seems fine.

16:55 Yeah, absolutely. And I feel like the big thing that's making the change here for a lot of people is emoji, which is being used even by people who are in English-speaking countries.

17:05 So the rest of the world is finally going, "Yay, everyone cares about Unicode now," all because of these emoji characters, which is sort of sad in a way, but also sort of good because it means we're actually caring about adopting this universally.

17:16 It's really interesting because like, it's a bit of a sad commentary on the world, like that's what pushes us forward.

17:21 At the same time, it is good that it's being pushed forward, right?

17:24 Absolutely, yeah.

17:25 Nice.

17:26 Alright, so this last one that I want to cover really has to do with thinking about alternate implementations or run times of Python, right? It's easy to say, well, there's CPython and, you know, there's just Python, right? I type Python and it runs, but there are actually many different run times. And I say run times because some of them are called interpreters, but only some of them interpret.

17:49 Right? So, like, we have IronPython, Python.net, Jython, Cython, a whole bunch of other ones that are like a little more niche.

18:01 And I'd say probably the most popular alternative one is PyPy.

18:06 P-Y-P-Y.

18:07 Have you done anything with this?

18:08 I have not done anything with, well, I played with PyPy, but only in one of those kind of like, maybe I'll pip it, or maybe I'll apt to get install it and see what happens.

18:15 Yeah, exactly.

18:16 Yeah, so PyPy is a JIT compiling implementation of CPython.

18:24 So the way CPython works, even though it's written in C, when you first hit your code, it all gets compiled to bytecode, but then those bytecodes don't get further compiled to machine instructions.

18:33 They just get interpreted in like this massive switch statement, while true loop.

18:40 And it just looks at things and it just interprets them as bytecode at runtime.

18:44 But that's not the same as compiling down to machine instructions, right?

18:47 So PyPy, surprisingly, does the same thing that CPython does until it finds a slow spot and then it switches it to a JIT compiled version, which is interesting.

18:57 Wow, that's really clever.

18:58 Yeah, it just finds the slow spots and go, "Did you call this a thousand times and it's slow?" All right, we're going to make that fast. We'll fix you up.

19:04 So there's a little bit of a slow startup.

19:06 So if this was like a script, it runs for half a second and it goes away, there might be no benefit or less benefit, maybe even a drawback.

19:13 I don't know. But if you use a web app that's going to be up for six hours and maybe get recycled or it's a game, say, but we're game that we're running, then that stuff would get jet compiled and be sort of warmed up, if you will.

19:25 - Yeah, so you're sure here, I think we're talking about games specifically, right?

19:29 Why would it be a game that this is nice for?

19:32 - Yeah, so there's a, one of the nice ways to make games is with Pygame, right?

19:37 And this guy, Rene Dunfield, he was at the PyPy Sprints in Switzerland, which were like in the Alps at some beautiful resort.

19:47 I'm like, man, I need to go to a sprint up here.

19:50 Anyway, they were working on it, and one of the things they got running is they got the new PyPy running PyGame perfectly well.

19:57 Apparently there were issues with it before, and they said on some of the games they got up to a 30-time speedup by just simply typing something different to start your game.

20:06 - Wow, so they got a 30-time speedup by instead of typing Python, they type just run it in PyPy, and they get 30 times faster.

20:12 That's amazing.

20:13 - Yeah, it's the same PyGame, same code that runs on, you know, CPython and so on.

20:18 So PyBi has a different way to implement C extensions and different APIs, so this is really nice, right?

20:25 It has this new way to do extensions for C or assembly language.

20:29 It's the CPyExt is getting faster, which apparently was important for the game, it should run quicker and so on.

20:36 So basically, the people are saying, like, it's really nice to have one code base that I don't have to try to, like, split across these different runtimes.

20:43 It just runs in both places.

20:44 Apparently, there's a few types of operations that are slower because transitioning into and out of PyPy down into the C level and back is slower than regular CPython, apparently.

20:56 But still, pretty cool.

20:57 - That is, that's very cool, especially if you're using PyGame.

20:59 This is exciting news.

21:00 - Yeah, for sure.

21:01 Like, who wouldn't want their game to run?

21:03 You think of how much people optimize video games to shave, get a few frames per second here or there.

21:10 - Absolutely.

21:11 - And this is massive.

21:12 One example they gave is ray tracing, and they had run it on Python 2.7.

21:17 It took nine and a half minutes, and then they ran it on PyPy, and it took 18 seconds.

21:22 So that's a pretty good speed up.

21:23 - That's not just a pretty good speed up, that's an incredible speed up.

21:26 If my tests took 10 minutes to run, and then they went down to seconds, I would be pretty amazed.

21:30 - Yeah, that would be so awesome.

21:32 So if you're playing with PyGame, or even if you're doing stuff on the web, or whatever, check out PyPy.

21:39 It could make a huge difference.

21:40 - Awesome.

21:41 - Yeah, it's good to see that project continuing to grow.

21:43 That's it for our items. You got any extra news or things you want to share with people?

21:46 Ooh, it's tough to decide what's new.

21:48 Going to, you know, conferences, going to find new things on the internet, making new things on the internet.

21:54 The one thing I think I'd want to share is I launched a thing a couple months ago, kind of quietly, and I've been slowly kind of testing it out with folks called Python Morsels, where I send an exercise out every week.

22:05 And this is my way of trying to serve the folks who are at my local study group and other people that I meet on the internet who want to learn Python and you know, they don't have a company that they're working in that does Python that could bring me in for training because I love teaching.

22:18 And I really want to serve individuals. So folks could check that out. It's pretty fun service. Nice. That's just python morsels.com.

22:25 Yes, yeah, python morsels.com and it redirects right now. But you know, one day in the future, it'll have its own domain that doesn't redirect.

22:30 It'll be all grown up on the internet.

22:32 Exactly.

22:33 Yeah, that's awesome. Yeah, so I'll put that in the show notes.

22:35 Yeah, I just have one piece of news to share. I've launched I kind of talked about this last week, but now it's in full swing.

22:41 I launched my 100 Days of Code course with the PyBytes, unrelated to this podcast, but similar name, PyBytesGuys, Bob Belderbose and Julian Securia.

22:50 So, really happy to have this course out.

22:52 It's an 18-hour course that guides people day by day through 100 days of Python. So, that's pretty epic.

22:59 Yeah, I scrolled through the page for it. It is a long page.

23:02 There's a lot in that course. It's super exciting to see that.

23:05 It gave us a chance to play with stuff that's really pretty far out there and really fun like building GUIs in Python or different ways to consume APIs and just lots of really interesting stuff that we got to play with.

23:17 So people can check it out at talkpython.fm/100days, 100 days.

23:22 Put that in the show notes as well.

23:23 Nice.

23:23 Yeah.

23:24 All right, Trey. Well, thank you so much for jumping in and being a co-host for this week.

23:28 It was fun to share these items with you.

23:30 Yeah, thanks for having me. This was super exciting. I hope to be back one day.

23:33 We'll have you back for sure. Thanks. Bye.

23:35 Thank you for listening to Python Bytes.

23:39 Follow the show on Twitter via @PythonBytes, that's Python Bytes as in B-Y-T-E-S.

23:44 And get the full show notes at PythonBytes.fm.

23:48 If you have a news item you want featured, just visit PythonBytes.fm and send it our way.

23:52 We're always on the lookout for sharing something cool.

23:55 On behalf of myself and Brian Aukin, this is Michael Kennedy.

23:58 Thank you for listening and sharing this podcast with your friends and colleagues.