Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


« Return to show page

Transcript for Episode #71:
We can migrate to Python 3, careful please

Recorded on Thursday, Mar 22, 2018.

00:00 Michael KENNEDY: Hello and welcome to Python Bytes, where we bring Python news and headlines directly to your earbuds. This is Episode #71, recorded March 22, 2018. I’m Michael Kennedy. Normally, I would introduce Brian but Brian is not here this week. I’m here with Trey Hunner. Trey, hello and welcome to the show.

00:00 Trey HUNNER: Hi, thanks Michael. How are you doing?

00:00 KENNEDY: I’m doing great. Thanks for dropping in. There’s a bit of a shuffle with the whole Spring Break travel chaos going on and so, it’s really great that you’re here to record with us and Brian will be back with me next week.

00:00 Probably a lot of people out there in the audience know you but maybe a really quick introduction just so people know who you are.

00:00 HUNNER: I’m super happy to be here because I’m a big listener of the show. I do onsite Python training for teams and I help folks learn Python on the Internet through weekly things and I tweet a whole bunch so folks have probably seen me on the Internet if anywhere.

00:00 KENNEDY: You work on the Internet. That’s awesome. And do things on that Internet. Alright, so before we get to the first item that you picked, I just want to say, “Thank you” to DigitalOcean. They’re sponsoring this episode like they do many. Check them out at do.co/Python.

00:00 Trey, what have you got for us this week?

00:00 HUNNER: So, my first pick is kind of related to a talk I’m doing at PyCon. I’ve been researching this stuff a whole bunch. It’s “The Conservative Python 3 Porting Guide.” This was written by a whole bunch of folks at Red Hat and it’s pretty much the best porting guide I’ve seen for how to take your code from Python 2 and switch it to

00:00 Python 3. It’s not perfect, but none of the guides out there really are.

00:00 KENNEDY: Yeah, this is something that there’s not as much material on as I would expect there to be. There are tools, we have six, we have two to three, we have Anthony Shaw’s Pluralsight course that he just launched, that’s brand new. Guido (van Rossum) just recently made a statement, which I thought was great, that really solidified the date. For a long time it was like, ‘Well, Python 2 is going to go out of support, out of security patches and all that stuff in 2020.’ That’s a whole year. We need a more of a timeframe than a year. So, when is it actually going out?

00:00 HUNNER: It’s January 1st, 2020, so it is less than two years away now.

00:00 KENNEDY: So, people should probably start paying attention to this, if they’re ever going to.

00:00 HUNNER: Yeah, this is a really big deal. A lot of my clients use Python 2 and switching to Python 3 is going to be a really big headache. I do wish this guide used the future library, which I know Anthony uses in his courses at Pluralsight, his one course there on switching from Python 2 to 3, instead of the modernized library. None of these guides are perfect. Some of them are a little bit older than others, this is just one of the best ones I found out there.

00:00 KENNEDY: Yeah, that’s really awesome. I really like it. It’s called, “The Conservative Python 3 Porting Guide.” Why “Conservative”?

00:00 HUNNER: Conservative because they don’t try to get you using new fancy Python 3 features, they’re just trying to get your code working on Python 3, ev if in kind of a begrudging way. You don’t want to switch, you have to out of necessity, this is the conservative guide.

00:00 KENNEDY: I see. So, it’s not like, ‘While you’re here, wouldn’t be awesome to throw a little async and await in there? Maybe some stars to force some keyword arguments and off you go…” No, this is just like, ‘You need to get your older code, your legacy code, to run on modern Python and that’s it. No changes.’

00:00 HUNNER: Absolutely. You just need to get it running.

00:00 KENNEDY: I think that’s a really good way to do it because, on one hand, I’m sure it’s super tempting. You’re in there, you want to make all these changes. But on the other, you don’t want to conflate making it run on Python with adding new features, re-architecting your code and so on. Maybe that’s a second project.

00:00 HUNNER: Absolutely.

00:00 So, I have a question for you. Which companies use Python?

00:00 KENNEDY: Well, there are a ton of them, actually, Some of the best companies on the web that you know of definitely use it. There’s a really cool article on RealPython.com by Jason Reynolds and I think a lot of people come to Python – give me your opinion on this – I feel like they come and they’re like, ‘Oh, I write a few scripts and I’ve tinkered with Python but I don’t write apps in Python, that’s a different thing.’ Do you get this experience?

00:00 HUNNER: Absolutely. This is a big thing with a lot of my clients. ‘We use Python. We don’t use it for making deployable software our clients us. It’s just for scripts.’

00:00 KENNEDY: Exactly. So, obviously that’s possible with Python but there’s so much more that you can do. I just wanted to pull this article up as some examples for how people might use Python. If you’re thinking of using Python at your company and they’re like, ‘No, no. We use C#. That compiles. That’s something we can use on large projects. We can’t use a toy language.’ Think about what these companies are doing.

00:00 The first one, Industrial Light and Magic, they do all the special effects for many of the sci-fi movies and so on, they use Python to track and audit pipeline functionality, maintain databases and basically move items and assets through a pipeline. That’s pretty awesome.

00:00 We have Google. So, in the beginning, of course, Guido worked there. Many of the luminaries in the Python space have or do. The founders said, “We’re going to use Python where we can and C++ where we must.” That’s pretty cool, right?

00:00 HUNNER: That’s really cool.

00:00 KENNEDY: I love that phrase. And of course, Python powers YouTube. Still to this day, handling millions of requests per second. That’s a pretty insane number.

00:00 HUNNER: That’s a ton of stuff to be putting through Python there.

00:00 KENNEDY: (Laughs) I know. It’s quite a bit. It’s really impressive how much is going on there. And actually, some really interesting comparisons. There used to be Google Video, which was a competitor to YouTube when YouTube was a a startup and they were doing all their work in C++ in Google Video. The people, 20 versus hundreds of engineers were writing Python and just running circles around Google Video. Google solved it by buying YouTube. Pretty interesting history.

00:00 HUNNER: I never knew that. I knew they acquired YouTube while having video. I didn’t know that Python was a deciding factor there.

00:00 KENNEDY: Yeah, it was one of the deciding factors there. This small team outperformed Google, because Google is in C++ for their video service.

00:00 So, speaking of large services, Facebook. They use this mostly on the infrastructure side but Facebook is really interesting in that they did a really interesting thing moving from Python 2 to Python 3 as changing the engineering culture. Jason Fried did this really awesome talk on that and the whole steps in which they took to make this happen are pretty amazing. Did you see that talk?

00:00 HUNNER: I did. I think he’s giving a similar talk at PyCon actually, this year.

00:00 KENNEDY: Oh, that’d be awesome. I’m definitely going.

00:00 Closely related in Instagram because kind of like YouTube, Instagram was its own thing and now it’s engulfed in to Facebook. But they run the largest deployment of Django in a framework in the world. And they have 800 million active users. I think maybe the most notable thing was Instagram’s keynote at the 2017 PyCon with Lisa Guo and Hui Ding, who have the most inspiring talk how they migrated from old version Django to new version Django and Python 2 to Python 3 without branching and continuing to ship features.

00:00 HUNNER: That was amazing. I really like that talk because in the beginning they started with, ‘Here’s how we did it a high level.’ And towards the end they dove right into the code that they used and the difficulties they encountered. If you are afraid of encountering difficulties, look at the ones Instagram encountered because they’ve got a big code base.

00:00 KENNEDY: Yeah, they have millions of lines. It’s insane. But they way they did it was truly inspiring. I felt that that was actually a really clear playbook of, ‘Hey, we could do the same thing.’

00:00 HUNNER: Absolutely.

00:00 KENNEDY: We also have Spotify. They’re doing a bunch of stuff with Python, primarily as backend services and data analysis. Quora. I think Quora is the best Q and A place on the Internet. It always seems to have super thoughtful questions and answers.

00:00 HUNNER: I don’t use Quora much because whenever I go they ask me to sign in. I mostly go to the other ones for that reason but I have actually found a lot of useful stuff on Quora before.

00:00 KENNEDY: Yeah, I’m surprised how much time and energy people are willing to put into answers there.

00:00 HUNNER: Absolutely.

00:00 KENNEDY: It’s interesting because that runs on Python, of course, and they were thinking about maybe C#, maybe Java, maybe Python, maybe C++. In the end they ended up doing kind of what Google did, Python where they can and C++ where they must.

00:00 Netflix, obviously Netflix is insane. In the evenings they account for one third of the bandwidth in the United States.

00:00 HUNNER: Wow.

00:00 KENNEDY: Yeah, and they run all their infrastructure and management operations and stuff over Python. Actually, I talked to those guys over on Talk Python, Episode #16, way back in the day. Really interesting stuff they’re doing.

00:00 Dropbox. When I think of the center of the universe of Python, I think Dropbox. What do you think?

00:00 HUNNER: Why’s that?

00:00 KENNEDY: Well, Guido works there.

00:00 HUNNER: That’s a big reason.

00:00 KENNEDY: It seems to be one of the companies that is this major, major technology companies that is all in on Python. Even their little widget thing that ships as a client app that is in your tray or your menu bar, that’s even Python.

00:00 HUNNER: Yeah, they definitely got a ton of deployed Python code on desktop computers that they don’t have control over, which is pretty amazing. I think they also have some Open Source packages up on GitHub that are Python.

00:00 KENNEDY: Yeah, I think they do have a lot of interesting stuff. They’ve had some stuff that they put out that’s kind of left to go as experiments. Very interesting.

00:00 So, the last one I’ll leave you with is Reddit. They have half a billion visitors a month. They had, let’s see, it was the fourth most visited site in the United States and seventh in the world. In 2015, they said they had 82 billion page views, which is kind of insane. And of course, that’s all driven on Python, SQLAlchemy, all sorts of goodness.

00:00 HUNNER: That’s pretty incredible.

00:00 KENNEDY: Yeah, awesome. Those are the companies that I wanted to highlight out of that article. I just thought these are really great examples. If people are saying, ‘Oh, we have a hundred thousand users. We can’t use Python, it’s not going to scale.’ Well, that’s probably not the real problem. It’s probably some architecture or infrastructure thing that is going to make this work easily.

00:00 HUNNER: For sure.

00:00 KENNEDY: Yeah, nice. Do you think we should stop writing classes, Trey?

00:00 HUNNER: Maybe. I don’t know of we should stop writing classes entirely but one of my favorite PyCon talks, I tend to tell people it’s my favorite because I recommend it so much is, “Stop Writing Classes” by Jack Diederich. This is from PyCon 2012, so it’s kind of an old talk but it’s definitely the days of Python 2. But all the code in it, all the advice is still applicable to today.

00:00 KENNEDY: That’s really awesome. So, give us the main takeaway from it.

00:00 HUNNER: One of the biggest bits of advice in it that I tend to tell people is if you have a class that only has two methods, one of those is your _init_ method and the other one is some other method, especially if it’s called call, you probably need a function instead of a class. So, this is something I recommend a lot to folks new to Python from Java and C++. I teach a lot of folks who are moving programming languages, and you know, Python is different from the other languages.

00:00 KENNEDY: Especially if what you’re going to have is effectively a static class, right? There’s just one copy of the variables. It’s very likely that you could get away with – you’re not doing any inheritance, you’re not doing any of that kind of stuff – you probably could just have a set of functions and module-level variables and you get, effectively, the same output.

00:00 HUNNER: Absolutely and that’s something I see a lot in code. On the flip side of things, one of the great things about this talk is it doesn’t just show you when not to write classes, it does show you a couple examples of when classes are going to be handy in Python.

00:00 KENNEDY: Yeah, that’s cool. I’m a big fan of object-oriented programming. It fades in and out. I still think it’s a great way to model stuff but when you’re creating more than one of the things, right?

00:00 HUNNER: Absolutely, it’s when you need to make more than one of them, when you need to use it more than once, that’s the time where classes can be abused.

00:00 KENNEDY: Yeah, absolutely.

00:00 HUNNER: So, what’s up with PyPi, Michael?

00:00 KENNEDY: I’m going to tell you about PyPi and I’m super excited about it, but before I do I want to tell you about DigitalOcean. So, there’s this thing coming up where I need a whole lot more power for my web server because there’s some big promotion and it could end up overwhelming my servers. So far, they’ve never had an issue. When you log in, it’s like three or four percent CPU. One thing that’s really cool about DigitalOcean is if you have one of their servers, you can go over there and you can flip a switch and within just a couple minutes it will take it offline for a second, resize, re-whatever, however you change the infrastructure. Say I want four CPUs and eight gigs of RAM, change it to that. And you can do that for one day and then go back to the one gig, one CPU the next day, right? Spend five bucks and get some super huge server to handle some spike that you know is coming. Pretty cool. A lot of the stuff that I do on the web is run through DigitalOcean, so I would definitely recommend them. Check them out at do.co/Python. They help make this show possible.

00:00 So, another thing that I think doing really awesome stuff on the web finally is, the new PyPi. Do you remember back when there was PyPi.python.org/PyPi, because it’s better if it ends up on both ends of the URL. That’s the original grey that’s been around forever, what the PyPi has been. Then for awhile there was PyPi.io, which was like the new one. Then there’s a PyPi.org which they redirected to, I think that’s the same site.

00:00 HUNNER: It’s getting confusing here.

00:00 KENNEDY: I know. So, there’s been almost a big problem with the old PyPi in that it was based on just super custom, hand-rolled socket processing and the entire PyPi was running out of a handful, two, three Python files thousands of lines long. Purely custom socket code, not based on Flask or Pyramid or any of those things. It’s just like, ‘Yeah, we’re going to start at the lowest level. I think it was more or less a prototype that just became what PyPi was for many years. And that was actually a big problem because people would come on, ‘Hey, I want to add a few ideas. Can I just plug that in as an extension?’ Or, ‘Oh my God, what is that? I’m not going to contribute to this project. I’m outta here.’

00:00 And so, finally, Donald Stuft and crew have released PyPi.org as an official thing. It used to have that red bar up like, ‘Warning, this is a testing area.’ There’s no longer a red bar. You go there and it’s properly a website.

00:00 HUNNER: I’m going there right now to make sure.

00:00 KENNEDY: Oh, wait, it’s back again. It was gone earlier.

00:00 HUNNER: There is a red bar, I think you can close it. However, I love that this is official. It’s coming up here, which is super exciting.

00:00 KENNEDY: Yeah, the big news is not the website itself, although that’s sort of the portal to it. The big news is when you pip install a thing, it now goes through PyPi.org.

00:00 HUNNER: That’s really exciting. So, this actually is not only the future, it’s the

00:00 presence of using pip with Python.

00:00 KENNEDY: Yes, that is where we are. There’s a really cool tweet – I’m sorry, I forget the guy’s name – it has graphs of the analytics of number of requests per second, errors per second, memory usage and latency; all that for the new PyPi.org handling the pip install requests. So, I’m really glad to see that. I guess I did just close that X and it stayed closed, you’re right. I opened it in an incognito window and it’s still there. But it’s almost gone.

00:00 HUNNER: It’s good that they at least cookie you.

00:00 KENNEDY: (Laughs) That’s right, it is good. So, one of the biggest challenges, you open this whole section with getting started and converting from Python 2 to Python 3, I would say the biggest thing that people complain about saying, ‘This is really hard to get this transition over’ is the old strings, byte strings, Unicode bytes array madness around that, right?

00:00 HUNNER: Yeah, absolutely. This is one of the biggest challenges is figuring out what is text and what is data and separating those because Python 2 didn’t make that distinction. Python 3 forces you to make that distinction.

00:00 KENNEDY: So, there’s a nice talk by Ned Batchelder that you pulled up for us.

00:00 HUNNER: Yeah, absolutely. It’s called, “Pragmatic Unicode: Or How Do I Stop the Pain?” It’s counter intuitively, this is from PyCon 2012 and it is mostly Python 2 code, so if you’re switching to Python 3, it might seem odd to watch this talk. The reason this is still relevant today is that Unicode hasn’t changed. The default way we see Unicode in Python 2 versus 3 has changed. There’s a big analogy in the is talk that I really like, a metaphor that Ned uses, the Unicode sandwich; I’ll let you go watch the talk to figure out what that is.

00:00 This really helps shape your mental model of how text works versus how bytes work in the Python interpreter. I think this is really important, especially if you’re not familiar with Unicode and you’re going to switch to Python 3. Or if you’re going to start using Unicode, you’ve got to understand a little bit about how it works.

00:00 KENNEDY: Yeah, this is really interesting. Unicode is definitely one of those tricky things. And I think growing up in the United States definitely, I felt like, put me at a disadvantage of appreciating Unicode, you know? It’s like, ‘Antsy, what’s wrong with Antsy?’ I don’t know. It seems fine.

00:00 HUNNER: Yeah, absolutely. I feel like the big thing that’s making the change here for a lot of people is emoji, which is being used even by people in English-speaking countries, so the rest of the world is finally going, ‘Yay. Everyone cares about Unicode now, all because of these emoji characters.’ It’s sort of sad, in a way, but also sort of good because it means we’re actually caring about this universally.

00:00 KENNEDY: It’s really interesting and it’s a bit of a sad commentary on the world that that’s what pushes us forward. At the same time, it is good that it’s being pushed forward, right?

00:00 HUNNER: Absolutely, yeah.

00:00 KENNEDY: Nice. So, the last one that I want to cover really has to do with thinking about alternate implementations or run times of Python. It’s easy to say, ‘Well, there’s CPython and there’s just Python. I type Python and it runs.’ But there are actually many different run times. I say run times because some of the times are called interpreters, but only some of them interpret. So, we have IronPython, Python.net, SHython, Cython, a whole bunch of other ones that are more niche. I’d say probably the most popular alternative one is pypy. Have you done anything with this?

00:00 HUNNER: I have not done anything it. Well, I played with pypy, in one of those, ‘Maybe I’ll apt-get install and see what happens.’

00:00 KENNEDY: Exactly. So, pypy is a JIT-compiling implementation of CPython. So, the way CPython works, even though it’s written in C, when you first hit your code it all gets compiled to bytecode. But then those bytecodes don’t get further compiled to machine instructions, they just get interpreted in this massive switch statement, while true loop. And it just looks at the thing and it interprets them as bytecodes at run time. But that’s not the same as compiling down to machine instructions, right? So, pypy surprisingly does the same thing that CPython does until it finds a slow spot and then it switches it to a JIT compiled version. It’s interesting.

00:00 HUNNER: Wow, that’s really clever.

00:00 KENNEDY: It just finds the slow spots and goes, ‘Did you call this a thousand times and it’s slow? Alright, we’re going to make that fast. We’ll fix you up.’ There’s a little bit of a slow start up. So, if this was like a script, it runs for a half second and it goes away, there might be no benefit, or less benefit. Maybe even a drawback, I don’t know. But if you use a web app that’s going to be up for 6 hours then maybe get recycled or it’s a game, say, or game that we’re running, then that stuff would get JIT-compiled and be sort of warmed up, if you will.

00:00 HUNNER: We’re talking about games specifically, right? Why would it be a game that this is nice for?

00:00 KENNEDY: One of the nice ways to make games is with pygame. And this guy René Dudfield, he was at the pypy Sprints in Switzerland in the Alps in some beautiful resort. I’m like, ‘Man, I need to go to a Sprint up here.’ They were working on it and one of the things they got running was they got the new pypy running pygame perfectly well. Apparently, there were issues with it before. And they said on some of the games they got up to 30 times speed up by just simply typing something different to start your game.

00:00 HUNNER: Wow, so they got a 30 times speed up, by instead of typing Python they typed, ‘Just run it in pypy.’ And that made it 30 times faster. That’s amazing.

00:00 KENNEDY: Yeah, it’s the same pygame, same code that runs on CPython and so on. So, pypy has a different way to implement C extensions and different APIs so this is really nice. It has new ways to do extensions for C or a similar language. The C py.exe is getting faster, which was important for the games to run quicker, and so on. Basically, people were saying it’s really nice to have one code base. You don’t have to try to split across these different run times, it just runs in both places. Apparently, there’s a few types of operations that are slower because transitioning into and out of pypy down into the C level and back is lower than CPython. BUt still, pretty cool.

00:00 HUNNER: It is, it’s very cool. Especially if you’re using pygame. This is exciting news.

00:00 KENNEDY: For sure. Who wouldn’t want their game to run… You think of how much people optimize video games to get a few frames per second here or there. This is massive. One example they gave is ray tracing. They had run out on Python 2.7. It took nine and a half minutes and they ran it on pypy and it took 18 seconds. So, that’s pretty good speed up.

00:00 HUNNER: That’s not just a pretty good speed up, that’s an incredible speed up. If my tests took ten minutes to run and then they went down to seconds, I would be pretty amazed.

00:00 KENNEDY: Yeah, that would be so awesome. So, if you’re playing with pygame, or even if you’re doing stuff on the web or whatever, chec pypy. It could make a huge difference. It’s good to see that project continue and grow.

00:00 That’s it for our items. You’ve got any extra news or anything you want to share with people?

00:00 HUNNER: Oo, it’s tough to decide what’s new. I’m going to conferences,finding new things on the Internet, making new things on the Internet. One thing I want to share is a thing I launched a couple months ago kind of quietly, I’ve been slowly testing it out with folks, called Python Morsels, where I send an exercise out every week. This is my way to serve the folks who are at my local study group and other people I meet on the Internet who want to learn Python. They don’t have a company that they’re working in that does Python that could bring me in for training, because I love teaching and I really want to serve individuals. So, folks could check that out, it’s a pretty fun service.

00:00 KENNEDY: Nice, that’s just pythonmorsels.com?

00:00 HUNNER: Yes, pythonmorsels.com. It redirects right now but one day in the future it will have its own domain that doesn’t redirect.

00:00 KENNEDY: (Laughs) It will be all grown up on the Internet. That’s awesome. I’ll put that in the show notes.

00:00 I just have one piece of news to share. I kind of talked about this last week but now it’s in full swing. I launched my #100 DaysofCode course with PyBites – unrelated to this podcast but similar name – guys, Bob Belderbos and Julian Sequeira. I’m really happy to have this course out. It’s an 18 -hour course that guides people day by day through 100 days of Python. It’s pretty epic.

00:00 HUNNER: Yeah, I scrolled through the page for it. It is a long page. There’s a lot in that course. It’s super exciting to see that.

00:00 KENNEDY: It gave us a chance to play with stuff that’s really pretty far out there and really fun, like building GUIs in Python, different ways to consume APIs. Lots of really interesting stuff that we got to play with. If people want to check it out, talkpython.fm/100days. We’ll put that in the show notes as well.

00:00 Alright, Trey, thank you so much for jumping in and being a co-host for this week. It was fun to share these items with you.

00:00 HUNNER: Thanks for having me. This is super exciting. I hope to be back one day.

00:00 KENNEDY: We’ll have you back for sure. Thanks, bye.

00:00 Thanks you for listening to Python Bytes. Follow the show on Twitter via @pythonbytes and get the full show notes at pythonbytes.fm. If you have a news item that you want featured, just visit pythonbytes.fm and send it our way. We’re always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page