« Return to show page
Transcript for Episode #9:
Walking with async coroutines, diving deep into requests, and a universe of options (for AIs)
Michael KENNEDY: This is Python Bytes, Python headlines and news delivered directly to your earbuds. It’s Episode #9, recorded Tuesday, January 17th, 2017.
This is one of your hosts, Michael Kennedy. I’m here with my other co-host, Brian Okken. Hey, Brian. How’s it going, man?
Brian OKKEN: It’s going very good. How are you?
KENNEDY: I’m doing well. I always look forward to our weekly chat about the cool stuff that goes on in the Python ecosystem.
I see that you found something quite cool for our first item.
OKKEN: Actually, it’s a series of things. The link that we have is to Python Asynchronous I/O Walkthrough. Here’s the story. In July, there was an open source book published called 500 Lines or Less: The Architecture of Open Source Applications. They do various open source free books, which is pretty cool. The 500 Lines or Less had one chapter in it, from A. Jesse Jiryu Davis and Guido van Rossum called, “A Web Crawler with Asyncio Coroutines.” It’s a pretty cool chapter about explaining asynchronous networking, showing how non-blocking sockets work and how Python 3 coroutines improve asynchronous network programs.
KENNEDY: Yeah, that’s really cool. I think that book is amazing. It’s not just this particular chapter, there’s many chapters. It’s also just not Python, but a lot of pieces are Python. Basically, it’s ‘Can you show me something amazing that I thought was really hard but can be explained or written in 500 lines or less? And then discuss the architecture.’ There’s like 20 projects or something; it’s very cool.
OKKEN: Yeah, the series is pretty neat. This is 500 Lines or Less. There’s previous books that are about architecture and about performance issues; it’s a neat thing they’re doing.
This particular chapter for me, since I’m kind of new to the asynchronous world, it still was a little bit heavy to get into. So, Phillip Guo, I think he interviewed you once?
KENNEDY: Yeah, he’s been on my show, on Talk Python to Me, Episode #22 and I also was on his show. He started a podcast recently; it’s got 25 episodes or something. I’ll be sure to link to that. That’s a really cool one as well. He interviews people just generally in the programming space.
OKKEN: So, what he did is pretty cool. It’s just walking through this article and taking about it and showing it on video. And then going through and doing some code examples. It’s about 90-minutes total to get through this but he split it up into 8 parts, so each one is not that hard to get through. It’s a pretty cool idea and I really appreciate it. I think it’s a good way to learn; to see somebody doing it, to hear him, to use all your senses.
KENNEDY: really like that. I think this is great work. Nice work, Phillip. Way to go. He’s done stuff like that before. The reason I had him on Episode #22 of Talk Python was, he did this thing called “CPython: a 10-hour Code Walk,” which was an exploration through the CPython internals, which I found really enlightening. I talked to Jesse about his article on Episode #69, as well. We talked about writing as a developer. He’s got some really cool thinking around design patterns on technical writing and stuff. This is really cool. I think seeing a problem unfold before your eyes in code is much different than seeing the finished product discussed. I find I learn really well that way. I’m happy this is here. I haven’t had a chance to go through it but I’m looking forward to it.
OKKEN: Also, both Jesse and Phillip are doing some cool things to help the entire community which I appreciate. Jesse’s been supportive of a lot of people including me.
KENNEDY: Yeah, absolutely. Jesse’s doing great stuff.
Do you want to think about the future of Python a little bit?
OKKEN: I thought I was thinking about the future of Python, with switching to 3.6 recently. What do you think the future has in store?
KENNEDY: We were living right on the edge of the front wave. There’s an article I thought was pretty interesting. I don’t necessarily agree with everything but I felt that it was certainly thought provoking. A guy named Serdar Yegulalp wrote an article and it’s called, “Four Likely Future Twists for Python” and I thought that was pretty cool. So, let’s go through the 4 twists.
The first one does not make me happy, but it’s certainly something that I can see coming into existence as a thing, so we’ll see. The first on that they put out there is that Python 2.x may live on.
OKKEN: Yeah, I actually can see that this might happen. It’s an open source project so there’s nobody stopping you from doing it.
KENNEDY: The thing that’s going to make a difference is what did the core developers do? Right now, the core developers are all focused on making sure that we make Python 3 as great as possible and they’re putting their energy into it and so on. I think Python 3 is certainly the right place for people to be, but I can see large banks… You think of places like Bank of America, they’ve got-I can’t remember the exact stats. I was talking to Mamud Hashemi from one of the Enterprise Python episodes I did and it’s something like, the number of lines of Python 2.6 or 2.5 code, is in the millions. Like 5 million lines of code or something crazy like that. I can see a company like that forking Python and just making some kind of ongoing effort.
OKKEN: Supporting your own Python, that just doesn’t sound fun.
KENNEDY: It sounds like a really bad idea. I would much rather people just upgrade and we can all work together and live together and do great stuff.
Here’s one of the predictions. There will probably be someone who forks Python 2.x and tries to keep it going. Whether that turns out to be beneficial or trading one set of problems for a worse set of problems, we’ll see.
The other one was one I had no visibility into; I didn’t see this coming. I read the title and it said, Requirements.txt may be replaced with something better.”
What? Okay, well, maybe.
It turns out the guys over at the Python Packaging Authority, that’s PyPI and Pip and all those projects. There’s an organization, a group of people that actually work on that. They’re working on this thing called Pipfile, and it’s a proposed new file format that’s supposed to replace Requirements.txt.
OKKEN: That’s interesting?
KENNEDY: Isn’t? The idea is that Pipfile is going to be better or superior to Requirements in that is has a TOML syntax, if I recall correctly. It’s a little like YAML. There’s one Pipfile that can have multiple configurations in it, so you don’t have a dev-requirements.txt, a test-requirements.txt, a prod-requirements.txt and so on. It’s cool. You can do a Pipfile.freeze on it and it will generate something that is a much richer document instead of just a .txt file with a package name, like a version. There’s a lot of configuration and stuff in there. For example, one of the things you can do in these Pipfiles is you can actually set the source of the packages, right in your Requirements file. So, you can say, ‘I want to use this version of Python. I want to use it against this source like pypi.org.’ Or you can create your own dev-pi server thing and run it internally and that would be part of your Requirements file. So, lots of cool enhancements there. I’m interested to see where it goes but it’s out there and you can actually check it out.
OKKEN: Yeah, I’d like to make sure it goes along the lines with other packaging issues like a setup file and stuff. Hopefully, we have one thing that replaces everything.
KENNEDY: Yeah, definitely.
Okay, so #3 is Python could get more enterprise editions. So, we’ve got things like Anaconda, Enthought, and Intel actually has their own Python implementation that is sort of a repackaging and tweaking of the Anaconda distribution. Just talking about how their maybe more variations of Python distributions focused on enterprise needs. Maybe. I certainly see it happening in data science, I could see it happening elsewhere.
OKKEN: I could see it, especially since everybody that’s relying on Anaconda has to wait for them to release something when they’re waiting on a new version.
KENNEDY: And the last one is Python Software Repository, so PyPI basically, could get enterprise-friendly package management. Cristian Medina wrote a really interesting article as a follow-up to my Episode #84 on the problems with PyPI and open source infrastructure finding this stuff. He put together a pretty interesting article talking about something called the Python Trusted Registry or Trusted Index, as a way to both fund PyPI and make it more enterprise-friendly. I’m actually thinking about this problem as well and I’m probably going to release something, like an article and a proof of concept, but I’m not ready to talk about it yet.
OKKEN: Okay, sounds interesting. Well, that was 4 predictions. Are we done with the future?
KENNEDY: That’s the future. Let’s go back to the present.
OKKEN: So, in the present, Numpy is used for a lot of data science and really just science and engineering. A lot of people use Numpy. There’s a new book, an open source book, called From Python to Numpy and it’s from a French person, Nicholas Rougier. The book is pretty interesting. It makes the assumption that you already know Python and you’re an intermediate developer and you just need to get your understanding of Numpy to apply it to whatever problem you’re on. I haven’t read a ton if it but it looks really well formatted. I like the topics that are in there; it just sounds interesting.
KENNEDY: Yeah, it sounds really cool and if you’re looking for scientific computation and you’re thinking, ‘Am I going to shell out $2,000 for MATLAB?’ Well, maybe look at some of the Python tools. It’s definitely growing.
KENNEDY: You know, it’s cool for us to think about the future but what if machines could think about the future? I really feel like 2016/2017 is the year where AI (Artificial Intelligence) became not something that’s always 5 years in the future but actually something that’s starting to take shape in the present.
OKKEN: Yeah, and now it affects everybody with all of their speaking devices.
KENNEDY: Yeah, we could be mean and we could say, ‘Hey, Amazon thing. Order us a…’ (Laughing) But we won’t do that.
The item I want to talk about, #4 here, is something called Universe. It’s a way to take games basically, all sorts of games and make them available as ways for AIs to experience the world. So, here’s the deal. The thing they put together where you can build an AI, put it into a Docker image, give it to the [her] UNSURE [11:20] environment, and it will play literally thousands of video games. It can learn from playing the games. You can play Flash-based browser games, other types of games, there’s a whole bunch. Like I said, over a thousand environments. The way it plays, you don’t have to integrate with all these different games; your Docker image is presented the visual screen that the humans look at and then they send mouse and keyboard events to the games.
OKKEN: Weird. Okay.
KENNEDY: Basically, just play as if they were a person playing the game. So, they have the same experience of the game as you, but here’s all these environments in which you can train and test AIs. What do you think?
OKKEN: It sound like Tic-Tac-Toe in global thermal nuclear war to me but...
KENNEDY: “Dr. Falcon, would you like to play a game?” (Laughing)
OKKEN: If you have an AI that has learned a whole bunch of stuff about games, like, why?
KENNEDY: Why would you care about this, right? Well, maybe you’re working on a robot and you want to teach the robot to walk around and interact with the world, but you don’t have to robot finished. You could take the smarts of the robot and let it play Counterstrike. (Laughing)
Or something for, until it gets really at good at not bumping into stuff, making it through paths, then you could just plug that into your robot. This is me totally speculating on the spot. I don’t know.
OKKEN: Just don’t give your robot a gun.
KENNEDY: Exactly, because it’s going to be really good at it at that point. (Laughing)
So, another thing that I’ve been doing a lot with lately is working with APIs and HTTP services and stuff. And your next item is a pretty interesting deep dive into literally the most popular Python package, Request, which for those of you who don’t know, is downloaded 7 million times a month.
OKKEN: Yeah, it’s titled “Python Requests Deep Dive” and it came from Anthony Shaw. He put it on Medium.com. Anthony’s been… I don’t even know why I know Anthony, I’ve just know about him for a long time. My first thought is that it’s going to teach people how to use Request, but it’s not. It’s a really cool article.
What happened was, he was on a large project to convert from httplib, so he wasn’t even using url-lib 2.x or anything, httplib into Requests. This project was Apache Libcloud with how it was handling stuff like HTTP, HTTPS REST/JSON, REST/XML; a whole bunch of bizarre APIs. He lists over 80 client libraries to handle different cloud services. It had one connection class that did the encoding and decoding of everything and converted this all.
This article is about all the stuff he had to deal with, like dealing with sessions, authentication, how he tests it, doing prepared request streams; it’s just chock full of stuff. It’s not even that long of an article, so I think he did a good job.
KENNEDY: Yeah, that sounds really cool. I haven’t had a chance to read it but I definitely bookmarked it. Anthony Shaw is a great guy, he’s doing cool stuff. He talked about Libcloud on Podcast._init; I’m afraid I don’t remember the number, but you can listen to him there. That’s really cool; it’s a cool project. This sounds like a pretty serious real world upgrade from library to another as an example and I think there’s probably a lot of lessons people can get from this.
OKKEN: One of the neat things he has in here is he uses a tool called Requests-mock. I looked through the API for this and it really is pretty cool. My philosophy on mocks is to really only mock what you absolutely have to do. Not everything but some of your testing should be using mocks, when you’re dealing with outside resources. An HTTP request id clearly dealing with an outside resource. The API for this is very simple and very powerful. I’ve got to try it out; it looks neat.
KENNEDY: I hadn’t heard of Requests-mock, and that sounds super useful. You’re right, that’s one of the places, if you’re doing testing, you don’t want to have to set up the service and configure it in test mode or whatever.
OKKEN: Now, that was dealing with APIs on the internet. How about desktop applications?
KENNEDY: Yeah, so, let’s talk about Python in a desktop application. Certainly, you can build interesting desktop applications. Dropbox is a real-world example of the most widely-deployed Python desktop app in the world because that client thing that I see spinning right now in my notification bar, that is all Python. Python 2.7, I believe; they’re working to upgrade it to Python 3.x but right now it’s 2.7. That’s interesting.
You can use a lot of different technologies, but I kind of feel like there’s just something missing to make Python a first-class environment for building desktop applications. So, we’ve had TKinter for some time and that’s sort of built-in, but I always feel like that looks super out of place. Especially on Mac or Windows, it looks much more like it belongs in a Unix/Linux space, but even there it doesn’t look super fabulous.
OKKEN: It looks like the TK stuff that I was using with TCL when I was in college a long time ago.
KENNEDY: Yes, exactly. It totally works and you can do it and so on.
There’s a thing I want to highlight, a Reddit conversation; it’s not a link it’s just a back and forth on Reddit. “What’s the communities favorite for developing OSX applications with Python?” There’s a couple people that weighed in. By far, I think, PyQt came in as the most highly-voted up. Have you done anything with Qt?
OKKEN: I did a small project with it a long time ago.
KENNEDY: The thing I like about Qt is the stuff that comes out of it looks native. It’s based on C++ so it starts super-fast. It feels like an app that belongs where ever it’s running and I think that’s cool. The challenge with this is, you’ve got PyQt the package and you’ve got PySide and these have been sort of competing, not entirely compatible, ways of writing this code. I think PySide is the better licensing model, but if I recall correctly, it only supports Qt 4.x and Qt is on Qt 5.x. So, it kind of hard to deploy Qt on the various machines, you’ve got a heavyweight install and so on.
This should be getting better. I think Qt, the company that runs Qt is bringing Pyside back in and going to modernize it but that’s not until the end of this year, probably. So, one vote was for Qt; that was cool.
Another one was for TK. Simple voice dependencies but we already talked about that.
Kivy definitely for a vote. Kivy is not sort of desktop app-feeling, it’s more like games. I think it’s origins were in games, so it doesn’t really have a native feel, so that’s not that great.
OKKEN: I think some people are using it for kiosks and stuff.
KENNEDY: Yeah, and that’s not bad. I know it does great stuff on mobile as well.
There’s some debate on whether that was the right choice. There’s Toga, which is really cool. Toga from the BeeWare guys. But Toga doesn’t have support for all the APIs and stuff.
So, I kind of want to point this out and just say, to keep raising the issue. Like, ‘Hey, Python Community. It would be awesome if there was something that was really clearly fantastic for building apps.’ Or people say, ‘Of course you should use this. It’s as good as Swift for OSX app; it’s as good as WPF for Windows apps, or whatever.’ It feels native and it’s really all coherent and I just don’t see that existing yet. I think the BeeWare guys are the closest to it. It was an interesting conversation.
OKKEN: You’ve brought up before, I think you brought it up on Twitter, that it’d be cool if we had something kind of like Electron.
I don’t really want my app to say, ‘Do you want this app to act as a server?’ when I try to run it. This is not supposed to be a server app. What kind of weird thing have I got a hold of here? I want something that feels like it drops in and it’s basically indistinguishable from a native app.
Ah, we can dream. One good step in that direction is cx_Freeze. It lets you take Python code plus its dependencies in the Python implementation and everything and package it into a redistributable. That just released version 5.0.1 in January, so that’s cool to see that coming along. That’s one of the important building blocks, at least.
OKKEN: That’s cool.
KENNEDY: Someday I dream of a future where, I don’t know, there’s some whizzy UI framework with a visual designer for Python.
OKKEN: It’s interesting that I don’t really think about that that much. You spend so much time interacting with elements on a desktop even though we don’t develop them very often.
KENNEDY: Not to say it can’t be done, I just think it can go up a couple levels and it would make a really big difference in the adoption of Python.
KENNEDY: It’s already growing super well and if you could take over the desktop market with it as well than who knows where it would go.
OKKEN: Maybe you should do that, Michael. You can take it on.
KENNEDY: Yeah, I have so much time. (Laughing) That’s a great idea.
OKKEN: Okay, so if you don’t have time for that, what are you doing?
KENNEDY: I’m working on my classes. I’m actually working on some stuff that has these things that we were talking about in mind. It’s at the top of my mind, but I’m not ready to announce anything yet. Cool projects that I’m working on and I’m working on a special episode of Talk Python to Me. We’ve got Episode #100 coming up. I have a special guest but I’m not going to announce it until I release it, but that one’s going to be big.
OKKEN: I can’t believe it’s been 100 already.
KENNEDY: I know. Two years. Amazing.
OKKEN: Speaking of podcasts, I’ve got a couple coming up, hopefully pretty soon. They just need to be edited. I talked to some really great people. And then I was interviewed on a podcast called Reflection is a Service. That was really fun to talk with somebody else about test-driven development. I think I talked about the testing column that I think I made up on the fly while I was talking with you one day.
KENNEDY: We’re going to make that a thing.
OKKEN: Yeah, and I’m still writing, so that’s what I’m up to.
KENNEDY: Yeah, you got some good reviewers for your book, right?
OKKEN: Yeah, definitely. I big chunk of the PyTest Core Team has agreed to help out and make sure I don’t say something stupid, so I appreciate that.
KENNEDY: That’s really excellent. I feel like we’ve definitely given people a lot of information this week.
OKKEN: Definitely. It’s good.
KENNEDY: And pretty drama-free, that’s always nice.
OKKEN: Well, thanks a lot for talking to me today.
KENNEDY: You bet. It was great to share this with you, Brian, and I’ll catch up with you next week.
Thank you for listening to Python Bytes. Follow the show on Twitter via @pythonbytes and get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbyes.fm and send it our way. We’re always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.