Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


« Return to show page

Transcript for Episode #50:
Bundling , shipping, and protecting Python applications

Recorded on Wednesday, Nov 1, 2017.

00:00 Michael KENNEDY: Welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode #50, recorded November 1st, 2017. I’m Michael Kennedy.

00:00 Brian OKKEN: And I’m Brian Okken.

00:00 KENNEDY: And, like always, we have some pretty awesome stuff lined up for you, but I want to say thanks to DigitalOcean.

00:00 So, DigitalOcean is sponsoring this episode and they’re going to share this new thing they’ve launched with you guys called Spaces. It’s like S3 but way, way better. We’ll talk more about that later.

00:00 Right now, I want to be a little philosophical and think like a Pythonista.

00:00 OKKEN: Yeah, think like a Pythonista. We’re going to highlight a video from Luciano Ramalho. He did a talk at PyBay 2017 called, “Think Like a Pythonista.” With most of my programming, I don’t really create new types that much in Python. I don’t really think I’m creating new types. I’ll use named tuples or something. But even so, I take advantage of all the built-in types and that’s kind of what part of his talk is about. He walks through talking about creating another container type called a Deck of Cards. So, he’s got a cards-type and a deck type. The Deck is a container and it has to do things like indexing and shuffling and stuff. He walks through how to do that so you can use a built-in iterate for loops and accessors. It’s really kind of a fascinating look at how little code it takes to make something look like any other Python type using some of the dunders and stuff.

00:00 KENNEDY: Yeah, for sure. I think he did a super job with this. This comes from his book, Fluent Python, which is absolutely one of the best books once you know a little bit of the language to really get the zen of it. He’s really big into the Python data model, the dunder methods and all that kind of stuff, so I’m sure this is a really great way to see how that comes out in a custom type.

00:00 OKKEN: Yeah, it is. And also, I think it’s a good talk for somebody to watch, even if you don’t think you’re ever going to create a container type yourself, because you learn about a lot of the things that your built-in types do for you that you don’t even really think about. There’s quite a bit of information in there that’s worth watching, even if you don’t feel like doing it. Plus, he does most of the talk with a Jupyter notebook and even changes some of the code and fixes a bug and does monkeypatching with live code. It’s pretty cool.

00:00 KENNEDY: Yeah, that’s pretty cool. I think definitely this concept of Jupyter notebooks, the whole scientific computing stuff is bleeding into all sorts of presentations. I think it’s great.

00:00 OKKEN: Near the end of the talk, he does an example of why operator overloading is awesome and he shows a little Numpy, a little tiny script that can graph the trajectory of a launched item. Like throwing a ball or something. It’s so clean and such little code because the Numpy library is allowed to take their data types and overload operators that multiplication and addition work on this large objects.

00:00 KENNEDY: Yeah, that’s awesome.

00:00 OKKEN: It’s pretty great.

00:00 KENNEDY: Luciano definitely has some special sauce that he sprinkles on what he does. It makes it really smooth. Check that one out, it’s a great one. That’s a live presentation basically derived from his book.

00:00 OKKEN: I’ll link to this of course, but there’s also a link to the rest of the PyBay 2017 playlist.

00:00 KENNEDY: Yeah, that’s cool. For sure.

00:00 So, we talked about how neither of us are really gamers last time, right?

00:00 OKKEN: Right.

00:00 KENNEDY: But one of the really interesting possibilities – Python is probably the biggest language for machine learning and AI – one of the big possibilities would be, what if you could take the AI you’re trying to teach about a world and immerse it into a game and just make it be in the game. So, if you wanted to have a self-driving car, could you stick it into a car simulator game and make it race around and try to win? Or you want to have a puzzle solving one, stick it in the middle of a Go game and make it try to play Go.

00:00 The thing I want to feature this week is something called Serpent.AI and that’s both the name of the product and the website. It’s a Game Agent Framework. So, this is pretty insane. What you can do is you can take any video game and you can plug in AI and Bot programming with Python. What do you think about that?

00:00 OKKEN: Wow. I think that looks fun.

00:00 KENNEDY: (Laughs) I know it looks super fun, oh my gosh. And the guy who created it – I’m sorry, I couldn’t find his name on here – he runs a Twitch channel. You can drop in on his Twitch channel, which is linked to Serpent.AI. I actually pulled up one particular video that he recorded on Twitch, which is like streaming gaming stuff. He’ll do like six-hour coding sessions where he’s like, ‘We’re going to take this game and plug an AI into it and solve it in real time.’ It’s pretty intense.

00:00 OKKEN: Wow. That’s cool.

00:00 KENNEDY: If you want to spend a day watching somebody plug in an AI into a game, you can do it.

00:00 OKKEN: That sounds neat. I do know that Twitch is being used by a lot of game people watching gamers and stuff. Do you know if it’s catching on much with live coding examples and stuff?

00:00 KENNEDY: I was thinking about that as I was watching. He does so live coding. He’s got PyCharm and Jupyter and the game going and console stuff in the video that I linked to. I honestly don’t know but I feel like there should be if people aren’t doing it.

00:00 OKKEN: There probably is, probably just not in my circles.

00:00 KENNEDY: Yeah, exactly. We don’t do enough gaming, right?

00:00 So, some of the things they talk about in this, it says there’s really no conventions based on this, you plug in whatever you want. So, if you want to use the latest reinforcement deep learning algorithm from Google, try that. If you want to use computer vision, we talked about one recently, LUMINOTH, you could plug that in there for example, to do image processing. Even if you wanted to randomly press buttons, you could try that as well.

00:00 OKKEN: So, I’m guessing this game has to be something that you can get at with Python library or something. It can’t play my XBox, for instance.

00:00 KENNEDY: No, I think it’s got to run on a computer. I think it simulates arrow keys and stuff like that. I’m not entirely sure how it determines whether it wins because you’ve got to give positive or negative feedback to the deep learning models. But something’s happening in there. It looks pretty cool.

00:00 OKKEN: Maybe you just have to hang out with a microphone and say, ‘Good job.’

00:00 KENNEDY: You have to pat it on the back. ‘Well done, AI. Well done.’ (Laughs)

00:00 So, you’ve been working with cookiecutter lately, right?

00:00 OKKEN: I have. Most of my work has been making some plugins for work but I’m utilizing the pytest cookiecutter plugin. One of the options was to generate some project documentation in MkDocs and I’ve never really used them before. I don’t if it’s pronounced ‘make docs,’ that’s how I’ve been pronouncing it.

00:00 KENNEDY: I think there’s a little bit of debate, but I think that totally works.

00:00 OKKEN: Okay. It’s MkDocs and I think it stand for Markdown docs. I know about Sphinx and I know that a lot of the Python documentation is with restructured text but I just don’t really like restructured texts. I just wanted to highlight MkDocs. I thought it was neat and I’ve been documenting some of my projects with it. Especially with cookiecutter it just pops it in there, at least this pytest login cookiecutter pops it in and created a yaml file. It’s kind of neat because you have this configuration of what your theme is and all that, but your writing is just a directory with a bunch of Markdown files.

00:00 KENNEDY: Markdown is a really great invention. I’m really glad it exists.

00:00 OKKEN: The one thing I was worried about is I do kind of want to keep some of the library API documentation with the code, not somewhere else. And I do want to document that somehow. I did stumble across one of the friends of our show, Christian Medina; he wrote a Medium article called, “How to Write Your Own Python Document Generator.” Essentially, it’s using MkDocs also but walking through your code and pulling out the doc strings. So, I’m going to give that a try next week and see how that goes.

00:00 KENNEDY: You’ll have to give us a report about how it came out.

00:00 OKKEN: Yeah.

00:00 KENNEDY: Awesome.

00:00 This audio file that you’re listening to right now came to you through DigitalOcean Spaces and, like I said in the beginning, they’re sponsoring this episode, as they are a number of them. You may know of Amazon S3 as a place you can put your files both for people to consume and browsers and stuff, but also for your apps to integrate with. Well, DigitalOcean decided to take it to another level with Spaces and create something similar but with a much better business model. You pay $5 a month, you get 250 gigs of storage, a terabyte of outbound bandwidth, which by the way is$93 just there on its own in S3. So, $5, that’s a super good deal and inbound data is always free. It’s way more predictable. You can save tons of money and it has the same API as S3 so all your tools and stuff still work as well. I definitely recommend it. I’m using it more and more everyday and, like I said, using it to deliver all the audio for this podcast.

00:00 OKKEN: That’s really cool and, like you said, they’re sponsoring this episode but they also save us money on every episode.

00:00 KENNEDY: That’s right, they’re kind of sponsoring everyone a little bit, in a sense, saving us some good money. Check them out at do.co/Python.

00:00 Now, you and I, I think we were kind of breathing the same air or something was I the air this week here in Portland because I chose an item that has to do with taking your Python applications and packaging them up for reuse in one way, and you chose the very next item, packaging up your Python applications in a slightly different way. Let’s do two interesting takes on this.

00:00 OKKEN: Neat.

00:00 KENNEDY: So, there’s a couple programs or libraries that you can use to build proper applications out of your Python code. I feel like deployment to end users is one of the things that’s really tricky for Python. This is one of its weak points. You’ve got a bunch of scripts, you’ve got to send those out, they have dependencies. You can’t get a regular user a Python thing and say, ‘Yeah, run this pip install this to configure it.’ That’s not going to work. You have to go, ‘Here’s an .exe, here’s a .app.’ And so on.

00:00 So, you can use cx_freeze, there’s a few others. PyDo app, PyDo.exe. But one that’s really nice is called PyInstaller. They just released version 3.3. This works on all the operating systems: Windows, Linux, Mac, even FreeBSD, Solaris, AIX. A bunch of them. The idea is that its primary goal is to do what I described, but also be compatible with third party packages out of the box. A lot of the weaknesses of the other ones that I’ve mentioned that run into trouble trying to use Matplotlib or Numpy or PyQt for example, which is super painful. It will take that and package all those libraries up for you. And they just released the 3.3 version which has Python 3.6 support.

00:00 OKKEN: Oh, that’s cool.

00:00 KENNEDY: Do you see yourself maybe using this?

00:00 OKKEN: I definitely could see using this. Especially, like you said, if you’re sharing an application with somebody that isn’t a developer, that’s good.

00:00 KENNEDY: If you’re going to ship some end product and you want to run it in Python, that’s pretty awesome. And you can use the --one-- file option and --noconsole and it could just be a GUI and you wouldn’t really know that it’s not built-in C++ or .NET if you run it on Windows or Cocoa on Mac. Things like that. It’s really nice.

00:00 OKKEN: Nice.

00:00 KENNEDY: So, what’s your take on this?

00:00 OKKEN: On this one, I think I definitely have to try it. But the item that I brought up and I was thinking about is PEX and it says it’s a library tool for generating Python-executable files, PEX files. The files are self-contained executables, but there’s executable virtual environments.

00:00 KENNEDY: Right, whereas the one I was thinking of kind of packages up all the Python things you need to run an .exe, this like, ‘Can we just package the virtual environment and somehow teach it to run?’ Right?

00:00 OKKEN: I’d have to look but I’m almost positive that it doesn’t package the Python with it. So, if you have everybody having Python installed, but you want to share Python programs, this might be a really cool thing. I just got introduced to it the other day, but it looks like it came from Twitter and it used to be part of the Twitter comments package, which looks like there’s some cool stuff in there. I’m going to have to check that out also.

00:00 KENNEDY: Yeah. When I was checking out the video that you’re about to talk about, I was like, ‘Woah, this Twitter comment thing looks awesome. There’s way more to be learning here.’

00:00 OKKEN: Yeah, but it’s now a stand-alone thing. It’s great. We linked to a video called, “WTF is PEX?” It's by Brian Wickman. It’s a little 16-minute video where he just pretty much talks about what it is. The insight is Python can import from a zip file.

00:00 KENNEDY: Even if you corrupt the zip file. (Laughs)

00:00 OKKEN: Even if you corrupt it. So, they corrupt the zip file by putting a main at the end and a header on there to tell the rest of the operating system that it’s a Python program. Your environment just tries to run it like Python and it imports all of its dependencies from itself. I kind of got lost as to exactly how that works, but it looks fun also.

00:00 KENNEDY: Yeah, it definitely looks cool. So, if your goal is more to share executable things with developers, it seems like PEX is really awesome. Maybe even with infrastructure. Like, ‘I want to deploy this thing to the server and I just want to drop it in and have it go and not have to deal with the dependencies and virtual environments and all that stuff.’ It looks super cool and I get the sense that they’re using this in production at Twitter, even though I don’t remember them exactly saying that.

00:00 OKKEN: Or using it just even for tools. Some of the examples were, you’ve got a little project, it’s got a virtual environment, or it’s got some requirements for it and some decencies. You just kind of make that into an executable and then you can just drop the whole thing into a bin directory or something and not really worry about it.

00:00 KENNEDY: That’s really nice.

00:00 OKKEN: It’s pretty great.

00:00 KENNEDY: Super cool. Alright, I want to keep with this theme of, ‘We’re going to package up some code and run it.’ Even with either of these, it’s great that PEX can zip up your source files, or that PyInstaller can zip up your source files, but you’re still sending your source code straight to people. And that can be bad, potentially. Maybe you’ve got some secret algorithm, maybe you’ve got licensing that you don’t want to just be able to like, ‘Well, will just come without the license check and it will be good.’ Something like that.

00:00 So, there’s this interesting article that came by this week called, “Using Cython to Protect a Python Codebase.” Cython is a fully to machine instruction compiled variant of Python. We’ve got Python, we’ve got PyPy, we’ve got Cython, we’ve got IronPython, all these variations. Cython is one that came the SageMath guys way, way back in the day, to compile this to faster machine instruction. It’s fully compiled code out of .py files, which is pretty interesting. That makes it way harder to mess with.

00:00 This article walks you through building a Python package and all the little extra tricks and tweaks that you’ve got to do so that you can compile your Python code with Cython and then ship it as a wheel, which is the current standard for pip installing a thing. You can create a binary distributable wheel that has zero source code that’s compiled for a particular architecture.

00:00 OKKEN: That’s very interesting.

00:00 KENNEDY: Yeah, and you can combine this with PyInstaller and make it an .exe and off you go.

00:00 OKKEN: Since the algorithm is open, it’s reversible then?

00:00 KENNEDY: Well, it’s like compiling C code or C++ or something. It is technically reversible and you can get it back, but it's way, way harder. They do say there are certain ways to hack the runtime. If you import it, you might be able to get it to give up some of its secrets but it’s definitely better than actually shipping the source code.

00:00 OKKEN: This reminds me of obfuscators. I can’t remember if that was in C# or Java. Do you remember obfuscators?

00:00 KENNEDY: Yes, I totally remember this. It was even such a weird word to say when I first learned about it. I remember that actually. I think it first came out in Java to sort of scramble up bytecode and .NET definitely has it as well, because they have IL, which both Java bytecode and .NET IL are super reversible. It’s almost exactly what you wrote gets pulled back, other than maybe where you put your curly braces – on the new line or the if line – things like that. It’s really reversible, so I think this protecting with Cython actually would be even better than obfuscating. Although, I haven’t looked at all the different angles. It’s really an interesting idea though. And it doesn’t have to be everything, right? You could take your core algorithm that you want to protect or whatever, Cythonize it and just import it into the rest of your code.

00:00 OKKEN: That’s a really great idea. For instance, a bunch of this stuff like reading config files or your user interface or something, you might not care about that part.

00:00 KENNEDY: Right, exactly.

00:00 OKKEN: They can be reverse-engineered pretty easily by looking at it. But your core algorithms and the real reason why people are buying your product, that’s not a bad idea. This is cool.

00:00 KENNEDY: Yep. I’ve never tried it but it’s a pretty cool idea and it’s a step by step sort of thing, and I thought people might find that useful out there.

00:00 OKKEN: I think so.

00:00 KENNEDY: Awesome. Well, that’s our six items.

00:00 OKKEN: Awesome. It went really fast.

00:00 KENNEDY: Yeah, like always.

00:00 OKKEN: So, I don’t have anything extra to add. Do you have anything you’d like to talk about?

00:00 KENNEDY: I have a little bit of a survey I’d like people to consider filling out, really only if you’re like super passionate about this. One of the projects I’m working on is, I want to create a course to make working with Python on Windows better and delightful, not frustrating and challenging. So, I’m trying to gather a bunch of information of what people find hard about working with Python on Windows. I’ve got a bunch of people to fill out a form, like 200 people filled this out already, but I put a link here to a Google Docs form. If you work with Python on Windows, you’ve had struggles with it and there’s some sort of feedback you can give that can help me maybe create a class or something. Some sort of class to help people be more successful with Python on Windows, fill out the form on the bottom of the show notes. That’d be awesome.

00:00 OKKEN: Oh, great. I’ll definitely fill this out and pass it around because there’s a ton of people that I work with that are kind of new to Python and new to Python on Windows.

00:00 KENNEDY: That’d be perfect, that’d be awesome. A lot of people are like, ‘Just use Linux or whatever’ but when you show up at a tutorial or to give a training class to a corporate group and they all have their corporate Windows 7 laptops, it doesn’t matter whether you recommend Linux or Mac OS, they have to make it work on their environments. So, my goal is to try to make that a smoother process.

00:00 OKKEN: Great. Cool.

00:00 KENNEDY: Awesome. Thanks, again for meeting up and pulling all these ideas together. It’s always fun.

00:00 OKKEN: Yeah, thank you. Talk to you next week.

00:00 KENNEDY: Thank you for listening to Python Bytes. Follow the show on Twitter via @pythonbytes.com. And get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We’re always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page