Transcript #50: Bundling , shipping, and protecting Python applications

00:00 Hello, and welcome to Python Bytes, where we deliver Python news and headlines directly to

00:04 earbuds. This is episode 50, recorded November 1st, 2017. I'm Michael Kennedy.

00:09 And I'm Brian Okken.

00:10 And like always, we have some pretty awesome stuff lined up for you. But I want to say thanks.

00:14 Thanks to DigitalOcean. So DigitalOcean is sponsoring this episode, and they're going to

00:19 share this new thing they've launched with you guys called Spaces. It's like S3, but way,

00:24 way better. Tell you more about that later. Right now, I want to be a little philosophical and think

00:28 like a Pythonista. Yeah, think like a Pythonista. We're going to highlight a video from Luciano

00:35 Romalo. And he did a talk at PyBay 2017 called Think Like a Pythonista. And I, one of the things

00:43 actually, when most of my programming, I don't really create new types that much in Python.

00:49 I take advantage of, I don't really think like I'm creating new types. Like I'll use named tuples or

00:56 something. But even so, I take advantage of all the built-in types. And that's kind of what part

01:02 of his talk is about. It's, he create, walks through talking about creating another container

01:08 type called a deck of cards. So he's got a cards type and a deck type. And the deck has to do,

01:15 is a container, and it has to do things like indexing and shuffling and stuff. And he walks through how to

01:22 do that so that you can use the built-in iterate for loops and accessors. It's just a really kind

01:29 of a fascinating look at how, how little code it takes to make something look like another,

01:35 any other Python type using some of the dunders and stuff.

01:39 Yeah, for sure. I think he did a super job with this. And this comes from his book,

01:43 Fluent Python, which is absolutely one of the best books once you know a little bit of the language to

01:47 really get the zen of it. And he's really big into the Python data model, the dunder methods,

01:53 and all that kind of stuff. And so I'm sure this is a really great way to see how that like,

01:57 comes out in a custom type, right?

01:59 Yeah, it is. And also, I think it's, it's a good talk for somebody to watch,

02:03 even if you don't think you're going to ever create a container type yourself,

02:06 because you learn about a lot of the, a lot of the things that your built-in types do for you that

02:12 you don't even really think about. There's quite a bit of information in there that's worth watching,

02:16 even if you don't feel like doing it. Plus, he does the entire talk or have most of the talk with a

02:22 Jupyter notebook and even changes some of the code and fixes a bug and does monkey patching

02:28 with live code. It's pretty cool.

02:30 Yeah, that's pretty cool. I think definitely this concept of Jupyter notebooks,

02:33 the whole scientific computing stuff is bleeding over into all sorts of presentations. I think it's

02:39 great.

02:39 And near the end of the talk, he does a, an example of why operator overloading is awesome.

02:45 And he shows a little numpy, a little tiny script that can graph the trajectory of a launched item,

02:55 you know, like throwing a ball or something. And it's so clean and such little code because

03:01 the numpy library is allowed to take their data types and overload operators so that multiplication

03:07 and addition work on these large objects.

03:10 Yeah, that's awesome.

03:11 It's pretty great.

03:12 Yeah. Luciano definitely has some special sauce. He sprinkles on what he does. It makes it really

03:16 smooth. So check that one out. That's a really great one. That's a live presentation

03:19 from basically derived from his book.

03:23 I'll link to this, of course, but there's also a link to the rest of the PyBay 2017 playlist.

03:27 Yeah, that's cool. For sure. So we talked about how neither of us are really gamers last time,

03:32 right, Brian?

03:32 Right.

03:33 Yeah. But one of the really interesting possibilities, you know, Python is probably the biggest language

03:40 for machine learning and AI. And one of the big possibilities would be like, what if you could

03:44 take the AI you're trying to teach about a world and like immerse it into a game and just make it be in

03:50 the game, right? So if you wanted to have like a self-driving car, like could you stick it into like

03:54 a car simulator game and make it race around and try to win? Or, you know, you want to have a puzzle

03:59 solving one, stick it in the middle of a go game and try to make it try to pay it go. So the thing I

04:05 want to feature this week is something called serpent.ai. So that's both the name of the product and the

04:10 website. And it's a game agent framework. So this is pretty insane. What you can do is you can take any

04:15 video game and you can plug in says any, that's kind of intense, any video game and you can plug in AI and

04:23 bot programming with Python. What do you think about that?

04:26 Wow. I think that looks fun.

04:28 I know it looks super fun. Oh my gosh. And they have a, the guy who created it. I'm sorry. I couldn't find his name on here. He runs a Twitch channel. So you can drop in on his Twitch channel, which is linked to from serpent.ai. And actually pulled up one particular video that he recorded on Twitch, which is like streaming gaming stuff. And he'll do like six hour coding sessions where he's like, we're going to take this game. We're going to plug in AI into it and solve it in real time. And it's pretty intense.

04:55 Wow. That's cool. If you want to spend like a day watching somebody like plugging in AI to a game, you can do it.

05:00 That sounds neat. So I do know that, I mean, Twitch is being used a lot by a lot of game people watching gamers and stuff. But do you know if anybody's doing or how, if it's catching on much with like live coding examples and stuff?

05:14 I was thinking about that as I was watching, he does do live coding. He's got like PyCharm and Jupyter and the game going and like console stuff in the video that I linked to, you know, I honestly don't know, but I feel like there should be if people aren't doing it.

05:26 Probably is probably just not in my circles.

05:28 Yeah, yeah, exactly. We don't do enough gaming, right? So some of the things they talk about in this, it says like, there's really no conventions based on this, you can just plug in whatever you want. So if you want to use like the latest reinforcement, deep learning algorithm from Google, try that.

05:42 If you want to use computer vision, we talked about one recently, Luminoth, you could plug that in there, for example, to do like image processing. And so even if you want to just randomly press buttons, you could try that as well.

05:54 So I'm guessing this game, it has to be something that you can like get at with Python library or something, right? Or I mean, I mean, I can't play my Xbox, for instance.

06:04 No, I think it's got to run on a computer. But I think it like simulates arrow keys and stuff like that. I'm not entirely sure how it determines whether it wins.

06:12 Because you got to give positive or negative feedback to the deep learning models. But something's happening in there. It looks pretty cool.

06:17 Maybe you just have to like hang out with a microphone and say, good job.

06:20 Yeah, just pat it on the back. Well done, AI. Well done.

06:22 So you've been working with Cookie Cutter lately, right?

06:26 I have. I've been, most of the, my work has been making some plugins for work, but I'm utilizing the pytest Cookie Cutter plugin. And that caused me to, I hadn't really, one of the options was to generate some project documentation with MakeDocs. And I never really used them before. I don't know if it's pronounced MakeDocs. That's how I've been pronouncing it.

06:49 I think there's a little bit of debate, but I think that that totally works.

06:52 Okay. So it's, it's MKDocs. And I think it stands for MarkdownDocs. And a lot of, I know about Sphinx and I know about that a lot of the Python documentation is with restructured text, but I just don't really like restructured text.

07:06 So anyway, I just wanted to highlight MakeDocs. It's, I thought it was neat. And I've been documenting some of my projects with it. And it's, especially with Cookie Cutter, it just pops it in there, or at least this pytest plugin, Cookie Cutter, it pops it in and creates a YAML file for you.

07:24 And really, it's kind of neat because you, you have this configuration of like what your theme is and all that, but your writing is just a directory with a bunch of Markdown files.

07:34 Markdown is a really great invention. I'm really glad it exists.

07:37 The one thing I was worried about is I do kind of want to keep some of the, like the library API documentation with the code, not somewhere else. And I do want to document that somehow. So I did stumble across one of the friends of our show, Christian Medina.

07:53 He wrote a Medium article called, how to write your own Python document generator.

07:59 And essentially it's, using MKDocs also, but walking through your code and pulling out the doc strings. So I'm going to give that a try next week and see how that goes.

08:08 Oh yeah. You'll have to give us a report how it came out.

08:10 Yeah.

08:11 Awesome.

08:12 All right.

08:13 So this audio file that you're listening to right now came to you through digital ocean spaces.

08:17 And like I said, the beginning, they're sponsoring this episode as they are a number of them.

08:21 And you may know of Amazon S3. It's a place where you can put your files, both for people to consume browsers and stuff, but also for your app to integrate with.

08:31 Well, digital ocean decided to take it to another level of spaces and create something similar, but with a much better business model.

08:38 You pay five bucks a month, you get 250 gigs of storage, a terabyte of outbound bandwidth, which by the way, is like $93 just there on its own in S3.

08:48 So five bucks, that is a super good deal. And inbound data is always, always free. So it's way more predictable. You can save tons of money and it has the same API as S3. So all your tools and stuff still work as well. It's definitely recommended. I'm using it more and more every day. And like I said, using it to deliver all the audio for this podcast.

09:08 That's really cool. And so like you said, they're sponsoring this episode, but they also save us money on every episode since we're...

09:14 That's right. They're kind of sponsoring everyone in a little bit of a sense, right?

09:17 Save us some good money. So check them out at do.co slash Python.

09:20 Now, you and I, we, I think we were kind of breathing the same air. Something was in the air this week here in Portland because I chose an item that has to do with taking your Python applications and packaging them up for reuse in one way.

09:37 And you chose the very next item packaging up for reuse in a Python application. So let's do some two interesting takes on this.

09:44 Neat.

09:45 So there's a couple of programs or libraries that you can use to build proper applications out of your Python code.

09:52 I feel like deployment to end users is one of the things that's really tricky for Python, right? This is one of its weak points.

09:59 You got a bunch of scripts. You're going to send those out. They have dependencies.

10:03 Like you can't give a regular user a Python thing and say, oh yeah, run this pip install this to like configure it, right? That's not going to work.

10:11 You have to go, here's an exe. Here's a .app and so on.

10:14 So you can use CX-free. There's a few others.

10:17 Pyto-app, Pyto-exe.

10:19 But one that's really nice is called PyInstaller.

10:21 And they just released version 3.3.

10:23 So this works on all the operating systems.

10:25 Windows, Linux, Mac, even FreeBSD, Solaris, AIX, a bunch of them.

10:30 And the idea is that its primary goal is to do what I described, but also be compatible with third-party packages out of the box.

10:39 So a lot of the weaknesses of the other ones that I've mentioned, they'll run into trouble trying to use Matplotlib or NumPy or PyQt, for example, which is super painful.

10:48 And it'll take that and package all those libraries up for you.

10:51 And they just released the 3.3 version, which has Python 3.6 support.

10:56 Oh, that's cool.

10:56 Do you see yourself maybe using this anymore?

10:58 I definitely could see using this, especially, like you said, if you're sharing an application with somebody that isn't a developer, it's good.

11:06 Right.

11:06 Yeah, if you're going to ship some end product, you want to run it in Python.

11:09 That's pretty awesome.

11:09 And you can use the .1 file option and .1 no console, and it can just be like a GUI.

11:15 And you wouldn't really know that it's not like built in C++ or .NET if you run it on Windows or Coco on Mac, things like that.

11:24 It's really, really nice.

11:25 Nice.

11:25 Yeah.

11:26 So what's your take on this?

11:27 On this one, I think I definitely have to try it.

11:29 But the item that I brought up, I was thinking about is PEX.

11:34 And it says it's a library and tool for generating Python executable files, PEX files.

11:40 And the files are, yeah, they're self-contained executables.

11:45 But there's executable virtual environments.

11:48 Right.

11:48 Whereas the one that I was thinking of kind of packages up all the Python things you need to run into in AXE.

11:53 This is like, can we just package the virtual environment and somehow teach it to run, right?

11:58 I'd have to look, but I'm almost positive there is no, it doesn't package the Python with it.

12:03 So if you have everybody already having Python installed, but you want to share things around with share Python programs, this might be a really cool thing.

12:12 And it looks like, I just got introduced to it the other day, but it looks like, yeah, it came from Twitter.

12:18 And it used to be part of the Twitter comments package, which looks like there's some cool stuff in there.

12:23 I'm going to have to check that out also.

12:24 Yeah.

12:24 I was, when I was checking out the video that you're about to talk about, I was like, whoa, this Twitter comment thing, it looks awesome.

12:29 There's way more to be learning here.

12:31 Yeah.

12:31 But the, it's now a standalone thing.

12:33 And there's this great, we linked to a video called WTF is PEX.

12:39 And it's by Brian Wickman.

12:41 And it's a little 16 minute video where he just pretty much talks about what it is.

12:46 And the insight is Python can import from a zip file.

12:51 Even if you corrupt the zip file.

12:53 Even if you corrupt it.

12:54 So they corrupt the zip file by like putting like a main at the end and some, like the header on there to tell the rest of the operating system that it's a Python program.

13:05 And it just tries, your environment just tries to run it like Python and it imports it.

13:12 It's all of its dependencies from itself.

13:14 And, you know, I kind of got lost as to exactly how that works, but it looks fun also.

13:20 Yeah, it definitely looks cool.

13:21 So if your goal is more to share executable things with developers, it seems like PEX is really awesome.

13:27 Or maybe even with infrastructure, like I want to deploy this thing to the server and I just want to drop it in and have it go and not have to deal with dependencies and virtual environments, all that stuff.

13:35 Looks super cool.

13:36 And I get the sense that they're using this in production at Twitter, even though I don't remember them exactly saying that.

13:42 Or using it in just even for tools because some of the examples were just to, you've got a little project, it's got a virtual environment with, or it's got some requirements for it and some dependencies.

13:54 You just kind of make that into an executable and then you can just drop the whole thing into your bin directory or something and not worry about it.

14:02 That's really nice.

14:03 That's pretty great.

14:03 Super cool.

14:04 All right.

14:05 So I want to keep with this theme of we're going to package up some code and run it.

14:08 And even with either of these, like it's great that PEX can zip up your source files or that PyInstaller can zip up your source files, but you're still sending your source code like straight to people.

14:21 And that can be bad.

14:22 Potentially.

14:22 Maybe you've got some secret algorithm.

14:25 Maybe you've got licensing that you don't want to just be able to like, well, we'll just comment out the license check and it'll be good.

14:30 Right?

14:31 Something like that.

14:31 So there's this interesting article that came by this week called using Cython to protect a Python code base.

14:39 So Cython is a fully to machine instruction compiled variant of Python.

14:45 Right?

14:45 We've got Python.

14:47 We've got PyPy.

14:48 We've got Cython.

14:49 We've got IronPython.

14:51 All these variations.

14:52 And Cython is one that came from the SageMath guys way, way back in the day to compile this to faster machine instructions.

14:58 But it's fully compiled code out of .py files, which is pretty interesting.

15:05 And so that makes it way harder to mess with.

15:07 Yeah.

15:08 So this article walks you through building a Python package and all the little extra tricks and tweaks that you've got to do so that you can compile your Python code with Cython

15:20 and then ship it as a wheel, which is the current standard for pip installing a thing.

15:26 Right?

15:27 So you can create a binary distributable wheel that has zero source code that is compiled for a particular architecture.

15:33 Wow.

15:34 Okay.

15:35 Isn't that cool?

15:35 So if you want to...

15:36 Yeah.

15:37 And then you could combine this with PyInstaller and make it an EXE and off you go.

15:40 Right?

15:40 Yeah.

15:41 Since the algorithm's open, is it reversible then?

15:44 Well, it's like compiling C code or C++ or something.

15:47 So it is technically reversible and you can get it back, but it's way, way harder, right?

15:52 Yeah.

15:52 So they do say there are certain ways to, like, hack the runtime.

15:56 If you import it, you might be able to get it to give up some of its secrets, but it's definitely better than actually shipping the source code.

16:03 This reminds me of Obfuscators.

16:05 I can't remember if it was in C# or Java.

16:07 Do you remember Obfuscators?

16:09 Yeah.

16:09 I totally remember those.

16:11 It was even such a weird word to say when I first learned about it.

16:14 I remember that, actually.

16:15 Yeah.

16:16 I think it first came out in Java to sort of scramble up bytecode, Java bytecode.

16:20 And .NET definitely has it as well because they have IL, which both Java bytecode and .NET IL are super reversible.

16:28 Like, it's almost exactly what you wrote gets pulled back other than maybe, like, where you put your curly braces on the new line, on the top, on the if line, you know, things like that.

16:38 It's, like, really, really reversible.

16:39 So I think this protecting with Cython actually would be even better than Obfuscating, although I haven't looked at all the different angles.

16:46 It's really an interesting idea, though.

16:48 And it doesn't have to be everything, right?

16:49 Like, you could take your core algorithm that you want to protect or whatever, Cythonize it, and then just import it into the rest of your code.

16:58 Yeah, that's a really great idea because, like, for instance, a bunch of the stuff, like reading config files or your user interface or something, you might not care about that part.

17:06 Right, exactly.

17:07 It can be reverse engineered pretty easily just by looking at it.

17:10 But your core algorithms and the real reason why people are buying your product, that's not a bad idea.

17:17 So this is cool.

17:18 Yep.

17:18 I've never tried it, but this is a pretty cool idea, and it's a step-by-step sort of thing.

17:22 So I thought people might find that useful out there.

17:24 I think so.

17:24 Yeah, awesome.

17:25 Well, that's our six items.

17:26 That's awesome.

17:27 Went really fast.

17:28 Like always.

17:29 So I don't have anything extra to add.

17:31 Do you have anything you'd like to talk about?

17:33 I have a little bit of a survey I'd like people to consider filling out, really only if you're, like, super passionate about this.

17:39 One of the projects that I'm working on is I want to create a course to make working with Python on Windows better and delightful, not frustrating and challenging.

17:48 So I'm trying to gather a bunch of information on what people find hard with working with Python on Windows.

17:53 I've got a bunch of people filled out a form.

17:56 Like about 200 people filled this out already, but I put a link here to a Google Docs form.

18:01 If you work with Python on Windows and you've had struggles with it and you have some sort of feedback you can give so you can help me maybe create a class or something, some sort of resource to help people be more successful with Python on Windows, fill out the form at the bottom of the show notes.

18:14 That'd be awesome.

18:14 Oh, great.

18:15 I'll definitely fill this out and pass it around because there's a ton of people that I work with that are kind of new to Python and new to Python on Windows.

18:23 Oh, that'd be perfect.

18:24 Yeah, that'd be awesome.

18:25 I mean, a lot of people are like, well, just use Linux or whatever.

18:28 But when you show up at a tutorial or to give a training class to a corporate group and they all have their corporate Windows 7 laptops, it doesn't matter whether you recommend Linux or macOS.

18:39 They have to make it work on their environments.

18:41 And so my goal is to try to make that like a much smoother process.

18:44 Great.

18:45 Cool.

18:45 Awesome.

18:46 Well, thanks again for meeting up and pulling all these ideas together.

18:48 It's always fun.

18:49 Oh, yeah.

18:49 Thank you.

18:50 Talk to you next week.

18:51 Thank you for listening to Python Bytes.

18:53 Follow the show on Twitter via at Python Bytes.

18:56 That's Python Bytes as in B-Y-T-E-S.

18:58 And get the full show notes at Python Bytes.fm.

19:02 If you have a news item you want featured, just visit Python Bytes.fm and send it our way.

19:06 We're always on the lookout for sharing something cool.

19:10 On behalf of myself and Brian Okken, this is Michael Kennedy.

19:12 Thank you for listening and sharing this podcast with your friends and colleagues.