Transcript #50: Bundling , shipping, and protecting Python applications

00:00 Hello and welcome to Python bites where we deliver Python news and headlines directly to earbuds. This is episode 50 recorded November 1st 2017 I'm Michael Kennedy and I'm Brian Okken and like always we have some pretty awesome stuff lined up for you But I want to say thanks. Thanks to digital ocean So digital ocean is sponsoring this episode and they're gonna share this new thing. They've launched with you guys called spaces It's like s3 but way way better tell you more about that later right now I want to be a little philosophical and think like a Pythonista.

00:29 Yeah, think like a Pythonista. We're going to highlight a video from Luciano Romalo, and he did a talk at PyBay 2017 called Think Like a Pythonista. And one of the things, actually, when most of my programming, I don't really create new types that much in Python.

00:49 I take advantage of, I don't really think like I'm creating new types. I'll use named tuples or something. But even so, I take advantage of all the built-in types and that's kind of what part of his talk is about. It's he create, walks through talking about creating another container type called a deck of cards. So he's got a cards type and a deck type and the deck has to do, is a container and it has to do things like indexing and shuffling and stuff. And he walks through how to do that so that you can use the built-in iterate for loops and accessors.

01:28 It's just a really kind of a fascinating look at how little code it takes to make something look like any other Python type using some of the dunders and stuff.

01:39 Yeah, for sure.

01:40 I think he did a super job with this.

01:41 And this comes from his book, Fluent Python, which is absolutely one of the best books once you know a little bit of the language to really get the zen of it.

01:50 He's really big into the Python data model, the Dunder methods, and all that kind of stuff.

01:54 And so I'm sure this is a really great way to see how that comes out in a custom type, right?

02:00 Yeah, it is.

02:01 And also, I think it's a good talk for somebody to watch, even if you don't think you're going to ever create a container type yourself, because you learn about a lot of the things that your built-in types do for you that you don't even really think about.

02:14 There's quite a bit of information in there that's worth watching, even if you don't feel like doing it.

02:19 He does the entire talk or most of the talk with a Jupyter notebook and even changes some of the code and fixes a bug and does monkey patching with live code. It's pretty cool.

02:30 Yeah, that's pretty cool. I think definitely this concept of Jupyter notebooks, the whole scientific computing stuff is bleeding over into all sorts of presentations. I think it's great.

02:39 Near the end of the talk, he does an example of why operator overloading is awesome.

02:45 And he shows a little NumPy, a little tiny script that can graph the trajectory of a launched item, you know, like throwing a ball or something. And it's so clean in such little code because the NumPy library is allowed to take their data types and overload operators so that multiplication and addition work on these large objects. Yeah, that's awesome.

03:11 It's pretty great. Yeah, Luciano definitely has some special sauce he sprinkles on what he does that makes it really smooth. So check that one out. That's a really great one. That's a live presentation from basically derived from his book.

03:23 I'll link to this of course, but there's also a link to the rest of the Pi Bay 2017 playlist. Yeah, that's cool for sure. So we talked about how neither of us are really gamers last time, right Brian? Right. Yeah, but one of the really interesting possibilities. Python is probably the biggest language for machine learning and AI.

03:41 One of the big possibilities would be, what if you could take the AI you're trying to teach about a world and immerse it into a game and just make it be in the game? If you wanted to have a self-driving car, could you stick it into a car simulator game and make it race around and try to win? Or you want to have a puzzle-solving one, stick it in the middle of a Go game and try to Try to pay it go.

04:04 So the thing I want to feature this week is something called Serpent.ai.

04:08 So that's both the name of the product and the website.

04:11 And it's a game agent framework.

04:13 So this is pretty insane.

04:14 What you can do is you can take any video game and you can plug in, it says any, that's kind of intense, any video game and you can plug in AI and bot programming with Python.

04:25 What do you think about that?

04:26 Wow.

04:27 I think that looks fun.

04:28 I know it looks super fun.

04:30 Oh my gosh.

04:31 And they have the guy who created it, I'm sorry, I couldn't find his name on here.

04:35 He runs a Twitch channel.

04:38 So you can drop in on his Twitch channel, which is linked to from serpent.ai, and actually pulled up one particular video that he recorded on Twitch, which is like streaming gaming stuff.

04:47 And he'll do like six hour coding sessions where he's like, we're going to take this game, we're going to plug in AI into it and solve it in real time.

04:54 And it's pretty intense.

04:55 Wow, that's cool.

04:56 If you want to spend like a day watching somebody like plugging in AI to a game, you can do it.

05:00 - That's neat.

05:01 So I do know that, I mean, Twitch is being used a lot by a lot of game people watching gamers and stuff, but do you know if anybody's doing or if it's catching on much with like live coding examples and stuff?

05:14 - I was thinking about that as I was watching.

05:16 He does do live coding.

05:17 He's got like PyCharm and Jupyter and the game going and like console stuff in the video that I linked to.

05:22 You know, I honestly don't know, but I feel like there should be if people aren't doing it.

05:26 - Probably is, probably just not in my circles.

05:28 - Yeah, yeah, exactly, we don't do enough gaming, right?

05:30 So some of the things they talk about in this, it says like, there's really no conventions based on this, you can just plug in whatever you want.

05:36 So if you wanna use like the latest reinforcement deep learning algorithm from Google, try that.

05:42 If you wanna use computer vision, we talked about one recently, Luminoth, you could plug that in there, for example, to do like image processing.

05:51 And it's even if you wanna just randomly press buttons, you could try that as well.

05:54 - So I'm guessing this game, it has to be something that you can like get at with Python library or something, right?

06:00 Or I mean, I can't play my Xbox, for instance.

06:04 - No, I think it's got to run on a computer, but I think it simulates arrow keys and stuff like that.

06:09 I'm not entirely sure how it determines whether it wins, 'cause you got to give positive or negative feedback to the deep learning models, but something's happening in there.

06:16 It looks pretty cool.

06:17 - Maybe you just have to hang out with a microphone and say, "Good job." - Yeah, just pat it on the back.

06:21 Well done, AI, well done.

06:23 (laughing)

06:24 So you've been working with Cookie Cutter lately, right?

06:26 - I have.

06:27 And most of my work has been making some plugins for work, but I'm utilizing the pytest Cookie Cutter plugin.

06:35 And that caused me to, I hadn't really, one of the options was to generate some project documentation with MakeDocs.

06:44 And I never really used them before.

06:46 I don't know if it's pronounced MakeDocs.

06:48 That's how I've been pronouncing it.

06:49 I think there's a little bit of debate, but I think that that totally works.

06:52 Okay.

06:53 mkdocs and I think it stands for markdown docs.

06:57 And a lot of, I know about Sphinx and I know about that a lot of the Python documentation is with restructured text, but I just don't really like restructured text.

07:06 So anyway, I just wanted to highlight makedocs.

07:09 It's, I thought it was neat and I've been documenting some of my projects with it and it's, especially with cookie cutter, it just pops it in there or at least this pytest plugin cookie cutter, it pops it in and creates YAML file for you. And really, it's kind of neat because you you have this configuration of like what your theme is and all that, but your writing is just a directory with a bunch of Markdown files. Markdown is a really great invention. I'm really glad it exists. The one thing I was worried about is I do kind of want to keep some of the like the library API documentation with the code, not somewhere else. And I do want to document that somehow. So I did stumble across one of the friends of our show, Christian Medina.

07:54 He wrote a Medium article called How to Write Your Own Python Document Generator.

07:59 And essentially it's using mkdocs also, but walking through your code and pulling out the doc strings.

08:06 So I'm gonna give that a try next week and see how that goes.

08:09 - Oh yeah, you'll have to give us a report how it came out.

08:11 - Yeah. - Awesome.

08:12 All right, this audio file that you're listening to right now came to you through Digital Ocean Spaces.

08:17 And like I said at the beginning, they're sponsoring this episode as they are a number of them.

08:21 And you may know of Amazon S3, it's a place where you can put your files both for people to consume, browsers and stuff, but also for your app to integrate with.

08:32 Well, DigitalOcean decided to take it to another level of spaces and create something similar but with a much better business model.

08:39 You pay five bucks a month, you get 250 gigs of storage, a terabyte of outbound bandwidth, which by the way is like $93 just there on its own in S3.

08:48 So five bucks, that is a super good deal.

08:50 And inbound data is always, always free.

08:52 So it's way more predictable.

08:55 You can save tons of money and it has the same API as S3.

08:59 So all your tools and stuff still work as well.

09:02 It's definitely recommended.

09:03 I'm using it more and more every day.

09:05 And like I said, using it to deliver all the audio for this podcast.

09:08 - That's really cool.

09:09 And so like you said, they're sponsoring this episode, but they also save us money on every episode since we're.

09:14 - That's right.

09:15 They're kind of sponsoring everyone in a little bit of a sense, right?

09:17 I'd say it was some good money.

09:18 So check them out at do.co/python.

09:21 Now, you and I, I think we were kind of breathing the same air.

09:26 Something was in the air this week here in Portland because I chose an item that has to do with taking your Python applications and packaging them up for reuse in one way, and you chose the very next item, packaging up your Python applications in slightly a different way.

09:42 So let's do some two interesting takes on this.

09:44 - Neat.

09:45 a couple of programs or libraries that you can use to build proper applications out of your Python code.

09:52 And I feel like deployment to end users is one of the things that's really tricky for Python, right?

09:58 This is one of its weak points.

10:00 You got a bunch of scripts, you're going to send those out, they have dependencies, like you can't give a regular user a Python thing.

10:07 So oh, yeah, run this pip install this to like configure it, right?

10:11 That's not going to work.

10:12 You have to go here's an exe, here's a dot app, and so on.

10:14 So you can use CX free, there's a few others, py2app, py2exe, but one that's really nice is called pyinstaller.

10:21 And they just released version 3.3.

10:23 So this works on all the operating systems, Windows, Linux, Mac, even FreeBSD, Solaris, AIX, bunch of them.

10:31 And the idea is that its primary goal is to do what I described, but also be compatible with third party packages out of the box.

10:40 So a lot of the weaknesses of the other ones that I've mentioned, they'll run into trouble trying to use matplotlib or numpy or pyqt for example, which is super painful.

10:49 And it'll take that and package all those libraries up for you.

10:52 And they just released the 3.3 version, which has Python 3.6 support.

10:56 Oh, that's cool.

10:57 Do you see yourself maybe using this anymore?

10:58 I definitely could see using this, especially, like you said, if you're sharing an application with somebody that isn't a developer, it's good.

11:06 Right.

11:07 If you're gonna ship some end product, you want to run it in Python, that's pretty awesome.

11:10 And you can use the one, the --one file option and --no console and it can just be like a GUI and you wouldn't really know that it's not like built in C++ or.net if you're running on Windows or you know, Cocoa on Mac, things like that.

11:24 It's really, really nice.

11:25 Nice.

11:26 Yeah.

11:27 So what's your take on this?

11:28 On this one, I think I definitely have to try it.

11:29 But the item that I brought up, I was thinking about is PECS.

11:34 And it says it's a library and tool for generating Python executable files, PECS files.

11:41 And the files are, yeah, they're self contained executables.

11:45 But there's executable virtual environments, right?

11:49 Where's the one that I was thinking of kind of packages up all the Python things you need to run into an AXE.

11:53 This is like, can we just package the virtual environment and somehow teach it to run, right?

11:58 I'd have to look but I'm almost positive there is no it doesn't package the Python with it.

12:03 So if you have everybody already having Python installed, but you want to share things around with--

12:09 share Python programs, this might be a really cool thing.

12:12 And it looks like--

12:13 I just got introduced to it the other day, but it looks like it came from Twitter.

12:18 And it used to be part of the Twitter Commons package, which looks like there's some cool stuff in there.

12:23 I'm going to check that out also.

12:24 Yeah, when I was checking out the video that you're about to talk about, I was like, whoa, this Twitter Commons thing, it looks awesome.

12:29 There's way more to be learning here.

12:31 But it's now a standalone thing.

12:34 And there's this great-- we link to a video called WTF is PEX.

12:40 And it's by Brian Wickman.

12:42 And it's a little 16-minute video where he just pretty much talks about what it is.

12:46 And the insight is Python can import from a zip file.

12:52 Even if you corrupt the zip file.

12:53 Even if you corrupt it.

12:54 - They corrupt the zip file by like putting like a main at the end and some like the header on there to tell the rest of the operating system that it's a Python program.

13:06 And it just tries, your environment just tries to run it like Python and it imports all of its dependencies from itself.

13:15 And you know, I kind of got lost as to exactly how that works, but it looks fun also.

13:20 - Yeah, it definitely looks cool.

13:21 So if your goal is more to share executable things with developers, it seems like PEX is really awesome, or maybe even with infrastructure, like I want to deploy this thing to the server, and I just want to drop it in, have it go, and not have to deal with dependencies and virtual environments, all that stuff.

13:36 Looks super cool.

13:36 And I get the sense that they're using this in production at Twitter, even though I don't remember them exactly saying that.

13:42 - Or using it just even for tools, because some of the examples were just to, you've got a little project, it's got a virtual environment with, or it's got some requirements for it and some dependencies, you just kind of make that into an executable and then you can just drop the whole thing into your bin directory or something and not worry about it.

14:02 - That's really nice.

14:03 - That's pretty great. - Super cool.

14:05 All right, so I want to keep with this theme of we're going to package up some code and run it.

14:08 And even with either of these, like it's great that Pax can zip up your source files or that PyInstaller can zip up your source files, but you're still sending your source code like straight to people and that can be bad potentially.

14:23 Maybe you've got some secret algorithm, maybe you've got licensing that you don't want to just be able to like, well, we'll just comment out the license check and we'll be good, right?

14:31 Something like that.

14:32 So there's this interesting article that came by this week called using Cython to protect a Python code base.

14:39 So Cython is a fully to machine instruction compiled variant of Python, right?

14:45 We've got Python, we got PyPI, we got Cython, We've got Iron Python, all these variations.

14:52 And Cython is one that came from the SageMath guys way, way back in the day to compile this to faster machine instructions.

14:59 But it's fully compiled code out of .py files, which is pretty interesting.

15:05 So that makes it way harder to mess with.

15:08 So this article walks you through building a Python package and all the little extra tricks and tweaks that you've gotta do so that you can compile your Python code with Cython and then ship it as a wheel, which is the current standard for pip installing a thing.

15:27 So you can create a binary distributed wheel that has zero source code that is compiled for a particular architecture.

15:34 Whoa.

15:34 Isn't that cool?

15:35 So if you want to--

15:36 It's very interesting.

15:37 Yeah, and then you could combine this with pyinstaller and make it an exe, and off you go, right?

15:41 Yeah.

15:41 Since the algorithm's open, is it reversible then?

15:44 - Well, it's like compiling C code or C++ or something.

15:47 So it is technically reversible and you can get it back, but it's way, way harder, right?

15:52 - Yeah.

15:53 - So they do say there are certain ways to like hack the runtime.

15:56 If you import it, you might be able to get it to give up some of its secrets, but it's definitely better than actually shipping the source code.

16:03 - This reminds me of Obfuscators.

16:05 I can't remember if it was in C# or Java.

16:08 Do you remember Obfuscators?

16:09 - Yeah, I totally remember those.

16:11 That was such, that was even such a weird word to say when I first learned about it, I remember that actually, I think it first came out in Java to sort of scramble a bytecode, Java bytecode, and .NET definitely has it as well, 'cause they have IL, which both Java bytecode and .NET IL are super reversible.

16:28 Like it's almost exactly what you wrote gets pulled back other than maybe like where you put your curly braces on the new line, on the top, on the if line, you know, things like that.

16:38 It's like really, really reversible.

16:39 So I think this protecting with Cython actually would be even better than obfuscating, although I haven't looked at all the different angles.

16:46 It's really an interesting idea though.

16:48 And it doesn't have to be everything, right?

16:49 Like you could take your core algorithm that you want to protect or whatever, Cythonize it and then just import it into the rest of your code.

16:58 - Yeah, that's a really great idea because like for instance, a bunch of the stuff like reading config files or your user interface or something, you might not care about that part.

17:06 - Right, exactly.

17:07 - It can be reverse engineered pretty easily just by looking at it.

17:10 but your core algorithms and the real reason why people are buying your product, that's not a bad idea.

17:17 So this is cool.

17:18 - Yep, I've never tried it, but this is a pretty cool idea and it's a step-by-step sort of thing.

17:22 So I thought people might find that useful out there.

17:24 - I think so.

17:25 - Yeah, awesome.

17:26 Well, that's our six items.

17:26 - That's awesome.

17:27 Went really fast.

17:28 - Like always.

17:29 - So I don't have anything extra to add.

17:31 Do you have anything you'd like to talk about?

17:33 - I have a little bit of a survey I'd like people to consider filling out, really only if you're like super passionate about this.

17:39 I'm one of the projects that I'm working on is I want to create a course to make working with Python on windows better and delightful, not frustrating and challenging.

17:48 So I'm trying to gather a bunch of information on what people find hard with working with Python on windows.

17:54 I've got a bunch of people filled out a form, like about 200 people filled this out already, but I put a link here to a Google docs forum.

18:01 If you work with Python on windows and you've had struggles with it and you have some sort of feedback you can give.

18:06 so you could help me maybe create a class or something, some sort of resource to help people be more successful with Python on Windows, fill out the form at the bottom of the show notes, that'd be awesome.

18:14 - Oh, great.

18:15 I'll definitely fill this out and pass it around because there's a ton of people that I work with that are kind of new to Python and new to Python on Windows.

18:24 - Oh, that'd be perfect.

18:25 Yeah, that'd be awesome.

18:25 I mean, a lot of people are like, well, just use Linux or whatever, but when you show up at a tutorial or to give a training class to a corporate group and they all have their corporate Windows 7 laptops, it doesn't matter whether you recommend Linux or macOS, they have to make it work on their environments.

18:41 And so my goal is to try to make that like a much smoother process.

18:45 - Great, cool.

18:45 - Awesome, well, thanks again for meeting up and pulling all these ideas together.

18:49 It's always fun.

18:49 - Oh yeah, thank you.

18:50 Talk to you next week.

18:51 - Thank you for listening to Python Bytes.

18:53 Follow the show on Twitter via @pythonbytes.

18:56 That's Python Bytes as in B-Y-T-E-S.

18:59 And get the full show notes at pythonbytes.fm.

19:02 If you have a news item you want featured, just visit pythonbytes.fm and send it our way.

19:06 We're always on the lookout for sharing something cool.

19:09 On behalf of myself and Brian Auchin, this is Michael Kennedy.

19:13 Thank you for listening and sharing this podcast with your friends and colleagues.