« Return to show page
Transcript for Episode #92:
Will your Python be compiled?
0:00 KENNEDY: Hello, and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds. This is Episode 92 recorded August 23rd, 2018. I'm Michael Kennedy.
0:11 OKKEN: And I'm Brian Okken.
0:12 KENNEDY: Hey, Brian, how are you doing?
0:13 OKKEN: I'm doing great.
0:14 KENNEDY: Yeah, same here, and I'm excited to be back together, and I can't wait to talk about some of these things. We have some amazing drop the mic sort of stuff coming up, so pretty excited to get into that, and also pretty excited that DigitalOcean is sponsoring this episode, so thank you, DigitalOcean. Check them out at pythonbytes.fm/digitalocean. Get $100 credit for new users. Right now, have we covered the fact that Python is popular?
0:37 OKKEN: Yeah, but I wanted to cover it again.
0:40 KENNEDY: Why not?
0:41 OKKEN: Python is pretty popular, and for the second year in a row, it came out as the number one popular programming language in the IEEE survey. So there's every year, IEEE puts out a popular programming survey, and that it's at the top isn't really news. What's really kind of cool about it though, and we'll have a link to the results, it's an interactive results thing, and I think we covered it last year, but this time Python is number one even in embedded, and that's the thing that stands out this year.
1:16 KENNEDY: Yeah, and it's worth pointing out that this survey is for electrical engineers. It's not like the same as say the Stack Overflow Survey, which is maybe more developer focused, so it's like you saw another area where Python has sort of taken over, which is awesome. Embedded, like you say.
1:31 OKKEN: The article that we're going to link to actually talks about whether or not this is real, and some people are doubting that Python really is that popular in embedded because I mean there's a lot of, in day to day job our embedded, the embedded work I do is in C++. There's a big range of what it means to be embedded, so they don't really define that. You have to kind of define that yourself. I define that mostly as you're developing software on a platform that's not the target platform, and the target platform doesn't really look like a computer.
2:04 KENNEDY: Like a Raspberry Pi or maybe something embedded in your car, something like that?
2:09 OKKEN: Yeah, definitely, but even it's a broad definition, so the embedded software I do actually runs on a Windows machine, but it doesn't look like a Windows machine. It looks like it's a testing equipment. It looks like a stereo component with dials and stuff. We don't run Python. We run C++. However, like you said, like Raspberry Pi, I think there's been a big push lately. I think that's where this is coming from. There's been a big push lately of trying to get people involved in more circuits and more real interactive programming with electronics through the use of MicroPython and CircuitPython on Raspberry Pis, and Arduinos, and micro:bits and all that sort of stuff, and including the Mu Editor that really tries to teach people right out of the get go. So there is some doubt of whether or not really Python's the top for embedded, but I don't doubt it at all. I think it's going to stay.
3:02 KENNEDY: Yeah, I think it's cool. I mean, you could have the debate of well, it's on professional shipping embedded devices that are for sale versus embedded in general, which like you say, could include $5 Adafruit type things that people play with or they're using in school, but it's not for sale, per say.
3:18 OKKEN: Right, but there's a lot of people taking those Adafruit things and turning them into kits for people or products for people to wear, and then the people that are learning on that are going to go into, might go into other embedded jobs, and I think since Python is faster to develop in, there's a big incentive, I think, for people to use a higher level language in embedded if they can, even in a production level.
3:43 KENNEDY: Yeah, and if they are, it's got to be Python, right? What else would it be?
3:46 OKKEN: Right.
3:53 OKKEN: Yeah. Is this drop the mic thing you were talking about?
3:56 KENNEDY: Unfortunately it's the second one, so I have to pick it back up, but let me work my way into this the way that it developed. So we have the mailing list, Python-Dev, and that's where a lot of the core developers talk. On this list, someone was talking about finding a better way to test the C APIs instead of writing some C test, some Python test, and those kinds of things. You can be sympathetic with this, I'm sure, and they were saying, look, we need to test the C code inside CPython, and one way to do this would, maybe we could use something like Cython, and we talked previously about how Cython lets you pretty easily integrate with C and Python at the same time. So there were some debates saying it might be a good idea, it might not be a good idea. On one hand, Cython is changing a lot. It's a pretty big dependency to take as part of the core development component of Python itself, things like that, but then something pretty incredible came along here, and I don't know how far along this is or whatever, but this guy named Yury steps in and drops a bit of a bombshell, and he says, hey you guys, this Cython thing is kind of cool, and I don't know if you want to do it, but over at Dropbox, we're working on a new compiler for Python called mypyc, and it just says, well this mypyc will compile type annotated Python to optmized C code, which ultimately compiles to machine instructions.
5:23 OKKEN: That's pretty cool.
5:24 KENNEDY: How about that? So it's like Cython, and we've seen where Cython can make things 20 or 50 times faster in certain circumstances because it's compiled. Really interesting stuff there, but the problem with Cython is it's a superset of Python. You have like the cdef, like you basically write in a separate language that if you squint, you're like, wait a minute, is that Python? No, no, no. There's a lot of weird otherness, other stuff going on around here. This is not Python, but it's Python-like, and it doesn't even use the same type annotations because it sort of predated that. So Cython is good, but it has like this you got to write in Cython, not Python. This mypyc thing they're talking about, this is straight Python 3.6 code. It runs on CPython, but you can compile it to machine instructions like Cython.
6:09 OKKEN: It's not all of Python. It's a subset, right?
6:13 KENNEDY: It's a subset. Exactly. So I think the main thing is you have to type annotate everything so it can actually compile it. That's more or less the difference with Cython if you type annotate everything, but it's a different type. It's using Cython types.
6:28 OKKEN: Right, but this is using just normal type annotations. That's...
6:33 KENNEDY: It's pretty incredible. It says, look, this is standard Python. It's a subset of Python, and you can easily integrate with C libraries because of CFFI. You can even run it on PyPy because it's a subset of Python. So it's pretty cool, and if Dropbox releases a full on compiler for Python, that would shock the world, I think.
6:53 OKKEN: Yeah, definitely. Like even with this with the subset to be able to develop it in Python and then do some little extra work to make sure you're able to compile it within this, this would be great.
7:06 KENNEDY: And the extra work is not much. It's like put type annotations on it. So maybe you have three or four functions that you call that are really called frequently or really a major portion of your time is spent there, and if you wrote it in C, it would be way faster. Well, if you just put type annotations on it, you could probably hit it with this thing. So pretty interesting little thread that got picked up there.
7:29 OKKEN: Well, yeah, have to watch this and update people if we see any more progress.
7:33 KENNEDY: Yeah, what I did not see is here's a link to the Git repo, or here's the announcement, or anything like that. So we'll have to see if anything comes from it, but I think it's really interesting that we've sort of gotten to this place because from my understanding is Guido is not a huge fan of compiling Python code and didn't see a need for this, but he must be involved in this because it's from Dropbox, and the main work he's doing at Dropbox is on mypy, and it's all about this, so I suspect he's involved somehow, but I have no confirmation on that.
8:05 OKKEN: Yeah, okay. Cool.
8:06 KENNEDY: Boom. Mic drop. Okay, I'll pick it up. Let's talk about Netflix. They are doing such cool stuff with Python.
8:13 OKKEN: Yeah, so it's Python and more. There's kind of a really cool article called Beyond Interactive: Notebook Innovation at Netflix, and as we've all known, Jupyter Notebooks have and using the Jupyter system has kind of revolutionized a lot of data analysis and machine learning and quite a bit of that interactive interacting with data environment, and that's true at Netflix as well, and that's what kind of this article's talking about is a lot of the cool things that they've done. Over the last few years, they've seen a larger and larger growth, and they've also said basically this is our supply data chain. I never really thought about all the different types of people that interact with data, but they listed, they have a little diagram they listed 9 or 10 different job roles that interact with data a little bit differently, but they're all interacting with it, like data experts, or data analysts, or algorithm engineers, or data scientists. There's all of these different things, and I'm not quite sure what they all do different, but they're all interacting with these huge data sets, and in their effort to use notebooks and Jupyter to kind of streamline and make things easier for people, in the process they've came up with some really cool extra projects that they've open sourced or developing in the open, and one of them is, I'm going to run through a few of them. One of them is called Interact, and it's an alternative user interface that has some cool things like inline cell toolbars and drag and drop cells. I didn't know that you couldn't do that in Jupyter, but that's neat, and a built-in data explorer. And then if you've got a Jupyter Notebook all set up with some data, wouldn't it be cool if you could have some of your states parameterized so that you can run the whole notebook with different parameters?
10:11 KENNEDY: This is almost like making a notebook like a function you can call and pass it inputs and get outputs.
10:16 OKKEN: Exactly, and that's what Papermill is, partly. So there's a project called Papermill that parameterizes notebooks, and then it also does things like if you've parameterized a bunch, you can analyze the different set of parameterized runs and do some analysis on the set, and then there's, it's kind of making a notebook kind of like you said, like a function that you can call. And then if you wanted to, if you're using that, and you want to share it with other people with the data and with the results, they've put together a thing called Commuter that lets people share them with each other, and then also scheduling. So if there's a separate article that's attached at the end about how if you've got this function with different parameters and stuff, maybe you want to test different live data, so you want to schedule it regularly or at different times. They've got a notebook scheduling system in place, so all of this is, it's not like a super easy thing to jump into, but they're trying to make it easier for people, and this is actually some pretty neat things that they've shared with everybody.
11:17 KENNEDY: This is incredible, and if you look at this big infrastructure diagram they have, it's just this entire architecture and cloud platform for running their style of executable notebooks using this new UI on containers you can configure and schedule. It's really crazy, and you can even come down and say, I'd like to run on some container with four CPUs and 30 gigs of RAM. Go. That's crazy.
11:47 OKKEN: It's kind of amazing, and I'm trying to start exploring a little bit more of what people are doing with Jupyter Notebooks and other parts of the Jupyter system because it is sort of opening up a lot of different data analysis into the open, which is neat.
12:01 KENNEDY: Yeah, that is really cool. Something else I just see in here, and I don't know quite where this shows up, but in that same organization that released all of these projects on GitHub, they're grouped into a bunch of projects under an org in the GitHub org style. There's a thing called VDOM for Python. What does it say? It says a virtual HTML DOM for Python, which is pretty interesting for creating HTML and whatnot, divs and paragraphs and stuff. That's also in here somewhere for whatever people want to do with that.
12:35 OKKEN: Interesting.
12:36 KENNEDY: Yeah, so I feel like this one is massive too. This is a really major thing, and organizations that depend on this notebook style, this could be a really big deal.
12:46 OKKEN: And I like that they split it out into different projects like the parameterization idea is powerful, and maybe you can use that by itself. If you don't quite like the user interface, so they've got the interact there that you can use that instead. These are cool that they've split them out a bit. I don't know how much they're tied together. I expect things like Titus and Commuter might depend on other pieces more than others, but it's neat.
13:13 KENNEDY: It's really neat. So another thing that is pretty cool that I want to talk about is DigitalOcean, and I've told you a bunch of things about it. Last time we talked about projects, and that's pretty awesome. Brian, you've heard of Heroku, right? Do you know what Dokku is? Dokku? I think it's Dokku, actually.
13:28 OKKEN: No, I don't.
13:29 KENNEDY: So Dokku is apparently a miniature Heroku powered by Docker. So if you want to have your own Heroku, you can spin up a Linux machine or machines and instill this thing called Dokku and treat it like Heroku for deploying and managing your code.
13:45 OKKEN: Oh, that's neat.
13:46 KENNEDY: That's neat. So over at DigitalOcean, they have a one-click deploy for creating Dokku servers. So you just go over there, create a new droplet, and you just choose I want a Dokku thing all set up. Click, boom. You're ready to make your own little version of Heroku.
14:00 OKKEN: Wow, that's really cool.
14:01 KENNEDY: Yeah, it's really cool. So they have all these cool one-click things to create different types of environments, and I just thought I'd highlight this Dokku one because I know people use Heroku a lot, and if you want some flexibility, this is a really awesome way to do it. So anyway, check them out at pythonbytes.fm/digitalocean. Get $100 credit if you're a new user to play with Dokku or whatever else you want to play with. All right, so remember a while ago we talked about running a Python script as a systemd service?
14:30 OKKEN: Yes, that was a while ago.
14:31 KENNEDY: It was. It was quite a while ago. I'll link to that when we talked about that as well, that episode, but the idea was I would like to take some Python code and make it turn on when I boot my Linux system. The idea was to take a Python script and make it just boot when Linux boots and run in the background. It's kind of like if you did a cron job but you have more control. Your app, your Python code is constantly running. If it crashes, it'll restart if you configure it that way and stuff. Well, that's great for Linux, but what about for Windows? The equivalent infrastructure in Windows is called a Windows service. It starts when Windows starts, if you want it that way. It can run when you're logged out, or it can run as a different user, it can run as a restricted account, all of these things. So if you want your app to run basically as Windows runs, then you have to create this Windows service. Well, it turns out it's quite easy to do this in Python, if you use the right libraries. So there's a cool article that just says how to create a Windows service in Python, and it's based on this project called pywin32, which seems to have no documentation. Maybe I just missed it, but it doesn't seem at all obvious how to use, but it is on GitHub and pretty popular. But this article talks about how to create a Windows service, and the idea is you just derive from a certain class, and you override a stop, a start, and a main method, and boom, you have Python code that runs as if it were part of Windows.
15:57 OKKEN: Oh, actually this is awesome. I'll use this right away.
16:00 KENNEDY: Yeah, it's really cool. I don't have a, I've written a couple of Windows services back in the day. I haven't had a chance or reason to do so recently. I did take that systemd idea and do some really cool stuff around my courses for automation there, but if people are on Windows or they have to deploy to Windows, I think this is a great little example. So if that appeals to you, if that sounds like something you want to try to do, this is a really accessible way to do it.
16:23 OKKEN: That's cool.
16:24 KENNEDY: Yeah, indeed.
16:25 OKKEN: I'm glad that this came out because I thought about doing this before. I'm like, oh man, I'm a smart guy. I can figure this out. I don't even know where to start looking.
16:31 KENNEDY: There's like weird registration stuff you have to do, but it's cool. You just say the name of the service is this, the description of the service is this, the way it starts is like on start, on system start or delay or whatever, and then you just run a command to install it. It's great.
16:44 OKKEN: Nice.
16:45 KENNEDY: Yeah, yeah. Probably want to package it up though, when you send it out.
16:48 OKKEN: Yeah, you should package it. Have we talked about packaging before? I think so.
16:52 KENNEDY: Not enough. It's one of the things that should be more accessible.
16:56 OKKEN: Okay, well, let's talk about it again. Mahmoud Hashemi, a while, I don't remember how long ago this was. I don't know if we covered it or not, but anyway, he wrote an article about different ways to package different levels of Python projects, and then he did a talk on it, and then that now it's been all this information has been shared with others and edited, and it's part of the PyPA, the Python Package Authority documentation, and it's called An Overview of Packaging for Python, and I kind of like how it's broken out because when you want to share Python code, there's different levels, so if you just want to share a module with somebody, there's that. It starts there, and talks about how to just share a simple module with somebody else, but then you quickly get into packages with source distributions, and wheels, and possibly binary distributions, and how to do that, but that's complete packages are different than if you want to share an application, so that's the part where I get fuzzy, but actually this is a good starting point that hopefully they'll keep current that talks about some of the different ways that you can share applications because sharing a web application is going to be different, and if you share it on Heroku, it's going to be different than sharing a desktop application or a command line application or something. Also whether or not you want to assume people have Python or whether you want to package Python with it, and what do you do with the dependencies and all of that. We've talked about all of this before, and there's a lot of different solutions, and so this is a good starting point to take a look at it if you want to share some code with somebody else. Start here, and then get lost.
18:41 KENNEDY: There you go. I like it. It brings a lot of the stuff we talked about together, like for example, in the bring your own Python executable section, it's like here's a whole bunch of ways to do this. You could use PyInstaller, cx_Freeze, constructor, which apparently I had never heard of, OSNAP, OSNAP. Another one, like there's a bunch of things in here, and I actually have only heard about half of them, so it's pretty cool.
19:02 OKKEN: Since applications kind of range, everything else is fairly consistent, and there's like one obvious right way to do it up through packages, and then it's with applications where it starts going off and feathering out into different solutions. I think it's because there's a lot of different requirements for applications different than just sharing source code. I don't think it's a bad thing that we have a lot of different solutions right now. We've talked about how it would be really nice if there was one obvious way. We're just not there yet, so here's a good list of some of them.
19:33 KENNEDY: Yep, and that's really cool. Well done, Mahmoud. So there was a little bit of drama a few weeks ago around PEP 572 about changing the Python language for an assignment, in-place assignment operator. Remember that? And that was actually, that was part of one of the straws that caused Guido to say, all right, I've had it with this. Tired of fighting over these things. I'm out. Well, here's a new PEP that I think proposes a bigger change to the Python language. This one is not accepted. It's in draft mode, so people can respond to it. However, I actually think the value this one brings is massive. It's a big deal though. It's a really big deal. So the idea is there's PEP 505 now, which is for None-aware operators, and the idea is there are several languages that are proven, there are some nice design patterns or language patterns that short circuit working with None or null or whatever it is in Swift. I forgot about what they call it there, but nothing. So there's basically two ways in which you work with this. One of these are called null-coalescing operators, which lets you substitute a value if you have a None object, and null-aware operators which lets you chain operations regardless of whether with or null. So the two languages that are most popular that come to mind have to be C# and Swift, and both of these have deep support for this concept. Swift takes it to a whole nother level to say you can't even assign the equivalent of None to a variable unless you mark it as nullable explicitly, which is way far out there. So the idea is Python could benefit from this. So there is these two cases, the null-coalescing and the null-aware. So I'll give you the null-coalescing. If you're going to do some kind of weird test like you'd say something like if value is not None, else missing. This big missing complicated thing could just be value question mark question mark missing. It's either going to be the value, but if value is none, then it's the other value missing. What do you think of that, Brian?
21:33 OKKEN: I'm still on the fence.
21:34 KENNEDY: Okay, this one to me actually doesn't offer a huge amount of value. It's okay. The next one to me is pretty killer. More with the fluent style of calling functions chained together. So the null-aware member access operator, it's the same basic PEP and the same basic structure, but it's used in a different case. It lets you chain these fluent interfaces together without testing for None. So suppose I have a user. The use has orders. The orders you can call a first operation, but if there's no orders, then it returns None, but if it does return one, the order has a name. So you would say if user is None, return None. User.orders.first(), if the first order is None, then we kind of bail out, but if you make through all those tests, you can get first order.name. So that's a lot of code to be spoken over the air, but you would now with this new proposal say user?.orders.first()?.name, and it would mean exactly the same thing. If you look at those two lines of code on digital paper, the amount of space on one and the other is ridiculous, right? Sorry, I didn't mean to cut you off.
22:38 OKKEN: First reading of the none-aware operators I was thinking it's just going to make the language more complicated to teach because when you're reading it isn't obvious what these things are doing. However, it does make the language smaller in that you can be more expressive in less code, and that's a good thing.
22:57 KENNEDY: I think it would make people more willing to effectively test for None when it could be none without cluttering the code. These two examples I gave you, it's like eight lines versus one line. You're willing to just go, oh I just think it's there, and we'll just go through it, and then you'll get None type does not have attribute such and such. Whatever the attribute is, you get that error all the, it's one of the most common errors in Python. But if you just put the question marks in there, that totally goes away.
23:23 OKKEN: That's the thing where if you don't think you need that problem, just think about how many times you see that. None does not have the index operator.
23:31 KENNEDY: Yes, exactly. Exactly. This basically provides a much nicer way to deal with it. My vote is for this feature, especially the null-aware member access, but I would be okay if Python never had it, but I do think the languages that do have it really make pretty good use of it. So it's in draft mode. People can give feedback before some sort of battle erupts over it.
23:54 OKKEN: Yeah, I think it's good.
23:55 KENNEDY: Yep, I do too.
23:56 OKKEN: Cool.
23:57 KENNEDY: It'll let you have a second edition or a third edition of your testing book because there's somewhere you got to put a question mark in there.
24:05 OKKEN: You're just adding work for me. Thanks.
24:07 KENNEDY: Anyway, it's up there for people to check out. Quite good. I don't have anything extra to share with the folks. Do you, Brian?
24:12 OKKEN: I've got a couple things. I just noticed the other day that the PyCascades that's going to be in February, February 23rd and 24th of 2019, and it's going to be in Seattle this time. It's the call for proposals is now open.
24:27 KENNEDY: Oh, that's sweet.
24:28 OKKEN: You got a lot of time too. You got it now through October 21st to get in your proposal.
24:33 KENNEDY: Oh, I'm super excited about that. The opening, the inaugural PyCascades in Vancouver BC last year was excellent, and I'm really glad I went. It was much smaller, and it was a really intimate event. You got to meet a lot of the people who were there, and I'm looking forward to have it in Seattle, and it's coming to Portland next year, so that'll be even better.
24:50 OKKEN: I'll try to go, and I'll also try to submit something. That'll be good. The other thing I wanted to highlight is I just wanted to brag because I got a Test and Code episode out a week or so ago that was with David Heinemeier Hansson, and it really went well. He's not a Python person, but it's interesting information. That was fun.
25:12 KENNEDY: All right, so for people who don't know who David Heinemeier Hansson is, what's his claim to fame?
25:17 OKKEN: Well, he's got lots, but he's the guy that made up Ruby on Rails, and he's one of the co-founders of Basecamp.
25:23 KENNEDY: Yeah, just that, so that's pretty awesome. Yeah, that's really cool that he was on your show, and I'm looking forward to checking that out myself.
25:31 OKKEN: It's definitely applicable to Python people as well because we talk about language agnostic things like just testing and stuff like that.
25:40 KENNEDY: Very cool. Well, thanks for the call out on PyCascades.
25:42 OKKEN: Sure, PyCascades, but I was catching up on your podcast, Talk Python, and they had a couple episodes that were people that learned Python after they didn't get CS degrees and learned Python later, programming later and just the whole concept of that, and listening to everybody's stories was very interesting. I liked those.
26:06 KENNEDY: Yeah, thank you. I did a two-part series on that, and that's I think it really connected to a lot of people because so many people find their way into our world of programming and Python without going through the traditional CS degree path, and I think it's just an example a lot of people doing really great in alternate ways in, so yeah, very cool.
26:26 OKKEN: Well, good talking to you, and we'll talk next week.
26:28 KENNEDY: Yep, you as well. Catch you later. Bye, everyone. Thank you for listening to Python Bytes. Follow the show on Twitter via @PythonBytes. That's Python Bytes as B-Y-T-E-S, and get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.