Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


« Return to show page

Transcript for Episode #81:
Making your C library callable from Python by wrapping it with Cython

Recorded on Friday, May 25, 2018.

00:00 KENNEDY: Hello and welcome to PythonBytes. Where we deliver Python news and headlines directly to your earbuds. This is Episode 81, recorded May 25th, 2018. I'm Michael Kennedy.

00:11 OKKEN: And I'm Brian Okken.

00:12 KENNEDY: And we've got a bunch of great stuff for you as always. Super excited to talk about it but before we do, Brian let's say thanks to DigitalOcean.

00:18 OKKEN: Thanks DigitalOcean.

00:19 KENNEDY: Thank you DigitalOcean. They're supporting the show by sponsoring this episode, and a number of others, and they are giving you all something awesome as well, A $100 free credit if you are a new user at pythonbytes.fm/digitalocean more about that later. I'd like to learn some stuff right now Brian. Learn about learning

00:39 OKKEN: We here a lot about machine learning and I came across a couple ways to learn it. So I've got a couple topics here. First one is a single page site, called Hello TensorFlow, and it's a kind of fun little demo with one machine learning example of a bad guess at the coefficients for a polynomial, and having the machine learning back end, I think it's TensorFlow, figuring out what the real answer is. And it's got a GUI, or a graph on there, where you can watch it zoom, or narrow in on it. So it's fun.

01:19 KENNEDY: That is cool. So it has the secret formula and it says, Here's some equation which you don't know. Machine learning system go learn how to predict where on that line it's going to be given a few points. And it's like this nice like animated thing, that rolls in. I love the realtime-ness of it. That's great.

01:37 OKKEN: And it doesn't quite get it right away, and if you run it again it's even better. So it increases it's guesses so it's kind of neat.

01:44 KENNEDY: Oh nice.

01:45 OKKEN: The other topic I wanted to bring up is I found out that there's, they are both an article about this that we'll link to, but there is a Google provided machine learning crash course that actually looks pretty slick. It's got fifteen hours of course material, and a bunch of lessons, and some exercises. So Google has put together a free course on getting started with machine learning.

02:08 KENNEDY: That's really cool.

02:09 OKKEN: That's kind of fun.

02:10 KENNEDY: So one it's kind of get you interested in the idea of it. The other is let's actually teach it to you.

02:14 OKKEN: Neat.

02:15 KENNEDY: So remember last week when we talked about pysystemd with Dan Bader? So he pointed out that it was interesting that pysystemd was implemented in Cython even though it had no real performance issues. I mean it just asked like hey is this service started? And it just delegated to the C API. So I came across an article called, Making your C library callable from Python by wrapping it in Cython by Stav Shamir. Which is a really nice short article.

02:43 OKKEN: Interesting.

02:44 KENNEDY: This is my guess, why they used Cython for pysystemd. So Cython is known for it's ability to increase performance in Python code of course, but it's also a really interesting way to sort of bring Python's syntax into the realm of C directly, right? And because of that it makes calling C code from what looks like Python and then exposing that part that looks like Python directly to Python really easy. So let my give you a quick example. Suppose I have a function, it's called hello. It takes a character pointer called name, right this is C, and then you do a printf("hello %s", name). So super simple. It's like a function. It takes a character pointer, and returns void. So if I wanted to call that function from Python, I could write just a couple of lines of code in Cython. So I would say c def external, right? Define, here's an external function, void hello const char, right? Just the signature of that C function. And then I'd write a function py_hello and say name:bytes goes to None, and just say hello(name), that's it.

03:50 OKKEN: Hmm

03:51 KENNEDY: And not all these crazy extensions or py objects whatever, just like It's basically Python code with type ants, or type annotations applied, right? I mean that's what Cython more or less is.

04:03 OKKEN: This is a really nice small example of how to get some. I thought I had to go through some, figure out how to link to the DLL and stuff like that, and yeah. Neat

04:12 KENNEDY: So the final step you actually have to build, you have to compile your Cython code so that it can be imported from regular Python code, and so they provide a setup.py which does that. And you just run Python setup.py build right? And it goes and does it's magic, and then you have it you can work with it. And by the way it happens fast, even if that's not the point

04:34 OKKEN: Okay I might have to go play with this. It's neat.

04:36 KENNEDY: It's cool isn't it? It's cool and it's easy. Its way better than flipping into C land and doing all sorts of stuff. I think if your point is only to consume this code, maybe even just to write regular Cython code there's a lot of good things going on there. Awesome

04:50 OKKEN: Cool

04:50 KENNEDY: So what's this next one you got, feature flags?

04:52 OKKEN: Feature flags and I can't remember who was the, there was um, now we're going back in time, about a year or so, but was it Instagram that talked about

05:02 KENNEDY: It was Instagram that moved from Python 2 to Python 3 on their Django deploy, deployment which I think the largest Django deployment in the world on the same branch and with out anything right? Just by using feature flags.

05:16 OKKEN: I don't remember if they used feature flags in that or not, but I know a lot of people that, a lot of teams that do have a model of of merging to the master frequently, or just working off the master for everybody, often use feature flags and I know how to do that in C++ but I wasn't quite sure, I could have hacked together something for Python, but this is a nice article about really how to do feature flags in Python well, and why you would do it, and how to do it. So a few of the benefits they talked about is improving teams response time to bugs, because if you add a feature with a feature flag, you can just turn it off by flipping the flag if you need to without having to redeploy everything.

06:00 KENNEDY: I really like this idea. It's great to be able to just keep everything in the same code base and also keep the database schemas in sync. Which is nice as well, so lot of cool stuff going on here.

06:12 OKKEN: You have just one code base to test. I mean you can turn on and off features to test different parts cause you want both with the feature and without the feature to still work, and you can migrate user groups with A/B testing or group splitting or however you want to migrate the feature in. But then it went on to talk about some of the ways to implement it nicely so that it's a maintainable flag system. How to measure your success with different analytics, and then using third party tools to make your flag support clean and not reinventing the wheel and other people have figured this out. And then also just at the end a comment to say, you know, once you've really decided that a feature's in place you have to go through and do feature flag clean up. So make sure that you remove the flags and have the features be permanent when you're ready to have them permanent. Clean up your code base. It was just a nice write up for this.

07:06 KENNEDY: It's really nice. I like it. And they have maybe one of the best visualizations of flat is better than nested, with some kind of Mortal Kombat type character.

07:17 OKKEN: It's crazy nested if statement. That's the clean up conversation right? Like don't do this.

07:23 KENNEDY: The example of how not to do feature flags, yeah definitely

07:27 OKKEN: Exactly Don't do this. Yeah its quite cool. Nice. Alright so speaking of quite cool, it's quite cool that DigitalOcean is sponsoring this podcast. And I wanted to just tell you guys quickly about them. So they're a hosting company. They've got data centers throughout the world, and I think one of the cleanest, nicest ways to create a set of virtual servers, and get them up and running and configure them. So if you want to create one that is already pre-configured for some infrastructure like discuss. You can just click a button. Say create me a discuss virtual machine based on a boon two, you know whatever version you're looking for, or create a fresh one and set it up however you like. So they provide a lot of the infrastructure for us, we actually pay for it, but they are the people we pay for getting you this podcast, which is pretty awesome. They've been really good. We're happy customers. And so if you want to be new happy customer, you can get $100 to try them out at pythonbyes.fm/digitalocean and that's for new customers. Check it out, and hopefully you create something cool and run it there.

08:28 KENNEDY: Nice

08:29 OKKEN: Hey Brian I got one, that I think you're going to like.

08:31 KENNEDY: Okay

08:32 OKKEN: It's about testing.

08:33 KENNEDY: I like testing.

08:34 OKKEN: Recently I had a TalkPython episode on the release of PyPI and the inside story of how that got revised and finally pypi.org is the official thing, not like a weird scary place that also matches to the same database. So that's really awesome. I can't remember who said it on the show, sorry, cause there were three folks, but they, one of the libraries brought up was pretend. Which is a stubbing library.

08:56 KENNEDY: Neat.

08:57 OKKEN: So stubbing is like mocking but it's different. How's that Brian? You know more about that than I do.

09:02 KENNEDY: Often times in mocking, you want to track the behavior of the code, if you're interacting with some system that's not really there, you want to make sure that you've called it in certain ways, or you called it a certain number of times, or the order of calls, things like that, and stubbing is really like, I just want to have my code be able to call something and have the return value be some pre-canned data. So it's more about pre-canning than about the behavior.

09:30 OKKEN: I see. So with mocking and maybe I'm going to say I'm going to call this login API and I want to make sure that it checks that I, my password is correct. Or something like that. That would be a mocking thing. Whereas for stubbing I just like, I need it to give back a password so it doesn't crash, cause it's like a, you know, non type attribute error type of crash if nothing comes back. So we've got to give it something. So let's create a stub to do that.

09:53 KENNEDY: Stubbing also is a great way to do things like um, if there's error conditions, like when you're connecting to a third party that goes out through some, you want if like the server crashes or something you'll leave an error code. So how do you simulate that? You can't go out and crash the server. It's a really great way to pretend bad things happen.

10:14 OKKEN: So let me give people the sense of the API it's real simple. So from pretend import stub. And then you just say stub and say here's a function name equal sum lambda, and now you can just start, you can pass that around. If somebody calls that function, it returns the value the lambda returns. Done. I don't that it could really be simpler. to be honest, right?

10:33 KENNEDY: Well that's one of the reasons that it's pretty cool is that, its simpler than using mock. Mock can do this to, but this is simpler.

10:39 OKKEN: Really nice yeah. You could probably use that with Flask couldn't you?

10:43 KENNEDY: You can probably use it with Flask. Yep. And my next topic is a surprise to me. I forgot that I put this down. But one of the things that I got out of PyCon, was I got to sit next to one of the people that, at dinner, that works on the Flask project, and he reminded me that they just went through and rewrote a bunch of the Flask tutorial. So I went through and took a look and yeah, the official Flask tutorial is got a lot of updates. The code that goes along with it's been updated. And just everything's been simplified, updated, it's a little cleaner. It includes a, I don't know if it had this before, but it includes a section on testing which highlights Py test of course, and coverage which is good. And one of the things I learned also with that discussion was Flask is a part of the a palette, I'm not sure what their entity is really but, Palette is a collection of people, that work on a collection of projects. And it's some pretty important stuff. We've got Flask, Click, Itsdangerous, I'm not sure what that is.

11:49 OKKEN: It's a request validation foundation of Flask.

11:52 KENNEDY: Oh okay. Itsdangerous, already said that, but Ginganingja2 Is there a Ginja? Or is it jus Ginja two?

11:59 OKKEN: I've not seen any Ginja in that wild lately but there probably was at some point.

12:03 KENNEDY: Okay but all, and then Mark Up Safe, which is a HTML mark up safe string for Python library, and then Werkzeug. Werkzeug?

12:14 OKKEN: Werkzeug

12:14 KENNEDY: I don't know how to pronounce that, Werkzeug?

12:17 OKKEN: With a V.

12:18 KENNEDY: Everybody relies on a lot, some of these things, even if you don't use Flask, and the Palettes project has a donate page now. So if you want to donate. And you can donate through their donation page. Pretty neat.

12:32 OKKEN: It's really nice. So we've had a lot of new training of Flask recently. Flask went 1.O a while ago. I talked to one of the guys, I'm sorry I so forget his name, because I meat a lot of people but, who was basically responsible for that whole progression. David I believe? And anyway he's like, I know people think because the Flask went one O the week after you guys did the zero ver episode actually that was in the works for like a year, I love you for doing that, but it was actually just a coincidence though. So anyway. Glad to see it go one O, but yeah this looks like Flask is getting some, a renewed love which is good.

13:06 KENNEDY: Yeah and I didn't know Click was by the same people. I use Click all the time so, that's pretty neat.

13:11 OKKEN: Yeah for creating CLIs very nice. Alright so let's round it out with a little bit of an internals look here as well, I feel like a lot of the stuff I'm covering this week is deep in the guts with the Cython, and if you're not doing Cython, you're doing regular Python then you're operating in the byte code space. So do you think people would be surprised if you told them that Python is compiled.

13:34 KENNEDY: Yes

13:35 OKKEN: A lot of them I think they would, but it's not compiled of machine instructions, or even jet compiled unless you're using Py Vy, but it's compiled to byte code. So those PYC files, those are like the instructions to the Python virtual machine, not instructions to your processor. But they're still compiled, and their still this byte code, and so understanding it's pretty interesting to just know how the internals of Python is working. So there's this nice article called, An Introduction to Python Byte Code, so if you know this byte code concept is kind of new to you, or you just want to play around with it a little, check it out it's pretty accessible. So I feel like there is a lot of hello world examples, in my topics so back to hello. So they have a function def hello, and it says print the static string hello world, right? Okay so what does this actually mean? And then they show you all the byte code. So okay we're going to load the global, which is the print function. We're going to load the constant, which is hello world. And we're going to call the function that is on the stack. So C Python uses a stack based virtual machine, and so they load these under the stack, if you have a function that takes two arguments, they might load two arguments on the stack, and then call the function, and things like that. So that's how your Python source code gets into executable form, and then these steps are actually sent down to the C Python run time and this giant wild loop with a switch statement, that's literally 3,000 lines of C code, that says what's the byte code, let's go do that, you know? So that's pretty interesting, and they talk about how you can take your Python code and look at this. So you just import dis, as in disassembly, and you say dis dot dis, and you pass it like a callable, or something like that and it'll show you the byte code, instructions that make it up. So it's pretty nice.

15:16 KENNEDY: New phone who dis? You can't actually just look at the PYC files though right?

15:20 OKKEN: No, they're like bytes. That's why you need this dis thing. I mean I guess if you can parse the bytes you can, but it's not strings I don't think.

15:28 KENNEDY: Okay.

15:29 OKKEN: I mean the pyc files are basically the compiled steps from your hello world text, into the byte code instructions in terms of bytes, and then, that's why those caps things are laying there. The next time you hit that app, right? It's just going to say, well let's just load up that pyc, so we don't have to reparse and validate it.

15:49 KENNEDY: This is kind of neat. I'm going to definitely go and check this out, because I just want to know more about this. It seems like something I should know about. Even though I probably don't need to on a daily basis.

15:59 OKKEN: I know you helpful it is, but its helping your conceptualization of how stuff works I think.

16:04 KENNEDY: Definitely. Very cool.

16:06 OKKEN: Yeah a little bit deeper down into the, is that the red pill or the blue pill? That takes you farther down. That's the red pill right? I don't remember. Awesome. But anyway it's definitely worth checking out if you haven't played with byte code before, it's a really nice simple way to get introduced to it. Brian you got anything? Any other news do you want to share with everyone this week?

16:22 KENNEDY: I don't think so, do you?

16:24 OKKEN: No, I'm all out of news this week, other than the ones I found so, I just want to say thank you. Thank you for being a part of this episode, and sharing everything with everyone.

16:32 KENNEDY: Thank you, bye.

16:33 OKKEN: You bet, bye. Thank you for listening to Python Bytes. Follow the show on twitter via @PythonBytes. That's PythonBytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm If you have a news item you want featured, just visit PythonBytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page