#81: Making your C library callable from Python by wrapping it with Cython
Overcast
Apple
Castbox
PocketCasts
RSS
RadioPublic
Spotify
YouTube
Sponsored by digitalocean: pythonbytes.fm/digitalocean
Brian #1: Learning about Machine Learning
- hello tensorflow
- one pager site with a demo of machine learning in action.
- “Machine Learning (ML) is the dope new thing that everyone's talking about, because it's really good at learning from data so that it can predict similar things in the future.”
- Includes a graphical demo of ML trying to learn the correct coefficients to a polynomial.
- Google Provides Free Machine Learning Course For All
- Machine Learning Crash Course (MLCC) is a free 15 hours course that is divided into 25 lessons. It provides exercises, interactive visualizations, and instructional videos. These can help in learning machine learning concepts.
- 40 exercises, 25 lessons, 15 hours, case studies, interactive visualizations
Michael #2: Making your C library callable from Python by wrapping it with Cython
- Article by Stav Shamir
- Cython is known for its ability to increase the performance of Python code. Another useful feature of Cython is making existing C functions callable from within (seemingly) pure Python modules.
- Need to directly interact from Python with a small C library
Want to wrap this C function?
void hello(const char *name) {
printf("hello %s\n", name);
}
Just install Cython and write this:
cdef extern from "examples.h":
void hello(const char *name)
def py_hello(name: bytes) -> None:
hello(name)
Then create a setup file (details in article), call python setup.py build_ext --inplace and you’re good to go.
Brian #3: Taming Irreversibility with Feature Flags (in Python)
“Feature Flags are a very simple technique to make features of your application quickly toggleable. The way it works is, everytime we change some behavior in our software, a logical branch is created and this new behavior is only accessible if some specific configuration variable is set or, in certain cases, if the application context respects some rules.”
def my_function(): if is_feature_active('feature_one'): do_something() else: do_something_else()Benefits
- Improving team’s response time to bugs. If a new feature causes a bunch of customer problems, just turn it off.
- Making possible to sync code more frequently. Merge to master with the feature turned off.
- Having a more fluid feature launching flow. Turn feature on in test/staging server.
- Validate your features easily with A/B testing, user groups, etc.
- Article discusses:
- how to implement flags cleanly.
- measuring success with analytics
- implementing flags with third party packages and recommends a few.
Michael #4: pretend: a stubbing library
- Heard about this at the end of the pypi episode of Talk Python and wanted to highlight it more.
- Pretend is a library to make stubbing with Python easier.
- Stubbing is a technique for writing tests. A stub is an object that returns pre-canned responses, rather than doing any computation.
- Stubbing is related to mocking, but traditionally with stubs, you don’t care about behavior, you are just concerned with how your system under test responds to certain input data.
- However, pretend does include a call recorder feature.
- Nice clean api:
>>> from pretend import stub >>> x = stub(country_code=lambda: "US") >>> x.country_code() 'US'
>>> from pretend import stub, raiser
>>> x = stub(func=raiser(ValueError))
>>> x.func()
Traceback (most recent call last):
File "[HTML_REMOVED]", line 1, in [HTML_REMOVED]
File "pretend.py", line 74, in inner
raise exc
ValueError
Brian #5: The official Flask tutorial
- Has been updated recently.
- simplified, updated, including the source code for the project.
- tutorial includes section on testing, including testing with pytest and coverage.
- Flask is part of Pallets, which develops and maintains several projects
- Click — A package for creating beautiful command line interfaces in a composable way
- Flask — a flexible and popular web development framework
- ItsDangerous — cryptographically sign your data and hand it over to someone else
- Jinja — a full featured template engine for Python
- MarkupSafe — a HTML-Markup safe string for Python
- Werkzeug — a WSGI utility library for Python
- You can now donate to pallets to help with the maintenance costs of these important packages.
- There’s a donate button on the pallets site that takes you to a PSF page. Therefore, donations are deductible in the US.
Michael #6: An introduction to Python bytecode
- Python is compiled
- Learn what Python bytecode is, how Python uses it to execute your code, and how knowing what it does can help you.
- Python is often described as an interpreted language—one in which your source code is translated into native CPU instructions as the program runs—but this is only partially correct. Python, like many interpreted languages, actually compiles source code to a set of instructions for a virtual machine, and the Python interpreter is an implementation of that virtual machine. This intermediate format is called "bytecode."
- These are your .PYC files
Example:
def hello()
print("Hello, World!")
2 0 LOAD_GLOBAL 0 (print)
2 LOAD_CONST 1 ('Hello, World!')
4 CALL_FUNCTION 1
- CPython uses a stack-based virtual machine. That is, it's oriented entirely around stack data structures (where you can "push" an item onto the "top" of the structure, or "pop" an item off the "top").
View and explore using
import dis
dis.dis(hello)
Episode Transcript
Collapse transcript
00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your buds.
00:05 This is episode 81, recorded May 25th, 2018.
00:10 I'm Michael Kennedy.
00:11 And I'm Brian Okken.
00:12 And we have a bunch of great stuff for you, as always.
00:14 Super excited to talk about it.
00:16 But before we do, Brian, let's say thanks to DigitalOcean.
00:18 Thanks, DigitalOcean.
00:19 Yeah, thank you, DigitalOcean.
00:20 They're supporting the show by sponsoring this episode and a number of others.
00:25 And they're giving you all something awesome as well.
00:27 $100 free credit if you're a new user at pythonbytes.fm/digitalocean.
00:33 More about that later.
00:34 I'd like to learn some stuff right now, Brian.
00:37 Learn about learning.
00:38 We hear a lot about machine learning.
00:41 And I came across a couple ways to learn it.
00:45 So I've got a couple topics here.
00:47 First one is a single page site called Hello TensorFlow.
00:52 And it's a kind of fun little demo with one machine learning example of a bad guess at the
01:01 coefficients for a polynomial and having the machine learning back end.
01:06 I think it's TensorFlow.
01:07 Figuring out what the real answer is.
01:10 And it's got like a GUI or a graph on there where you can watch it zoom or narrow in on it.
01:17 So it's fun.
01:19 That is cool.
01:20 Yeah.
01:21 So it has a secret formula.
01:22 And it says, here's some equation which you don't know.
01:25 Machine learning system.
01:27 Go learn how to predict where on that line it's going to be given a few points.
01:31 And it's like this nice little animated thing that rolls in.
01:34 I love the real timeness of it.
01:36 That's great.
01:37 And it doesn't quite get it right away.
01:39 And if you run it again, it's even better.
01:42 So it increases its guesses.
01:43 It's kind of neat.
01:44 Oh, nice.
01:45 The other topic I wanted to bring up is I found out that there's both an article about this
01:49 that we'll link to.
01:50 But there is a Google-provided machine learning crash course that actually looks pretty slick.
01:58 It's got like 15 hours of course material and a bunch of lessons and some exercises.
02:03 So Google has put together a free course on getting started with machine learning.
02:08 That's really cool.
02:09 It's kind of fun.
02:09 Yeah.
02:10 So one is kind of like gets you interested in the idea of it.
02:12 The other is like let's actually teach it to you.
02:14 Yeah.
02:14 Neat.
02:15 Yeah.
02:15 So remember last week when we talked about PySystemD with Dan Bader?
02:19 Yeah.
02:20 Yeah.
02:20 So he pointed out that it was interesting that PySystemD was implemented in Cython even though
02:26 it had no real performance issues.
02:28 I mean, it just asked like, hey, is this service started?
02:31 And it just delegated to the C API.
02:32 So I came across an article called Making Your C Library Callable from Python by Wrapping
02:38 It in Cython by Stav Shamir, which is a really nice short article.
02:43 Interesting.
02:44 Yeah.
02:44 This is my guess why they use Cython for PySystemD.
02:47 So Cython is known for its ability to increase performance of Python code, of course.
02:52 But it's also a really interesting way to sort of bring Python syntax into the realm of C directly.
02:59 Right.
03:00 And because of that, it makes calling C code from what looks like Python and then exposing
03:06 that part that looks like Python directly to Python really easy.
03:09 So let me give you a quick example.
03:11 Suppose I have a function.
03:12 It's called hello.
03:13 It takes a character pointer called name.
03:16 Right.
03:16 This is C.
03:17 And then you do printf hello, percent S name.
03:20 Right.
03:21 So super simple.
03:22 It's like a function.
03:23 It takes a character pointer and returns void.
03:25 So if I wanted to call that function from Python, I could write just like a couple of lines
03:31 of code in Cython.
03:33 So I would say C def external.
03:35 Right.
03:36 Define.
03:37 Here's an external function.
03:38 Void.
03:39 Hello.
03:39 Cons car.
03:39 Right.
03:40 Just the, there's the signature of that C function.
03:42 And then I'd write a function like pi underscore hello.
03:45 I'd say name colon bytes goes to none.
03:48 And just say hello name.
03:50 That's it.
03:50 Not all these crazy extensions or pi objects or whatever.
03:53 Just like it's basically Python code with type hints or type annotations applied.
04:01 Right.
04:01 I mean, that's what Cython more or less is.
04:03 This is a really nice small example of how to get some.
04:06 I thought I had to go through some like, like figure out how to link to the DLL and stuff
04:11 like that.
04:12 And yeah.
04:13 Neat.
04:13 Yeah.
04:13 And so the final step you have to actually build, like you have to compile your Cython
04:18 code so that it can be imported from regular Python code.
04:21 And so they provide a setup py which does that.
04:25 And you just run Python setup py build.
04:27 Right.
04:28 And it goes and does its magic.
04:30 And then you, then you have it.
04:31 You can work with it.
04:31 And by the way, it happens to be fast.
04:33 Even if that's not the point.
04:34 Okay.
04:34 I'm going to have to go play with this.
04:36 It's neat.
04:36 Yeah.
04:36 It's cool.
04:37 Isn't it?
04:37 It's cool.
04:38 And it's easy.
04:38 Yeah.
04:38 Yeah.
04:39 It's way better than like flipping into sea land and doing all sorts of stuff.
04:42 I think if your point is only to consume this code, maybe even just a regular Cython code.
04:47 There's a lot of good things going on there.
04:49 Yeah.
04:49 Awesome.
04:49 Cool.
04:50 So what's this next one you got?
04:51 Feature flags?
04:52 Yeah.
04:53 Feature flags.
04:54 And I can't remember who was the, there was, now we're going back in time, but a year or
05:00 so, but was it Instagram that talked about?
05:02 Yeah.
05:03 It was Instagram that moved from Python 2 to Python 3 on their Django deploy deployment, which
05:08 is I think the largest Django deployment in the world on the same branch and without anything,
05:14 right?
05:14 Just by using feature flags.
05:16 I don't remember if they used feature flags in that or not.
05:19 But I know a lot of people that, a lot of teams that do have a model of merging to the
05:25 master frequently or just working off master for everybody often use feature flags.
05:31 And I know how to do that in C++, but I wasn't quite sure.
05:34 I could have hacked together something for Python, but this is a nice article about really how to
05:40 do feature flags in Python well and why you would do it and how to do it.
05:45 So a few of the benefits they talked about is improving Teams response time to bugs, because
05:51 if you add a feature with a feature flag, you can just turn it off by flipping the flag if
05:57 you need to without having to redeploy everything.
06:00 I really like this idea.
06:01 It's great to be able to just keep everything in the same code base and also keep the database
06:07 schemas in sync, which is nice as well.
06:10 So a lot of cool stuff going on here.
06:11 You have just one code base to test.
06:14 I mean, you can turn on and off features to test different parts because you want both
06:18 with the feature and without the feature to still work.
06:20 And you can migrate user groups with A-B testing or group splitting or however you want to migrate
06:27 the feature in.
06:29 But then it went on to talk about some of the ways to implement it nicely so that it's a
06:35 maintainable flag system, how to measure your success with different analytics, and then using
06:41 third-party tools to make your flag support clean and not reinventing the wheel.
06:47 And other people have figured this out.
06:49 And then also just at the end of comment to say, you know, once you've really decided
06:53 that a feature's in place, you have to go through and do feature flag cleanup.
06:57 So make sure that you remove the flags and have the features be permanent when you're ready
07:02 to have them permanent clean up your code base.
07:04 So it was just a nice write-up for this.
07:06 Yeah, and it's really nice.
07:07 I like it.
07:08 And they have maybe one of the best visualizations of flat is better than nested with like some
07:13 kind of Mortal Kombat type character.
07:15 It's a crazy nested if statement.
07:20 That's the cleanup conversation, right?
07:22 Like, don't do this.
07:23 Yeah, and the example of how not to do feature flags.
07:26 Yeah, definitely.
07:27 Don't do this.
07:29 Yeah, it's quite cool.
07:29 Nice.
07:30 All right.
07:31 So speaking of quite cool, it's quite cool that DigitalOcean is sponsoring this podcast.
07:35 And I want to just tell you guys quickly about them.
07:37 So they're a hosting company.
07:38 They've got data centers throughout the world.
07:41 And I think one of the cleanest, nicest ways to create a set of virtual servers and get
07:48 them up and running and configure them.
07:49 So if you want to create one that's already pre-configured for some infrastructure like
07:53 Disqus, you can just click a button and say, create me a Disqus virtual machine based on
07:57 Ubuntu, you know, whatever version you're looking for.
08:00 Or create a fresh one instead of PowerView like.
08:02 So they provide a lot of the infrastructure for us.
08:05 We actually pay for it, but they are the people we pay for getting you this podcast, which
08:10 is pretty awesome.
08:11 They've been really good.
08:12 We're happy customers.
08:13 And so if you want to be new happy customer, you can get $100 to try them out at pythonbytes.fm
08:20 slash digital ocean.
08:22 And that's for new customers.
08:24 Check it out.
08:25 And hopefully you create something cool and run it there.
08:28 Nice.
08:28 Hey, Brian, I got one that I think you're going to like.
08:31 Okay.
08:31 It's about testing.
08:32 I like testing.
08:33 Recently, I had a Talk Python episode on the release of PyPI and the inside story of how
08:39 that got revised.
08:40 And finally, pypi.org is the official thing.
08:43 Not like a weird, scary place that also matches to the same database.
08:46 So that's really awesome.
08:47 And I can't remember who said it on the show.
08:49 Sorry, because there were three folks.
08:51 But one of the libraries brought up was pretend, which is a stubbing library.
08:56 Neat.
08:56 Yeah.
08:57 So stubbing is like mocking, but it's different.
08:59 How's that, Brian?
08:59 You know more about that than I do.
09:01 Oftentimes in mocking, you want to like check the behavior of the code.
09:05 Like if you're interacting with some system that's not really there, you want to make sure
09:11 that you've called it in certain ways or you called it a certain number of times or the order
09:15 of calls, things like that.
09:17 And stubbing is really like, I just want to have my code be able to call something and have
09:23 the return value be like some pre-canned data.
09:26 So it's more about pre-canning than about the behavior.
09:30 I see.
09:31 So with mocking, maybe I'm going to say, I'm going to call this login API and I want
09:34 to make sure that it checks that I, my password is correct or something like that.
09:39 And that would be a mocking thing.
09:40 Whereas for stubbing, I just like, I need it to give back a password so it doesn't crash
09:44 because it's like a, you know, none type attribute error type of crash if, if nothing comes back.
09:51 So we got to give it something.
09:51 So let's create a stub to do that.
09:53 Stubbing also is a great way to do things like if there's error conditions, like when, when you're
09:59 connecting to a third party that goes out to some, if you want to like, if the server
10:04 crashes or something, you'll get an error code.
10:06 So how do you, how do you simulate that?
10:08 You can't go out and crash the server.
10:09 It's a really great way to, to pretend bad things happen.
10:14 Yeah.
10:14 So let me give people a sense of the API.
10:16 It's real simple.
10:16 So from pretend import stub, and then you just say stub and say like, here's a function name
10:22 equals some Lambda.
10:23 And now you can just start, you pass that around.
10:25 If somebody calls that function, it returns the value of the Lambda returns.
10:28 Done.
10:28 It's like, I don't think that it could really be simpler to be honest.
10:33 Right.
10:33 Well, that's one of the reasons why it's pretty cool is the, it's simpler than using mock.
10:37 Mock can do this too, but this is simpler.
10:39 Yeah.
10:40 Really nice.
10:40 Yeah.
10:41 You could probably use that with flask, couldn't you?
10:42 Yeah.
10:44 You could probably use it with flask.
10:46 Yep.
10:47 And my next topic is a surprise to me.
10:50 I forgot that I put this down.
10:51 But one of the things I was, I got out of PyCon was I got to sit next to one of the people
10:56 that at dinner that works on the flask project.
11:00 And he reminded me that they just went through and rewrote a bunch of the flask tutorial.
11:05 So I went through and took a look and yeah, the flask, the official flask tutorial is,
11:10 got a lot of updates.
11:11 It's, the code that goes along with it's been updated in the, just everything's been
11:16 simplified, updated.
11:17 It's a little cleaner.
11:19 It includes a, I don't know if it had this before, but it includes a section on testing,
11:23 which, highlights pytest, of course, and coverage, which is good.
11:28 And, one of the things I learned also with, with that discussion was flask is a part
11:33 of the, a pallets.
11:35 I'm not sure what their entity is really, but pallets is a collection of people that work
11:41 on a collection of projects.
11:42 And it's some pretty important stuff.
11:45 We've got flask, click.
11:47 It's dangerous.
11:48 I'm not sure what that is.
11:49 It's a request validation foundation of flask.
11:51 Oh, okay.
11:52 It's dangerous.
11:53 I already said that, but Jinja and Jinja2.
11:56 Is there a Jinja?
11:57 Is it just Jinja2?
11:59 I've not seen any Jinja in the wild lately, but there probably was at some point.
12:03 Okay.
12:03 But, and then, markup safe, which is an HTML markup safe string for Python library.
12:10 And then workzug, workzug, I don't know how to pronounce that.
12:15 Workzug.
12:16 With a V.
12:17 Everybody relies on a lot, some of these things, even if you don't use flask and the pallets,
12:23 project has a donate page now.
12:25 So if you, if you want to donate and you can donate through their donation page, pretty
12:31 neat.
12:31 It's really nice.
12:32 So we've had a lot of news around flask lately.
12:34 Flask went one Oh, a while ago.
12:36 I talked to one of the guys.
12:38 I'm sorry.
12:39 I'm so forgetting his name, but because I met so many people, but who is basically responsible
12:43 for that whole progression.
12:44 David, I believe.
12:46 Anyway, he's like, I know people think because the flask went one Oh, the week after you guys
12:51 did the zero over episode actually that was in the works for like a year.
12:56 I would love for you to be doing that, but, it was actually just a coincidence.
12:59 So anyway, glad to see you go one Oh, but yeah, this looks like flask is getting some renewed
13:05 love, which is good.
13:06 Yeah.
13:06 And, I didn't know click was by the same people.
13:09 I use click all the time.
13:10 So that's pretty neat.
13:12 Yeah.
13:12 For, creating CLIs.
13:14 Very nice.
13:15 All right.
13:16 So let's round it out with a little bit of an internals, look here as well.
13:20 I feel like a lot of stuff I'm covering this week is like deep in the guts with the Cython.
13:24 And if you're not doing Cython, you're doing regular Python, then you're operating in the
13:29 bytecode space.
13:30 So do you think people would be surprised if you told them that Python is compiled?
13:34 Yes.
13:34 A lot of them, I think they would, but it's not compiled to machine instructions or even
13:38 JIT compiled unless you're using PyPy, but it's compiled to bytecode.
13:42 So those PYC files, those are like the instructions to the Python virtual machine, not instructions
13:48 to your processor, but they're still compiled and there's still this bytecode.
13:52 And so understanding it's pretty interesting to just know like how the internals of Python
13:56 is working.
13:56 So there's this nice article called an introduction to Python bytecode.
14:01 So if you know, this bytecode concept is kind of new to you, or you just want to play around
14:05 with a little, check it out.
14:06 It's pretty accessible.
14:07 So I feel like there's a lot of hello world examples in my topics.
14:12 So back to hello.
14:13 So they have a function def hello, and it just prints the static string hello world, right?
14:19 It's just, okay, well, what does this actually mean?
14:21 And then they show you all the bytecode.
14:24 So, okay, we're going to load the global, which is the print function.
14:26 We're going to load the constant, which is hello world.
14:28 And we're going to call the function that is on the stack.
14:31 So CPython uses a stacked based virtual machine.
14:34 And so they like load these things under the stack.
14:36 If you have like a function that takes two arguments, they might load two arguments on
14:40 the stack and then call the function and things like that.
14:43 So that's how your Python source code gets into executable form.
14:48 And then these steps are actually sent down to the CPython runtime and this giant wild loop
14:54 with a switch statement.
14:55 That's like literally 3000 lines of C code that says, what's the bytecode?
14:59 Let's go do that.
14:59 You know, so it's pretty interesting.
15:01 And they talk about how you can take your Python code and look at this.
15:05 So you just import this as in disassembly and say this dot this and you pass it like a callable
15:11 or something like that.
15:12 And it'll show you the bytecode instructions that make it up.
15:15 So it's pretty nice.
15:15 New phone who dis you can't actually just look at the PYC files though, right?
15:20 No, they're like bytes.
15:21 That's why you need this dis thing.
15:23 I mean, I guess if you can like parse the bytes, you can, but it's not strings.
15:28 I don't think.
15:28 Okay.
15:29 Yeah.
15:30 I mean, the PYC files are basically the compiled steps from your hello world text into the
15:36 bytecode instructions in terms of bytes.
15:38 And then that's why those cache things are laying there.
15:42 The next time you hit that app, right?
15:44 It's just going to go and say, well, let's just load up that PYC so we don't have to reparse
15:48 and validate it.
15:49 Yeah.
15:49 This is kind of neat.
15:50 I'm going to definitely going to go check this out because I just want to know more about
15:54 this.
15:55 It seems like something I should know about, even though I probably don't need to on a
15:58 daily basis.
15:58 Yeah.
15:59 I mean, I'm not sure how helpful it is, but it's helpful in your conceptualization of how
16:03 stuff works.
16:03 I think.
16:04 Yeah, definitely.
16:05 Very cool.
16:05 Yeah.
16:06 A little bit deeper down into the, is that the red pill or the blue pill that takes
16:10 you farther down?
16:11 That's the red pill, right?
16:12 I don't remember.
16:13 Awesome.
16:13 Well, anyway, it's definitely worth checking out if you haven't played with bytecode before.
16:17 It's a really nice, simple way to get introduced to it.
16:19 Brian, you got anything, any other news you want to share with everyone this week?
16:22 I don't think so.
16:23 Do you?
16:24 No, I'm all out of news this week other than the ones I found.
16:27 So I just want to say thank you.
16:28 Thank you for being part of this episode and sharing everything with everyone.
16:31 Thank you.
16:32 Bye.
16:33 Yeah, you bet.
16:33 Bye.
16:35 Thank you for listening to Python Bytes.
16:37 Follow the show on Twitter via at Python Bytes.
16:39 That's Python Bytes as in B-Y-T-E-S.
16:42 And get the full show notes at pythonbytes.fm.
16:46 If you have a news item you want featured, just visit pythonbytes.fm and send it our way.
16:50 We're always on the lookout for sharing something cool.
16:53 On behalf of myself and Brian Okken, this is Michael Kennedy.
16:56 Thank you for listening and sharing this podcast with your friends and colleagues.
