Transcript #128: Will the GIL be obsolete with PEP 554?

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to

00:04 your earbuds. This is episode 128, recorded April 30th, 2019. I'm Michael Kennedy.

00:11 And I'm Brian Okken.

00:12 And this episode is brought to you by DigitalOcean. They're great supporters of the show.

00:16 Please check them out at pythonbytes.fm/digitalocean and get $100 free credit.

00:20 Brian, I am super excited.

00:22 Are you? What about?

00:23 We are on PyCon Eve.

00:26 Yeah, but when is this going to go out, though? I mean...

00:30 I think this is going out before PyCon. At least it'll be out during PyCon, if nothing else. We're

00:35 going to rush.

00:36 Yeah, I'm excited that I don't have to pack my banner thing this time.

00:40 I know. We're going to be at the JetBrains booth, and they're going to have all that stuff set up

00:44 for us. So we can just roll in like a normal attendee with a relatively small amount of gear

00:50 and tow. It'll be great.

00:50 Yeah, and I want people to show up at the 630 at the Thursday night thing, because I'm going to be

00:55 recording at 630 live. So stop by there.

00:57 Yeah, I feel like with the JetBrains stuff and just independent of that, we're going to be doing

01:02 a lot of live recording there for test and code, for Talk Python, and for Python Bytes. And we'll

01:07 try to get the word out about that, but it's going to be a lot of fun.

01:09 Yeah.

01:09 For sure. So let's just jump right in the first one here. I see this one as well within your

01:14 wheelhouse.

01:15 Yeah. I forgot about the order. Yes. There's an article from Adam Johnson called Solving Algorithmic

01:21 Problems in Python with pytest. And yes, pytest is definitely my wheelhouse. I like the highlighting

01:27 of this. Here's the idea. You've got coding challenges, especially algorithmic ones like

01:32 Project Euler or Advent of Code or other. There's lots of coding challenges projects. And doing

01:39 those goes through one example and shows how to translate the project description, the problem

01:45 description and the specification into some quick little tests. And they're like showing

01:51 it how small tests can be. Just testing one aspect of the answer. So the example goes through

01:57 basically, yeah, it's a little bit of a TDD practice thing of coming up with some tests to

02:02 test the test case, going through an exercise, translating all the specification into tests,

02:07 and then just working through it and looking at all the failures, failure cases, especially.

02:11 So one of the, his example was to just create a stub answer, just the function you're trying to

02:17 actually write, just have it return some constant and then build that up. And it's a little bit of a

02:23 TDD practice, but also just practice writing tests to do these sort of projects. And it's a short article

02:29 and I really liked it.

02:30 Yeah. It's a cool way to explore solving some problems with the whole Project Euler thing.

02:35 And it's also a cool way to get some experience working with pytest. And I feel like these types

02:41 of problems are pretty good entry level pytest type problems. It's not, well, we have the website and we

02:48 want to make sure the user can register and like, okay, well, how do I mock out the database call and

02:52 then like stub out the email service? You know, like all that complicated stuff, right? Where here,

02:57 it's like a pretty constrained problem, right? Like find the minimum path, you know, amongst these bridges

03:03 or, you know, whatever the problems, like these sort of mathematical algorithmic ones.

03:07 Yeah. And his example is like super easy. It's just returning the minimum value in a list as long

03:12 as it's positive. Nice. And it's easy to get your head around. And actually I love doing little

03:17 practice things like that. So cool. Yeah, I do too. And I'm a fan of Project Euler. So quite cool.

03:23 Now this next one that I have, I feel like maybe we haven't really discussed it enough,

03:28 but it has to do with Python packaging. Do you think? Oh yeah. We haven't covered packaging much.

03:33 No. So it's good that it's finally coming up on the show. Yeah. This one is interesting because it's

03:38 kind of like a, a meta packaging tool. So we've talked about the pyproject.toml. We've talked about

03:44 the pipfile.lockfile, requirements.txt, poetry, all these things, right? Well, this one that I found

03:51 or was actually sent to us by Dr. Igleby on Twitter, it's called DepHell as in dependency hell.

03:59 So it just comes out and it says it, right? Like there's flames on the logo. Come on.

04:03 So the idea is that it will let you work in these different modes and even automatically translate

04:09 between them. So one of them is like, Hey, we have a setup.py for managing this particular project.

04:17 I would rather have a pyproject.toml file that expresses the same thing. And you just run a

04:24 command line against that project and it'll generate the pyproject.toml based on the current setup in your

04:32 project. Oh, nice. Yeah. So if you want to say, switch it to poetry, you can run a command. It'll

04:36 switch to poetry. If you want to switch it over to pipenv, fine. You run that and it'll do it. So

04:41 that's one of the things it does. It's pretty cool in that it'll let you use all these different

04:46 things. It doesn't really try to replace them, but it more tries to like tie them together,

04:50 right? Like I've grabbed a project, but maybe it's not in my, the tooling that I like. It's pretty easy

04:56 to extend. It's really quite nice. I like this ability to translate between them. It also has some

05:01 of the features of like pipenv. So it'll create like a shell. It'll install your command line

05:07 utilities into their own isolated virtual environment. Like pytest could be in its own

05:13 environment that has nothing to do with your project, but is available to run against your

05:18 project. Things like that. Yeah. Nice. Yeah. So yeah, not too much else to say, I guess it's based on

05:24 ASyncIO. So that means it only supports modern Python. That's pretty awesome. So it's super fast. All of its

05:31 network calls and stuff are made asynchronously and yeah, it's pretty good. I think something like this

05:36 would be great for like a project that has, has a maintainer that doesn't care about the pen for

05:42 something is fine with requirements or whatever they're using, but there's a couple maintainers that

05:47 really like pipenv and want to want to have the pip file around and the lock file. And yeah, that's

05:53 interesting. Like you could have one of them that's the source of truth and then use this tool to generate

05:58 the others if you'd rather work that way. Right. Yeah. Yeah. You could even do this part of a

06:02 automated build, right? Delete the pip file and then recreate it as part of the build and then

06:08 check that back in. Who knows? Yeah. Okay. Cool. Yeah. Yeah. Pretty cool. This next one that you

06:12 found is a bit of a rant, huh? Yes, definitely. Mike Croucher, which, I'm not sure what he does.

06:17 I think he works. He writes a lot of great articles and come across his name every once in a while. So

06:22 thanks Mike. This rant is a, he even says it in the name, Python rant from Foo import star is bad,

06:29 but basically just import star is bad. And I thought this was just done that nobody did this anymore,

06:36 but I actually see quite a bit of code that still has this in it. And I was actually in looking for

06:43 a different blog posts. There's some blog posts that have great advice in Python, but the example code has

06:49 import star. And I'm not going to point people to that because it's just a bad practice. And his

06:56 example is, for instance, is, the square root function SQRT. If you just have like result

07:03 equals square root of minus one, what does that mean? You don't know what it means because you don't know

07:09 where that came from. And there's some really confusing examples that he's showing how it may have

07:15 been from the math library. It may have been from numpy or cmath or scipy or simpy. All of them have

07:21 the same function name and we like namespaces, but when you use import star, you throw away the namespace

07:28 ability. You just import everything into your current namespace. So don't do that.

07:33 Yeah. I like the hat tip to a full metal jacket. This is my rant on import star. There are many like

07:39 it, but this one is mine. This one is my own. You know, I've, I totally, like this as well.

07:45 I'm a big fan of having like super explicit namespaces to like really tell where something

07:51 comes from. In fact, I typically try to shy away from, from thing, import something.

07:57 Yeah. Not just import star, just even that it's more like import this module than module dot

08:03 function, module dot class. You know, sometimes if it's like deeply nested, I might do import thing as

08:10 like just the last part of that name or, but you know, something to give you a hint, like where the

08:14 heck did this come from?

08:15 Right. And like, for instance, the, numpy is the convention is to import numpy as np.

08:21 Because numpy is really long to type.

08:24 I know. However, you're using a ton of it. And so, yeah, there, there are conventions. And if you

08:30 can notice the conventions, you follow those.

08:33 Yeah, absolutely.

08:33 I'm with it too. So sometimes when I'm refactoring some code or trying to understand it,

08:38 it's frustrating when somebody has a bunch of from library import, like five different functions.

08:44 Yeah. And you know, it's, it's not terrible if you've got something like visuals to do your code,

08:49 or you've got PyCharm and you can like go to definition or you hover over it and it'll say more.

08:53 But if you see it in a blog post or you see it printed or you see it in like a gist or

08:58 somewhere that doesn't have like understanding of the environment, then you're like, okay,

09:02 well, what is that? Right? Like, so just, you know, think about it.

09:05 Yeah. I actually kind of wish they deprecated, but that's probably never going to happen.

09:08 Yeah, probably not. I talked about digitalization at the top. Let me just tell you about something

09:13 that's new and cool. Brian, are you familiar with GitHub actions?

09:16 I just heard a little bit about them.

09:19 Yeah. Same. I haven't really done anything with them, but it's basically GitHub actions are like a

09:23 series of workflows that can be triggered when you do like a push to a repository, you create a release or

09:30 you create an issue, right? And it runs a series of actions that then, you know, can kick off CI or do

09:36 other sorts of tests. Well, digital ocean has come out with GitHub actions for digital ocean. So you can do

09:43 really cool things like I would like to upgrade my Kubernetes cluster anytime I push to the release

09:50 branch or the master branch and my GitHub repo and just bake that straight into GitHub, right? Just the

09:56 fact of you doing a commit or like, you know, the integration test passing or whatever, you know,

10:01 whatever you want to do with the GitHub actions, you can set that up. So there's special GitHub actions

10:04 for digital ocean and yeah, check them out. Really cool stuff. They have on their blog posts,

10:10 they have something about working with the Kubernetes service there and then using the

10:15 GitHub actions to sort of keep it always up to date. So check them out at pythonbytes.fm slash

10:19 digital ocean, get a hundred dollars free credit for new users. And I feel like GitHub actions are

10:24 something I just want to learn in general. How about you? Definitely. Yeah. It seems like something

10:27 that could help out with workflows. Yeah, absolutely. Cool. Well, this next one that I want to talk

10:32 about is not super new, but I don't know how we've gone this long really without talking about it.

10:38 So Dask. So Dask is a way to natively scale Python. And when I first thought about it,

10:45 I first heard about it, I thought, okay, well, Dask is like this thing that takes data science

10:49 workloads and runs among clusters. And the reason I didn't get super excited is like, well, I don't do

10:54 that much data science and I don't have enough that require clusters to run. Now they're usually

10:59 pretty small little graphy things or something if I'm doing any data science. However, I recently had

11:04 Matthew Rocklin, who's behind Dask on Talk Python in episode 207. And we talked a lot about it and

11:10 there's actually some really cool stuff. And I think more applicability to more people than I first

11:15 realized. So basically the idea is like Dask will take the NumPy, SciPy sort of stack and scale it out.

11:23 All right. So you have NumPy, you have Pandas, you have side kit, learn code, all that.

11:27 So there's Dask versions of like NumPy arrays and Panda data frames. So there's like Dask data frames.

11:34 And what you can do is just work with those basically the same API, but instead of working

11:39 just locally, it will work with them on the cluster. So suppose you have like three terabytes of data you

11:45 need to process and it can't fit into RAM. So you can't just load it up into a NumPy array or a Panda data

11:50 frame, but you can tell Dask to process and it'll share it across the cluster and do all the work and

11:55 the computation and the cross server communication that you need. Isn't that cool?

11:59 That is neat.

11:59 So it sounds really neat. And that workload, like to me, doesn't really help that much because I don't

12:03 have to do a whole lot with that. I know some people that'll be super valuable for, but you can

12:07 also just run Dask locally on your machine and it'll create like a little mini cluster that runs locally

12:13 and it'll use like threads and processes and whatnot. And it'll even let you process more

12:18 data locally than will fit into RAM and do like, you know, lazy loading and all sorts of interesting

12:23 stuff there. So pretty cool. It even lets you escape the GIL. So you get better like parallels

12:29 and even on your own computer and it runs arbitrary Python code, not just NumPy and Pandas, even though

12:35 that is main, that's its main use.

12:36 That's actually pretty darn cool.

12:38 Yeah, that's what I thought. So I decided to make it at one of our topics for today.

12:41 Yeah. The large file stuff, I definitely hit that occasionally.

12:45 Yeah. And I don't really want to think about it just for special cases, but being able to use

12:50 running it under Dask might just speed it up. So that's cool.

12:53 Yeah. You basically just create like a Dask client or something and it'll like locally create a little

12:57 server cluster that'll process it all. It's pretty cool.

13:00 Nice.

13:00 Or run on a thread pool or something like that. Last thing I thought that was kind of interesting,

13:04 maybe do this for me. Click on the Dask thing to open it up and just go to the bottom.

13:08 Notice the supported by there?

13:10 Wow.

13:11 Isn't that cool? Supported by the NSF, supported by NVIDIA, supported by DARPA, Anaconda Inc.,

13:17 things like that. So here's a really interesting example of not just a project that's cool,

13:22 but a project, an open source project that's like really supported by, you know, some neat

13:28 companies or organizations. So anyway, I just kind of thought that was a cool thing that jumped

13:33 at me as well as like the kind of the proper support this project's getting.

13:37 Oh, I definitely need to check this out. Yeah.

13:39 Yeah.

13:39 Neat.

13:39 Yeah. Pretty cool. It might tie in with graphing.

13:42 Yeah. Actually, there's some pretty graphs on the Dask website. Yeah. But if you don't want to,

13:48 I don't know how they're using it, but it's possible to do animations even within Matplotlib.

13:53 Nice.

13:54 And I'm highlighting an article by Parul Pandey, sorry if I'm getting that name wrong,

13:59 called Animations in Matplotlib. I thought we'd already covered this, but we haven't yet.

14:04 Just the fact that you can do animations. And I guess I hadn't realized when I first started

14:09 working with plots in Python that Matplotlib did it. And you can do some, lots of different ways you

14:15 can simulate or do animations within Matplotlib. And the top picture is just, is this, of the article,

14:23 is this raindrop simulation. And I could just sit and watch this for like an hour.

14:29 I was thinking the same thing. It's like the equivalent of white noise, but visual. It's just

14:34 like, yeah, it really does look like raindrops hitting, you know, a little pond or a puddle

14:39 or something. It's quite cool.

14:40 Right. So it has these random circles that appear dark. And then as they get bigger,

14:44 they get lighter and then they eventually disappear. And that just happens all over the

14:49 page. And it's pretty neat. But that's all using Matplotlib animations. And there is a link to the

14:56 source code for that. But the tutorial goes through and talks about, there's a couple different ways to

15:01 do animations. And the author prefers funk animation and has a tutorial for animating a sine wave.

15:08 The confusing part of that, to me, was that the x-axis doesn't really mean anything at that point.

15:13 Because the sine wave keeps moving.

15:15 Right.

15:16 But it's a pretty small, concise example of how animations work.

15:20 Yeah, it's super easy. So basically, you create a figure, set it up in Matplotlib.

15:24 You create an initialize function that sets the data however it's going to be. And then you have an

15:29 animate function that's given a frame and it just computes the change and, you know, sets it again.

15:34 Yeah, it's quite nice.

15:35 Yeah, just a few lines of code. She goes on to, I think it's a she, goes on to talk about live

15:41 updates. So if you've got a graph or a plot that is using data that is changing, you can live update

15:47 those. Animating, turning a 3D plot. And that's really pretty.

15:51 Yeah, yeah. There's a bunch of cool graphs here.

15:53 Yeah.

15:54 Yeah, I could see if I had stuff to graph, I would be all over this.

15:56 I guess there's a third-party package called celluloid that makes some of the animations

16:01 a little bit more concise. So she gives some examples of that too.

16:04 Cool. Yeah, that's a good one.

16:05 All right. The last one here for us is pep554, multiple interpreters or sub-interpreters in the

16:14 standard lib.

16:14 Oh, wow.

16:15 So this is kind of meta and interesting and possible. So this, I don't believe is approved

16:20 yet. This is potentially, possibly coming. So I don't think it's out. Yeah, proposed in Python 3.9.

16:27 And we'll just see if it's, yeah, I don't see whether it's approved or not. But, you know,

16:32 maybe, probably in 3.9 coming is this pep554, which allows for multiple sub-interpreters in the

16:38 standard lib. So apparently CPython already had this capability to have multiple sub-interpreters

16:45 run, but it was never exposed. Right? So deep, deep down, there's some module you could use. And

16:50 here's like a public API on top of that.

16:52 Okay.

16:53 So why do you care about it? Well, it says the proposal introduces the standard lib interpreters

16:59 module, like import interpreters. And currently it's provisional. And it basically exposes this

17:06 core functionality as sub-interpreters already provided by the C API along with new functionality.

17:11 And here's the most important part for sharing data between interpreters. So the idea is you're

17:16 going to set up some kind of channel, which is like a queue or a name pipe or something like that to pass

17:21 data back. Like I can't take an object I've created in one part of my program and share it with one of

17:26 these sub-interpreters. I've got to like JSON serialize it or pickle it and then bring it back or like

17:31 there's no data sharing, which is really interesting for isolation. So the main use case of this,

17:36 are well, one running code in isolation. Like if you want to work within your process and you don't

17:41 want to kick off another process, say with multi-processing or something, you can still

17:45 run code that you don't necessarily trust with some restrictions here because it won't have access

17:51 to your memory structures or anything like that. It'll just be like a little isolated Python,

17:56 but you don't have to result to multi-processing.

17:59 Oh, that's cool.

18:00 So that's kind of cool. Maybe plugin systems or something like that, or maybe even incompatible

18:04 versions of modules. Like maybe, you know, I run into this with doc opt all the time for

18:10 some reason, like MailChimp will only use this version, but something else requires another

18:14 version. Like one's less than something and some other has to be greater than that. You know what I

18:19 mean? So maybe you could run like that part of the code in one of these sub-interpreters and have it run

18:24 on a different version. I'm not sure that might take a little bit of juggling, but another one that the one that stood out to me, I think is pretty interesting here is the guild, the global interpreter law.

18:34 is there because, you know, basically the way people perceive it is, is it blocks parallelism.

18:41 But the reason it blocks parallelism is around memory management of shared objects, right? So any PI object

18:47 you have, it has to have reference counting and whatnot to keep its memory managed. Well, these sub-interpreters

18:53 don't share objects, so they don't share the gill, which means you could have like true computational

18:59 parallelism in your code. So they all have like their own GIL. Effectively, yes. Yeah, exactly. So it's still the gill, but if you have a bunch of them, then it doesn't really matter. And you don't have the overhead of multiple processes or passing data from multiple processes or all that.

19:14 The case that I'm thinking about is people that have tried to write their own little IDE or even a big IDE

19:20 in Python to run Python. You've got this issue that you've still only have one gill, so you've got to do

19:27 launch another thread, you have to have another task or something.

19:31 And this would allow something like that to be easier.

19:34 Yeah, so I don't actually, yeah, I agree. I don't know what actually would come out of this,

19:37 but it looks like it has some interesting potential.

19:39 And it's also interesting that it basically just formalizes what was already there.

19:44 So that's pretty cool, too.

19:45 Yeah.

19:45 Awesome. All right. Well, that's it for our main topics.

19:48 You got any extra stuff you want to throw out there?

19:50 Other than I'm just super excited about PyCon.

19:52 Yes, it's PyCon Eve. I'm so excited.

19:54 It's going to be good. I'm looking forward to seeing you and everyone in Cleveland.

19:58 When are we going to be around? Do you remember our times?

20:01 Middle of the day.

20:03 Yeah.

20:04 After lunch. So come try to find us.

20:07 Yeah, we won't be at the booth all the time. We're going to be doing other events like open

20:11 spaces and live recordings and other places and maybe even attend to talk. Who knows?

20:15 The times that we will be there, it should be posted on the booth. So there'll be at least

20:20 three hours each day that we're doing something interesting there that you can come by and see us.

20:24 Yep. I'm getting stickers.

20:25 And stickers. Definitely find us at PyCon.

20:27 How about you? You've got some big news.

20:29 Yeah, I got some big news.

20:30 Big news is my iOS app is finally out after negotiating, let's call it, with the App Store folks

20:38 who were better than Google Play, but still it was quite the back and forth to get everything right.

20:44 So finally, the Talk Python training iOS app is out.

20:47 Check it out at training.talkpython.fm/apps. If you install it, log in, you can get two free courses in addition to the ones you might already have.

20:55 So that's pretty cool.

20:55 Yep. I installed it this morning.

20:57 You did? You're already on top of it?

20:59 Yeah. Yeah. It looks great.

21:00 Right on. Thanks. That's very cool.

21:02 And then lastly, you know, like I've said, our listeners are awesome. Anytime we say, here's something that's kind of unique, they're like, and the five other amazing ones.

21:10 So here's one called Blessings. We talked about Bullet, and we talked about Cooked Input, and Blessings is kind of in that realm of making terminal input nicer.

21:20 And this is also output. It's not exactly the same, but Blessings lets you do things like bold your terminal output and move the cursor around and do all sorts of cool stuff.

21:29 So this is from Eric Rose, and Preycin Daniel sent this over and said, hey, you should check this out in addition to the ones you're talking about.

21:35 So here it is. It looks pretty cool.

21:36 Ooh, I have the exact use case for this in mind. So yay.

21:39 Nice. What are you going to do with it?

21:41 I want to do, like I just got finished with reading the TDD by example.

21:45 Yeah, I know. You would have thought I would have learned that beforehand.

21:49 But yeah, I finally read it. And one of the things is a to-do list that is bold for stuff you're working on and like not.

21:57 Nice. Yeah.

21:58 Anyway, he uses that.

21:59 Super cool. Yeah, there it is. Awesome. All right. We have a joke coming in from Topher Chung.

22:04 Do you want to do it or should I do it?

22:06 Oh, you do it.

22:07 All right. Knock, knock. Race condition.

22:09 Who's there?

22:10 Knock, knock. Race condition. Who's there?

22:16 Oh, perfect.

22:16 Man, these never get old.

22:18 We can do...

22:19 I am starting to notice that the pie joke well is starting to run a little dry. We've been emptying it a lot. So people have to start sending in their jokes. That was a good one. Thank you, Topher.

22:29 Yeah, thanks.

22:30 All right. Well, Brian, we shall reconvene in Cleveland.

22:33 Yeah, talk to you there.

22:35 All right. See you, everyone there. See you, Brian.

22:36 Bye.

22:37 Thank you for listening to Python Bytes.

22:38 Follow the show on Twitter via at Python Bytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.