Transcript #128: Will the GIL be obsolete with PEP 554?

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.

00:05 This is episode 128, recorded April 30th, 2019. I'm Michael Kennedy.

00:11 And I'm Brian Okken.

00:12 And this episode is brought to you by DigitalOcean. They're great supporters of the show.

00:16 Please check them out at pythonbytes.fm/digitalocean. Get $100 free credit.

00:20 Brian, I am super excited.

00:22 Are you? What about?

00:23 We are on PyCon Eve.

00:26 Yeah, but when is this going to go out though?

00:29 I mean, I think this is going out before PyCon.

00:33 At least it'll be out during PyCon.

00:34 If nothing else, we're going to rush.

00:36 Yeah, I'm excited that I don't have to pack the, my banner thing this time.

00:40 I know we're going to be at the JetBrains booth and they're going to have all that stuff set up for us.

00:45 We can just roll in like a normal attendee with a relatively small amount of gear and tow.

00:50 It'll be great.

00:50 Yeah.

00:50 And I want people to show up at the 630 at the Thursday night thing, because I'm going I'm going to be recording at 630 live, so stop by there.

00:58 - Awesome, yeah, I feel like with the JetBrains stuff and just independent of that, we're going to be doing a lot of live recording there for test and code, for Talk Python, and for Python Bytes.

01:06 And we'll try to get the word out about that, but it's going to be a lot of fun.

01:09 - Yeah, yep. - For sure.

01:11 So let's just jump right in the first one here.

01:13 I see this one as well within your wheelhouse.

01:15 - Yeah, I forgot about the order.

01:17 Yes, there's an article from Adam Johnson called "Solving Algorithmic Problems in Python with pytest." And yes, pytest is definitely my wheelhouse.

01:26 I like the highlighting of this.

01:28 Here's the idea.

01:29 You've got coding challenges, especially algorithmic ones like Project Euler or Advent of Code or other, there's lots of coding challenges projects.

01:38 And doing those goes through one example and shows how to translate the project description, the problem description and the specification into some quick little tests.

01:50 And they're like showing it how small tests can be.

01:53 just testing one aspect of the answer.

01:56 So the example goes through basically, yeah, it's a little bit of a TDD practice thing of coming up with some tests to test the test case, going through an exercise, translating all the specification into tests, and then just working through it, looking at all the failures, failure cases, especially, so one of his example was to just create a stub answer, just the function you're trying to actually write, just have it return some constant, and then build that up.

02:22 and it's a little bit of a TDD practice, but also just practice writing tests to do these sort of projects.

02:28 And it's a short article and I really liked it.

02:31 - Yeah, it's a cool way to explore solving some problems with the whole Project Healer thing.

02:35 And it's also a cool way to get some experience working with pytest.

02:40 And I feel like these types of problems are pretty good entry-level pytest-type problems.

02:46 It's not, well, we have the website and we want to make sure the user can register.

02:50 and like, okay, well, how do I mock out the database call and then like stub out the email service, you know, like all that complicated stuff, right?

02:57 Where here it's like a pretty constrained problem, right?

03:00 Like find the minimum path, you know, amongst these bridges or, you know, whatever the problem is, like these sort of mathematical algorithmic ones.

03:07 - Yeah, and his example is like super easy.

03:09 It's just returning the minimum value in a list as long as it's positive.

03:13 - Nice.

03:14 - And it's easy to get your head around.

03:15 And actually I love doing little practice things like that.

03:18 Cool.

03:19 Yeah, I do too, and I'm a fan of Project Healer, so quite cool.

03:24 Now, this next one that I have, I feel like maybe we haven't really discussed it enough, but it has to do with Python packaging.

03:30 Do you think?

03:31 - Oh yeah, we haven't covered packaging much.

03:33 - No, so it's good that it's finally coming up on the show.

03:36 This one is interesting because it's kind of like a meta packaging tool.

03:41 So we've talked about the pyproject.toml, we've talked about the pipfile.lockfile, requirements.txt, poetry, all these things, right?

03:49 Well, this one that I found, or was actually sent to us by Dr. Igelby on Twitter, it's called DepHel, as in dependency hell.

03:59 So it just comes out and it says it, right?

04:01 Like there's flames on the logo, come on.

04:03 So the idea is that it will let you work in these different modes and even automatically translate between them.

04:10 So one of them is like, hey, we have a setup.py for managing this particular project, I would rather have a pyproject.toml file that expresses the same thing, and you just run a command line against that project, and it'll generate the pyproject.toml based on the current setup in your project.

04:32 - Oh, nice.

04:33 - Yeah, so if you wanna say switch it to Poetry, you can run a command, it'll switch it to Poetry.

04:37 If you wanna switch it over to pipenv, fine, you run that, and it'll do it.

04:41 So that's one of the things it does.

04:43 It's pretty cool in that it'll let you use all these different things.

04:47 It doesn't really try to replace them, but it more tries to tie them together.

04:50 Right?

04:51 Like I've grabbed a project, but maybe it's not in the tooling that I like.

04:55 It's pretty easy to extend.

04:57 It's really quite nice.

04:58 I like this ability to translate between them.

05:00 It also has some of the features of like pipenv, so it'll create like a shell.

05:05 It'll install your command line utilities into their own isolated virtual environment like pytest could be in its own environment that has nothing to do with your project, but is available to run against your project, things like that.

05:19 Yeah.

05:20 Nice.

05:20 Yeah.

05:20 So yeah, not too much else to say, I guess it's based on async I/O.

05:25 So that means it only supports modern Python.

05:27 That's pretty awesome.

05:28 So it's super fast.

05:30 All of its network calls and stuff are made asynchronously and yeah, it's pretty good.

05:35 I think something like this would be great for like a project that has, as a maintainer that doesn't care about PIPenv or something is fine with requirements or whatever they're using.

05:46 But there's a couple of maintainers that really like PIPenv and wanna have the pip file around and the lock file.

05:52 - Yeah, that's interesting.

05:53 Like you could have one of them that's the source of truth and then use this tool to generate the others if you'd rather work that way, right?

06:01 Yeah, you could even do it as part of a automated build, right, delete the pip file and then recreate it as part of the build and then check that back in.

06:08 Yeah.

06:09 Okay.

06:09 Cool.

06:10 Yeah.

06:10 Yeah.

06:11 Pretty cool.

06:11 This next one that you found is a bit of a rant, huh?

06:13 Yes, definitely.

06:14 Mike Croucher, which I'm not sure what he does.

06:17 I think he works.

06:18 He writes a lot of great articles and come across his name every once in a while.

06:22 So thanks, Mike.

06:23 This rant is, he even says it in the name, Python rant from foo import star is bad.

06:29 But basically just import stars bad.

06:31 And I thought this was just done that nobody did this anymore.

06:36 but I actually see quite a bit of code that still has this in it and I was actually in looking for different blog posts there's some blog posts that have great advice in Python, but the example code has import star and I'm not going to point people to that because it's just a bad practice and his example is for instance is the square root function sqrt if you just have like result equals square root of minus one. What does that mean?

07:07 You don't know what it means because you don't know where that came from and There's some really confusing examples that he's showing how it may have been from the math library It may have been from numpy or cmath or scipy or sympy All of them have the same function name and we like namespaces But when you use import star you throw away the namespace ability you just import everything into your current namespace So don't do that. Yeah, I like the hat tip to a full metal jacket. This is my rant on import star There are many like it, but this one is mine This one is my own. You know, I've I totally like this as well I'm a big fan of having like super explicit name spaces to like really tell where something comes from. In fact, I Typically try to shy away from from thing import something. Yeah, not just import star just even that It's more like import this module than module dot function module dot class, you know Sometimes if it's like deeply nested I might do import thing as like just the last part of that name or but you know something To give you a hint like where the heck did this come from?

08:15 right and like for instance the numpy is the convention is to import numpy as NP if numpy is really long at a time. I Know however, you're using a ton of it. And so yes Yeah, there are conventions and if you notice the conventions, you follow those.

08:33 Yeah, absolutely.

08:33 I'm with it too. So sometimes when I'm refactoring some code or trying to understand it, it's frustrating when somebody has a bunch of from library import, like five different functions.

08:44 Yeah. And you know, it's not terrible if you've got something like Visual Studio Code, or you've got PyCharm, and you can like go to definition or you hover over it and it'll say more.

08:53 But if you see it in a blog post or you see it printed or you see it in like a gist or somewhere that doesn't have like understanding of the environment, then you're like, okay, what is that?

09:03 Right?

09:03 Like, so just, you know, think about it.

09:04 Yeah, I actually kind of wish they deprecated, but that's probably never gonna happen.

09:08 Yeah, probably not.

09:09 I talked about digital ocean at the top.

09:12 Let me just tell you about something that's new and cool.

09:14 Brian, are you familiar with GitHub actions?

09:16 I just heard a little bit about them.

09:19 Yeah, same.

09:19 I haven't really done anything with them, but it's basically GitHub Actions are like a series of workflows that can be triggered when you do a push to a repository.

09:29 You create a release or you create an issue, and it runs a series of actions that then can kick off CI or do other sorts of tests.

09:37 Well, DigitalOcean has come out with GitHub Actions for DigitalOcean.

09:42 So you can do really cool things like, I would like to upgrade my Kubernetes cluster anytime my push to the release branch or the master branch and my GitHub repo and just bake that straight into GitHub.

09:55 Right, just the fact of you doing a commit or like the integration test passing or whatever.

10:01 You know, whatever you wanna do with the GitHub actions, you can set that up.

10:03 So there's special GitHub actions for DigitalOcean and yeah, check them out, really cool stuff.

10:09 They have on their blog post, they have something about working with the Kubernetes service there and then using the GitHub actions to sort of keep it always up to date.

10:17 So check them out at pythonbytes.fm/digitalocean.

10:20 Get $100 free credit for new users.

10:23 And I feel like GitHub actions are something I just want to learn in general.

10:25 How about you?

10:26 Definitely.

10:27 Yeah.

10:28 It seems like something that could help out with workflows.

10:29 Yeah, absolutely.

10:30 Cool.

10:31 Well, this next one that I want to talk about is not super new, but I don't know how we've gone this long really without talking about it.

10:38 So Dask.

10:39 So Dask is a way to natively scale Python.

10:43 And when I first thought about it, I first heard about it, I thought, "Okay, well, Dask is like this thing that takes data science workloads and runs them on clusters." And the reason I didn't get super excited was like, "Well, I don't do that much data science and I don't have enough that require clusters to run." Now, they're usually pretty small, little graphy things or something if I'm doing any data science.

11:03 However, I recently had Matthew Rocklin, who's behind Dask, on TalkByThon at episode 207, and we talked a lot about it.

11:10 And there's actually some really cool stuff.

11:12 And I think more applicability to more people than I first realized.

11:15 So basically, the idea is like, Dask will take the NumPy, SciPy sort of stack and scale it out.

11:24 Alright, so you have NumPy, you have pandas, you have scikit-learn code, all that.

11:28 So there's Dask versions of like NumPy arrays, and panda data frames.

11:33 So there's like Dask data frames.

11:35 And what you can do is just work with those, basically the same API, but instead of working just locally, it will work with them on the cluster.

11:42 So suppose you have like three terabytes of data you need to process and it can't fit into RAM, so you can't just load it up into a NumPy array or a Panda DataFrame, but you can tell Dask to process and it'll share it across the cluster and do all the work and the computation and the cross-server communication that you need.

11:58 Isn't that cool?

11:59 - That is neat.

12:00 - So it sounds really neat and that workload, like to me, doesn't really help that much 'cause I don't have to do a whole lot with that.

12:04 I know some people that'll be super valuable for, but you can also just run Dask locally on your machine And it'll create a little mini cluster that runs locally.

12:13 And it'll use threads and processes and whatnot.

12:17 And it'll even let you process more data locally than will fit into RAM and do lazy loading and all sorts of interesting stuff there.

12:24 So pretty cool.

12:25 It even lets you escape the gill.

12:27 So you get better parallels even on your own computer.

12:31 And it runs arbitrary Python code, not just NumPy and Pandas, even though that's its main use.

12:36 That's actually pretty darn cool.

12:38 Yeah, that's what I thought.

12:39 So I decided to make it one of our topics for today.

12:41 >>Yeah, the large file stuff, I definitely hit that occasionally.

12:45 And I don't really want to think about it just for special cases, but being able to use running it under Dask might just speed it up.

12:53 >>Yeah, you basically just create a Dask client or something and it'll locally create a little server cluster that'll process it all.

12:59 It's pretty cool.

13:00 Or run on a thread pool, something like that.

13:03 Last thing I thought that was kind of interesting, maybe do this for me.

13:05 Click on the Dask thing to open it up and just go to the bottom.

13:09 Notice the supported by there?

13:10 - Wow.

13:11 - Isn't that cool?

13:12 Supported by the NSF, supported by NVIDIA, supported by DARPA, Anaconda Inc., things like that.

13:18 So here's a really interesting example of not just a project that's cool, but an open source project that's really supported by some neat companies or organizations.

13:30 So anyway, I just kind of thought that was a cool thing that jumped at me as well as the proper support this project's getting.

13:37 I definitely need to check this out. Yeah. Yeah, pretty cool. It might tie in with graphing. Yeah, actually there's some pretty graphs on the Yeah, the desk website. Yeah, but if you don't want to I don't know how they're using it But it's possible to do animations even within matplotlib nice and I'm highlighting an article by Carol Pandy, sorry if I'm getting that name wrong called animations in matplotlib I thought we'd already covered this but we haven't yet Just the fact that you can do animations and I guess I hadn't realized when I first started working with plots in Python that Matt Plotlib did it and you can do some lots of different ways you can simulate or do animations within.

14:18 Matt Plotlib in the picture is just is this of the article.

14:24 Is this a raindrop simulation.

14:27 And I could just sit and watch this for like an hour.

14:29 I was thinking the same thing. It's like the equivalent of white noise but visual.

14:34 It's just like, yeah, it really does look like raindrops hitting a little pond or a puddle or something. It's quite cool.

14:40 Right. So it has these random circles that appear dark and then as they get bigger, they get lighter and then they eventually disappear and that just happens all over the page.

14:50 And it's pretty neat.

14:52 But that's all using Matplotlib animations.

14:54 And there is a link to the source code for that.

14:57 But the tutorial goes through and talks about, there's a couple different ways to do animations, and the author prefers funk animation, and has a tutorial for animating a sine wave.

15:08 The confusing part of that, to me, was that the x-axis doesn't really mean anything at that point, because the sine wave keeps moving.

15:15 But it's a pretty small, concise example of how animations work.

15:19 Yeah, it's super easy.

15:21 you create a figure, set it up in Matplotlib, you create an initialized function that sets the data however it's gonna be, and then you have an animate function that's given a frame and it just computes the change and sets it again, yeah, it's quite nice.

15:35 - Yeah, just a few lines of code.

15:37 She goes on to, I think it's a she, goes on to talk about live updates, so if you've got a graph or a plot that is using data that is changing, you can live update those.

15:48 animating turning a 3D plot and that's really pretty.

15:52 >> Yeah. There's a bunch of cool graphs here.

15:53 I could see if I had stuff to graph, I would be all over this.

15:57 >> I guess there's a third party package called celluloid that makes some of the animations a little bit more concise.

16:02 So she gives some examples of that too.

16:04 >> Cool. Yeah, that's a good one.

16:06 The last one here for us is PEP554, multiple interpreters or sub-interpreters in the standard lib.

16:14 >> Oh, wow.

16:15 >> So this is meta and interesting, possible so this I don't believe is approved yet this is potentially possibly coming so I don't think it's out yeah proposed in Python 3 9 and we'll just see if it's I don't see whether it's approved or not but you may be probably in 3 9 coming as this have 5 5 4 which allows for multiple sub interpreters in the standard lip so apparently see Python already had this capability to have multiple sub interpreters run but it was never exposed So deep, deep down there's some module you could use, and here's a public API on top of that.

16:53 So why do you care about it?

16:54 Well, it says the proposal introduces the standard lib interpreters module, like import interpreters, and currently it's provisional.

17:03 And it basically exposes this core functionality of sub-interpreters already provided by the C API, along with new functionality.

17:11 Here's the most important part for sharing data between interpreters.

17:15 So the idea is you're going to set up some kind of channel, which is like a queue or a name pipe or something like that to pass data back.

17:21 Like I can't take an object I've created in one part of my program and share it with one of these sub interpreters.

17:27 I've got to like JSON serialize it or pickle it and then bring it back.

17:30 Or like, there's no data sharing, which is really interesting for isolation.

17:34 So the main use cases of this are well, one running code in isolation.

17:38 Like if you want to work within your process and you don't want to kick off another process, say with multi-processing or something, you can still run code that you don't necessarily trust with some restrictions here because it won't have access to your memory structures or anything like that.

17:54 It'll just be like a little isolated Python, but you don't have to result to multi-processing.

17:59 - Oh, that's cool.

18:00 - So it's kind of cool, maybe plug-in systems or something like that, or maybe even incompatible versions of modules, like maybe, you know, I've run into this with DocOpt all the time for some reason, Like MailChimp will only use this version, but something else requires another version, like one's less than something, and some other has to be greater than that.

18:18 You know what I mean?

18:19 So maybe you could run like that part of the code in one of these sub-interpreters and have it run on a different version.

18:26 I'm not sure, that might take a little bit of juggling.

18:28 But another one, the one that stood out to me, I think is pretty interesting here, is the GIL, the Global Interpreter Lock, is there because, you know, basically the way people perceive it is, is it blocks parallelism.

18:42 But the reason it blocks parallelism is around memory management of shared objects, right?

18:46 So any Pi object you have, it has to have reference counting and whatnot to keep its memory managed.

18:51 Well, these sub-interpreters don't share objects, so they don't share the GIL, which means you could have true computational parallelism in your code.

19:02 So they all have their own GIL?

19:04 Effectively, yes.

19:05 Yeah, OK.

19:05 Exactly.

19:06 So it's still the GIL, but if you have a bunch of them, then it doesn't really matter.

19:09 and you don't have the overhead of multiple processes or passing data from multiple processes or all that.

19:14 - The case that I'm thinking about is people that have tried to write their own little IDE or even a big IDE in Python to run Python.

19:23 That you've got this issue that you still only have one GIL so you gotta launch another thread, you have to have another task or something.

19:31 And this would allow something like that to be easier.

19:34 - Yeah, so I don't actually, yeah, I agree.

19:35 I don't know what actually would come out of this, but it looks like it has some interesting potential.

19:40 And it's also interesting that it basically just formalizes what was already there.

19:44 So that's pretty cool too.

19:45 - Yeah.

19:46 - Awesome.

19:46 All right, well that's it for our main topics.

19:48 You got any extra stuff you want to throw out there?

19:50 - Other than I'm just super excited about PyCon.

19:52 - Yes, it's PyCon Eve, I'm so excited.

19:55 It's gonna be good, I'm looking forward to seeing you and everyone in Cleveland.

19:59 - When are we gonna be around?

20:00 Do you remember our times?

20:02 - Middle of the day-ish.

20:04 - Yeah, after lunch.

20:06 So come try to find us out.

20:07 - Yeah, we won't be at the booth all the time.

20:09 We're gonna be doing other events like open spaces and live recordings in other places and maybe even attend a talk, who knows.

20:16 The times that we will be there should be posted on the booth.

20:19 So there'll be at least three hours each day that we're doing something interesting there that you can come by and see us.

20:23 - Yep, and get stickers.

20:25 - And stickers.

20:26 Definitely find us at PyCon.

20:27 - How about you?

20:28 You've got some big news.

20:29 - Yeah, I got some big news.

20:30 The big news is my iOS app is finally out after negotiating, let's call it, with the App Store folks, who were better than Google Play, but still, it was quite the back and forth to get everything right.

20:44 So finally, the Talk Python Training iOS app is out.

20:47 Check it out at training.talkpython.fm/apps.

20:50 If you install it, log in, you can get two free courses in addition to the ones you might already have.

20:55 So that's pretty cool.

20:56 - Yep, I installed it this morning.

20:57 - You did, you're already on top of it?

20:59 - Yeah, yeah, it looks great.

21:00 - Right on, thanks.

21:01 That's very cool.

21:02 And then lastly, like I've said, our listeners are awesome.

21:05 Anytime we say, here's something that's kind of unique, they're like, and the five other amazing ones.

21:10 So here's one called Blessings.

21:12 We talked about bullet, and we talked about cooked input.

21:15 And Blessings is kind of in that realm of making terminal input nicer.

21:20 And this is also output.

21:22 It's not exactly the same.

21:23 But Blessings lets you do things like bold your terminal output and move the cursor around and do all sorts of cool stuff.

21:29 So this is from Eric Rose and Preston Daniel sent this over and said, "Hey, you should check this out in addition to the ones you're talking about." So here it is.

21:36 It looks pretty cool.

21:36 >> Ooh, I have the exact use case for this in mind.

21:39 So yay.

21:40 >> Nice. What are you going to do with it?

21:41 >> I want to do, like I just got finished with reading the TDD by example.

21:45 Yeah, I know. You would have thought I would have learned that beforehand.

21:49 But yeah, I finally read it.

21:51 One of the things is a to-do list that is bold for stuff you're working on and not.

21:57 >> Nice.

21:58 So anyway, he uses that.

22:00 - Super cool, yeah, there it is.

22:01 Awesome, all right, we have a joke coming in from Topher Chung.

22:05 Do you wanna do it or should I do it?

22:06 - Oh, you do it. - All right.

22:08 Knock, knock, race condition.

22:09 - Who's there? (laughs)

22:12 - All right, knock, knock, race condition, who's there?

22:16 Perfect, all right, well--

22:17 - Man, these never get old.

22:19 We could do-- (laughs)

22:21 - I am starting to notice that the pie joke well is starting to run a little dry.

22:25 We've been emptying it a lot, People have to start sending in their jokes.

22:28 That was a good one.

22:29 Thank you, Topher.

22:29 Yeah, thanks.

22:30 All right.

22:31 Well, Brian, we shall reconvene in Cleveland.

22:33 Yeah.

22:34 Talk to you there.

22:35 All right.

22:35 See you, everyone there.

22:36 See you, Brian.

22:36 Bye.

22:36 Thank you for listening to Python Bytes.

22:38 Follow the show on Twitter via @pythonbytes.

22:41 That's Python Bytes as in B-Y-T-E-S.

22:44 And get the full show notes at Python Bytes.FM.

22:47 If you have a news item you want featured, just visit Python Bytes.FM and send it our way.

22:51 We're always on the lookout for sharing something cool.

22:54 On behalf of myself and Brian Okken, this is Michael Kennedy.

22:57 Thank you for listening and sharing this podcast with your friends and colleagues.