Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


Transcript #96: Python Language Summit 2018

Return to episode page view on github
Recorded on Thursday, Sep 20, 2018.

00:00 KENNEDY: Hello, and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is Episode 96 recorded September 19th, 2018. I'm Michael Kennedy.

00:10 OKKEN: And I'm Brian Okken.

00:10 KENNEDY: Hey, Brian, how ya doing?

00:12 OKKEN: I'm doing really good, how are you?

00:13 KENNEDY: I'm doing great. There's always something to talk about in the Python space. It never gets boring. No, it definitely doesn't. And we're here bringing this to you today in part because of DigitalOcean sponsoring this episode. So, check them out at pythonbytes.fm/digitalocean and get $100 credit for some server goodies. Tell you more about that later. Brian, you found an interesting one that I don't even know what to make of it here. Tell us about it.

00:36 OKKEN: I can't even remember where I came across this. I guess it doesn't matter. There's a Python Library called Plumbum. That's spelled P-L-U-M-B-U-M. I have no idea where they came up with that name. But it's a toolbox of goodies to do shell-like things in Python. It says shell combinators and more. I'm not sure what a combinator is. Oh, there was that Y-combinator on Back to the Future. Other than that, I don't know what a combinator is.

01:03 KENNEDY: Is it like the pipes between things?

01:05 OKKEN: Maybe.

01:06 KENNEDY: Perhaps.

01:07 OKKEN: Anyway, the motto of the library, this is direct quote, is "Never write shell scripts again." And thus, it attempts to mimic the shell syntax. Oh, shell combinators is shell syntax, where it makes sense while keeping all the pythonic and cross-platform goodiness of Python. And it's kind of fun. It's kind of cheating actually, I think, but it's cool. We have an example in the Show Notes, but there's a bunch of examples on their website, on the Read Me and Stuff. You can do things like chaining commands, with pipes and there's a built-in grep, and word count, and head counts within this library. So if you've got a Bash shell script and somebody says, "Hey, you should convert that to Python" this'll take you like five minutes to convert it to Python using this. And that's not necessarily a bad thing. If you know kind of how to transform something here, you can convert it right away. And then, maybe if you want to go figure out how to do that with other ways, you could. But for a lot of quick and dirty things, why not do something like this.

02:12 KENNEDY: It looks really cool. Like certainly, writing Bash stuff, the more tests you have to do and the looping over things, it's just like how do you do that in Bash again? It's not so nice. But there's certainly things you can do in the shell that are like painful in Python, like show me the files here. It's like, okay, well fine. We're going to import OS and want to do a lister, and loop over that, right? Like, ls, that's super easy. And this kind of brings those things together. It's really nice.

02:37 OKKEN: Yeah, so I've got a couple things. I'm going to throw this out right away is in Jenkins, you can do shell commands within the little command steps of Jenkins. But if you're doing pipelines on Windows, you can't do Bash for some reason. Or I can't figure out how, a way to do Bash. But I can do Python. So, I could write a little Python script to do the thing that I wanted to do in Bash, and do it right there. So, I'll try that.

03:02 KENNEDY: Yeah, nice. So, this is a really cool project. And certainly if you're thinking about some kind of Bash script or something like that, this is a really nice option.

03:10 OKKEN: Maybe, I should just try to use the Linux Subsystem on Windows 10, instead of...

03:14 KENNEDY: Yeah, so if you have a problem with getting Bash to work, it doesn't work on Windows, right? But you can definitely bring in the Windows 10 Linux subsystem. And obviously, that's Windows 10. I'm sure that it's available on the Windows Server equivalent of Windows 10, which I honestly don't even know what version that is. But there's probably a Server equivalent that has the Linux subsystem as well. People can throw that in the show comments if they want. But there's an article by Marcus Sherman called The Windows 10 Linux Subsystem for Python Developers. First of all, have you heard of this Windows 10 Linux Subsystem? Do you know what this is?

03:51 OKKEN: Yeah, I've been trying to keep an eye on it, and as they make fixes to it, and add features, I've been trying some of these things out.

03:59 KENNEDY: Yeah, it's interesting. So, it's basically Linux embedded inside of Windows. And you've always been able to run, in Windows, a VM ware, Virtual Machine, or something, and run, say Ubuntu. But that's not what this is. It's not like a entirely separate running VM, but it's like a little, tiny, micro VM type thing that is the command line shell for Ubuntu. So, you can basically have Ubuntu commands and a Ubuntu terminal in your Windows. The idea is if somebody says, "Hey, here is some library you want to use, and the way you get it is you compile it from source and run it, or you need to have this other weird utility, or type these commands, right?" Like, how often have you seen a tutorial that says, Okay Brian, the next step is to type python3 -m venv venv and then `you say source . venv/bin/activate. That's awesome, right, it works fine on your Mac. Works fine on my Mac. It works fine on my Linux server. Does it work fine on Windows?

04:55 OKKEN: No, you have to use the scripts directory.

04:57 KENNEDY: Scripts directory. There is no Python3. There is no source. What is all this stuff, right? It's like why is it different. And so, the motivation that Marcus had for writing this is like, hey, if you're going to learn Python and you're on Windows, you might consider using the Ubuntu subsystem, because then all those commands that everybody tells you to do, they just work.

05:17 OKKEN: Oh, that's a good idea.

05:18 KENNEDY: You could use, say, Anaconda, and you'd get all the tools, but you still can't type Python3. You still can't type source. All these things don't work, unless you run Ubuntu for Windows. So, it's a really nice step-by-step way to go through it specifically for setting up for Python development on Windows.

05:35 OKKEN: Okay, cool, neat.

05:36 KENNEDY: Yeah, I know, pretty sweet, right? So, you might say it's a little cheat sheet for getting Ubuntu working there, right?

05:42 OKKEN: Yeah, hey, we're doing really good on the whole transition thing.

05:45 KENNEDY: Yeah, it's working.

05:46 OKKEN: Yeah, we've got a cheat sheet on my next item. I've been trying to do type hints a little bit more. So, we've got a lot of people at work and myself are using PyCharm. And the little popup dial, I don't know what that called.

06:02 KENNEDY: The auto-complete dropdown, when you type dot?

06:03 OKKEN: Yeah, all that stuff. It works really great and it also works better if you have type hints in your API to things. So I've been trying to do, at least for the external functions, I've been trying to do type hints in there. But I forget some things. So, I ran across this type hints cheat sheet. And it's actually on the mypy documentation site. I mean, I remember some of the easy things, like how do you represent string, and hint, and stuff like that.

06:32 KENNEDY: Colon hint.

06:33 OKKEN: Yeah, but what about duck type stuff? If I have a thing that acts like an array, but it might be a tuple, or it might be a list, or it might be something else. I just know that I need to be able to use that bracket index operator and find the link. That's the sequence for type hints, but I don't always remember that if that's an iterator, iterable, or a sequence, or things like that. Those duck type things are what I'm using it for the most, but I have other people that I work with that are just sort of learning this typeahead thing, so all of it's useful.

07:06 KENNEDY: Yeah, that's not that long. Scroll bar estimation is probably seven or eight pages. Flip through that once, and you pretty much have mastered the type hints. There's a couple interesting things in here that I thought was nice. Talking about adding type hints for asynchronous function, so async def. One that I had never seen was class level variables, so Class var looks pretty interesting. It's not on the instance, but it's on the type.

07:32 OKKEN: Oh, interesting.

07:32 KENNEDY: Right, you can use Class var annotation to declare a class level I've never done that. That's pretty interesting. Then also, specifying the type for a say a generator or something that yields some king of iterator. There's also an actual generator type decoration that takes three arguments. That's not covered in here, so that's all potentially worth noting. But yeah, very cool stuff. I think this a nice find. Now before we get on to the next thing, I want to tell you quickly about DigitalOcean. Like I said, they're sponsoring the show, and they have lots of good reasons to go check them out. Something that they just announced that's coming soon and is available for early access, is Kubernetes. If you have some kind of site that you would like to run on a Kubernetes cluster, you can now go. First visit pythonbytes.fm/digitalocean. Create an account so you get the free $100 credit, then go to digitalocean.com/products/kubernetes, and you can sign up for early access to check that out. Then you can submit containers to run on the Kubernetes cluster and have simple, zero down time deployments.

08:38 OKKEN: That's actually very, very cool.

08:40 KENNEDY: Yeah, I'm pretty excited about that come along. That's going to be fun. Every now and then we cover these sort of higher level topics a little softer. Heres one that tries to look a little bit farther in the future. I'm just going to put these out there, and then I'll let you react to this article. The overall theme here is that Python is driving the creation and foundation of new languages. This was an article and it says, "Here are five predictions for what programming will look like in 10 years." Okay, so number one, programming will be more abstract. When I first read that I'm like, "Please don't tell me it's drag and drop." or something like this right? But no, they say that trends like serverless technology containers, we just talked about Kubernetes, and low code platforms suggest that developers may be able to work at higher level abstractions in the future. Not thinking about servers for example, that sounds reasonable.

09:34 OKKEN: Yeah, to some extent.

09:36 KENNEDY: Yeah, conditionally. "AI will become a part of every developers toolkit, but it won't replace them." What do you think?

09:42 OKKEN: I don't buy that actually, but okay.

09:44 KENNEDY: Give 'em some comment I can say to some of the voice assistance, but I'm pretty sure I'll set off thousands of them, so I won't do it. All right, "A universal programming language will arise."

09:54 OKKEN: Well Python, it's already arisen.

09:56 KENNEDY: So yeah, the reason this whole article was pretty interesting is it has a pretty strong Python influence. For example, the beginning says just in the last few years some languages have risen and some have fallen. The have risen links to an interesting article about Python. The have fallen links to an interesting one about Ruby for example. Then under this section there's a nice quote that says, "Python may be remembered as being the great, great, great-grandmother of languages of the languages of the future, which underneath the hood may look like English but are actually far easier to use."

10:28 OKKEN: Okay, yeah, you ready for my reaction yet or you got more?

10:32 KENNEDY: I'm ready, bring it.

10:35 OKKEN: One of the jokes about Python sometimes is that it's runnable pseudocode. I don't actually want it to get closer to English, because English has so many subtleties and weird things that you can do that have implied meaning. The closer we get to the actual English language, the harder it is to actually debug something. Having it not be that close, I think is better, because it's less up for interpretation.

11:05 KENNEDY: Right, please read that sentence. The people over there have already read the red sign, you know, things like that. I'm not so sure that a universal language will arise either. There's always something you may want to take most advantage of and by doing so you may get less good for other things. For example, look at say the data science tool set. It's better working with sets of data than individual things. I don't know, it focuses on one thing and then other things become less easy with it.

11:38 OKKEN: One of the changes, which is nice, is the languages are not being developed in the vacuum that they used to be. They're learning from each other.

11:46 KENNEDY: That's for sure.

11:47 OKKEN: We're getting the mega cool feature in one language that we would like to have in another. We're already seeing that. Python's talking some features from other languages and pulling it in. Other features are taking cool languages. I've heard from a lot of people that hardly any other languages have good testing tools like pytest. Specifically, they don't have pytest for other languages. That would be cool if we could have some better testing tools in other languages.

12:15 KENNEDY: Yeah, I mean there's certainly examples of that. Apple's Swift is very much Python inspired. You see the optional, null coalescing operator being moved towards Python in 3.8 potentially. Things like that. So I agree. I definitely see that as a blending. I don't know if there'll be a single one. All right, final two. "Every developer will need to work with data." I think that they're doing that now, right?

12:36 OKKEN: Yeah, I think that we always have.

12:39 KENNEDY: Yeah, and then, "Programming will become a core tenant of the education system.", like Reading and Writing.

12:45 OKKEN: I believe that.

12:46 KENNEDY: I do too.

12:47 OKKEN: It isn't necessarily like Reading and Writing. I think it's more like learning Biology and basic Chemistry. It's not that you actually need it everyday, but it's good that most people kind of understand how it sort of works.

12:59 KENNEDY: I'm totally with you. The thing I think it will eject from the education system is Geometry. I feel like the reason Geometry is taught in middle school and high school is so that critical thinking and proof type of logical thought can be taught. You don't really need to prove this thing about isosceles triangles. That doesn't help you in life. But the thought process does. I feel like programming, you have the same type of thought processes but you get a skill not just a grade. That's my predication.

13:28 OKKEN: That's interesting, yeah.

13:30 KENNEDY: We'll see about that. All right, so another one on asyncio, I have something I've been digging into a ton lately.

13:35 OKKEN: Yeah, I wasn't even aware that this was a thing going on, but apparently the standard library asyncio documentation was completely rewritten from scratch recently.

13:47 KENNEDY: That's awesome, that's big news.

13:49 OKKEN: Yeah, it even says it in the tweet announcement, "This is big news." with two flames on the side. I haven't actually read through this a whole bunch, but it does look like there's quite a bit of documentation there, so that's neat.

14:01 KENNEDY: That's really cool.

14:01 OKKEN: And it says they're not done. There's a comment that this is just the beginning. There going to add some tutorials for high level concepts in APIs and then they're also going to keep going, because apparently there's some new things planned for 3.8.

14:17 KENNEDY: That's awesome, yeah you highlight some of the things coming for asyncio itself. The library and capabilities async and await are getting better right?

14:24 OKKEN: Yeah, I think this sort of talks to the importance of the async in the future. If not only are people still working on improving the async story within Python, but they're also trying to improve the documentation to make sure that it's usable. I'm glad to see that sort of documentation going side by side with new concepts. That's a good thing. But that kind of highlights the importance of async going forward.

14:49 KENNEDY: Yeah, I think async is just becoming increasingly important. I'm really glad to see it really gain some momentum, because if you look at apps written with and without async in a library that can take advantage of it, it's like night and day. It's ridiculous.

15:03 OKKEN: Like you said again, it's a different concept sort of thing. You can think in higher level concepts and break problems up into more logical breaking points.

15:12 KENNEDY: Right, you don't have to think about threads, and processes, and coordination, and all that. You just say, "I'm going to do this part, then I'll wait, then I'll do that part, then I'll wait, then I'll do that part." Then, you know, that's sort of that higher level abstraction. You want to get caught up on all the places that Python is headed? All the directions, all the threads?

15:28 OKKEN: Yeah, definitely.

15:31 KENNEDY: There was a 2018 Python Language Summit, okay? That's where all the core developers get together and they talk about what they're doing, they do proposals to each other like, "Hey, I think we should focus on this or we need to address that and so on." There's some really interesting items on lwn.net, which is a pretty interesting site. They've got a lot of great stuff mostly focused on Linux. It's kind of like the Craigslist of Linux news or something. I don't know, it's just got to interesting design, but basically this is the meta-article about the Python Language Summit this year. It lists off a whole bunch of sessions. Each of which are covered in detail in another article, so what I'm going to do is cover the high level bits of each one of these main sessions, okay? So the first one is, could we have subinterpreter support for Python. We were just talking about asyncio and stuff, one of the main problems. I don't know if I'd call it really a problem, because there's a lot of ways around it actually, but conceptions is that Python's not great for threading specifically around taking advantage of multiple course for CPU stuff because of the GIL, right? It could have 100 threads and only one will run at a time if it's pure Python cause of the GIL. So the suggestion is, could we take this feature, which is apparently already in Python called subinterpreter support, to create lightweight subprocesses that run within the same processors. Subinterpreters will allow multiple Python interpreters per process, and there's potential for zero copy data sharing between them, and things like that. Which is pretty cool.

17:02 OKKEN: That's interesting.

17:03 KENNEDY: It might be interesting even for versioning. Right, like this library requires this version or that library requires that version. Well you can run maybe multiple subinterpreters, I don't know. That's a whole different angle. It wasn't part of this presentation, but they say the problem is that subinterpreters currently share the GIL, so the potential work is could we make them not share the GIL so they actually can take advantage of multiple cores. It's becoming a thing. My new MacBook here has 12 cores as far as the OS sees it. If I write Python code, and it takes full advantage of all the processor that it can in single thread is only 8.3% of the capacity of a CPU. I heard that's a lot less than 100%. Having features like this to let you do more computation will be great. All right, I don't want to go too too deep in these cause it'll make this an hour long show, so I'll try to go a little quicker. The next one is modifying the Python object model, looking at CPython's structure. So Carl Shapiro from Instagram talked about how they might rework the way that interpreter interacts with classes for better performance and memory cause Instagram is all about that. One of the interesting things in this article is that there was some controversy around the presentation and some other stuff, so I'll let you read about it. I won't bring it up here, but it's worth looking at. We just talked about the GIL for parallelism. Larry Hastings has been on and off working on this thing called The Gilectomy, the removal of the GIL. The GIL basically is a thread protection around allocation in Python, which means you don't have to take a bunch of fine grain locks when you're allocating new objects. But because of that, it means only one instructions run at a time, so could we take that out and have a Python that does true parallelism? Basically the update is not much has happened. So for better or worse. Marietta talked about using GitHub issues for Python. I didn't realize this wasn't already happening, but apparently it's not. Apparently there's another site that's currently being used and trying to move over to GitHub issues. That's pretty cool. Lucas talked about shortening the Python release cycle, so right now it's on an 18 month release cycle. So in 18 months from whenever 3.7 was out, 3.8 will be out. Could we make that yearly as possibly the conversation she's going to read in that? One you might find interesting, is unplugging old batteries. There's encoding libraries for formats that predate Mime and other weird type of things that are in there that nobody likely uses anymore. So what is the process for taking stuff out of the standard library? There's a process for it getting in, but how do you get it out?

19:40 OKKEN: Oh, that's interesting. Yeah, a lot of things could probably be not just completely eliminated but moved to a pip package or something.

19:48 KENNEDY: Exactly, this is now out, but if you need it back you can pip install it separately. But you're job is to maintain it. We're not shipping it anymore. Something to that effect is kind of cool. They talked about Linux distributions and Python 2, with Python 2 coming to life, what is the process for basically getting those out of the Linux systems that they ship with? How do we do that? He talked about Python's static typing. There's an update on that. Mostly focused on stub files. Have you heard of stub files? It's like the ... right? I've looked at them, and they basically are the method signature and then ... They're not actually the implementation. They're just saying here is the actual shape of this method. So what you do, is they get imported alongside of your code by things like mypy. They're kept in this thing called the Typeshed, which is a place for third party libraries to be in there. You can look through there and see things like what type of stuff, like the context variable, and what is its type definition for example. There's a bunch of things with ... all over it. That are just empty, not implemented sort of structures. This is where a lot of the third party stuff and standard library goes, but they're saying, "Look, a lot of packages are not putting things here. Namely things like Django and SQLAlchemy." So there's a proposal of finding a way to bind the type information to the package, and not to the central location so that package containers can ship those stubs along with their libraries.

21:25 OKKEN: Oh yeah, that would make more sense. Or at least it would make some sense.

21:28 KENNEDY: Yeah, like it'd be easier cause then you don't have to say do a PR to some other repo. You just put it along with your library and send PIP. Send it in PyPI. Okay so, Python virtual environments. There's some debate about whether their useful and what we might do instead, primarily around if your trying to teach Python you shouldn't have to start out with a 20 minute conversation of how to do virtual environments both on Linux, Windows, and MacOS. Then you can actually talk about Hello World, but how do we just do that? An interesting debate there. There was some conversation around the whole new decision making process now that Guido stepped down. Another one about getting along in the Python community, trying to make things like the dev mailer list be more interesting and welcoming as well. Also, finally, mentoring and diversity for Python with Marietta and Victor Stinner as well. Two separate presentations in that general area. Every single one of these things I've talked about there's an article about it and more references. So if you want to dive into it, it'll keep you busy for an afternoon.

22:28 OKKEN: Okay, one of the things that you wrote down but we didn't talk about was TkInter.

22:33 KENNEDY: Oh, so the one about getting along and being more civil on dev with somebody complaining that TkInter is so badly out of date that it should be removed from Python, but it's shipped with Python? Which kind of brings us back to how do we unplug old batteries cause if people don't maintain it, and it doesn't work for anyone than why would TkInter be there? We might get back on another GUI kick if we go down that too far.

22:56 OKKEN: Okay, all right.

22:57 KENNEDY: Anyway, that's what that was about. The sort of rudeness on the mailing list was focused around TkInter and where it needs to be upgraded, removed, whatever. All right, well that is it for the Python Language Summit. I thought that was a super interesting. There's just so many looks at where people are putting their effort. This is where the core developers are focusing their time, so I think it's definitely worth paying attention to.

23:18 OKKEN: Yeah, definitely, cool.

23:19 KENNEDY: All right, you got anything else you want to share with folks while we're here.

23:22 OKKEN: Yeah, one thing. I noticed this from, I'm going to get his name wrong, Anthony Sottile, codewithAnthony on Twitter, mentioned that precommit had a huge jump in downloads on July 30th. I at first assumed that it was because we covered it or something, but apparently it didn't line up with us covering it.

23:43 KENNEDY: Should of caught this earlier. Could've gotten credit, but no.

23:46 OKKEN: But I looked at other packages and there is a weird jump in almost all packages, downloads starting in at the end of July. That don't have any news around that? That's weird, and I wonder what's going on?

24:01 KENNEDY: You know, I wonder if that has to do with when the new PyPI launched?

24:06 OKKEN: Oh yeah, I wonder. It's almost coincident with the pip 18 as well.

24:11 KENNEDY: The new PyPI launched in April, so yeah it could be with pip 18. Who knows, I bet it's some low level infrastructure thing like caching got expired and everything that had cached everywhere got a new one or something.

24:22 OKKEN: Yeah, maybe they changed the way they're caching it or something.

24:25 KENNEDY: I do remember seeing that from pip that every now and then if you run a sufficiently old pip it'll say at it's offline cache that's not compatible, and it'll redownload. I'm wondering if in the reverse room would a new pip came up. That may happen.

24:40 OKKEN: I bring that up so that if anybody knows what's going on they can get ahold of us and we can find out.

24:44 KENNEDY: Yeah, throw it in the short notes, pythonbytes.fm/96.

24:46 OKKEN: Yeah, and that's one of the things that I love about this is cause we can talk about things that we don't quite know about, and people will correct us.

24:53 KENNEDY: But generally pretty nice about it too, so thank you for that.

24:56 OKKEN: One of the things that somebody mentioned, we've talked about PyCUDA, did I get that right?

25:02 KENNEDY: Think it was CUDAPy. CUPy, .cupy.

25:05 OKKEN: CUPy, we were guessing that maybe if you had a video card on your laptop maybe you could use it, be we were corrected that it would have to be an NVIDIA card, because CUDA only runs on NVIDIA.

25:17 KENNEDY: Apparently CUDA is a NVIDIA thing, which I didn't realize. I thought it was just a graphics card standard, but no, it's an NVIDIA thing. This is all triggered by your question, could I do this on my MacBook or on my laptop? If you have a Windows or Linux laptop that has an NVIDIA card, yes. Otherwise no apparently is the answer, so a couple people wrote us that and communicated that to us. Thank you all for that. I guess if you're on a MacBook, you could get an external. On the new ones you can do external GPUs and you could get an NVIDIA one there. But even the ones sold by Apple, is actually a Radeon, so it doesn't help you. Anyway, just thanks for the clarification.

25:54 OKKEN: Yeah, do you have anything else to add?

25:57 KENNEDY: Nothing else, not right now. I'll have some good news next week I'm going to share, but right now nothing. Nothing to say.

26:02 OKKEN: All right, well thanks a lot.

26:03 KENNEDY: Just working away on projects. Yeah, thank you. Great to chat with you as always. Thank you for listening to Python Bytes. Follow the show on Twitter via @PythonBytes. That's Python Bytes as in B-Y-T-E-S, and get the full show notes at pythonbytes.fm. If you have a news item you want featured just visit pythonbytes.fm and send it our way. We're always on the look out for sharing something cool. On behalf of myself and Bryan Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page