Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


Transcript #208: Dependencies out of control? Just pip chill.

Return to episode page view on github
Recorded on Wednesday, Nov 11, 2020.

00:00 Hello, and welcome to Python bytes where we deliver Python news and headlines directly to your ear buds. This is Episode 208 recorded November 11 2020. I am Brian Aachen, and I'm Michael Kennedy. And it's getting cold outside. It is getting cold outside. I feel like winter is coming. I went out on the deck and I'm like, well, there's something wrong with my deck. It's really slippery. Oh, that's ice. What a weird time of year it is. Right call to get this fixed. Exactly. there'd be some sort of contract here you can get I'm sure there's something wrong with the sun. We're gonna need some help here. Yeah, this episode is brought to you by us. We'll tell you more about what we're doing other than how you can support us a little later. But first, I want to speak of cold I want to talk about something called chill. So icy. Have you heard of this before? I have heard of this. this. I've just heard of it. But it looks fantastic. It drives me crazy. I have this whole new change with PIP where PIP is super picky about the dependencies. And then having things like depend a bot or other automatic tooling upgrade stuff as things come out. If they ever get out of sync, then people break in some ways like for example, Boto three and Boto core have been driving me crazy. And it sounds like this would fix it. Normally, you build up, you got like a repair requirements, that txt file or you got your versions that you're working with, you want to put those in a setup or some sort of get way to capture it, one of the ways you can do that is PIP freeze. And if you already have an environment, like virtual environment setup, with everything you need, you say PIP freeze, it spits out all of the things that you have installed, and all the versions that you've got. Now, there's different variants that you can do with PIP freeze. Also, you know, people often talk about that as having stability, like you don't accidentally get a change that's pushed in the future, that's a breaking change. And to me, I never appreciated that like that never seemed to be something I cared about. But once things like GitHub started saying, there's a new version of this or the thing you have installed, there's a security vulnerability that if you don't upgrade, you're subject to on the web, which is a bad place to be, then putting that version information in there explicitly allowed me to know, oh, I need to go and update the server because there's this critical vulnerability that GitHub told me about. That's what won me over to putting the PIP freestyle with versions in there. Oh, yeah. Yeah. And I think that's good. But you might want to just have the version, the thing that you're really using, like, let's say you're using black, do you want to list black? Or do you want to list black and all of its dependencies? Yeah, exactly. And prese just gives you everything. And so what PIP chill does is it just shows you the stuff that you installed. So like, let's say, if you installed just black, and you ran PIP freeze, you'd see a whole bunch of stuff. But if you run if you now install PIP chill, Pip dash chill, and run PIP chill, you'll get PIP chill. Also, it'll tell you that it's it's there. But it also just shows you black and what version of black you have. So if you've, hopefully, black itself is specifying enough dependencies that make sense. There's definitely times where you want to have everything, all the versions nailed down, like you said, for security updates and stuff like that. But there's all a bunch of times where like, for instance, I've got internal projects where we actually have, we actually vet all of the versions and put them in a different repository. But different combinations, we wouldn't be, you don't need to be that specific. So PIP chose a way to just list the ones that you've installed. And I think it's really cool. I think it's really cool to one of the things that's neat is he toggles between this, here's what you installed. And here's everything, which is what you're talking about with PIP freeze. But there's another mode where you do dash V, and it will show you the stuff you installed, and then it'll show you commented out but present, these are all the things, the libraries, and they're dependent their versions that were installed, but are not pop levels to their commented out. And then there's a little comment, like installed as a dependency of Jinja to time or the installed as a dependency of cookie cutter. So you can do that to get a look at say, hey, well, what is my virtual directory look like? Why is this thing here? Oh, I see it came from this other place. Yeah. And then also a few you can have that with or without versions. And so you can if you know, there's a dependency vulnerability for particular thing you can say, well, which would version Am I getting? So good? Yeah, this is really nice. Yeah. And like if you've got like a version checker thing that tells you if there's, like vulnerabilities, right? Yes, we're using Well, if for vulnerabilities, I think just having a pin version in a requirements. txt file is sufficient for GitHub to say there's a problem. So like a new requirement file is pin. But if you sign up with dependency, which is there's some things that are super duper annoying about depend a bot, but what is nice is that it will show you anytime there's a thing so you say like weekly notify me of updates to my requirements file.

05:00 So every Monday morning, I get a list of like, here are the updates that I could upgrade for, like my web framework or database access or whatever. And like put in automatic PR for it. It'll show you like the change log, and then you can accept that and it'll update your, your requirements. Yeah. Okay. And there's also a workflow that some people use that have a, like a smaller, like a requirements in or something, it's just a list of List of packages they're using. And then they'll just get the latest of whatever those few things and then test it and then freeze it. So you've got like a, like a two stage thing. So you that what you deploy is a frozen set of package lists, but and all the versions, but you've got some other way to do just how to generate that which ones you should install. So right, yeah, yeah, that's interesting way to do it. Because then you basically have just pi pi and Pip. Just giving you the latest. Yeah. Yeah, just to round out the thing that is super duper annoying about depend a bot is like, on top Python training, let's say it has 3040 packages it's using if you count up the dependencies, Pip freeze, not PIP chill, freeze. And every change is its own PR. And I swear, almost weekly, I am merging, manually merging, merge conflicts between dependable and dependable. It's like, well, there's your current red line, and that requirement line changed. And like they won't give it to as one just, here's the changes for the week. It's like, No, no, here's a bunch of changes. And you'll probably be in here merging this manually. So enjoy that. Anyway, it's still really cool to have it automatically. I do like this idea of it just kind of like some part of the CI going, this is what I depend upon, install what I need generate the pinned versions, because then when you go to production, you say install PIP dash R, it will actually, you know, upgrade the stuff that needs upgrading. Yeah, rather than so bovill, one of the requests I have for the people working on PIP chill is, I think an option to not list itself would be good, because I'm just installing it so I can run it. So that actually makes a lot of sense to just omit self.

07:09 You can like pipe it through a yes B or something. But maybe it's like some kind of worm and its goal is to just get into all projects. And by listing itself, just gonna slowly make its way out. Maybe speaking of problems you might want to avoid not that that is one but one that you might is if you work on Windows, which from the Stack Overflow survey and a piece of survey, we found a very significant number of people in the Python space do that. And I've heard people in Python talk about this thing called num P. You heard of NumPy. NumPy is something that definitely is the foundation of the many, many, many data science libraries. Right? Well, big news and by Daniel Mulkey. Recent Windows Update broke open BLS, which is a dependency of NumPy. Speaking of dependencies, dough effectively if that doesn't work, NumPy doesn't work. That sounds less good. Yeah, yeah. So there's a whole long developer community thread I mean long, both talking about it, suggesting fixes that may or may not be fixes etc, etc. It starts like this, I'll just read the the quick overview so people know what to be on the lookout for I'm a NumPy. developer, we've been trying to track down some strange issues, where after updating a Windows 10 to zero or four, suddenly code that worked no longer works. Here's the NumPy issue and the corresponding issue in open vlas BLS. The problem can be summarized as when calling f mod something is changed so that much later calling open blast assembly routine fails. And we're way down here, Brian, the only difference I can see is that in the register that Visual Studio exposes after the call to F mod St. Zero is Nan.

08:56 The fact that people yeah, geez. But yeah, good luck. But the fact that that doesn't work is not ideal. So Steve dour and some of the other Microsoft people commented there. There's a fix later to take until January 2021 to be released. We know that Matt p posted workarounds is for all those at home following along and looking for a quick fix. NumPy has released a bug fix 119 three to work around this issue. The bug fix broke something else on Linux. So we had to revert the fix in release 119. Four. So the fix on Windows, the workaround on Windows broke the Linux version. But as a Windows person, you may not care. So on Windows, you could pin it NumPy equal equal 119 dot three for now. And just don't put that into production on Linux. Sounds good, I guess. Yeah. I mean, if you're a package maintainer, this seems ugly. But if you're like an end user, I'm either installing it on Windows or I'm installing on Linux. It's not a problem. Yeah. I don't know if PIP actually has a

10:00 Form flag like Pip. In the dash R, you know, the requirements. You specify this platform gets this requirement that platform gets, I'm guessing No, well, no, I mean, it's part of the wheel. So the wheels can say this wheel is appropriate for only certain platforms. Yeah. But if you're building an application, not something that's on Pip, I don't know. I don't know how to do it. But you can. This is like a fix. But just be aware that that will make it not work on Linux. So yeah, anyway, I just want to put this on people's radar. So they're not down looking at registers going, why is st zero Nan, That's odd. Glad I don't have to do things like that. We can all be thankful. We don't have to do things like that. I mean, that's why we work in Python. Right. Let's reverse it. I'm really grateful that other people are looking at that. So I don't have to. Yes, thank you, people. Speaking of Thank you, Brian and I are both doing a bunch of stuff online. And people often ask how they can support us support the podcast and just get better at Python. And we're trying to help people do that. Right. Right. We are Yeah, lots of ways. One of the things we're doing is this podcast. And if people want to just support us directly, there's a Patreon. Link in all of the shownotes, you can donate we have like 60, I just checked, we have 65 people doing this now. That's awesome. Thank you, everyone. And you're teaching some classes I am the most recent one that I worked on, should maybe maybe be out, it's completely done, I just put my head down and did the whole thing as like a four and a half hour fast API course for building API's with fast API. So I'm really excited about that one. And hopefully, it may even be out. But people would just talk out, check out training talk python.fm. And it'll be over there soon. If it's not by the time you hear this. Yeah, nice. Indeed. I'll just keep plugging away at it, man.

11:40 This is this gonna be the 30th course that we've created over there, and we're not done, we're gonna keep plugging away the plugin, maybe need a plug in architecture.

11:49 Tell me about it. Well, so one of the things I love about pi test is the plugin system. So you can mean plugins are, can provide extra fixtures, extra flags, they can modify the output, they can do all sorts of fun things. Anyway, that plugin architecture that I test uses is not built into pi test. It's something called pluggy. And there weren't, I mean, there are a couple little how to guides. But there, they were kind of sparse, and I was having trouble. I wanted to do a login in a little application before and, and struggled. But now here's a use apparently two Python talks, one of them held that by Gotham and one at pike on India, and an article around it. So the article is build plugins with pluggy. It's kind of a nice article, it starts with a small command line application that you like looks for books or something. And it has one output format. And the argument is like, wouldn't it be cool if you could have more different kinds of output? And one of the ways to do that in an application is to have allow plugins so that anybody can extend the application to with for a different format? Yeah, one thing that's really nice about that is you don't have to understand the whole program and all the code to like contribute to it, you just have to understand that little tiny plugin interface. And if you can handle that, then you basically can extend that. Yeah, and one of the beauties of those sorts of things. So I've written plugins for pi test. And I don't understand the internal architecture of pi test, I don't really understand how to do the plugins system that Well, I mean, pretty good. But you can take something that's already done, and copy it. So you can say, oh, here's another one that also changes the output. I want to I could copy that and change it myself. Yeah, yeah. This is kind of how that works. It took part of the system of the application. And he walks through how you would change the architecture, to instead of doing the formatting, internal, you could do it as a plugin. And then it talks about the different architecture of pluggy. Because you've got a host and a plugin, and a hooks and hooks, specs, and all sorts of stuff like that. And it's it's actually easy to get lost through this if you're just reading it. But I suggest walking through the code as he's doing it and actually running the stuff and looking at it. And there's not really any super easy ways to do plugin systems in Python. I don't know if any language has a better system. But plugins pretty good. You know, if you follow along, you can kind of get with it. So yeah, very cool. I haven't looked to see if the talks are online yet. But I'm looking forward to, to watching this. If they do go up. I suspect they probably will, in the modern age of everything being online. Yes. Yeah, probably. Awesome. So let me talk about a little queries syntax that I've been wanting in the Python space for a long time. And you almost have it like it's so close in some ways, but to foreign others, for example, with list comprehensions, generator expressions, all those types of things we can do a lot of in memory data like things right, we could go and say, if I got a list of numbers, I could get like the square of the number for all the numbers

15:00 If the number is like, you know, every number that's odd, or you have some weird thing like, right, that's sort of a query type of thing. There's like a where clause, there's a FROM clause, there's a select transformation bit. You could even do like paging with slices on the end of it. So like, skip three pages of five, so you know, slice 10, comma 15, or something like that, except, there's a few things that are just not there. The one that drives me the craziest the most is like a sorting, by the way, like, if you could just do like an order by in a list comprehension. All that'd be nice, but you can't. And you also can't put that sword on the end, because that returns none. Ah, so if there was some way to put like, a little bit more together, that would be great. And one of the probably the best implementations of this, I would say, across all the languages is this thing called link language integrated query in C sharp, yeah. And I noticed not about C sharp. But studying that there's a really interesting way of basically taking lambda expressions, applying them to collections and doing those database like things but much more where you can do like joins. And you can do other sorts of operations that are like paging or like filtering by type and all kinds of neat stuff there. So Adam sent over this project that he ran across, called link it, which adds link type of in memory query syntax, almost database syntax to Python lists, okay. So like, for example, if you had a list of programmers, you could say programmers where he you know, lambda v goes eat out experience, great and 15, except for Ilan mosque, but it's the type of this type of person or, you know, category of developer, take three to get only three of them skip, you know, you could skip like two times three, to skip two pages, and three, to get the third page of them, you can select just out there variables, and then apply more queries to that do this really cool, fluent interface on top of these collections, which I think is just super neat. And I really like this interface. So if people like this fluent style of programming, and they kind of think in data database type of thinking, here's a really cool way to apply that to Python. This is kind of cool. Yeah, I like it a lot, too. I like it a whole lot. So previously, and by previously, I mean, really previously, I don't remember exactly. Let me see, use our little search over a Python bytes back in Episode 106. So what is that like? Three, four years ago, or something like that? We talked about this thing called a rescue from 60. North, which is a cool project by Well, the guys at 16 North who do a lot of Python. And it basically is a similar type of thing. It adds linq queries to it. But what I didn't like what didn't spark joy in me, I guess on that one is, you had to like, convert everything to this query object. And then you can do queries on it, then you could convert it back to a list, which is like, so close, you know. But what's neat about this other project, this link project is it is the things that you do the queries against are a superclass of list are derived from list. So anytime a list is expected anywhere, you could just pass the result of this around. Okay, do you like conversions like it? Now? It's a query? Well, they Oh, now it's not anymore. Now it is again, right? It's kind of always this thing that can see it in his list, but actually has this query capability, which sounds minor, but it seems like a big usability benefit, actually. Yeah, definitely. Because, yeah, you want to be able to write functions that can return this and send it to maybe you want one of these objects again. So you can do further queries Exactly. Or you can pass it to something that expects a list. That's good. Yeah, it's pretty good. And if you want to upgrade a regular list your list you just, you know, pass it to the constructor of this, like drive less thing. And then now it's like it's Coriolis. So it's not quite as nice as some of the like, through language built in functionality. But still, this is quite neat, actually. Yeah, I think I'll play with it. Because I think that there's some use cases that are obvious until you start playing with it. It's really legible, right? Like, I would like to, from this group, where these are true. Select this thing. where that is true about the sub thing you got, right? Like it's it's pretty natural way of reading codes. I kind of like it. Yeah, I'm just I'm chuckling about your example, though. I'm sure you didn't make it up. But now this comes from the documentation. What is it? So I've got programmers, I'm looking for somebody with greater than 515 years of experience. I don't want to Elon Musk, something AV type. I don't know what it is. I don't know what that is either. Take three random people, I guess. There's three. I think the format three, yeah, the first three, if there could be thousands, just give me the first three. First three, find out what they had for lunch. And if it was a hot lunch, and it was not from Pizza Hut, get the last one and get the last slice of the pizza by the Yeah, that's it for

20:00 So what you end up with, I believe it's a list of pizza slices, which correspond to the last slice of pizza each programmer ate. Okay, I think I haven't read it yet. I got to write some code to make that effort. But yeah, I think that's what you end up with. Okay, cool. Anyway, yeah, pretty, pretty neat. But this idea of having this sort of natural query language that mimics databases, but is in memory, could have some legs, I get out what you got for the last one. The last one? Well, this one is, um, I just thought we should announce this because it sounded pretty neat. Although the details of it are a little a little over my head. But so there's a Spotify deals with music, right? There'll be a lot of music. So one of the things they talked about in I'm going to cut to the chase, it's a application or a framework, called Clio. I think I'm pronouncing that right K. Li O, for processing audio files, or they say any large buyer or any binary files, especially large ones that scale things like, you know, maybe like pictures or, or any other binary data. Anyway. So it's a it's an application. It's used for pipelines. It's intended for large scale input and output of all these files, scalability, reproducibility, working with pipelines, and streaming and batching. And to try to get it to be easy to use and easy to read, so that you can have a close collaboration between researchers and engineers. It's Python based. It looks pretty cool. And for a lot of people doing research in data pipelines for these a either audio files or vision or pictures or something like that, this might be cool. Yeah, it could be all sorts of stuff, right? I could imagine anytime there's a bunch of computation and a bunch of steps, people collecting the data, someone's gonna analyze it the output on the other end, it might be a pretty neat use case. Apparently, it's built on top of Apache Beam. And so it's also cloud agnostic. It does intense intended to work with clouds services, but it, it's a lot, you can use lots of different ones. I'm linking to the a couple announcements. And an article about it looks like a large effort and probably a large effort for Spotify to make it open source. I think that's pretty cool that they're trying to have this be around. One of the articles says this might be useful for people doing, like even comparing, you know, dolphin songs or something. There's a lot more use cases other than just music to deal with audio files. And this can be quite cool. Is this the project that Leonard was working on? I saw her posting about, I think it was her she was posted about something? I don't know. Anyway, it looks like a really cool project from Spotify. And does she work on Spotify? She used to I think she still does. I just tried to pull up her GitHub repo, but I couldn't find it for her Twitter account, but I couldn't find it for some reason. Anyway, yeah, this is neat. I'm a little bit surprised that open source this but at the same time, it seems useful. Why are you surprised? Just seems fairly proprietary or no? No, it seems, I guess, like a little mix of a somewhat specialized for their world a little bit. And then you're not at all related to like what Spotify does. You know, like, it's not? I don't know, it doesn't seem like it feeds back anything to Spotify, necessarily other but it? I mean, it definitely looks useful. So quite cool. Okay. Yeah. Yeah. Nice and nice to see Spotify doing that. Also nice is getting feedback from listeners, right? Just say like a good quarter of our entire history of our show has been like, and did you know about this? I'm sure you do. You just didn't cover it. Now, we didn't know about that. But thank you. Now we do. And so Marco garelli sent in something and says, Hey, recently, you've mentioned that it would be really cool. If you could have a Jupiter notebook that just was more of a report thing. You could reveal the code, but by default, the code is code cells are collapsed and hidden. So you can just go through it. And if you say like, Oh, I actually want to see the code for this particular part. Show me Yeah, yeah. So he sent in Article, I guess, walking through how you would do that with a notebook. So he's like, Hey, I actually wrote something about that. So here you go. In case, people care about that. So the idea is basically set up a Jupiter notebook, I'm reading this right set up a Jupiter notebook, you can tell it to build an HTML file that is like a static HTML file with all the data that's in there, or saved like you would see on GitHub or something where it's not actually live in computer, but you see the output and you see the cells, right, and then go tweak that HTML file. Thank you tweak the HTML file to just add a little bit of code that will auto use some JavaScript to collapse those. So then you can just share that HTML file. And people can, you know, toggle stuff open and closed and they get more of an article style output. Oh, nice. Yeah. So nothing like super deep or whatever. But it sounds like you're in that space. You're like, I really wish I could share this notebook. But that 30 lines of Python, it needs to be there. But it's not exactly what I wanted people to see. I just want them to see the steps. I think this is a pretty neat. Oh, that's cool. Yeah. So hopefully useful. Short and sweet. Indeed. Hello. When that happens when people we come up with we're like, I wish this could do this. And I wish it did you know it does. Yeah, exactly. Exactly. Well, that's our six items.

25:00 anything extra the talk about I do have a couple of things that are all small little things. One, there was this tweet I was mentioned in let me just read the tweet to you. And you just tell me how you feel about this. Okay, okay. Okay, so Michelle a Renard tweeted, the difference between Western and Chinese societies can be compared to him Kennedy's difference between software 1.0 and software 2.0. And it links to the Sanders media showdown article. What? So I saw this, I was like, Okay, this is weird like that we get copied all sorts of things like, Hey, we're gonna copy 20 people that talk about stuff on Twitter, because we published this article, and we want you to cover it. And I'm always like, Yeah, not so much. But if you hover over this, it says this is actually an AI bot pundit trained to sound informed and profound. The content mine that links is popular, and no tweet is the opinion of the offer author of the bot. I just thought it was really weird that there's like this AI bot going around on the internet, you know, using like, machine learning and stuff to just randomly be a pundit. Weird, right? Yeah. Anyway, Michelle, a Renard is out there. People do all sorts of weird stuff on Twitter. Yeah, but randomly like mentioning just other random people. Like a bad person over here. We're good to go. I just thought it was funny and amusing. So I put it up there. Yeah, it would have been even I guess more to the point if they would have said the difference between like Python two and Python three. I know. Well, I'd be one of the bot. Even if it was software two and software three, it would have been closer. Yeah.

26:31 Yeah, just takes off. And also Apple just had their big event a little bit ago. Have you got your new silicon? Mac? He ordered it. So Apple silicon don't have not Now have you? I ordered a new Mac, I actually was thinking about getting one of those, which was really interesting, because it's just, I think it's gonna have a lot of knock on implications. Like, for example, if you're shipping C code, as part of say, like a wheel, will it also do you now have to deal with like, different platforms on the Mac? While you're where your package actually work on the silicon? Mac? I don't know. I don't know. I don't either. But it's, it's gonna be interesting. I mean, this stuff needs to recompile. So I'm mostly wondering what this these apples, silicon max will mean for Python and its packaging ecosystem? Are we going to run into a place where like, we can't use NumPy on the new Macs for like, three months? I have an idea. Okay. I think if we get a whole bunch of Patreon followers, they can help us buy each of us one of these things. And then we can test stuff for people. Yeah, there you go. Perfect.

27:34 I actually, I did buy a new ish, 16 inch MacBook Pro, that's still Intel, because I was hoping to get one of these these new shiny ones. But well, if you're not going to make a new one, my current Mac is sort of dying, I don't have to buy something. So actually, the whole talk about that actually encouraged me to buy not Apple, so kind of mine, but I'm pretty excited to see where that goes. It's got some like huge ml performance speed ups and a lot of a lot of interesting things like the new MacBook Pro, I think has 17 hours of battery life. One day for video playback, or something insane like that. That's incredible. I've got like, six I know, the MacBook Air doesn't even have a fan like it literally, it can't make a sound because it has no mechanical pieces besides typing, I guess. Anyway, just wanted to shout that out. put that out there and see what people thought about the new Apple silicon have been like a different app compilation type and what that's gonna mean for Python and think about it. I mean, I guess it really comes down to what happens to C Python, right? Because it'll run under emulation mode if it's not. But if they upgrade c Python to run natively on Apple silicon, then what does that mean for packaging? Could be like a whole a whole deal? I don't know. Yeah, well, big thing. Yeah. I'll say that. You shouldn't joke about it, though. Now, fine.

28:45 Everything's funny. All right. They tell us a joke. This was sent to us that sent us I was just sent out by data, eta HQ to new database. They're doing some interesting things. They just posted out 10 ee engineers are the future

29:01 engineers with 10 E's. Beautiful, beautiful. I've got one for you as well. This one was comes to us by Richard karns. I don't know this one's a little bit political, but we could probably pull it off to anything substantive. Anyway, why did the data scientists get in trouble with animal welfare? Because she was trying to cut trying to import pandas?

29:24 directly? Good, right. Yeah, I think thanks, Richard. For right now, and that was a good one.

29:30 About training for pandas. You know, it says PD Come on.

29:36 Indeed. Well, thanks again. Mica. Yeah, I find is always same time next week. Yeah, I'll do it again. Yeah, thank you for listening to Python bytes. Follow the show on twitter at Python bytes. That's Python bytes as in V yts. And get the full show notes at Python byte set FM. If you have a news item you want featured just visit by thumb bytes.fm and send it our way. We're always on the lookout for sharing something cool. This is

30:00 Brian knockin and on behalf of myself and Michael Kennedy, thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page