Transcript #106: Fluent query APIs on Python collections

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.

00:05 This is episode 106, recorded November 30th, 2018. I'm Michael Kennedy.

00:10 And I'm Brian Otkin.

00:11 Hey Brian, how you doing?

00:12 I'm good. It's early in the morning, man.

00:15 Yeah, good morning, man. We're recording a little earlier than our average lunchtime get-together.

00:21 But, you know, this is what we do for our audience. We get up when it's dark and we talk about Python.

00:26 Yes, we do.

00:27 So before we get on to the topics, I just want to say thank you to DigitalOcean.

00:31 They're sponsoring the show like they did for much of this year.

00:34 So check them out at pythonbytes.fm/digitalocean.

00:37 Get $100 free credit for new users.

00:39 Speaking of cloud hosting and whatnot, one of the main innovations that's been coming along here the last couple of years is this whole idea of DevOps, right?

00:47 I don't know.

00:48 DevOps has been around for a while and really important for a lot of people.

00:52 But I think Python is getting involved more and more with DevOps.

00:56 We've also talked about packaging tools and dependency management a lot on This podcast and others are others as well This is actually we've talked about poetry and a band I don't know if we've covered pip tools before but I don't think we have pip tools is a package that Combines like it has a couple commands pip compile and pip sync which are ways to work with pip based usage based packages and I'm not gonna really get into that too much. But what what I found was interesting was there's a an article recently written by Enix that is called Python application dependency management in 2018. So Basically, he's got a use case. He's working in DevOps or a DevOps problem space right here and he wants the requirements he's got for doing packaging is He wants to be able to specify immediate dependencies like a requirement text or something But then and then be able to have a tool resolve the dependency tree So I know I need Django, but if Django needs a bunch of stuff I don't want to specify all that and then hopefully have hashes like the lock files and then integrate these into with talks so that you can run tests.

02:17 And then also the last one is in deploying servers, I don't want to actually have to activate them like a virtual environment and then install everything.

02:27 I want to be able to install dependencies and application and dependency into a virtual environment without having to actually go in there and do it.

02:36 Those all seem reasonable, but so far, he's tried pipenv, he's tried poetry, And right now he's sticking with pip-tools based solution.

02:46 Like using the pip-tools thing itself or just PIP?

02:48 A combination of them because being able to, you can't just use pip alone for a lot of this stuff.

02:55 So like for instance pip-tools is one of the things that you can create hashes with.

02:59 You can create like the lock files essentially.

03:03 And working with multiple requirements files like a requirements text or like a requirements.

03:08 He has a work model which I haven't seen before which is sticking multiple requirements files in a requirements directory.

03:15 So you've got a main, a requirements/main.txt and then a dev.txt for development requirements.

03:23 And then you also have to be able to update them. So occasionally, you've got pinned requirements, but you want to be able to update them to the newest and test that.

03:33 So all these things, it requires like a work model that should, these are reasonable requests.

03:39 One of the things I really liked about this was it was just basically we have an environment now that we want to be able to, I think it's a good thing, we want to be able to not slam people for like make fun of or bash on different projects too much.

03:59 But you also want to be able to be honest and say, "Hey, I tried this thing and it didn't work for me." And so one of the things I liked about this is I think it's a good example article of of saying hey This was my use case. I tried this thing this thing didn't quite work For me and here's why but also Don't slam the people who made that thing because I think it's cool that they're working on the problem It just didn't work quite right for me yet. Yeah, this whole package management package tooling stuff dependency tooling It feels a little bit like the editor wars like don't tell me use them I want to use emacs or I want to use by charm or whatever not quite But it seems something like people get into their workflow And you know they they really like their workflow, and they kind of want to stick with it Yeah, I think it's also some limitations of some of these tools They're built to address different problems than the one you're trying to solve right there Not one size fits all things and I found it refreshing because I I actually really like the idea of poetry Actually, I like the idea of pitman van poetry, but I in I don't have any specifics right now But I jumped in and tried to use it in some of my workflows Like he Nick Some of the things I tried just didn't quite work. So I have to go back to old ways Yeah I mean, I guess that makes sense it the more that those tools try to do for you all at once The more likely they are to not exactly fit your workflow. Yeah. Yeah, so One way that's really cool to extend applications would be to have the ability for someone to just drop a file into a directory, accessing a common API and have that just become part of your app or part of the capabilities of your app.

05:39 Right, like a plugin, right?

05:40 - Yeah, yeah.

05:41 - That'd be cool.

05:42 So there's a cool library called PluginLib.

05:46 PluginLib.

05:47 P-L-U-G-I-N, Lib.

05:51 Yeah, so really nice.

05:52 And it addresses this problem in a couple ways.

05:55 And when I first heard about this, I'm like, well, can't you just put a file in a directory and be good with it, just try to load it?

06:02 But the more I looked at what this library is doing, it's actually solving a lot of problems for us.

06:09 For one, it's an interesting example of metaclasses and programming with metaclasses, but you don't really have to deal with it because you just use the API and it works.

06:18 But one of the things, plugins are validated when they're loaded instead of when they're used, right?

06:24 Right away, they're validated.

06:25 Make sure everything fits the API.

06:28 They can be loaded through either a file pass, through entry points, through module names, and so on.

06:35 You could have multiple versions of the same plugin.

06:37 Which is pretty interesting.

06:40 You can blacklist them.

06:42 So if somebody puts some kind of plugin there, you're like, no, we're not gonna allow you to load this, even if we find it, you know, and configure something like that.

06:50 You can group them, you can have conditionals like this plugin works on macOS but not on Linux for example.

06:57 All sorts of cool stuff like that.

06:59 So if you were thinking about having this extensible plugin model, give plugin a look.

07:04 It looks like it handles a pretty simple problem but has a lot of support for you while it's doing it.

07:10 Yeah, this is pretty neat.

07:11 Just check this out.

07:12 Yeah, I just don't work on any apps that really, that makes a lot of sense for me but if I If I did, I would definitely want to have a look at that.

07:20 - Yeah, and it looks like it's based on, or partly based on abstract methods of class hierarchies.

07:27 - Yeah.

07:28 - And I think I remember looking at this, and this doesn't scare me off, but the particular application I was working with, we didn't have any classes uses anywhere, and I didn't really want to introduce them for this.

07:40 - You're like, let's throw in some abstract-based classes and hierarchies to a thing that doesn't even use classes.

07:45 Yeah, yeah, that makes sense.

07:47 But at the same time, writing a plugin for an application is a very specific need.

07:51 And I think it's fine for didn't match the rest of the model of the rest of your system.

07:56 So yeah, something to look into.

07:57 Yeah, for sure.

07:58 So what do you got for us next?

08:00 For some reason, I'm a little interested in testing sometimes.

08:03 Really?

08:04 Yeah.

08:05 One of the PyBytes guys, Bob Belderboss, he wrote an article called how to test your Django app with Selenium and pytest.

08:14 And yeah, pytest is one of my favorite things too.

08:17 It's a really nice write-up of, I'm going to quote the beginning of it.

08:22 It says in this article, I will show you how to test a Django app with pytest and Selenium.

08:27 One of the things I was intrigued by is that they don't use a toy app.

08:31 They use their CodeChallenges platform and compare the logged out homepage versus the logged in dashboard.

08:39 And I think that's really cool and brave of them to publish.

08:43 This is how we're testing our own stuff.

08:45 Publicly, on the internet.

08:47 Yeah, no, it's really cool.

08:49 Yeah, and it talks through, I probably wouldn't have done it, part of the project setup, they're using the activate and deactivate of a virtual environment to load environmental variables.

08:59 I don't have specific links to it, but I know there's a ways to do this within pytest itself as well, using something else.

09:09 Yeah, I know, it's kind of cool But at the same time, it makes it hard to--

09:15 makes it a little bit hard to--

09:16 I think it's kind of hidden in the whole setup, right?

09:19 There's nowhere in the code that you see that.

09:21 And you basically accomplish that by editing the activate script to set some environment variables when you activate the environment.

09:27 But if you go to another machine, you check it out again some other location, that kind of stuff is not there.

09:32 So it's hard to store passwords and secrets.

09:35 You don't want them on--

09:36 they definitely don't want to put them into this repo either.

09:38 So what are you going to do?

09:40 but it is tricky.

09:41 >>Yeah, and you wouldn't be able to use Tox with this because Tox is going to create its own virtual environment.

09:46 But yeah, so I'll have to, maybe I'll have to write up an example of how I would tackle that.

09:50 >>Yeah, that actually would be interesting.

09:51 >>But the rest of it, even though they're not using mocking out a database itself, they do show how to do that, how to get the database set up.

10:02 And then, yeah, anyway, it's just kind of a cool example.

10:05 I love working with Selenium, and this is a neat write-up.

10:09 It's cool, you know, maybe not everyone knows about Selenium.

10:12 They maybe know about web scraping and maybe trying to test it that way.

10:16 They also maybe know about testing inside of apps.

10:19 Maybe just quickly tell folks who are not totally familiar, what is this like?

10:24 Like, is this like requests or what is this?

10:26 What Selenium does is it really just opens up while it's running.

10:30 You don't have to interact with what you see is what you get or anything.

10:35 You're using Python code to control a web browser.

10:39 And you can use it in a couple of ways.

10:41 You can have like a, a web browser that you can actually see.

10:44 So while it's running, it pops up on your screen or like for instance, Chrome.

10:49 And I believe Firefox also has a headless version where you can, where you can, it can operate without actually, they, they just call it headless because it doesn't pop anything on your screen.

10:59 But other than that, it interacts just like it, as if it was open and you can, you know navigate the DOM and you can select things and activate things and and one of the benefits of going through this using Selenium is that it all the JavaScript stuff is loaded also through loading everything. You can interact with it as if it were just literally in your browser because it's literally in a hidden browser in the headless mode so yeah it's definitely an interesting way to do it like his example you're like let's take these things and put it in the username put in the password and then click this button you know find this button and click it, the login button, and stuff like that is really unique.

11:37 Okay, so before we get on to FluPi, which is the next thing I want to talk about, let's talk about a digital issue real quick.

11:44 So you probably built a lot of apps, have a lot of cloud resources running so far this year, but which one goes with which project?

11:54 That's always been something that's driven me crazy about other places, like I go to AWS and there's just tons of virtual machines running and stuff, you're like, what is this?

12:02 Do we need these?

12:03 I don't even know anymore.

12:04 So it would be really nice if we could group them together.

12:05 And at DigitalOcean, they create this idea called Projects, where you can group droplets and load balancers and IPs and all that into application categories that they work with.

12:16 Like you could have your app, you could have your app sandbox testing Q&A environment, and you could have multiple ones of those and so on.

12:25 So really nice, super cool way to organize your environment in DigitalOcean.

12:29 check them out at pythonbytes.fm/digitalocean.

12:32 And like I said, for people who are new users, you get $100 free credit to play around.

12:37 So that's also really nice to have.

12:39 - Very nice.

12:40 - So I'm a big fan of fluent APIs in general.

12:42 And maybe not everyone knows the term.

12:44 It's like when you take a thing and you call a function on it, and that function itself returns back the same object.

12:52 So instead of saying I've got a variable and I first initialize it to a list, And then I do some operation on that list, and it changes it.

13:01 And I do another operation.

13:02 Maybe it turns it into a set.

13:03 Then I pass that set somewhere else.

13:05 I could just say, let's say customers--

13:08 or let's say numbers.

13:10 This is an example.

13:10 It's easier.

13:11 So say numbers.map.filter.skip.take.orderby.

13:18 Just like one after another, all in one line.

13:20 That's the Fluent API in general.

13:23 So the thing that I want to point out is actually two libraries.

13:26 So you can take your flavor of this Fluent API and apply it to collections.

13:32 Because there's often more than one thing you might want to do to a collection.

13:37 You might want to change its values.

13:38 You might want to filter it.

13:39 You might want to only take a couple of them or put them into groups and things like that.

13:44 So there's one called--

13:46 I originally wanted to pronounce it as Fluffy, but FluPy.

13:49 FluPy is in Fluent Python.

13:51 And it lets you do, well, exactly what I was describing on collections.

13:55 So given any collection, you can upgrade it to a fluent collection by just calling flue and passing the collection.

14:04 And then it has all these operations.

14:06 Like you say .map and pass it a lambda.

14:09 And then on that result, you say .filter, pass another lambda, .chunk to break it into pages, I believe.

14:16 And .take will give you only three chunks of five back, for example.

14:22 So you could do that all in effectively one line.

14:24 Now maybe you want to wrap it for readability, but you don't have to use a bunch of intermediate variables or intermediate collections or anything like that.

14:31 - This is nice, yeah.

14:32 - It's slick, right?

14:32 And the thing that's really cool is it's all based on yield and generators.

14:37 - Really?

14:38 - Yeah, so every step in this chain is some generator type of operation, which means you could pass it a billion items and it's not gonna filter a billion items, map a billion items and all that.

14:51 It's just gonna take them one at a time through this sort of data pipeline that you build.

14:55 - Nice.

14:56 - Yeah, so the example that I put in the show notes uses count, like just all of it.

15:01 (laughing)

15:02 - All of it.

15:03 - But you know, it ends with a break it into pages of five and then take three pages, right?

15:09 So it only takes 15 items out.

15:11 - Okay.

15:12 - Even though you pass it, and the way that that works of course is with generators 'cause you can just pull through 'til like the last part stops pulling and then it basically stops iterating its infinite collection, really nice.

15:21 - In the little example you gave, you do like a for item in pipeline at the end too, so you can use the result as a generator also.

15:30 - Yeah, exactly, because it's a generator, like that whole sequence of operations doesn't actually execute until you iterate it, until you pass it to something that itself does, like pass it to a list or to a dictionary or something like that, yeah.

15:43 Or use it in another one of these pipelines.

15:45 So it's really cool to build up these data pipeline, data transformation things using Flupo.

15:51 - Yeah, and they also have a CLI, so you can do this kind of stuff from the CLI for, apparently, this guy's data science team.

15:59 Not everyone necessarily is great with Linux, but they would like to do a little bit more with these tools and with these ideas, these Python data pipelines, so you can actually execute these sorts of operations on the command line as well.

16:13 - I like it.

16:14 What's, and then do you wanna cover Ask?

16:16 - I do, I wanna cover Ask.

16:18 So, ASQ.

16:20 So this is another one.

16:21 I don't think we've covered it, but I guess it's possible that we have.

16:25 There's kind of two philosophies here.

16:27 They both do the same general thing, flupy and ask, A-S-Q.

16:34 But the operations that you get to apply these fluent collections changes or vary.

16:40 So the flupy is like map and filter and so on.

16:44 And the other idea is if we work with this link operation, this link which comes from the C# space and is actually, I think, one of the better parts of C#.

16:54 It really maps very closely to the SQL query language with WHERE, SELECT, ORDER BY, and so on.

17:01 So ASK is very, very similar to this for Python.

17:06 It's exactly the same thing, a fluid API on top of collections.

17:10 But here's a collection of words, like 0, 1, 2, 3, as written out.

17:15 And you say query of words dot ORDER BY len, then by and you pass it just here is just ordering alphabetically but you could pass something else and take five select string dot upper to list and once you get back as a list of the first five and that ordered by length one so the the short ones like one six ten and so on to me this the database like operators of ask is a little bit nicer than the map filter reduce syntax of the other one, but they're both valuable and they both have the same idea.

17:50 So I kind of thought it'd be fun to just cover them, cover them both here.

17:53 I think it's good there's both of them because it kind of depends on where, what sort of mindset you're thinking about.

17:59 It's a different way to think of similar problems.

18:01 Yeah, but they both solve similar problems in really interesting ways.

18:05 You know, I mean, you could say like a list comprehension.

18:07 Okay, well, list comprehension could do a filter and it could do a map, but it doesn't.

18:14 I wish there was just a little bit more on those types of things.

18:17 Like, for example, wouldn't it be great if there was a order by clause in a list comprehension instead of having to do the list comprehension and then do a sorted on top of the result, you know, things like that.

18:28 It's just, it's like you could almost not do these things with list comprehensions, but not quite.

18:33 You can't do a paging and stuff.

18:35 So it's like, it just adds that little bit extra that often you really need in practice.

18:38 Yeah.

18:39 Yeah.

18:39 True.

18:39 Nice.

18:40 So there's a pretty exciting blog that's kicking back into gear.

18:43 Yeah.

18:44 It's a blog called Neopythonic, and it's Guido's blog.

18:48 And the last time we saw him, it looks like the last blog post before November was in July of 2016.

18:57 So one of the benefits of him stepping down as a BDFL is that he can blog now, so this is a good.

19:05 - Yeah, I think it's, and you know, the one from July was just him turning off comments 'cause people are spamming us.

19:10 So it was even further back than that.

19:12 It's great to see him getting back into action.

19:15 So we have this.

19:16 We have him doing some more interviews with the MIT podcast video series and whatnot.

19:23 So yeah, it's really great.

19:24 And a couple of questions.

19:25 So he's just answering listener questions or student questions, really.

19:30 And a couple of questions are whether to choose a 9 to 5 job or be an entrepreneur.

19:35 And then the other one is, will AI make human software developers redundant?

19:41 And of course the answer to the second one is no, because software development is not boring and AI is to replace boring tasks.

19:50 And like driving a car, yeah.

19:52 But the nine to five versus entrepreneur is interesting because as we know, he's worked for other people for a while, so.

19:59 - Yeah, some really interesting places, right?

20:01 Like Google and Dropbox and stuff like that.

20:04 - Yeah, I mean, and they kind of tie together because at the end of the nine to five question, He said, "Just be sure to not take a job "that can be immediately replaced by a computer." - Yeah, for sure.

20:17 Did you ever see the posters called Demotivators?

20:19 I think that's what it's called.

20:21 - No, it sounds great.

20:22 - So you know those posters that corporations will hang up that just make you cringe?

20:26 It'll be like an eagle flying over ocean with a sunset and it'll say, "We all soar together," or, "We can all soar higher." Like something you're just like, "Whoa, no, come on.

20:39 Who's that supposed to act?

20:40 So demotivators have pictures like that, but the caption isn't something super positive.

20:45 Like one example would be there's like a hang glider flying over an ocean into a sunset, and the caption is something to the effect of like, it's so great to dream big and to fly out, you know, like soar towards your dream.

21:01 It's even better to sit on the shore and watch people sink into the ocean when they, you know, it's just like stuff like this, right?

21:07 - So there's one about what you're talking about here, and it says, see, do you have a boring, it's got some picture of people working together.

21:16 It says, "Your job is boring.

21:17 "So boring that it'll probably be replaced "by robots in 10 years.

21:21 "You should find a new job." But it's got this beautiful picture.

21:24 Maybe I'll try to find it and link to it, but yeah, they're really good.

21:27 - It's dark, nice.

21:28 - Yeah, it's great.

21:30 Great humor.

21:32 All right, so the last thing I wanna talk about today is something called Anvil.

21:37 And I'm surprised I haven't talked about on the show.

21:39 I've had Meredith on Talk Python who created this.

21:42 And I think there's several interesting aspects.

21:45 In and of itself, it's interesting.

21:47 But then also how it works, pretty unique, and maybe inspires some other folks as well.

21:53 So Anvil is a visual web building tool.

21:58 So what you do is you go there and you log in and it gives you kind of this design surface, right?

22:05 You can go and you've got buttons and text boxes and combo boxes and stuff like that.

22:11 And you can drop them down and interact with them, set some properties.

22:14 And you've got this visual bit, and you've got this sort of code part that runs as you interact with it.

22:18 So you've got a visual piece you can work with and a code behind, or code view, they call it.

22:23 And what's really interesting is all of that stuff happens in Python.

22:28 So if you've got a combo box, you want to hook the change event, you don't go and write JavaScript.

22:34 you double click the little combo box and it takes you to a method that's Python, you know, def combo box dot click or underscore click.

22:43 And that little comma that says, write your code here.

22:45 - Oh, nice.

22:46 - That code is Python code and it runs in your browser.

22:49 - That's cool.

22:50 - So, you know, we've talked a lot about WebAssembly and some of these other things about how do we get Python in your browser.

22:55 Here is like a full web development framework where the majority of your logic runs on the client side, written in Python, but then behind the scenes, you know, translated to JavaScript.

23:08 - Okay, well this would be fun to play with.

23:10 - Yeah, it's really interesting.

23:11 And then they also have like a backend database and a backend services where you can write some code that runs in like a Docker container or something.

23:18 So you have a server side bit and a front client side bit.

23:20 But most of the code you write by default would probably land up in, you know, if you don't restructure things, probably land up as Python running in the browser.

23:28 And it's really quite nice.

23:30 So here's a cool way for people who don't really know that much about web apps, but they know Python, a really nice way to quickly get up to speed with sort of what I think of as like forms over data type web apps, right?

23:43 Here's a grid, here's a bunch of input boxes I need to enter that data, save it, put it into a list and show it to you.

23:50 Like really quite common types of apps for sort of intranet type apps.

23:55 So check that out, it's pretty cool.

23:57 And it does have a paid version, but there's also a free version.

24:01 So maybe it's interesting to people, but I think it's just interesting as a case study of how they're using Python in the browser as well.

24:08 - Yeah, cool.

24:09 - Cool, yeah, all right, well, that's it for our six items this week.

24:12 Brian, you got anything else you wanna share with everyone?

24:15 - Yeah, I wanna share, I've got big news, big, big news.

24:18 Anyway, not huge news, but I wrote a book called "Python Testing with pytest." Did you know that? - You did the book?

24:25 Yes, I did, it's awesome.

24:26 (Brian laughs)

24:27 - Yeah, so we talked about it a lot for the first year we were doing this.

24:31 But the news related to that is there's a second printing out.

24:35 So I updated it, ran everything with Python 3.7 and the modern version of pytest at the time.

24:43 And then also fixed a whole bunch of the errata of people that pointed out things like just little typos and things like that.

24:51 And updated the source code, stuff like that.

24:53 So it's--

24:54 >> Nice, congratulations.

24:55 >> Available, P2 is available.

24:57 They call it-- P2 must be second printing, P2.

25:01 How about you?

25:01 Any news?

25:02 No, nothing.

25:03 Nothing right now.

25:04 I'm in just massive podcast recording mode, trying to get ready for the holidays, both so that I can take a week or two off.

25:11 And it's also hard to get guests to come on over the holidays, because obviously they want to do just chill out stuff.

25:19 It's been all recording all the time.

25:21 So I have cool things I've recorded, but they're not out yet.

25:24 We've even done one together, right?

25:26 That'll be fun.

25:27 - Yeah, we-- - At the year end.

25:28 - And then I had you on mine too because I have the same problem you do.

25:32 - That's right.

25:33 Yeah, we got a lot of fun stuff coming out on Testing Code and Talk Python, and of course, as always, on Python Bytes.

25:39 - So we're not taking a break over Christmas break.

25:42 We're gonna try to go, just go all the way through.

25:44 - That's right.

25:45 We may physically be taking a break, but due to the magic of time shifting and podcasting, the podcast will not be taking a break.

25:51 - Yes, wonderful.

25:52 - Yeah, awesome.

25:53 - Well, thanks for talking to me this morning.

25:54 - Yeah, you bet.

25:55 Bye. - Bye.

25:56 for listening to Python Bytes. Follow the show on Twitter via @pythonbytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Auchin, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.