Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to earbuds.
This is episode 106, recorded November 30th, 2018. I'm Michael Kennedy.
And I'm Brian Otkin.
Hey, Brian. How are you doing?
I'm good. It's early in the morning, man.
Yeah, good morning, man. We're recording a little earlier than our average lunchtime get-together.
But, you know, this is what we do for our audience. We get up when it's dark and we talk about Python.
Yes, we do.
Yeah. So before we get on to the topics, I just want to say thank you to DigitalOcean.
They're sponsoring the show like they did for much of this year.
So check them out at pythonbytes.fm/DigitalOcean. Get $100 free credit for new users.
Speaking of cloud hosting and whatnot, one of the main innovations that's been coming along here the last couple of years is this whole idea of DevOps, right?
I don't know. DevOps has been around for a while and really important for a lot of people.
But I think Python is getting involved more and more with DevOps.
We've also talked about packaging tools and dependency management a lot on this podcast and our others as well.
This is actually, we've talked about Poetry and Pipenv.
I don't know if we've covered pip-tools before, but...
I don't think we have.
pip-tools is a package that combines, like, it has a couple commands, pip compile and pip sync,
which are ways to work with pip-based, usage-based packages.
And I'm not going to really get into that too much, but what I found was interesting was there's an article recently written by Hynek that is called Python Application Dependency Management in 2018.
So, basically, he's got a use case.
He's working in DevOps or a DevOps problem space right here.
And the requirements he's got for doing packaging is he wants to be able to specify immediate dependencies, like a requirement.text or something.
But then be able to have a tool resolve the dependency tree.
So, I know I need Django, but if Django needs a bunch of stuff, I don't want to have to specify all that.
And then, hopefully, have hashes like the lock files.
And then integrate these with talks so that you can run tests.
And then, also, the last one is in deploying servers, I don't want to actually have to activate, like, a virtual environment and then install everything.
I want to be able to install dependencies, an application and a dependency, into a virtual environment without having to actually go in there and do it.
Those all seem reasonable, but so far, he's tried PIPM, he's tried Poetry, and right now he's sticking with pip Tools-based solution.
Like using the pip Tools thing itself or just PIP?
A combination of them.
Because being able to, you can't just use pip alone for a lot of this stuff.
So, like, for instance, pip Tools is one of the things that you can create hashes with.
You can create, like, the lock files, essentially.
And working with multiple requirements files, like a requirements text or, like, a requirements.
He has a work model, which I haven't seen before, which is sticking multiple requirements files in a requirements directory.
So, you've got a main.
Oh, interesting.
Like a requirements slash main.txt and then a dev.txt for development requirements.
And then you also have to be able to update them.
So, occasionally, you've got pinned requirements, but you want to be able to update them to the newest and test that.
So, all these things, it requires, like, a work model that should...
These are reasonable requests.
One of the things I really liked about this was it was just, basically, we have a very...
We have an environment now that we're...
We want to be able to...
I think it's a good thing we want to be able to not slam people for, like, make fun of or bash on different projects too much.
But you also want to be able to be honest and say, hey, I tried this thing and it didn't work for me.
And so, one of the things I liked about this is I think it's a good example article of saying, hey, this was my use case.
I tried this thing.
This thing didn't quite work for me and here's why.
But also, don't slam the people who made that thing because I think it's cool that they're working on the problem.
It just didn't work quite right for me yet.
Yeah, this whole package management, package tooling stuff, dependency tooling, it feels a little bit like the editor wars, like, don't tell me to use Vim.
I want to use Emacs or I want to use PyCharm or whatever.
Not quite, but it seems something like people get into their workflow and, you know, they really like their workflow and they kind of want to stick with it.
Yeah.
I think there's also some limitations of some of these tools.
They're built to address different problems than the one you're trying to solve.
Right.
They're not one-size-fits-all things.
I find it refreshing because I actually really like the idea of poetry.
Actually, I like the idea of Pip-Am-Vam poetry, but I don't have any specifics right now, but I jumped in and tried to use it in some of my workflows.
Like, some of the things I tried just didn't quite work.
So I have to go back to old ways.
Yeah.
I mean, I guess that makes sense.
The more that those tools try to do for you all at once, the more likely they are to not exactly fit your workflow.
Yeah.
So one way that's really cool to extend applications would be to have the ability for someone to just drop a file into a directory, accessing a common API and have that just become part of your app or part of the capabilities of your app.
Right.
Like a plugin, right?
Yeah.
That'd be cool.
So there's a cool library called PluginLib.
PluginLib.
P-L-U-G-I-N-L-I-B.
Yeah.
So really nice.
And it addresses this problem in a couple of ways.
And when I first heard about this, I'm like, well, can you just put, you know, like a file in a directory and be good with it?
You know, just try to load it.
But the more I looked at what this library is doing, it's actually solving a lot of problems for us.
For one, it's an interesting example of metaclasses and programming with metaclasses.
But you don't really have to deal with it because you just use the API and it works.
But one of the things, plugins are validated when they're loaded instead of when they're used, right?
Right away, they're validated.
Make sure everything fits the API.
They can be loaded through either file paths, through entry points, through module names, and so on.
You could have multiple versions of the same plugin.
Oh, interesting.
Yeah, pretty interesting.
You can blacklist them.
So if somebody puts a, you know, some kind of plugin there, you're like, no, we're not going to allow you to load this even if we find it, you know, configure something like that.
You can group them.
You can have conditionals like this plugin works on macOS but not on Linux, for example.
All sorts of cool stuff like that.
So if you were thinking about having this extensible plugin model, you know, give plugin them a look.
It looks like it handles a pretty simple problem but has a lot of support for you while it's doing it.
Yeah, this is pretty neat.
Just check this out.
Yeah, you know, I just don't work on any apps that really, that makes a lot of sense for me.
But if I did, I would definitely want to have a look at that.
Yeah, it looks like it's based on, or partly based on abstract methods of class hierarchies.
Yeah.
And I think I remember looking at this, and this doesn't scare me off, but the particular application I was working with, we didn't have any classes uses anywhere, and I didn't really want to introduce them for this.
You're like, let's throw in some abstract-based classes and hierarchies do a thing that doesn't even use classes.
Yeah.
Yeah, that makes sense.
But at the same time, writing a plugin for an application is a very specific need, and I think it's fine for if it didn't match the model of the rest of your system.
Yeah.
Something to look into.
Cool.
Yeah, for sure.
So what do you got for us next?
For some reason, I'm a little interested in testing sometimes.
Really?
Yeah.
I'm surprised.
One of the PyBytes guys, Bob Belderboss.
Belderboss?
He wrote an article called, How to Test Your Django App with Selenium and pytest.
And yeah, pytest is one of my favorite things, too.
It's a really nice write-up of, I'm going to quote the beginning of it.
It says, in this article, I will show you how to test a Django app with pytest and Selenium.
One of the things I was intrigued by is they don't use a toy app.
They use their Code Challenges platform and compare the logged out homepage versus the logged in dashboard.
And I think that's really cool and brave of them to publish it.
This is how we're testing our own stuff.
Publicly.
On the internet.
Yeah, no, it's really cool.
Yeah.
And it talks through, I probably wouldn't have done it.
Part of the project setup, they're using the activate and deactivate of a virtual environment to load environmental variables.
I don't have specific links to it, but I know there's ways to do this within pytest itself as well, using something else.
Yeah, I know.
It's kind of cool to do that, but at the same time, it makes it hard to, it makes it a little bit hard to, I think it's kind of hidden in the whole setup, right?
There's nowhere in the code that you see that.
And you basically accomplish that by editing the activate script to set some environment variables when you activate the environment.
But if you go to another machine, you check it out again, some other location, that kind of stuff is not there.
So it's, you know, it's hard to store passwords and secrets.
You don't want them on, they definitely don't want to put them into this repo.
Right.
So what are you going to do?
But it is tricky.
Yeah.
And you wouldn't be able to use Tox with this because Tox is going to create its own virtual environment.
But yeah, so I'll have to, maybe I'll have to write up an example of how I would tackle that.
Yeah, that's actually would be interesting.
But the rest of it's, even though they're not using, mocking out a database itself, they do show how to do that, how to get the database set up.
And then, yeah, anyway, it's just kind of a cool example.
I love working with Selenium.
And this is a neat write-up.
It's cool.
You know, maybe not everyone knows about Selenium.
They maybe know about web scraping and maybe trying to test it that way.
They also maybe know about testing, like, inside of apps.
Maybe just quickly tell folks who are not totally familiar, what is this like?
Oh, yeah.
Is this like request or what is this?
What Selenium does is it really just opens up while it's running.
You don't have to interact with, like, what you see is what you get or anything.
You're using Python code to control a web browser.
And you can use it in a couple of ways.
You can have, like, a web browser that you can actually see.
So while it's running, it pops up on your screen.
Or, like, for instance, Chrome and I believe Firefox also has a headless version where it can operate without actually –
they just call it headless because it doesn't pop anything on your screen.
But other than that, it interacts just like it, as if it was open.
And you can, you know, navigate the DOM and you can select things and activate things.
And one of the benefits of going through this using Selenium is that all the JavaScript stuff is loaded also through loading everything.
You can interact with it as if it were just literally in your browser because it's literally in a hidden browser in the headless mode.
So, yeah, it's definitely an interesting way to do it.
And, like, his example, they're like, let's take these things and put it in the username, put it in the password, and then click this button.
You know, find this button and click it, the login button and stuff like that.
It's really unique.
Okay, so before we get on to FluPi, which is the next thing I want to talk about, let's talk about DigitalOcean real quick.
So you probably built a lot of apps, have a lot of cloud resources running so far this year.
But, you know, which one goes with which project, that's always been something that's driven me crazy about other places.
Like, go to AWS and there's, like, just tons of virtual machines running and stuff.
You're like, what is this?
Do we need these?
I don't even know anymore.
So it would be really nice if we could group them together.
And at DigitalOcean, they create this idea called projects where you can group droplets and load balancers and IPs and all that into application categories that they work with.
Like, you could have, you know, your app.
You could have a your app sandbox testing Q&A environment.
And you could have, you know, multiple ones of those and so on.
So really nice, super cool way to organize your environment in DigitalOcean.
Check them out at pythonbytes.fm/DigitalOcean.
And like I said, for people who are new users, you get $100 free credit to play around.
So that's also really nice to have.
Very nice.
So I'm a big fan of fluent APIs in general.
And, you know, maybe not everyone knows the term.
It's like when you take a thing and you call a function on it and that function itself returns back the same object.
So instead of saying I've got a variable and I first initialize it to a list and then I do some operation on that list and it changes it.
And I do another operation.
Maybe it turns it into a set.
Then I pass that set somewhere else.
I could just say, you know, like, let's say customers or let's say numbers.
This is an example.
It's easier.
So say numbers.map.filter.skip.take.orderby, right?
Just like one after another all in one line.
That's the fluent API in general.
So the thing that I want to point out is actually two libraries.
So you can take your flavor of this fluent API and apply it to collections.
Because there's often more than one thing you might want to do to a collection.
You might want to change its values.
You might want to filter it.
You might want to only take a couple of them or put them into groups and things like that.
So there's one called, I originally wanted to pronounce it as fluffy, but flupy.
Flupy is in fluent Python.
And it lets you do, well, exactly what I was describing on collections.
So given any collection, you can upgrade it to a fluent collection by just calling flu and passing the collection.
And then it has all these operations.
Like you say .map and pass it a lambda.
And then on that result, you say .filter, pass another lambda, .chunk to break it into like pages, I believe.
And .take will like give you only three chunks of five back, for example.
So you can do that all in effectively one line.
Now maybe you want to wrap it for readability, but you don't have to like use a bunch of intermediate variables or intermediate collections or anything like that.
Nice.
It's like, right?
And the thing that's really cool is it's all based on yield and generators.
Really?
Yeah.
So every step in this chain is some generator type of operation, which means you could pass it a billion items.
And it's not going to like filter a billion items and map a billion items and all that.
It's just going to take them one at a time through this sort of data pipeline that you build.
Nice.
Yeah.
So the example that I put in the show notes uses count.
Like just all of it.
All of it.
But, you know, it ends with a break it into like pages of five and take three pages.
Right.
So it only takes 15 items out.
Okay.
Even though you pass it in the way that that works, of course, with generators, because you can just pull through till like the last part stops pulling.
And then it basically stops iterating this infinite collection.
Really nice.
In the little example you give, you do like a four item in pipeline at the end too.
So you can use the result as a generator also.
Yeah, exactly.
Because it's a generator, like that whole sequence of operations doesn't actually execute until you iterate it, until you pass it to something that itself does.
Like pass it to a list or to a dictionary or something like that.
Yeah.
Or use it in another one of these pipelines.
So it's really cool to build up these data pipeline, data transformation things using FluPyde.
Nice.
Yeah, and they also have a CLI.
So you can do this kind of stuff from the CLI for this guy's data science team.
Not everyone necessarily is great with Linux, but they would like to do a little bit more with these tools and with these ideas, these Python data pipelines.
So you can actually execute these sorts of operations on the command line as well.
I like it.
And then do you want to cover ask?
I do.
I want to cover ask.
So ASQ.
So this is another one.
I don't think we've covered it, but I guess it's possible that we have.
There's kind of two philosophies here.
They both do the same general thing.
FluPy and ask.
ASQ.
But the operations that you get to apply to these Fluent collections changes or varies.
So the FluPyde is like map and filter and so on.
And the other idea is if we work with this link operation, this link API, which comes through the C# space and is actually, I think, one of the better parts of C#.
It really maps very closely to the SQL query language with where, select, order by, and so on.
So ask is very, very similar to this for Python.
It's exactly the same thing.
A Fluent API on top of collections.
But here's a collection of words like 0, 1, 2, 3 as written out.
And you say query of words dot order by len, then by, and you pass it just here.
It's just ordering alphabetically, but you could pass something else.
And take five, select string dot upper to list.
And what you get back is a list of the first five in that ordered by length one.
So the short ones, like 1, 6, 10, and so on.
To me, the database-like operators of ASQ is a little bit nicer than the map, filter, reduce syntax of the other one.
But they're both valuable, and they both have the same idea.
So I kind of thought it'd be fun to just cover them both here.
I think it's good there's both of them because it kind of depends on what sort of mindset you're thinking about.
It's a different way to think of similar problems.
Yeah, but they both solve similar problems in really interesting ways.
You know what I mean?
You could say like a list comprehension.
Okay, well, a list comprehension could do a filter, and it could do a map.
But it doesn't – I wish there was just a little bit more on those types of things.
Like, for example, wouldn't it be great if there was a order by clause in a list comprehension?
Instead of having to do the list comprehension and then do a sorted on top of the result, you know, things like that.
It's just – it's like you could almost not do these things with list comprehensions, but not quite.
You can't do a paging and stuff.
So it's like it just adds that little bit extra that often you really need in practice.
Yeah, yeah, true.
Nice.
So there's a pretty exciting blog that's kicking back into gear?
Yeah.
It's a blog called Neopythonic, and it's Guido's blog.
And the last time we saw him, it looks like the last blog post before November was in July of 2016.
So one of the benefits of him stepping down as a BDFL is that he can blog now.
So this is a good –
Yeah, I think it's – and, you know, the one from July was just him turning off comments because people are spamming us.
So it was even farther back than that.
It's great to – it's great to see him getting back into action.
So we have this.
We have him doing some more interviews with, like, the MIT podcast video series and whatnot.
So, yeah, it's really great.
And a couple of questions – so he's just answering listener questions or student questions, really.
And a couple of questions are whether to choose a 9-to-5 job or be an entrepreneur.
And then the other one is, will AI make human software developers redundant?
And, of course, the answer to the second one is no, because software development is not boring and AI is to replace boring tasks.
And, like, driving a car, yeah.
But the 9-to-5 versus entrepreneur is interesting because, as we know, he's worked for other people for a while.
Yeah, some really interesting places, right?
Like Google and Dropbox and stuff like that.
Yeah, I mean, they kind of tied together because at the end of the 9-to-5 question, he said, just be sure to not take a job that can be immediately replaced by a computer.
Yeah, for sure.
Did you ever see the posters called Demotivators?
I think that's what it's called.
No, it sounds great.
So you know those posters that, like, corporations will hang up that just, like, make you cringe?
It'll be, like, an eagle flying over, like, ocean with a sunset, and it'll say, we all soar together, or we can all soar higher.
Like, something you're just like, whoa, no, come on.
Who's that supposed to act?
So Demotivators have pictures like that, but the caption isn't something super positive.
Like, one example would be there's, like, a hang glider flying over an ocean into a sunset.
And the caption is something to the effect of, like, it's so great to dream big and to fly out, you know, like, soar towards your dream.
It's even better to sit on the shore and watch people sink into the ocean when they, you know, it's just, like, stuff like this, right?
So there's one about what you're talking about here, and it says, see, do you have a boring, it's got some picture of, like, people working together.
It says, your job is boring.
So boring that it'll probably be replaced by robots in 10 years.
You should find a new job.
But it's got, like, this beautiful picture.
Maybe I'll try to find it and link to it.
But, yeah, they're really good.
Yeah, it's dark.
Nice.
Yeah, it's great.
Great humor.
All right.
So the last thing I want to talk about today is something called Anvil.
And I'm surprised I haven't talked about it on the show.
I've had Meredith on Talk Python who created this.
And I think there's several interesting aspects.
In and of itself, it's interesting.
But then also how it works.
Pretty unique.
And maybe inspires some other folks as well.
So Anvil is a visual web building tool.
So what you do is you go there and you log in.
And it gives you kind of this design surface, right?
You can go and you've got buttons and text boxes and combo boxes and stuff like that.
And you can drop them down and interact with them, set some properties.
And you've got this visual bit.
And you've got, like, this sort of code part that runs as you interact with it.
So you've got a visual piece you can work with and a code behind or a code view, they call it.
And what's really interesting is all of that stuff happens in Python.
So if you've got, like, a combo box, you want to hook the change event, you don't go and write JavaScript.
You, like, double-click the little combo box and it takes you to a method that's Python, you know, def, combo box.click or underscore click.
And that little comment says, write your code here.
Oh, nice.
That code is Python code and it runs in your browser.
That's cool.
So, you know, we've talked a lot about WebAssembly and some of these other things about how do we get Python in your browser.
Here is, like, a full web development framework where the majority of your logic runs on the client side written in Python, but then behind the scenes, you know, translated to JavaScript.
Okay.
Well, this would be fun to play with.
Yeah, it's really interesting.
And then they also have, like, a back-end database and a back-end services where you can write some code that runs in, like, a Docker container or something.
So you have a server-side bit and a client-side bit.
But most of the code you write, by default, would probably land up in, you know, if you don't restructure things, probably land up as Python running the browser.
And it's really quite nice.
So here's a cool way for people who don't really know that much about web apps but they know Python.
A really nice way to quickly get up to speed with sort of what I think of as, like, forms over data type web apps, right?
Here's a grid.
Here's a bunch of input boxes.
I need to enter that data, save it, put it into a list, and show it to you.
Like, really quite common types of apps for sort of intranet type apps.
So check that out.
It's pretty cool.
And it does have a paid version, but there's also a free version.
So maybe it's interesting to people, but I think it's just interesting as a case study of how they're using Python in the browser as well.
Yeah.
Cool.
Yeah.
All right.
Well, that's it for our six items this week.
Brian, you got anything else you want to share with everyone?
Yeah, I want to share.
I've got big news.
Big, big news.
Anyway, not huge news, but I wrote a book called Python Testing with pytest.
Did you know that?
Yes, I did.
It's awesome.
Yeah.
So we talked about it a lot for the first year we were doing this.
But the news related to that is there's a second printing out.
So I updated it, ran everything with Python 3.7 and the modern version of pytest at the time.
And then also fixed a whole bunch of the errata of people that pointed out, things like just little typos and things like that.
And updated the source code, stuff like that.
Nice.
Congratulations.
So it's available.
P2 is available.
They call it P2.
It must be second printing P2.
Anyway.
How about you?
Any news?
No, nothing.
Nothing right now.
I'm in just massive podcast recording mode trying to get ready for the holidays, both so that I can take a week or two off.
And it's also hard to get guests to come on over the holidays because obviously they want to do just chill out stuff.
It's been all recording all the time.
So I have cool things I've recorded, but they're not out yet.
We've even done one together, right?
That'll be fun.
Yeah.
And then I had you on mine too because I have the same problem you do.
That's right.
Yeah, we've got a lot of fun stuff coming out on Testing Code and Talk Python.
And of course, as always, on Python Bytes.
So we're not taking a break over Christmas break.
We're going to try to just go all the way through.
That's right.
We may physically be taking a break, but due to the magic of time shifting and podcasting, the podcast will not be taking a break.
Yes, wonderful.
Yeah, awesome.
Well, thanks for talking to me this morning.
Yeah, you bet.
Bye.
Thank you for listening to Python Bytes.
Follow the show on Twitter via at Python Bytes.
That's Python Bytes as in B-Y-T-E-S.
And get the full show notes at pythonbytes.fm.
If you have a news item you want featured, just visit pythonbytes.fm and send it our way.
We're always on the lookout for sharing something cool.
On behalf of myself and Brian Okken, this is Michael Kennedy.
Thank you for listening and sharing this podcast with your friends and colleagues.