Transcript #144: Are you mocking me? It won't work!

00:00 Hello and welcome to Python Bytes, where we deliver news and headlines directly to your earbuds.

00:04 This is episode 144, recorded August 20th, 2019. I'm Brian Oggen, and Michael's on vacation, but I have two awesome guest hosts. Welcome, Chris May.

00:15 Hey there.

00:16 And also welcome Mahmoud Hashemi.

00:18 Hello, everybody. It's good to be back.

00:20 So both of you have been on Test and Code, but have either of you been guests on Python Bytes before?

00:26 I think I actually guest hosted once in a similar situation where Michael was on vacation or you were on vacation.

00:33 I think it was me and Michael that time.

00:34 So it's good to get the reverse.

00:36 Cool.

00:37 And I definitely never have been on Python Bytes, but it's been something I would have been excited about ever since I thought I had the opportunity.

00:43 Dreams do come true.

00:44 Indeed.

00:45 I really appreciate both of you coming on.

00:48 As a reminder to everybody, this episode is brought to you by DigitalOcean.

00:51 Check them out at pythonbytes.fm/digitalocean.

00:55 a $50 credit for new users, lots of good stuff.

00:58 More about that later.

01:00 Let's just jump in.

01:01 I'm going to put Chris on the spot and have him go first since he's new here and we can heckle.

01:06 Yeah, bring it on.

01:07 No.

01:08 Yeah, my first topic I selected was "Why Your Mock Doesn't Work" by Ned Batchelor.

01:15 And it's an article that just came out like perfect timing for my team and I because there There are members on my team who are building an application that's syncing or bringing data from one database server to another across cloud platforms.

01:32 Essentially, they're trying to do a test-driven development and they came to me and they're like, "How are we supposed to do this?

01:41 There are all these third-party clients that we're connecting to and trying to get authorization from and all these different things." They quickly found out that mocking these third party tools was hard.

01:49 I was just so thankful that Ned wrote this article to show us that one problem is that sometimes mocking, the way that they're setting up their mocks, the item that they were mocking never actually mocked the object that they were trying to mock.

02:05 It's a little hard to explain in video, but, or in audio rather, but Ned actually has some great illustrations to show exactly how mocking works and when it works and how it can be messed up.

02:17 - I saw those illustrations.

02:18 I don't know if he's using graphvis or what, but we're definitely getting the point across.

02:22 - Yeah, I was wondering, 'cause I thought he did such a great job with those illustrations.

02:26 I was like, how did he take the time to design those?

02:29 It's good to know that there may be a tool behind that.

02:32 - Well, I mean, and he's got, yeah.

02:34 I mean, he used them in his Python names and values talk and blog post, and I always, like especially for beginner to intermediate programmers, I always send them to that Python names and values talk.

02:43 It's a really good use of the half hour.

02:46 - Nice.

02:47 Hey, look, we're getting bonus content already.

02:48 - Yeah, and that actually, that names and values talk actually makes you a better mock user as well, 'cause it helps you understand how that all works.

02:59 Plus Ned's just really good at explaining stuff.

03:01 - Absolutely.

03:02 The other thing I love about this article is he also links to two other resources, another article and a video where people gave examples of when you don't wanna use mocking and also good practices when you do.

03:18 And one of the takeaways I took away from my team is that, especially if you're trying to mock third-party code, you're going to need to create tests that of course test your own application against the mock, but also test the mock against the third-party tool.

03:31 And so this is a bunch of stuff that you're going to invest in if you want to test with mocks.

03:37 If I can jump in, what I'll say, whenever you're integrating and there's an SDK or a library for integrating, chances are you're not gonna use 100% of that library or that service.

03:46 You're probably gonna use a little sliver of it.

03:47 I know it seems like, oh, great, they wrote all the code for me.

03:50 Let me jump right in and integrate directly with their library.

03:54 But it really makes sense a lot of the time to use a gateway or some other pattern to kind of like make explicit which parts of that library you use.

04:02 Because eventually when you have to integrate with another service or you wanna throw it away, it's just gonna be a lot easier to sort of like maintain that side of the integration.

04:10 So yeah, great posts.

04:12 >>Yeah, I also like to, I think it was in this post here where you mentioned the kind of what you were saying, usually there's only a thin sliver of the functionality you need and to use, you know, to create a mock that only does that, but also to make sure that across your organization or across your team to only have one mock object that handles like that one third party tool.

04:33 >>Mm-hmm, mm-hmm, definitely.

04:34 >>Well, Mahmoud, what do you got for us?

04:36 So what I've got for the group here is a little library/application that I ran into while doing some research for a talk that I'll get into later.

04:46 But basically it's called "vermin".

04:49 And the name's not really explained, but the only reason I can see behind it is that Python has snakes, and snakes are kind of vermin, and vermin kind of sounds like "version".

04:58 And so what vermin is, that's like, you know, V-E-R-M-I-N.

05:03 Oh wait, I just got it. It's like version minimum. Okay.

05:06 So what it does is it's a rules-based Python version compatibility detector with a pretty clever name.

05:13 So what it does, maybe you heard the news, but basically now all 360 top downloaded PyPI libraries now support Python 3.

05:23 There's like a website for it.

05:25 And the way that that's determined is, I think, Brett Cannon has a library or application which basically looks at the setup.py and then looks at the index classifiers.

05:38 And so if the person who published the package says it supports Python 3, it takes it at its word and says that this library says it supports Python 3.

05:46 Now, the top 360 libraries have done so, which is a pretty good milestone for us, especially as we approach 2020.

05:52 But the thing is that it's based on what people say.

05:54 So if you, when it comes to your own code, right?

05:57 Like if you're like a team at a company or a maintainer of an application that doesn't necessarily have a setup.py, there's, you know, can I use Python 3?

06:07 It's not really gonna help you 'cause you don't have those classifiers.

06:10 So basically, vermin, what it does is it scans over every single py file, every single module, and it tells you the minimum Python version necessary to import that module.

06:21 And it even gets down to the function level.

06:24 So basically you can use this to scan your code and say like, look, there's a Python 2.7 like feature being used by this function over here in this module. And then you can also certify that, Hey, this module over here uses like Python 3.5 syntax, or, Hey, that one actually is using a 3.7 thing, but we're only on 3.6.

06:43 So it's rules-based, it's got over a thousand rules. It's pip installable, you pip install vermin, and it has a little command line entry point. And, Yeah, when I found it, it had like less than 80 stars on GitHub, and I'm hoping, you know, I saw how much work went into this, I'm hoping that maybe it blows up a little bit, because it's a very useful tool, and I'll explain why in my next segment.

07:03 But basically, yeah, definitely check it out, Vermin.

07:06 That's really cool. I mean, I could definitely see...

07:08 Yeah, it sounds really cool. It sounds like a lot of work went into that.

07:11 Exactly, a lot of work went into it, and it's still a little bit quirky, I'll tell you.

07:14 Like, I ran it with just plain Vermin as described, and I got one result, And then I ran it with -v, which made it kind of like, it's supposed to make it more verbose, I thought, but it actually made it look harder and it came back with a more accurate result.

07:31 So maybe when you run it, run it with -v.

07:33 Or maybe help the guy out, I think his name is Morton Christensen.

07:37 Maybe help him out and give him a PR, make the verbose act the same way as the non-verbose.

07:42 But no, great tool regardless.

07:43 - Ah, okay.

07:44 The verbose might be more picky.

07:46 Very interesting.

07:47 Well, I think vermin looks cool and I'm excited about going from not just from two to three, but for instance, we now are excited about Python 3.8 features or 3.7 features, and I want to make sure that it's possible to move things around.

08:02 >> Exactly.

08:03 >> There's a lot of use for that and forever, I think.

08:06 Anyway, I don't want to talk about a new feature.

08:09 I actually want to talk about something that's been around for a while, but I don't see it a lot in code.

08:13 and I just saw this recently, an article called the non-local statement in Python.

08:19 I'm going to attempt this name, Abolash Raj, I think that's it.

08:24 But the idea is if you use a local variable, like if you assign to a local variable, it creates a new thing in the namespace, a local name.

08:35 The normal thing that everybody knows is unless you add the global.

08:40 But global doesn't just make it so that you can assign a variable that's not declared in this function.

08:49 It also makes it so that it's a global scope variable.

08:53 And sometimes that's not really what you want.

08:56 You just want it to reference, like for instance, if you got a local function defined within another function, and you want to reference or assign to an outside variable, It's really easy to see when you look at the code, but I forgot this was around.

09:12 I don't use it very much, but it's pretty cool.

09:14 I think I run into this occasionally and I usually just make things global.

09:18 So Mahmoud, you probably have more experience than me.

09:21 Do you run into this ever?

09:22 Oh my goodness.

09:23 So back when I was like really learning the ropes in Python, and I remember it so clearly back in like 2011, I was playing with the concept of like dynamic scope, which is in contrast to lexical scope.

09:34 So that's like, like Scopuscope, you sort of look within your block or your function, right?

09:38 And Dynamic Scope is this older concept where it's like, you would look up the stack for something.

09:44 I think Emacs Lisp still uses this. Kind of glad that Python doesn't.

09:49 But you know, in like the mad science, like, you know, mode of learning all of Python's cool features, I wanted to, you know, do something like that.

09:57 Yeah, somewhere in there, I was like messing with generators and Dynamic Scope, and I found myself just hitting right into a wall and I talked to Raymond Hedger and he's like, "Oh yeah, what you want is non-local and that's going to be sometime in the far future." Yeah, anyways, I mean, here we are and it's there and frankly if I use it, I'm over that phase. So if I use it, I feel like I'm doing something wrong and I should probably like factor this in a clearer way. But that said, I'm sure it still comes up and, you know, I think everyone's probably going to have a little mad science phase of their own And so let them use non-local and deal with the consequences.

10:31 Well, right. I mean, it's like one of these features that it's used so little that you're just adding complexity to the code because most people won't know what it means.

10:39 Right.

10:40 You got to be careful.

10:41 Yeah.

10:41 You know, it's still like it has its own little place and I'm sure some, like someone has a legit use for it beyond like just messing around. So I'm glad it's there still.

10:51 Yeah, definitely. I'm going to take a little bit of a sponsor break and thank DigitalOcean for sponsoring this show.

10:58 DigitalOcean has been a long time supporter of Python Bytes and we really appreciate it.

11:03 And not only that, all of the infrastructure for pythonbytes.fm runs on DigitalOcean.

11:08 We've got our website, our database, and even the audio file hosting is done through DigitalOcean.

11:13 It's been wonderful, super affordable and solid.

11:17 There's enough features to get us going and to let us grow, but not so many that it's too complicated.

11:24 So I think it's a great platform, easy to start, easy to grow, and highly recommended.

11:31 So if you'd like to give it a shot, visit pythonbytes.fm/digitalocean.

11:35 It'll give you $50 free credit for new accounts, so you can give it a try.

11:41 I think it's a good place to be.

11:42 Get yourself a droplet.

11:43 Yeah, a droplet.

11:45 So let's start this all over again, and not back from the beginning, but it's Chris's turn again.

11:51 So Chris, what you got?

11:53 Well, this week, Microsoft announced that they've improved Python support in their Azure platform.

12:00 I think in particular, they've announced general availability for a bunch of new Python things, some of which I've...

12:06 So the company I work for, we've primarily used Microsoft Azure, and I've used some of these tools in beta.

12:13 But so now it's available to everybody.

12:15 And I'm excited because reading through the announcement from Microsoft, I wasn't 100% sure what was new.

12:22 But what I was really thankful for is Brett Cannon tweeted a link to it and said, "Here are the specific points that he was excited about." The first off is that it's debuting with Python 3.6, but 3.7 support is currently being worked on.

12:36 And actually, I can already see it in preview, which is nice.

12:40 And especially good news for me is that 3.8 support won't take nearly as long.

12:44 So I'm hoping that very shortly after they release Python 3.8, it'll be available in Azure.

12:50 And then the second thing is that there's native async and await support in there.

12:54 And that's pretty exciting, but really the other thing that I wanted to pull from that, that Brett didn't mention in his tweet, is that it's actually pretty impressive, like the amount of stuff that Microsoft does for us.

13:05 In particular, the Azure functions, it's just amazing to me because they have a way of deploying to the web where you essentially just create a new Git remote, and you just push your code up there, there and when it goes, when your code arrives there, it essentially sticks into a Docker image and runs it for you and it's a pretty fast startup and all these things that's like so complicated I can't imagine what's all going on and you don't have to worry about it at all.

13:33 You just have something that runs in your computer, you push it up and it's done and it's really, really impressive.

13:38 I've never played with Azure functions.

13:40 So are those kind of like Lambda functions on AWS?

13:44 Yeah, they sure are.

13:45 and I hear they're probably about as easy.

13:48 I've never actually, I've intended to play with the Azure Lambdas.

13:52 In fact, at the local Python meetup, we've had a couple people talk about how easy it is to do, and I was just thankful that here at work, I've been able to try the Azure functions.

14:02 So, it's really nice.

14:04 - Nice.

14:05 I'll have to give that a shot sometime.

14:06 - Yeah.

14:07 Actually, I think Microsoft offers a free month or two month thing if you want to do it, and on top of that, the Azure functions are incredibly cheap.

14:14 I just, they have a calculator up and you can check to see how much it'll cost to do certain things.

14:19 And I maxed it out, like I have a function that calls a bunch of API endpoints and saves them to the database.

14:26 And we realized if we did it 48 times a day against every single website that the company owns, we still would be in the free tier.

14:33 So it's really impressive what you can do.

14:36 - That's cool.

14:37 Well, I'm kind of excited about this next topic.

14:40 Like super excited because we talked about, I'll have to dig up the episode, but we did bring up awesome Python applications before.

14:49 But we have an update.

14:50 Well, yes.

14:51 I don't remember what it was.

14:52 It was testing code with Python bytes.

14:53 But I remember accidentally dropping this idea on the podcast and it blew up a little bit ahead of when I expected to launch it.

15:01 But yeah, so I'm here bringing an awesome Python applications update.

15:07 And so basically, awesome Python applications, for those who haven't heard of it, is sort of like your standard awesome list where it's like you're on GitHub and there's a readme and there's a ton of links you know and the all these links point to applications that are free and open source Python and so at first it looks like a normal awesome list one it's actually generated from some structured yaml and you know when I started out it was like 25 applications like Zulip and Sentry sort of came to mind but then like the more I look the more I found and it grew to something like 180 by the time it went a little viral.

15:43 And then these days it's like 255.

15:46 Frankly, it'll hit 260 by the end of the week.

15:49 And so the list keeps growing, almost faster than I can keep up with it.

15:52 Thank goodness it's somewhat automated.

15:54 And so this last weekend I was at the local Python conference, PyBay.

15:59 Maybe see you, PyBay 2020.

16:01 It was a pretty big hit there.

16:03 I basically put in the show notes a link to my slides and where the video will be and whatnot. But what I did for my talk there was I took, like I said, structured YAML. So I have all of the links to all of the repos. Technically any awesome list could do this and I'm surprised that more haven't. But I went and I cloned all of the repos. And this isn't just GitHub.

16:25 This is things that are on GitLab, things that are sort of like off any particular software forge. There are things that are on Bitbucket, things they're in Bazaar, you know, it's a whole mix of things and I cloned them all and it was something like 30 gigabytes of code Yeah, and these packages they date all or these applications rather date all the way back to like 1998 like mailman and G edit You know if you're gnomes or G that uses Python and 95% of them have had a commit this year 2019 So these are all very active and the reason I've compiled this isn't so much that we can go use them like of course I actually used several of them without even realizing that they're Python.

17:05 But the thing is that as application developers we need exemplars we can go look to for patterns in testing and documentation and architecture and packaging. Things that people do without necessarily writing a blog post about them or you know going on a podcast or giving a conference talk. So basically I went and I cloned it and I did all this analysis and this is how I found And so, Vermin, the good news here, let me just, I should have led with this, right?

17:32 But the good news is that, I was surprised by this, fully two-thirds of these Python applications, which are on median, like, you know, 10 years old, two-thirds of them support and use Python 3.

17:46 Wow.

17:47 So, I was expecting to find that a lot of these had sort of been, you know, it's tough to maintain open source.

17:53 And I thought that they would sort of have maybe fallen behind a little bit.

17:58 Goodness knows that most corporations I run into, well I don't know about most, but a lot of them are still using Python 2 for their Python applications.

18:08 Even Google, everyone's favorite, even Google hasn't fully ported over to Python 3 yet.

18:14 So yeah, but 66% support Python 3.

18:19 And I've had a bunch of other interesting findings.

18:22 If you go to my slides, you can check them out.

18:24 And I'll also be giving the talk at PyGotham 2019.

18:27 So if you're in New York, early October, I think tickets are available.

18:30 So there's so many cool things in here.

18:32 And there's stuff that I knew about once and forgot about, like Eric, one of the Python IDEs.

18:40 So that's fun.

18:41 Yeah, I mean, the newest one I found, someone at the conference came up and told me about it, is actually, I think, nia.is or .si or something like that.

18:49 it's a torrent tracker for anime. And so this particular anime fan informed me that this is one of the largest torrent trackers out. And you know, it's written in Python, it's right there on GitHub. And you know, it's not on PyPI, it's not really discoverable. You need this word of mouth to kind of find these applications and all the lessons contained therein. Like I calculated it out, over 2,000 years of project maintenance, right? Like in these repos that was just waiting to be learned from. And you know, I think the list should grow even further. So if any of the listeners have applications that they happen to know are written in Python, you know, hit me up on GitHub. I'll definitely like, you know, consider and curate. Nice. Well, this is cool. Thanks.

19:31 Absolutely. It's amazing to see how much that's grown. I remember when you were on whichever podcast you're on and I was just inspired and now I'm even more so again to see like how many things have been built with Python and I still haven't reached the bottom yet. That's amazing. My dream is really to like, you make a static site for this so that you can filter because basically like I can I now have the data and I'll look at publishing this data data set somewhere but basically it's like you know show me the projects that use Docker show me the project that use PI installer show me the projects that use twisted versus whatever you mentioned asyncio right earlier and I was kind of surprised there are quite a few older projects that have actually adopted async like asyncio rather you know like and it's pretty even split between concurrent futures, twisted, tornado, G event, and async I/O.

20:18 They're basically all around 20% on this list.

20:21 - How about that? - Yeah, nice.

20:22 - Yeah, so, check it out.

20:24 - Well, I'm gonna wrap this up with a really easy one.

20:28 We've talked about pre-commit several times, I think, on the show, but if you, and pre-commit is a tool to allow you to run some code before you check in your code to make sure things that are whatever, you've got a style guide or some checks or whatever you wanna do.

20:43 It really is general purpose, but how do you get started?

20:46 And actually this is something that I needed myself, the pre-commit documents now that just recently added a quick start guide.

20:56 And I love it.

20:57 I mean, it just kind of walks you through installing pre-commit, configuring it for the first time, taking a look at what the hooks are, installing the hooks, 'cause you have to install the hooks also, which is something I wasn't aware of.

21:11 And then the initial attempt to just run your hooks against your project.

21:15 That's awesome, but I wanted to add, and one of the reasons why I wanted to bring it up on the show wasn't just because, hey, go check this out, but I'd like to add that doing this slowly is a good idea.

21:26 So I tried this with like three or four or five different hooks, like Flake 8, Black, and some other things, and then let it loose on my code base, and it changed like everything.

21:38 And I thought, I don't really know what's going on here, So I'd like to modify it a little bit to say, if you're adding this to a project, for each new hook, add them one at a time to your YAML file, install it and run it against your files first, and then commit that if you're happy, review the changes if you're happy with it, your test pass and all that, check them in and then move on to the next one.

22:02 I think it'll be easier for the rest of your team to swallow the fact that you just changed all the files.

22:11 That's great news. This is one thing when I saw it in the news, I was super excited to see, because we want to start doing this here at work.

22:19 And now that I know that there is a quick start guide to do it, it makes me so much more excited to do it, because I tried messing with the GitHub hooks or the GitHooks themselves, and I'm like, "God, I really need to learn Bash." Well, not just that. They're just basically unusable in a team environment without something like this to manage them.

22:37 I think Anthony Sotow kicked this off and he's really just kept running with it, you know, right out the park He's done a great job and the community as a whole actually too because anyone can contribute Sort of useful pre-commit hooks there too. That's awesome. All right, so we're that's done with our six Does anybody have any extra information they'd like to shout out? I have a couple of things I'd like to share first off the humble bundle humble bundle easy for me to say by the no starch press is throwing a special with the Python Software Foundation, where if you buy, you can get a collection of books for a certain price and the PSF gets money, which is wonderful.

23:17 They can use as much money as they can get, I think.

23:21 And I'm all about supporting them.

23:23 And the second thing I wanted to share was, I know friends of the show, PyBytes, have released their NewbieBytes, which I know they have put in a ton of hard work into.

23:29 and I'm excited for them to have got it up and running.

23:32 I'm excited to, like, as somebody who runs a local Python meetup group, I'm excited that there's another resource out there to help people who are very new to Python to learn how to use it.

23:43 Nice. That's cool.

23:44 I didn't have anything prepared, but I mean, I guess I'll just shout out again, like, you know, I'll be doing an East Coast tour, like PyGotham and the Maintainers Conference in Washington, D.C.

23:53 You know, if you want to get a ticket to one of these, I'd be happy to meet anyone out there, chat about Python and versioning and all this stuff. I should also like, you know, shout out my wife.

24:01 She helped me make all the graphs for this awesome Python applications presentation.

24:05 And yeah, she was using like real Python, I think, to learn a lot of the pandas to do so. She really, again, just an amazing job. So thanks, Maya. Sweet. Well, I was going to plug the Pi 3 readiness, but you already brought it up. So we'll drop a link to the other one.

24:20 that the top 360 packages are now Python 3, which is great.

24:25 And I was hoping that one of you had a joke, so I only have one and it's not very good.

24:30 So either of you got a joke?

24:31 - Joke?

24:32 Oh my goodness.

24:33 You go first, I'll see if it's an act I wanna follow.

24:36 - Okay, well just, I was actually looking for some, so I was looking for programming one-liners, looked at a Reddit thread, read a great answer, which was, "Any joke can be a one-liner "with enough semicolons." I think that joke is obfuscated enough that I'm not going to try to follow it.

24:56 I've got a good bad one.

24:57 So a SQL statement walks into a bar up to a couple of tables and says, "Can I join you?" Ah, yeah.

25:04 Just dropping that one on us.

25:07 Okay.

25:08 All right.

25:09 All right.

25:10 Well, let's leave it at that.

25:12 So thank you everyone for sharing all these awesome items with us and we'll catch up with you later.

25:20 All right.

25:21 Sounds great.

25:22 Thanks Brian.

25:23 Bye bye.

25:24 Thank you for listening to Python Bytes.

25:25 Follow the show on Twitter via @pythonbytes.

25:26 That's Python Bytes as in B-Y-T-E-S.

25:29 And get the full show notes at pythonbytes.fm.

25:32 If you have a news item you want featured, just visit pythonbytes.fm and send it our way.

25:37 We're always on the lookout for sharing something cool.

25:39 On behalf of myself and Brian Okken, this is Michael Kennedy.

25:42 Thank you for listening and sharing this podcast with your friends and colleagues.