Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


« Return to show page

Transcript for Episode #181:
It's time to interrogate your Python code

Recorded on Wednesday, May 6, 2020.

00:00 Hello, and welcome to Python bytes where we deliver Python news and headlines directly to your ear buds. This is Episode 181. Recorded may 6 2020. I'm Michael Kennedy. And I'm Brian. And this episode is brought to you by data dog tell you more about them later, Brian, this first item that you picked here, I saw this going around the twitterverse. And I was like, Oh, that looks really cool. Tell us about it. Yes, interrogate. So it's a new little tool, I think it's kind of new, new to me, at least, it checks your codebase for missing doc strings, or the presence of doc strings. So it's written and maintained by Linwood. And we were notified by Herbert beemster suggested to us I like doc strings, I don't like them, I don't have a policy of having memory that were but doc strings can help you understand the code quite a bit. I really love how like a function doc string is utilized by VS code and pi charm, and other editors to do things like if you hover over a function, it just pops up a little window with like, you know what parameters you can send to it. But it also sends the doc string shows the doc string, if there's one. There's also other tools like Sphinx or pie doc or doc utils. That can use that amongst other things to generate documentation. I don't really use those I know a lot of people do, though. So it's a problem. If you've got some missing if you really need to have doc string someplace and they're missing. So interrogate is a command line tool that you can use to check your code to make sure everything has a doc string that needs to, I think this is great. Yeah, it's super cool. And well done. Len, nice work on this one, I think this is really nice. Because if you have a public library, that should be documented, right? That should be covered with nice doc strings that give you all sorts of help in your editor, like you mentioned, but also, if you generate the documentation, you don't want to have just some random empty, either completely missing or just the, you know, like the function name and just an empty body or whatever it gets generated there. So this is really cool. And I could see if you were doing a package that was public, having this as part of your ci CD process that if somebody introduces something into say like a PR that here's a new function, we wrote like, great, it failed the CI because it doesn't have a doc string. That'd be awesome. Yeah. And then there's also some common, there's some common like caveats. Like Yes, all public functions should have doc strings. But what about like, private functions and stuff? There's options to be able to talk to turn off things where, you know, might not need a doc string for just an internal helper function or something. Yeah, I totally agree. Like, if it's not really meant to be included, you know, in the public API, then it shouldn't have, it shouldn't necessarily have a doc string. Yeah, it can if it helps to understand it, but you maybe don't need it. So this is, yeah, this this is super cool. Well, then Len. Yeah, I wanted to mention also, there's information on the readme, about how to use the tool within talks and within ci workflows. So that's helpful. Yeah, this one seems like a super easy thing to adopt. If you already have a CI process in place. And you're working on something that's public, whether that's public to your people within your company, or you know, it's a package that is used by other projects, right, or it's public as in on pi.org, I still think it makes sense to just document it for whoever is consuming it. Who did not write in it. Yep. Cool. Now, this next one I've been holding off on talking about for a long time, because it's really interesting, but I didn't feel like I was able to do it justice. So the thing I want to talk about is streamlet. With Streamlight. The idea is it allows you to build much simpler data science, applications and web apps, I guess you would say, so many people come to Python from, I've got a little bit of computation that I got to do. And so I'm going to write a script that does like 10, things maybe doesn't even have a function, it just does do X, you know, load data, transform data, pivot data, show graph, or something like that, right? Really, really simple. They're not like pro web developers that are like busting out view or react or something like that to build cool apps. But they've got this really cool outcome with Python, and they're happy. But at the same time, they would like to have this publicly shareable. They may be wanting to put it on the internet. They want to make a web app out of it and the gap from I've got 10 lines of code that outputs a graph to I have a fully interactive, dynamic, server backed JavaScript front end type of visual website. That's a big gap. Right? Yeah. So streamlined for me. Yeah, yeah. Well, yeah, for me as well. Now, for Streamlight. The idea is that you write the code in the simple straightforward style that I talked about, but then it turns it into an interactive web application. SNI. That sounds neat, right? And it does a whole bunch of really interesting tricks to let you continue to

05:00 write code in this simple, straightforward way, and yet interact with like callbacks and stuff. So you might have a function that comes along it says, Okay, what I need to do is I want to show, I don't know, a drop down list in a web app, or a slider bar or something like that. And then you write your code that says, get the value from the slider bar. And you might think, Okay, well, normally, that's wait for the user to click on the slider bar, and when there's a change, or some kind of callback, and then you have some function that runs on the callback, and it updates the DOM, right. But this is like straightforward procedural code, or imperative code that doesn't have any of those ideas in it. So what it does is it uses interesting caching on your function so that if it were to rerun it with the same inputs, it gives you the same answer instantly, like LRU cache type of stuff. And then if any of those controls on the webpage change, it just reruns the whole script. And when you say give me the value from the slider, it says, well, currently, the slider is set at this. And if it happens to change, it just reruns the entire thing from beginning to end. But because everything's cached, it only re computes the parts that are needed. So you don't have to think about callbacks, you don't have to think about user interaction, you just write top to bottom code. And here, you've got multiple parts of your application that can be changed dynamically that just continue to refresh, and it refreshes it by using JavaScript. So it doesn't even reload the page, it just looks like parts of the page are dynamic and coming alive.

06:27 This is neat. Yeah, so it's super cool. Basically, if you can write like straightforward, top to bottom code that works with like graphene, or pandas or something like that, you now can create a single page, a spa app. And I don't remember what JavaScript framework it uses, like react or view or something like that. I think the might be react anyway, it's not super relevant. But it does all of that work with like, posting it, and creating the server callbacks and setting up the JavaScript and the UI widgets and all that kind of stuff. And all you do is write basically procedural code. One thing that I think is a danger of these types of applications is, it's easy to say if you use our magic graphing library in our magic UI widgets in our magic, you know, data processing library, not a real database, but our our magic server database that we create, then it works, right? What I like about this is it doesn't do any of those things. It just says you want to use pandas, and NumPy, and plotly, or something like that, use those, and then use them in your script. And then we just make it this single page app, which is pretty awesome. Yeah, that was my question. Is this a hosted thing? Or can I do it like behind a firewall or local thing? Or, you know, yes. So with streamlet, you have a couple of options. I haven't done anything, myself, but you can self host it, or they'll host it for you, I believe. And if you they have this thing called for teams, which is like, they automatically deploy and manage the apps and you just create them and say, here's the code and run it. It's like kind of a platform as a service for streaming apps type of thing. That's a paid part of what they're doing. But the stuff I described up until then, that is free and open source. So it's pretty, pretty usable. Yeah. So you have to basically deploy them yourself. But they talk about how to do it, you know, it's not super hard. Okay. Thanks. Yeah. So I had the one of the guys Adrian on who is behind this, and has also done some other interesting technical programming stuff, and self driving cars and folding at home those types of things. On talk Python, just a couple episodes ago, I don't remember exactly what number it was, but I guess 260 ish, something like that. So just check out that episode, if people want to go much deeper. But this is really cool. If you've writing these imperative, simple little scripting apps, and you're like, I would really like to share this or make it better streaming. That's cool. Yeah. Nice. I feel like we both have themes. For ours, your thing is continually documenting and whatnot. And you'll see my theme develop as well. This is interesting, because I ran across this interrogate second, but after I ran across this an article by by he, Nick, it says, why you should document your tests. He's gonna makes a lot of good points. So I believe and we have this policy where I work of tests should have met doc strings on them to tell you kind of what the test is doing and and why it's there. There's a lot of reasons that I have never really been able to describe clearly. Because you're I mean, ideally, your test name should be descriptive and in the code should be clear. So why do you need a doc string? It's not. So that's not something that you're going to call from an API or anything, you probably won't type help function, your test file. Yeah, right. Right. And you also don't call it I mean, like test methods that you run pi test with PI test calls it you don't call those tests, but you can get confused in the future. And one of the explanations is, it has a good example where he shows some simple

10:00 code with a simple test name. But the thing he's asserting on doesn't really make sense. And the reason if you don't know what's going on under the hood. And the reason is, because tests often are testing for a side effect of an action. If you're testing the output of a function, that's obvious you, it's kind of obvious over the five pass in functional tests of view, passing of certain parameters, and you get a certain expected output you can assert against that. But what about actions that have side effects and you need to test for those that might need some more explanation, and rewrites This is quite common in testing very often, you can't ask questions directly. Instead, you verify certain properties that prove that your code is achieving its goals, by checking side effects and stuff like that. It's a short little article, and I was just happy to have some that expressed of why you should document and it might be you that it's confused in the future. And then it ties back into the interrogate, because so I was just thinking when I was reading about interrogate, and like, Oh, awesome, I really want to put something in place to just make sure all my tests have docstrings. And he lists in Oracle, the exact command line that you need to make sure that all that just text, make sure that your tests have doc strings, and nothing else. So is it cool? Yeah, it looks really nice. And I can see how this would lead you down to finding a library that searches for or evaluates these. Yeah, so the test example he has here is testing that the hash, like creating the same object instance of a class twice, and taking the hash and making sure that those are not the same, because he's trying to test that attars is not implementing false on the class or something to that effect, right. So like, what the heck is going on with this? Like, why would you check the hash of a thing is not equal to the hash of a thing. But it's looking like you said, for this sort of, like, hidden layer that's operating below it. Right. Yeah. And then also, my reasoning, and I think he brings it up is that that, often, when you're looking at a test, it's because you really got to get a release out and one of the tests is failing. So you got to go look at the test and figure out what it's testing for and why it's failing. And if you're don't have it clear why the test is important, you run the risk of the person at the time going well, I don't even understand what this test does. Maybe some important is commented out, commented out, passes, don't want to see eyes, good will go home.

12:34 I don't do that. But I've heard it done before. Of course, I would never do anything like that. But I could imagine that it has been done. Now I actually have I have some experiences in the past where like, there was some project and had like a beautiful set of tests, and then and ci and stuff. And it wasn't working on anymore. And whoever took it over. It's like, Oh, those tests were really annoying. So I turned them off. Like, Oh, my gosh, those were there for a reason. What just happened? So at least like if, if this gives you some description about what's up, that would be good. Is there anything where like pi test will show the doc string as part of the error report? I think there was a somebody that wrote a tool that would do, I don't remember the name of it. I remember running across a plugin that would pop out the doc string for failed tests, which would be pretty cool. Yeah, that would be pretty cool. Hopefully, the person evaluating the test failures has access to the code, they can go. Yeah, I'm thinking more of like a CI CD. Like you're not really you don't have it, like loaded up. Right there. Right? You're just like, Oh, I'm cruising on GitHub, checking a PR, what the heck is this? Yeah, I actually wrote a was an internal thing. But a quick plug in for pi test that would just before every test function, print out the test name and print out the the doc string. If we can look at the logs. It's there. Yeah, that's cool. Very nice. Also, cool data dog. So this episode is brought to you by data dog. And let me ask you a question. Do you have an app in production that's slower than you like its performance all over the place? Maybe sometimes fast, sometimes slow? Here's the important question. Do you know why it does that with data dog, you will, you could troubleshoot your app performance. With their end to end tracing, check out their detailed flame graphs to find bottlenecks and latency and that finicky app of yours. be the hero that got your app back on track at your company. Get started with a free trial at Python, bicep, FM slash data dog and get a cool little t shirt as well. Alright, so your theme is so far, it's been pretty clear. Next one of mine, we're going to talk about a project called hollow ww is be heard of hollow is not I'm sure you've heard of Sai pi, and those types of things. So there's kind of a grouping of other projects right under that general banner with like NumPy, matplotlib and whatnot. But hollow is is kind of like that, for data processing and visualization. In one of its features acts very much like streamlet but in a more general, probably a little more work to set up and

15:00 Right got to work more in its framework, but in a very similar way to what streema is doing as well. So it's a coordinated Hello visits a coordinated effort to make browser based data visualization and Python easier to use, easier to learn and more powerful. So, yeah, so what does it do? So it has a bunch of tools that make it easier to apply Python plotting libraries to your data, bunch of tutorials, conda, like metapackage, that when you you know, and conda install all of his, it actually installs a whole bunch of things I'll tell you about a second even has some sample data set. So you can like go through the tutorials and actually get the outcome, right. So it's made up of a bunch of different things, one called panel, and that one makes creating apps and dashboards for your plots, using any of the supported plotting libraries, which is pretty awesome. It has hv plot to quickly generate interactive plots as hollow views to make your data, instantly visualizable, geo views for visualizing geographic data, data shader for rendering huge data sets puram for creating user configurable objects, config stuff, and then color set, or like color maps. So yeah, you want to have like nice colors that flow together for your plot. And panel is the main thing that I was thinking of that lets you create these single page apps around interactive sliders and drop downs, render graphs and other other kind of stuff. And yeah, very cool project as well. Yeah, I'm just looking at the data shader. And then like some really pretty pictures, because the quick examples of overlaying Oh, yeah, data plots over the map of the US things like that. Cool. Yeah. Or, you know, you can plot attractors with 10 million points each. Oh, is that all?

16:50 And these graphs, I mean, these the resulting pictures are I don't know what they're like 400 by 400 pixels, or smaller as i get i came from 10 million points, or here's the United States, plotting each person in the United States where they're physically located from the 2010. Census. So 300 million points of data overlaid on to the United States map. Yeah, it's like, like a son really is pretty interesting. Actually. I think I can see you over there, Brian, here on the left.

17:21 You are too. That's right. Awesome. Anyway, yeah. So this is a cool project. There's a whole bunch, a bunch of different little libraries that are pretty neat. So if you want to try to visualize your data, and we'll come back and talk some more about both these projects towards the end of the show, as well, but bunch of cool libraries, like all brought together under this hollow this project to make them work together.

17:41 I want you guys next, I have another command line tool. So this was by Oh, gosh, Rosario, and he wrote a little blog post about it. But it's a there's a project called a live progress. And it's a progress bar for Python. And it's really cool. So it is pretty cool command line interface, a progress bar. So often, I mean, like, even, like pi test has added progress bars for, you know, watching your test finish and things like that. And it's nice to have progress bars, it gives you good feedback, and whatnot. But there, there's limits to what you can do, except for this seems unlimited. It's got a whole bunch of different animations, and spinners and things you can combine to make a more entertaining progress bar. So it's fun. Yeah, these are really nice. And you know, these little touches, they probably don't seem like much, but they can definitely make your app feel more professional and more polished, rather than just, you know, the answer seven, or like, whatever we're done. Yeah, some of the like, what was it, Pip and I don't use it, but I appreciated some of the fun, Command Line Interface stuff that they've added to it. And I think adding some fun to a tool is nice. One of the things I wanted to comment on also was just the about the the repo itself. So the code is up on GitHub. And the readme has some nice thing features I wanted to call out. So the the animated pictures of what it does is a nice touch. We've said this before, we love things like this, I liked that there was a to do list. So it encourages having a short to do list encourages contributions. And I think even listing things I've gotten that I've gotten done recently, so that people that might not know about those features, don't try to go work on them. And then just a short list of things you'd like to have done. And then has an interesting facts section, which is kind of cool. The code is in a functional style uses closures in generators. So I mean, actually, a lot of times I'll look at code not as an example of how to do something. So if somebody's proud of like their use of generators, maybe it's worth checking out if you try known generators. And then another feature that I'm definitely going to pick up for things that I work on is changelog highlights. So not the entire change log in the readme but just one or two lines of semi colon separated features per version.

20:00 Have change. So this is kind of a nice thing to add. Oh, yeah, that is really nice. Cool. I think it's a great project. If I need a progress bar, I'm definitely gonna consider this one. But like I said, like, I do think they're neat ideas. Um, sometimes. So yeah, that's cool. Very nice. All right, last ones a quick one. So we talked about how love is talked about streamlet, and panel and how those are kind of similar. So there's this other project called awesome panel, which is kind of like an awesome list for a panel that way to build interactive data science, single page apps, or not even single page apps. This is run by Mark skov. Madsen look at that closer, right, Mark. And yeah, it's a cool project to just show a bunch of examples and a curated list of these. So yeah, if you just open that up, you can go see there's a app that comes up, it takes just a second, come to life. And then on the left, there's like a gallery. There's all sorts of cool pictures. There's also a nice talk, remember, who gave the talk. But yeah, there's like a talk from one of the conferences. If you go to the gallery, you'll see like, as you navigate around, it's kind of like this single page app. So you click gallery, you'll see like the main part of the app sort of spin for a second and then come up. And then you can go and say, oh, there's like one of the things that's really cool that I'll point you guys out is this thing called the image classifier in there. So I went in there. And it's lets you if you just scroll down, it says, upload an image. And you can grab some JPG and upload it. I grabbed some like random microphone picture that I had laying around and threw it in there. And then it ran, you can pick different neural networks to run against it, you want to run NASA net, mobile, NASA net, large mobile net v2, like, I don't know what these are these classifiers. But you can pick from them. And then it'll tell you in a cool little graph at the bottom, like what it thinks that is, and it thought the microphone was most likely a microphone, but it could have been a shield, because it had like a big metal guard on the front. It's just really cool. And so this whole website is built, I got it right in panel to show you how you can build a cool panel app. But then, as you dive into the gallery, each one of these pieces is like an interactive sub panel or something like that. So it's kind of it's a little bit meta in that regard. But it's pretty cool, actually. Yeah, it's really, it's really cool. Yeah. And, yeah, it's apparently super easy to work with. Again, I haven't built anything with it. But it seems like a much simpler way to get your interactive data science stuff on the internet, then learning Vue js, JavaScript, flask, SQL, alchemy, etc, etc. Yeah. So anyway, pretty cool. I recommend if you want to check this out, just go browse the gallery. That's the best way to see what this thing can do. I love visual pretty pictures. I know. Yeah, this one definitely passes our, our tests for you must have pretty pictures

22:46 in order to talk about graphical stuff. Yeah. So Brian, really quick on this next one, click on the app for it, because it's take just a second to load up. So that's information items, I want to tell you about a couple of things, then we'll get here as well. First, one of the features that I really like about Visual Studio code, I haven't really used because I don't use Visual Studio code like a meaningful way. I use it all the time for like little bits of editing, but not if I'm like doing real projects, I probably use PI charm. But it has, it has live share, which is a pretty killer feature. So you can say I would like to share my coding environment with someone else who's going to like look over my shoulder, do pair programming, just code review, whatever. And they can actually debug and step through and like see your code in their editor. And it could even be like a different editor. I think there's other stuff support like proper Visual Studio vs. vs. code and whatnot. So that's really cool. I've always thought like, well, that would be awesome for pi charm. But no. So someone on Twitter, sorry, I don't remember who sent this to me. But thank you, I sent up this link to this project called code together. So code together is a freemium product. So this is a modern open source thing. It's like a paid project, you can go sign up pay dollars a month for certain features, or you can just use the free version, but it has that type of experience that I just talked about, but for many different editors, right? So it comes for all the IntelliJ stuff. So like WebStorm, pi charm, and so on. also works with VS code. It works with other things. I don't know exactly all the things that it covers, but certainly Eclipse IntelliJ and VS code, which covers quite a bit. So if you're looking for that, and you weren't using live share, because it didn't exist for what you're doing. You could check this out. I've not used it. I'm not endorsing it. I'm just saying it looks interesting, and it might help people. Cool. Now. Cool. And then the other one that I want to just quickly mention is related to the first thing these are totally different things. But Kevin Vander Veen sent over a message a week ago and said, Hey, I built a cool Data Explorer to help you understand the whole COVID pandemic stuff in a

22:46 Local way. And he built it using streamlet. So I thought, hey, here's an app that someone just sent over. That's like a cool running on Heroku streamlet app. That would be a nice example of what I talked about at the beginning. And he also has the GitHub repo there as well. So if you go and check out that app, like, for example, Brian, you scroll down to the second to the third graph below the third graph, there's like a drop down that lets you pick Oregon, and you can go pick that. And Wow, it looks like we're flattening the curve really well, that's pretty awesome. Although the last two days have been rough, I guess. Or you can go down a little farther and compare it against different states, you could go in there and type like Colorado, you could type Oregon, or New York, or whatever you want. And then it'll like autocomplete that out of the list and then regenerate. And this is exactly like what I say like, there's some line in the stream like code, that's just saying, Get the tags from the tag selector. And then if you type in there, it just reruns the whole thing. But most of the stuffs cached and then it'll redirect, read, rather grab. It's super cool. So this is both useful, I think for if you want to try to like understand that. It's a cool data science project. The source is on GitHub. But also, it's just a cool example of Streamlight if people want to see that going, and the graphs are fun, too, because you can do like buck select and zooming and stuff like that. Yeah, these are just plotly. So they do all the standard plotly stuff, which is pretty cool. That's kind of what I like about the streamlet thing, not forcing you to use their random graphing thing, but just other graphs that you might like it. Yeah. All right. Well, that's it for my extra What else do you got? Well, I wanted to remind people that pike on 2020 Online is available. And new content is still being posted through the first few weeks of May. I think that's when they're wrapping things up as well. I just saw some new videos go up there. Yeah, yeah, one of the new videos that just came up was my talk. So multiply your testing effectiveness with parameterised. Testing. Oh, yeah. There it is right at the bottom with a couple of new tags on it. Yep. So I'm excited to have people get feedback from people to see what they thought. So that's good. I like it. Yeah. Well done. While I was there, I was looking around. There's tons of great talks. There's a bunch of tutorials there. There's charlas. Is that new this year, and they did it last year, maybe over the last few years. But it's the Spanish language track. Yeah, it's nice. I'm glad they're doing it. Sponsor workshops. That's kind of nice to have. And then even an online poster Hall. So that's there. So it's cool. The other thing I wanted to bring up quick extra. If anybody's following the drama and pi test, the drama is over. And I'm happy with the resolution. So hopefully there will be peace in the family. I hope so a link to the Twitter announcement from pi test. Yeah, they got the whole public statement there. So the folks who had dropped out, are they dropping back in or what's ours, they're just permitted a follow up for that. I don't know that. Any more details. I know at least one person is back in. Okay. All the people are was worried about are signers of this message. So yeah, so probably kind of seems like maybe they're back in? Yeah. Well, that's awesome. I'm really glad to hear that go solved. And it sounds like exactly the outcome that I would have voted for as well. Okay, so, you ready for some jokes? And out of these jokes? How about you? Are you ready for humor? Yes. humors good. Humor is definitely good enough. So I found a couple of funny pictures. And I'm gonna put the pictures in the show notes. I don't know if they'll come up in your podcast players. Some of them do. Some of them don't. Some of you guys, they allow pictures in this podcast feed, just like in the player as you're looking in the show notes. But certainly on the website, there'll be there. And these are a really book covers. I love these. They're like O'Reilly and just if you've forgotten the O'Reilly books always have the title. And then they have an animal that goes with it. Right. So if you wrote a book for O'Reilly, the thing is like, what's your animal on your book and whatnot? Yeah, so these are like, take those ideas, put an analog animal on it, but make it silly.

22:46 There's also usually like some sort of saying at the top of the book. So for these, you definitely have to read the top also. Yeah, it's right, like sort of the subtitle. Okay, so how about this all read the first one, you do the second one, and so on. So this one has like a badger or something on it on the front of it. And it's kind of like sneaky head down. And the title is pointless meetings, the survival guide how to survive all pointless meetings.

22:46 I feel like many, many things that used to be meetings these days are now in email. And people are like, Oh my gosh, they really can't just be emails like why we've been going to all these meetings, right? Yeah. All right, you want to get the next one. The next one is overriding your teammates code. My code is better than yours. Anyway.

22:46 It's got a horse on the front.

22:46 there either. I really love this one. This one is the essential semicolon parenthese sorry, quote, semi colon, parentheses, DROP TABLE animals, semi colon dash dash and there's no

22:46 No animal, because while this is a SQL injection, deleted it, and it says now with user generated content.

22:46 That ones that may be the fastest now with the security hole. Exactly. related. What's next? Okay, this one is a fish on the cover expert hoping nobody hacks you. Security by optimism in prayer. What the fish has to do with it, I must be missing but that's good. This was people's security solution. Right? That's true. So the next one is an octopus. Obviously many legs, I can type many things in the title, the entire book is exiting them eventually. just memorize the 14 contextual dependent instructions.

22:46 Excellent. Give him eventually. Yeah, I love it. I love it. Yeah. So you have to visit that website and check out these because they're pretty sweet. Yeah. Awesome. Speaking of sweet, it's been great to be here with you, Brian. Thanks. It's been great. Thanks. Yep. Bye bye.

Back to show page