Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


« Return to show page

Transcript for Episode #192:
Calculations by hand, but in the compter, with Handcalcs

Recorded on Wednesday, Jul 22, 2020.

00:00 Hello, and welcome to Python bytes where we deliver Python news and headlines directly to your earbuds. This is Episode 192. Recorded July 22 2020. At the look that one up, I am Brian knockin and Michael Kennedy, and I can't believe we're heading close to 200. This is crazy. Oh, yeah. been at this for a while. That's gonna be like four years almost. Yeah. Again, this episode is sponsored by us. And we will tell you a little bit more about other things that we're doing later in the show. But first, some of the ways that people tell each other what they're up to is their personal GitHub README on their GitHub profile, right? Yeah, that's right. So I was impressed by something that I saw recently, Simon Willison, he's the CO creator of Django, he posted something called the blog post saying the how to do a, building a self updating profile README for GitHub. So at the top of it, I'm going to quote this, it says, GitHub quietly released a new feature at some point in the past few days, profile readmes. This is news to me. Yeah. So if you create a repository with the same name as your GitHub account, so in Simon's case, assignment W. So github.com, slash Simon w slash Simon W. So to go too deep, and then add a readme.md README markdown file, to it, GitHub will render the contents at the top of your personal profile page. So that's neat. In that case, it's just one up. So if you go to github.com, slash Simon W, you see his, but his looks really awesome. It's got a whole bunch of cool stuff in it. Because he took it one step further. It's not a static markdown file. He's got another article that talks about it. But this article here, walks through exactly what he does. And also it's all open source. So you can see his code. He uses GitHub actions, there's both a button that he can push to make it happen. But there's also any post to his own Simon w repo will cause this to happen. But the GitHub actions run, he contributes to a lot of open source projects. So he takes a certain set of repos that he has, and pulls the latest releases and have like, like latest release notes using the GitHub graph qL API. So there's an an example of that. There's an example of using feed parser to pull blog entries off of his blog. And an example of using a SQL query to grab this. He's got a site called t l for today, I learned grabs a few links off of that. So he's got a three column setup for a readme that is kept up to date using GitHub actions. How cool is that? That is awesome. Yeah. So normally, you go to your GitHub repo, and you have your picture on how many followers you have, and whatnot. Some other cool stuff we'll talk about later. But then has your pinned repositories and that green ish heat map of how frequently you contribute to various projects, or just to GitHub in general. But now you can have right at the top, whatever you want to write, which is pretty awesome. I think I might have to do this. Yeah. I mean, you still get all that other stuff, but it's just that other stuff is below this README. info. That's pretty neat. Yeah, very cool. And it's super simple, right? Anyone can write a readme.md file. Yeah. And one of the reasons why I brought this up is I think there's a lot of people trying to utilize, I mean, this day of COVID, and quarantine and stuff. I'm glad I'm not looking for a job. And I think that if you are looking for a job making your GitHub profile, look professional, and show the content that you want to show off, and having things like, you know, blog posts on your GitHub profile. That's pretty cool. It is really cool. And just you know, you know, that people, employer say they check people's GitHub profile accounts. Yeah. Right. So how many people have are going to have these unique special ones that show they care, right? Not too many. Well, the people that listen to our podcast, exactly, yeah. All the awesome people. Okay. So that's really cool. I definitely didn't know about that. Thanks for sharing that looks neat. So we got this next one from Connor Foster. And he works in engineering, but also does data science, he things and he sent over this project that he works on. That is incredibly cool. So a lot of times what people want to do is they want to take symbolic math or math that you might write out by hand, turn it into Python code, through pandas and NumPy, and whatnot, psychic learn, or psychic in general, and then run it through Jupiter and get an answer. But he says he works in design engineering, and you have to do a lot of calculations. And those have to be kept as part of the legal records to show the project design history. And one Yeah, one thing you do is doing by hand that's kind of crazy. A lot of people use Excel. That's a nightmare. like Excel is like unbounded, go twos you can't see which is always tricky. So you could do with Jupiter, but then you just got this pile of code and here's the answer.

05:00 All right, so you want to like the theoretical view to verify the formula you're using, right? Yeah. So he created this thing called hand calcs and ca LCS like cow, keyless calculations anyway, hand calc. And the idea is you type in Python code into a Jupiter cell. And then you can do a percent percent render from the Hancock's project, and it will turn it into symbolic math. This is beautiful. Yeah, as if you'd written out by hand. Yeah, with like, as an example, in the little video demo. We've said before we like those and everybody does. But as as like, square root symbols with bunch of symbols underneath it and all sorts of symbols that Yeah, looks like what you would have had to show if you were in math class, right? Yep, exactly. And it will show steps like symbolic steps from step a, step B, step C. And you can say, show it shorthand or expand it out longhand, and show me all the steps you use to like solve these problems and all kinds of cool stuff. Oh, yeah, the reason it looks so good is basically converts. Symbolic Python math over to low tech. And low tech is like the de facto math representation language for academic paper. So you know, you want to have like integral signs, you want to have infinite summations, all that kind of stuff. No problem. This is really cool. And Nicole, and then you can also use the symbolic tag to get it to do other, like show more symbolic stuff. You can do longhand shorthand, you can have it do units, they'll fit units, like millimeters cubed, or whatever. And it'll carry the units through the calculation in symbolically Yeah, but looking at all these formulas. It's giving me nightmares. Don't look anymore. Okay. Well, I guess the thing you would you would want to think about the trade off is, would you rather look at them in their proper mathematical form or in like programming,

06:52 meaning that you know, you where you turn it into like, Star Star pow, instead of, you know, proper exponents and stuff? No, no, I was just kidding. This is beautiful stuff. But when we got into integrals, that's where basically that's where my brain left, and I never really caught.

07:10 Yeah, cool. All right. So people have to take programming math, but they want to represent it more nicely. Check out handicaps. That's awesome. Yeah. Nice. Oh, I'm next.

07:24 Actually, I'm not. I'd like to talk with all of us about our sponsor, and their sponsor is a talk Python training and testing code today. Tell me about talk Python training, I'll tell you about what I'm working on this week, I started writing a new course, we have a couple of new courses that are fun that are coming. And the one that I started working on is called Python memory management and tips. I mean, more. Yeah. So if you ever wondered, like, what happens, like how does it free up memory? What algorithms make like work better with Python memory? And what algorithms can make it more expensive? or slow? What are some of the tips and tricks you can do to like dramatically decrease the memory consumption like two or three times with almost exactly the same code type of thing? Well, I'm writing a course on that. Oh, that's neat. Yeah. Especially for people like talking about doing some more, but we can get Python on smaller operating system architectures like circuit Python and stuff. That's important. So yeah, that's that's a really good point that on the small memory constrained pieces, you might care a lot for sure. Yeah. How about testing code? Well, I was interviewing somebody recently, David, Lord, his actual interview will come out sometime in August. But he said, I was looking at testing code. And a lot of the recent episodes really haven't been about testing. What's up with that? And then say, yeah, it's and code test and code. But yes, so there is it there's a lot of testing focus, because primarily because I think that software engineering doesn't talk about testing enough. But I do cover a lot of stuff. I'm gonna highlight a few of the last episodes. I talked to Sebastian Ramirez on episode 120. About first API and typar talked with Brett cannon on episode 119. About packaging and pipe Project tommo. And what's going on there. 120. One's a diversion. It's a completely different sort of talk we talked to I talked to somebody about 3d printing and finite state machines and stuff. And it's just sort of a fun people doing Python in cool things. Very cool. And then again, talking to thinking about people possibly looking for jobs. In Episode 122. We talk about better resumes for software engineers. So there's your there's a lot of stuff for everybody. Even if you cringe when you think about testing, please check out testing code. We are still putting out episodes if you want to hear more. I'd love to hear what people want to hear about. Yeah, it makes our job so much easier when we get suggestions. Yeah, suggestions and questions and things that can blow into things. But yeah, like a suggestion to return the print statement. So you have to put the parentheses Yeah, so this is crazy, and I don't really have much

10:00 comment here. But I saw the thing by Guido and then I saw this article by Jake edge on Lw in dotnet. I don't know what Lw instance were, but it doesn't matter. Anyway, the non Return of the Python print statement. So this is odd. I thought, we have talked about the new peg parser. In Python it's going on. But one of the things that happened with that is, I guess one of the reasons why Python two to three, they went from a print statement to a print function was it made the purchasing easier, but with the pig parser, you can do all sorts of crazy things. And you can have functions that syntactically look like statements and have it work just work, sort of. So as an example, we could use a print statement instead, instead of you having to be put the parentheses in, you could avoid the parentheses. Anyway, he put he just put it out there as an idea. And essentially, people said, No, yuck, what do you think about this, it's interesting, it would be one fewer things that has to happen to move to the next stage for from a two to three conversion. But on the other hand, this looks like one of the easiest conversions for that step. To me, I'm not a fan of having statements and functions in the language, because it looks to me like functions basically solve the same problem with a little more clarity. You know, they're a little more functional, you can span the multi line, if the arguments are super long. You need to like with this print statement, you'd have to use like a continuation backslash and other weirdness like that. So you know, just just because you can doesn't mean you should, I guess that's probably how I feel about it. But yeah, I wouldn't use it. If it were in the language. Let me put it that way. I yeah, I'm for the No, yuck, camp. I think it's thick. That print statement shouldn't have been a statement in the first place. And I think Python three fixed it, having it be a function is the right thing to do. I wish there were more statements that were functions instead. Also, I think, I wish a cert was a state function instead of a statement, because people doing, thinking that it's a function and putting a parentheses around assert causes problems. But that's not what this is about. It's interesting. I brought it up because people should know about this weird, wacky discussion. Yeah, that's funny. I'm glad that the it got thumb down. And I don't think it's gonna be willing to make a statement about it. Yes.

12:24 All right. Well, I'm gonna make a statement about flask. I think flask you just had david Lord on the show, right? That's not out yet. But yeah, pretty cool. Yeah. And he's lead maintainer of flask these days. So flask is at least at the API level got to be the most popular web framework there is because it's slightly more popular than Django, if you look at some of the recent surveys, but if you look at the other frameworks, many of them are flask esque, if you will, right? Things that are like responder, or CNA, or whatever, they have this idea of like sort of the same style, right. So there's an article called fast API for flask users. And I'm actually a big fan of fast API, I'm hoping to have some opportunity to use it soon, like, the API's that I've worked on, they've been around for a while they predate fast API. And I don't really want to go create a whole new site just so I could use a different framework. That sounds like maintenance to me. So I haven't gotten a chance to use it in production yet. But fast API looks awesome. So there's an article called fast API for flask users. And it says, look, you probably know the flask API. Here is the equivalent for fast API. Okay, yeah. And so there's talk about some of the advantages. And they're pretty awesome. So automatic data validation. And fast API doesn't exist in flask, generally speaking, automatic documentation generation, built in best practices, like type annotations and pedantic scheme schemas and whatnot. It comes ships or recommend, I guess says terms of like a requirement, you have to have a, a wsgi server. So it comes with UV corn, which is one of the it's like cheese, corn plus UV loop racing stuff. And in a lot of ways, it's super similar. So if you want to create a view method, instead of app dot route, you would say, app dot get and so fast API, would you imagine the name indicates it's mostly for building API's? Yes. So when you talk about functions, and what they're going to do, you say, not just here's a URL, but here's a URL, and an HTTP verb. So app dot get forward slash or app dot, put slash account or something like that, which is pretty cool. In the route, you can specify variables. So in flask you would, you could have a user ID and in the string route, you would say, int colon user ID, if you want flask to convert that to an integer, right? That's fine. It works. Okay, but that the rest of the tooling doesn't help you know, it's an integer just because flask

15:00 Notice it's going to be an integer, right? So in fast API, you put the variable up there as well. But then in the function, you put the variable name has a type, and then it will actually convert that to an integer using the Python language, tools or specification rather than the string API thing. That's handy. If you want to query string and flask, you just have a URL, you can go to request arcs, and you can get the value out of the query string. In fast API, you just put the query string values, or sorry, the keys as arguments and they just get passed in. That's pretty cool. Yeah, if you have an API that takes a JSON post, like you know, it's accepting a JSON document, you can just say it takes a dictionary. And that gets posted in but you can go way, way further, which is awesome, you can define a pedantic model, which is a class that has types and validation on the class, right. And then you can say, my view method, or my API method takes like in the example they have as a sentence that has got like, various components, nouns, verbs, and whatever. You can say, here's a function, and it has an argument called sentence. And it'll take that JSON document, parse it into the pedantic model and pass it to you pre validated this definitely one of the benefits of best API is this data validation. Yeah, this data, this is like built in data validation, because how much how many times you spend like effort, I got a string, but I got to convert it to an integer, I got to make sure that this value is here, I got to make sure that this one is, you know, like, whatever like it, it matches some subset of a sub strings or whatever, just let let the framework handle it also has the equivalent of blueprints, which it calls routers, and this automatic validation I talked about. So anyway, there's a nice article that says, you know, flask, let's teach you fast API's real quick by just doing this equals that. Yeah, I love this. Because there's a lot of people that have been writing API's in flask for a long time. And so it's just second nature to them. And having something to say, hey, I want to try fast API. But is the learning curves can be a problem. Well, it was something like this decoder ring, and it is set up for you just sort of skim through and go, Well, how do I do URLs? Oh, this is how you do it. And your variables and different things. It's set up really nice. Yeah. Yep. Definitely fun. Definitely useful. So do you use Twitter? I do use Twitter. Sometimes happily. Sometimes I get dragged into stuff. Sometimes I use it in write only mode where I want to make a statement, but I don't really want to go read it. But yeah, definitely. Yeah. Well, I have a probably common sort of a love hate relationship with Twitter. I use it a lot in like keeping track of other people. But but sometimes I don't really like that it's a pain to delete old stuff, because I don't I think of it as a current conversation. I don't really look at somebody what somebody wrote a year ago. And I don't really care what I wrote a year ago. So I have used Twitter deletion tools before. They seem kind of weird that I have to go out and give my credentials to some other website or something, though, so but I know, I'm sure that'd be fine. It'll be fine. Yeah, anyway, but there's API, so you could use the Twitter API. But how and so I thought it was really cool that Chris Albin is somebody that tweets about data science a lot. And he posted a little snippet that he said he uses, at least he did it first at one point, but it's a it's a cool little example of using a library called tweepi. To interact with Twitter, and to delete old tweets for your account. So it's just this really short little Python script. But it deletes tweets, there has defaults, but you can change those obviously, it's a just Python scripts, you can change whatever you want. But it's set up to delete tweets that are older than 62 days. And that have likes less than 100 people, and that you haven't liked yourself. So the idea being, if you go through some of your old tweets, and the ones that you you're like, Oh, yeah, that was cool. I want to keep that around. Just like it like your own tweets, and then run this script, and it'll delete some old stuff. I would definitely have to change that hundred count to something else, because I don't think I've ever had a tweet liked by 100 people. That's a big number. You know, Twitter used to show how many tweets you actually had. And I don't think it shows it anymore. On my profile, at least. I don't see it immediately how many tweets I had just followers and following and and likes and stuff like that, but yeah, pretty, pretty cool. It's like, keep the highlights, right. Just keep my highlights. I don't need every random thing of Oh, oh, my God had a hamburger today. People don't need that as a piece of history. Yeah. So when you don't know what's gonna stick and what's good or not, and I was actually reading an article recently.

20:00 about, about Twitter about how that what that says to you. If somebody like for instance, you're trying to get a job, and somebody looks at your Twitter account, having the junk in there that like nobody really related to having that automatically called out and just having the highlight reel. That's not a bad idea for some of the data. And you can turn away down. You could say, look, if there's no likes, or no retweets, just drop it. Yeah. Yeah. It might even be good for me to like, just to go back a couple days. But if nobody's liked it a couple days, maybe just take it away. That didn't happen. Right? Yeah, that didn't happen.

20:38 So people could end up clinging to their old tweets, but they probably shouldn't. Right. Yeah. So I want to talk about an article by Mr. Turner traveling. Now, we spoke about him. Sort of, not by name, I don't think but we talked about Phil, the data science memory profiler a little while ago, okay. All right. So he's the guy who wrote that actually had him on talk Python on episode 274, as well talking about that. So that was pretty cool. But he independent of that he wrote this article that I came across that I liked, called clinging to memory, how Python function calls can increase your memory usage. And this is part of my research for working on that course, that I was talking about that memory, Python memory management stuff, okay, so he talks about like, Hey, we're going to have this thing, it's going to load up some NumPy data, and then it's going to pass it to a function, the function is going to make some changes, take the return value that and pass it to another function, it's going to make some more changes. So basically three steps, and said, Look, we'd expect that we've loaded two gigs of memory. And yet, when you run Phil against it, you end up with three gigs of maximum memory usage, which is a little bit weird. And the reason is those initial like intermediate values that you're working with, on step one, and step two, the way Python decides a variable goes out of scope, is in this case, the function returns, not like it's never used again. But it's just the function returns, in which case, it's gonna hang on to all the intermediate copies all the way to the end. Interesting, right? Like some languages, they determined that and they get rid of it, like it's C sharp, the JIT compiler will notice like, okay, a variable is not used after half the way so we're going to make it eligible for GCC. Basically, unless it's in debug mode and keep it around in case somebody sets a breakpoint, they want to see it. So there's a lot a lot of the tricks that things can do. Python doesn't do them. And so it's going to stick around for this length of the function. So what can you do to make it not stick around as long, because maybe you only have two gigs, and you don't wanna use three gigs or whatever. Right? So he talks about three different solutions. One is to don't hold on to the intermediate variables and just chain into one massive function call, like, pass the results of one to two to step two, pass the results of step two to three, and there's no variables holding on. So it'll be gone, right? That's an option. Another one is to like iteratively, change the variable, say, like data equals load data from first step, data equals step two, processing of data, data equals Step three, a processing of data. And that way, you're dropping the reference count to the first to the intermediate steps along the way. Right, so that's an option. And then there's a third one that's more complicated about creating like a sort of a ownership management type of thing that people can check out as well. But I just thought it was interesting to think about, you know, when How long do these things stick around? And what techniques might use that are incredibly simple, like, just reuse the variable name problem solved? In terms of having too much memory usage? Interesting. Yeah. When I look at these, they all look kind of like the same. But having having the answer be that they use different amounts of memory is not obvious, right? It's not obvious at all. But it's, you could easily look at this one where you're iteratively, changing the variable and say, Oh, you shouldn't do that. You should name it more clearly. Because maybe the type is changing along the way. And it would be weird. But you could say, Yeah, but

20:38 this one works, because it will fit into RAM and the other one won't. So we're willing to accept this like slightly imperfect, imperfect code, because it works better. Anyway. There's a lot of interesting trade offs you can make here, but I just it's it's only the tip of the iceberg for things like this. You could do I think, but it's interesting to just put it on your radar. Yeah. That is interesting. Yeah, actually, then, like we said, I think that more and more as we start using Python for other applications, or non desktop kind of things, like when we're non server things, if we're using it for, there's a couple ends of it. If you're using small devices, like in circuit Python or something, you're going to care about this stuff. But also if you're using very large data sets, then we care about it again, and it doesn't matter how much memory your computer has, having multiple copies of gigabytes of data when you don't have to will slow things down.

20:38 Yeah, for sure. Or even if it's like an API and you just happen to be doing is not that extreme, but you happen to be doing 1000 of them at a time. Sorry. Yeah, exactly. And as we use Python more and more and more applications, we are going to start caring about that again. Yeah, absolutely. That's the end of our six. I actually have been just so swamped with stuff. I don't have anything extra to talk about. Do you have any extra items? I do. And this is just a follow up email we got from a listener named Adam. Thank you, Adam. And you had talked about pickling things. Apparently, you're a fan of dill pickles and wait, pickled strings, pickled lists, pickle dictionaries. Now we were talking about pickling and how it didn't make sense most of time. But there might be some use cases, you know, like, what might be a use case that we really need, right? So Adam said, Hey, I got a use case. That worked for us. I worked on an API that spoke to a third party service that was wonky, and it was over raw sockets. So you'd have to create these byte arrays and send them along. And the thing was also not available 24. Seven, it would sometimes crash, things like that. So what they would do is they could set a flag in their app, and it would pickle all the messages that it would would have sent. And if the site comes back, it can like rehydrate those things and then ship them along. Or you could pull them up for debugging, and look at their details and whatnot. So it was like, Oh, we got to save this exactly as we would have sent it. Let's just pick it. That's pretty cool. Yeah, it seems like a pretty good one. Yeah. And there was a feature flag, they could turn on and off, which was kind of cool. Yeah, they could also do that for the messages they got from the service. Pretty cool. Real quick, Python 384 is out. I've already brew upgraded mine. So that's all good. And big news, big news. I can't believe it. I've been selected. I, if I go to my GitHub repo, I don't have the cool README thing that you're talking about. But under my picture, it says I have a pro account because I had to pay for him stuff. But I noticed that I'm an Arctic code vault contributor. Wow. So remember, we spoke about the Arctic code vault where GitHub is taking a bunch of the popular projects. And then like sticking them over in some vault in Norway or somewhere like that, Greenland, to preserve it. And if the code that you've contributed to GitHub was selected, then now you get this cool little highlight. That's like a snowflake. This is Arctic code, Vol contributor, and you can hover over and I'll say why. So yeah, I've contributed apparently, to a couple of things. And you might be as well, well, yeah, you were the listener might, but I just checked mine. And I am too. So that's neat. Yeah. Awesome. Neato. Yeah. So I think we covered that once. That. Yeah, yeah, we definitely covered the code vault. But yeah, I think this, you are a contributor thing. Little badge is new. And I don't know, it makes me happier than it probably should.

20:38 Yes, it's so cool.

20:38 It's cool. Yeah. Yeah, it's super neat. Testing is cool. I love testing and having good code coverage is cool. Yeah. So I've got a joke for you a cartoon, if you will, okay, from this place called geek and poke. They have all sorts of good stuff there. And you can click on the picture, and it'll take you to the actual comic, to there's a two people, a woman developer and a man developer, staring at each other. And the woman is the more senior one. They're looking each other and it says QA best practices. She's looking, looking at the guy and says, never just remove a failing test. Guy stares back blankly for a second says, only remove the assert statements. Yep.

20:38 does how does it stay in decent code coverage?

20:38 Yeah, you can fix the test. You can make a test not fail. Remove the cert statements. It's good. Yeah, that's funny. You said you actually like test for failure, though, engineers that they potentially could fail? Yeah. Well, I think that's one of the reasons why we do code coverage on all or not code. We do code coverage. But we also do code review. What's the word again? review? Yes, code reviews? Yes, we do code reviews on tests, because we have had tests show up before that exercise. We, you know, with test equipment, we do a lot of complicated things. You set up everything and run some stuff. And then we've often have people forget to check anything at the end. And in the process, it is important to look at the end to see Is there any way this can actually fail? Is it or is it just exercising things? I mean, actually exercising things isn't a bad thing, because you can get asserts in your code or accept an exception. Yeah, you still test something but you're not testing very much. Yeah.

20:38 You're testing it runs basically. Yeah. So awesome. Well, yeah. Just never move affiliate tests only the start statements. It's terrible. We should not give that idea to people. No, we should totally delete this joke. It didn't happen. It wasn't funny.

20:38 It was.

20:38 Thanks a lot, Mike. Yeah, you've had great to see you. As always. Thank you for listening to Python bytes. Follow the show on twitter at Python bytes. That's Python bytes as in be yts and get the full show notes at Python bytes at FM. If you have a news item you want featured, just visit Python bytes out FM and send it our way. We're always on the lookout for sharing something cool. This is Brian Aachen and on behalf of myself and Michael Kennedy, thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page