Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book

« Return to show page

Transcript for Episode #199:
Big news for a very small Python runtime

Recorded on Wednesday, Sep 9, 2020.

00:00 Hello, and welcome to Python bytes where we deliver Python news and headlines directly to your earbuds. This is Episode 199, almost 200. Brian, I noticed almost recorded September 9 2020. I'm Michael Kennedy. And I'm Brian McCann. And this episode is brought to you by us, we'll tell you more about the things that we're doing and what we have to offer later. But for now, I want to talk about not any huge announcement, but a very, very small announcement. That is also kind of huge, very cool. Yeah. So micro, you might say, you might say it's microscopic, but it's a big deal to micro Python has just had a new release. And this is significant, because the last release from micro Python was December 19 of last year. So it's been, what is that almost 10 months since the last release. So if you care about micro Python, this is like a pretty big deal to version 113 is out. And one of the major things is that has improved async IO async and await support. And it has cool Bluetooth improvements. And it even comes with the pep five to six. That walrus operator Oh, nice. Yeah. So there's a lot of stuff going on here. They basically, micro Python, as you can imagine comes with a micro async IO module, you a I'm guessing the U is really a new you async IO module that's supposed to be more compatible with C pythons. async IO. So the main idea is that the task object, you can use a task object for each co routine, which allows you to have like unbounded queues of work and stuff like that. Whereas the previous I think you had to pre allocate it for some reason. I never tried to do async IO on micro Python, but apparently it was a little bit wonky. And so now it's more consistent. Another thing that I thought was interesting about this release, Brian was they went through an auto formatted all the code in the entire repository. And if you look at their repo, there is Python and micro Python, but a lot like C Python, it's a lot of C as well, right? I think it's like 80% C or something. Okay, so they auto formatted the Python code with black and now it probably won't surprise to many people, but they auto formatted. C with it's that communities equivalent of black and get ready for the name. uncrushed defy they UNHCR justified the C code, okay, do you know this? No, actually, I'm writing it down. Because I'd like to check this out. Isn't that great? So you can basically configure how it formats your C code, and it's like black, it's like, you know, it just runs automatically and fixes your C code. And boy, if you thought Python code needed some fixing, I bet you the C code. And there's more variety in there, right? Yeah, there's I've seen some ugly C code in my day. Yeah, for sure. For sure. Because with the like the curly braces and semicolons, you have a lot more flexibility in how you format stuff. Whereas Python forces some of that structure upon you. Or they also added blue kitchen. Apparently, blue kitchen is a Bluetooth stack. And so they added a micro m you Bluetooth module as an alternative to the nimble stack as well. And there's a Unix port, and you can build the Unix port with Bluetooth support for either these bindings, that's pretty cool. That comes with new events. So if you're doing like Bluetooth discovery, and things like that, you can like scan for Bluetooth stuff more easily. It also came with a big memory leak fix for arm chips, which is big in the small space, ironically. So apparently, there's certain memory that was not reclaimed by the garbage collector and other types of things. So if you've, you know, and at the same time on these very small devices, you care a lot about not wasting memory, right? It's not like you got 32 gigs, like my laptop does. So like whatever, right? It's gonna matter if you lose memory on like a little micro Python chip. And also they set it up. So you can run parallel tests against multiple micro Python targets. So I'm guessing like different chips and different devices, you can just say go test them, run it synchronize across two or more micro Python targets. There no, that's to speed it up or to like, test the whole thing on different platforms in parallel or whatever. But it sounds like a good improvement. And it does come with a few breaking changes. So people can check that out. Anyway, if you care about micro Python, and I'm guessing this may make its way over to circuit Python as well, at some point, knowing that those projects are really close. So anyway, this looks really good if you're doing big stuff on small things. Yeah, it's good to see this going forward. And thanks to Matt Contini for sharing this in bringing to our attention. I wish I was better at remembering names, but somebody I had on testing could brought up respects and or respects I should stop Are you SPX. It's a utility for marking out the Python HTTP library. So that somebody brought that up, and I had never heard of it before. So I figured we'd bring it up here. So with requests if you want to mock it, there's a whole library called responses.

05:00 And cleverly named a bit hard to search for. And then with HTTP x, if you want to market there is respects, I'm going to name it response x. But we just the other part, you just don't write it. It's like the opposite of a silent E. It's, it's not written, maybe.

05:18 So I get, it sounds like a great idea. Well, interestingly enough, when you write the two imports out, there's the same number of characters. And maybe that's part of the reason why they did that. Anyway, that quickstart is like chosen, it's really quick. It's just, it's a really just a few lines of code to mock out a request response call response call, you can do things like things that you would expect, like an assert that something was actually called the, and what the status code was, and things like that. But you can also have custom content. And you can return JSON content, of course, there's some examples. What I like is the documentation is pretty nice you, there's some examples on how to use it with both pi test and unit test, including how to set up like a mock API fixture for pi test. So you can have multiple tests using the same mocked endpoint. And there's a bunch of nice things about it. Like for instance of if you want to mock out a whole bunch of different URLs, you can use some regular expressions to just set what what URLs you want to catch with that. And then also, the content that you return from your mocked object or marked request, is you can have a callback that can generate that on the fly based on what you pass in it. If you have a HTTP x based application, you definitely need to know about this for testing. Yeah, absolutely. And one of the things that's challenging about testing the ACP x is the async stuff, right? And so this is a really cool way to get in there and just make it work right there. Just put a decorator there, and it just mocks it out for you, like, like you would expect. Looks very clean. Yep. Pretty easy. And there's a lot of magic going on under the hood to make that a clean interface. But I think they did a good job. Yeah. So HTTP x comes from sort of an async derivative of requests, which I think is a really fantastic library. And I've done some cool stuff with HTTP. So yeah, I'm a fan of this is great. Now, before we move on, let's talk about a couple of things we got going on, that people can check out that will definitely help support the show. And they might find interesting, I know that you've got some updates for your book, as I write Yeah, the Python testing with PI test book, just a minor, the book itself has a minor update. So there's, it wasn't a big enough change to change the hardback version, but the ebook will be updated. I don't think the hardback is going to be updated. But it's, um, couple lines of code in a couple of code examples in chapter five. So it's a very minor thing. But the big difference is the code download. So we've updated the code download. So even if you're working with the hardback book, or using the old version, and don't update the ebook, please redownload the code, it'll make the your learning experience better. And the main changes that I've pinned the dependencies in the target project, or the example project, so that so that everything worked better. Good, there was a tiny DB is a database that I use in the project. And it had some incompatible changes. And instead of trying to update everything to use the new, the new TDP, I just pinned it to an old version. Yeah, that makes sense. I mean, it's not like you're trying to teach people tiny dB, you're like, here's a dependency, we're trying to get a renter around or something. Yeah. So it's that's not important to the content of teaching people how to write tests. Yeah, that's the problem of creating content the world moves on. And they're not always compatible with what you did. I'm about to Yeah, over talk Python, we've got a whole bunch of stuff coming. And so what I'd like to encourage people to do is just to go to talk, or training dot doc And right there, you can at the front of the training site, you can just enter your email. So if you get a bunch of announcements, because I believe we are working on five or six courses right now are under active development. We've got all sorts of great stuff and other going through it just sign up to get notified when those come out. Very cool. Indeed, indeed. So a while ago, I went on this Twitter

09:25 journey, let's say I don't know how to really explain it was like I posted a quick question that led to a ton of feedback. And then, Wow, so many pieces of information and ideas and variations were sent around what I was trying to say like I'd like a dictionary that contains objects that I can access, say with different keys, like I'd like to access by, put a bunch of users and access them by ID, but maybe also by email or by city and that will come with a bunch. So it was it was like this sort of exploration there and one of the recommendations that came over sort of around that is like hey, you should

10:00 Check out this thing called get pie. Okay, I don't know where the name comes from, but it doesn't tell a lot about what it is. But it's a vectorized Python dictionary and set implementation. And vectorized. As in, it matches up perfectly with NumPy. And so pandas and all the things that are built upon NumPy, to plug straight into them. So imagine I want to have a dictionary that has data in NumPy. But lets me treat it like a regular Python dictionary, or a set and things like that. So that's what this is, it basically brings a super high performance Python dictionary and set implementation that automatically integrates into the Python scientific ecosystem, which I think is pretty cool. Yeah, yeah. And it's built upon this thing called parallel hashmap. So parallel hash map is apparently the current state of the art unordered map set with minimal memory, overhead. And super fast runtime. So like a C binding. And so this is just a Python wrapper on top of this C library, that's a super fast dictionary and set. So that's pretty awesome. And here's another one for you, Brian, for your c++ adventures. The integration between get py and parallel hash map is this thing called pi bind 11.

11:19 So pi bind 11 is a compatibility layer between c++ 11 and Python. So if you want to write like modern c++, and then plug it in easily to Python, here you go. Nice. Yeah, that's pretty cool. Okay, there's two classes, GP, so get pi GP dot dict, and GP dot set. And they're designed to be basically similar to the standard dictionaries and sets from Python. But there's a few differences. So check out the docs. And then I threw in a quick little example here that has two NumPy arrays. And you can say, here are the keys here, the values, put them together. And then you can just access different values. And it's also typed, which I think is kind of interesting, right? Like it has like an unsigned eight bit integer or something like that. Much like NumPy is, but it's more like a Python array, where you specify the numerical type than it is just an unbounded list. Right? So there's some interesting stuff going on. does it solve your problem that you're looking for? No, not at all.

12:22 It's very interesting. There's still neat. Yeah, yeah, it's so neat. Speaking of neat, we already talked about Black Swans, but it's pretty neat. Let's talk about it again. Yeah, this was sent in by john Hagen. And he mentioned that I mean, I pretty sure we've mentioned eyesore before. I know we've mentioned black. But a lot of projects use both. So I sort will search your imports. So you don't have to. And then just so it's consistent, they're consistent. And they're alphabetically sorted. And then there's some others, right. And there's Pep eight recommendations about that I believe about like grouping stuff that comes from the library from standard library versus externals and whatnot. Yeah. And so it's so you don't have to think about it, you can just use nicer. Black also is becoming more and more popular, I think, or it's just constantly very popular. for reform. It's all of your code, but it includes the import statements as well. So there was this issue that some projects wanted to use both eyesore and black, and there is in they kind of fight with each other right out of the box, if you run ice, or you'll have one answer, and Black will do something different in some cases. So what do you do? Well, there's black hat, a configuration page. And we'll link to this in the show notes that has documents what settings you have to set for eisert, so that it's compatible with black. They also do that for like eight and piland, which is nice. But flaky and pilot are not that complicated. It's the eyesore that there's like six settings you have to change. But I sort five just came out. And I sort of five has what they call profiles. And so if you run eisert with profile, black, black is one of the built in profiles, it will sort the imports such that they're compatible with black. Nice. So now if you use ice or with the black profile, and black also you they won't fight with each other anymore. That's very cool. And black holes also came up with some new changes. So if you're interested in black, check those out. They're huge changes, but some minor fixes. The profile feature is pretty cool. And even if you don't care about black, I think you should check it out. They do apparently, Django pi charm, Google OpenStack clone adders and hug are other profiles that are included. And also they're just good examples so that you can look at how different projects are configuring their I sort. You can configure yours if you want. Yeah, I really like it. That's super cool. And it is nice to have the top of your files nice and clean and organized in some certain way. It's interesting. There's a lot

15:00 Have a lot of people do different linters or code reform matters. But this combination of eyesore plus black is something that's becoming more and more common in a lot of projects. Yeah. Yeah. Very cool. So I have something interesting for you, if you were to think of the influence of Microsoft Excel, relative to the influence of like all of in human genetics and the genome science of the world, and they got into a fight, who would win, like a superhero? This seemed like apples and oranges. I have no idea what they did get into a fight and excel one. So here's the story. There's an article that was sent over by Chris Moffitt. He's the guy who wrote the moving from Excel to Python and pandas course over talk Python. So we did a bunch of research into like, all these funny things, and weird things around Excel. So he sent this one over, and I thought I'd cover it because it's fun. So, on the verge, there's an article that talks about how scientists have renamed human genes to stop Excel from Miss reading them as a date.

16:02 Okay, is that crazy? Yeah, yeah. So you think it was just like, formatting and stuff like that. So there's like 10s of thousands of genes in the human genome, and each gene has given a name and a numerical code, and then use to talk about it. So like, this one controls like what color of hair you have or whatever, right? So over the past year or so, 27 human gene genes have been renamed all because Excel kept rereading them as symbols, their symbols and states. Okay. Well, the examples are important. Yeah. So like, March one.

16:38 No spaces. March one is a one of the genes, right, but it gets converted to Oh, three slash, oh, one slash the year? Or if you're in Europe, oh, one slash, oh, three slash the year, right. Either way, that's not what you want it. So

16:55 March one actually stands for a membrane associated ring, CH type finger one, which of course, is the first of March.

17:03 Now it sounds funny, like we're making fun or whatever. There's a study in 2016 that examined the genetic data shared alongside 3600 published papers, and found that one fifth of them, which, what is that that's like 700 papers were affected by accelerators. Yeah. Isn't that messed up? So there's a scientific body in charge of naming genes called Hugo gene nomenclature committee, who knew there's a committee for naming genes. But apparently, they published new guidelines for naming genes. And they said, you have to consider what happens if you type into Excel. If it gets reformatted. That's not okay. better pick a new name.

17:48 So they put like weird suffixes and stuff on the things to make them work like March one is like March one in one or something like that. Anyway, so why do I bring this up on the Python show, you know, so much of this work that people are doing there can totally be solved by pandas and Jupiter notebook, and stuff. And so I, you know, the guy who was quoted in there is like, Hey, I made these mistakes, when I was in grad school, because all I knew how to use was Excel. So here's a bit of a, an example, along with a plea to help folks are over using Excel to take a step forward, and use you know, something like pandas and Jupiter, and you know, you're gonna be able to do a lot more cool processing. Anyway, which I think is great. Yeah. Also, one other thing, if you think the geneticists have a exclusive right to these mistakes, there's a really cool article, blog post over on Oracle's blog actually called the 10 costliest spreadsheet, boo boos in history. And they're really hilarious in the like, thing. God, this didn't happen to me. Well, in just like, awesome stock photo that they have this too. Yeah. It's so bad. Yeah. It's like perfect. It could just come out of Excel. Yeah. So some of them are fairly mundane. But others like in my five, the British intelligence agency bugged over 1000 wrong phone numbers, because the 000 in the last three digits of the number got, you know, Miss stated, oh, no. Yeah. Or Eastman Kodak was forced to reinstate financial results for two quarters by from, yeah, 2 million and 13 million respectively, due to a spreadsheet mistake. Yeah, there's just all these crazy examples of stuff just going wrong. Like, oh, sorry, that was a billion dollar mistake, because we were off by, you know, some random thing. Anyway, there's a bunch of errors like this. And it's really interesting to think how you might use the Python tools to not have these errors. Yeah, I think Excel has too much influence over the world. It does. But if you were going to use Python, and you had a Jupiter notebook, would you run it in your downloads folder.

20:00 That's definitely not. Yeah, so this is another this a kind of a, an interesting and shocking really think about it before, but glyph has written an article called never run Python in your downloads folder. It's not just about that. Okay, so there's your advice don't run Python in general. But I think it's a good article to spread around and in read, because it's a nice short tutorial on how sis path works, how it's populated. So Python has this thing called sis path. That is, that's where it looks up. So if you say import something or mostly that where it's importing things, also the dash M for something, if you say dash m, and then some module, where does it find that, and it finds it in lots of places. One is, is the normal system include directory, or the system packages. But there's other places too. And one of the places where Python uses is the location you're in when you run Python. So little example he talks about, a lot of people are now using instead of running PIP directly, they run Python dash m Pip, which I also am including that in my advice, usually, because I'm tired of trying to fix people's errors when they're Python and their PIP are pointing to two different places. that's frustrating. So what happens if you run Python dash m pip install something, some wheel in your Downloads directory? Well, normally, everything's fine, except if there's a malicious, fake IP py in your Downloads directory that happens to have gotten there, because somebody wrote some malicious JavaScript code to stick it there. It's possible. I don't know if it would happen, but it's possible. It's more likely now. Yeah, especially now. Thanks, thanks, Cliff. But then it's gonna run that instead of the PIP that you expected to run, which is, is bothersome. So this is hidden, there's some extra advice in here, I encourage everybody to read the whole article, understanding how the Python PATH variable works as well. Because sometimes other, you know, applications will, in the end, if the installer of an application can change things and change your Python path, it shouldn't. But sometimes they'll do that out of convenience. So occasionally, look at your Python path and make sure that there's nothing weird in there. And maybe contact application developers or if they're doing something odd. If you are mucking with Python path recommendation is, but absolute path, don't put anything relative in there. You want to be able to have complete control over that the problem with the Downloads folder is not that the Downloads folder has weird permissions, or higher permissions or anything of that nature. It's just that web virus malicious ness might, it's most likely to drop the payload there. And the Python path plus that operation or that aspect is what is likely to lead to trouble. Yeah, right. Like you wouldn't say don't run it in your documents folder, unless you copy virus Python files in there.

23:10 Yeah, in the example, it was real of like, sometimes, like, let's say you're behind a firewall or something like that. And pip install just doesn't work. And you haven't figured out how to do proxies yet. And you really just need some package, you might just download the wheel somewhere. Okay, you might have that in the wheel might be fine. But put it somewhere else. Don't leave it in your Downloads directory, put it someplace else. And please use virtual environments that'll help things as well. Yeah, absolutely. Good advice. And definitely another thing to be aware of. Internet. It's a scary place.

23:44 Yes, but we also good place. It goes both ways, because we do love it. That's it for our main topics. But I do want to share two quick things. And they both have to do with the internet. Number one is we just passed 5 million downloads, Brian. Wow, that's awesome. Yeah. And we're in the top 150 or so of all tech podcasts in the world. So just want to say thank you, everyone that's helped to make that happen, because that's kind of a milestone. So that's really cool. Yes, thank you, indeed. Also, I recently finally broke down and got a Wi Fi mesh router. Or routers, can you say singular router. It's got to be plural, right? So one of those routers where it has a bunch of different stations and they all work together. But because of the way it works, you don't actually have to connect to different ones at different places. It's just all one Wi Fi. And man, I got this a Linksys Wi Fi six mesh router, which is quite pricey, but man it is so so awesome. So people are like suffering from being home all the time that I Gosh, I gotta get better internet. My internet was actually pretty good here but we started getting a ton of radio interference from different sources and that would degrade it and stuff. So I have my recording area office above my garage. It's like little studio

23:44 apartment thing we built above the garage, which is a separate building across from the house. In over here, if I go speed test, I get 400 megabit measured off my Wi Fi now, that is solid. Okay, so how far was your, or the mesh nodes? It's probably 50 feet, but it goes through like several outside walls through like a couple inside wall. It's like there's layers, it's kind of go through. But now with the mesh, I could put one of the nodes much closer to the wall that I'm close to, that spans like that gap between houses, okay. And how are they talking to each other? They have a like, their own channel, and they do some kind of like, back channel Wi Fi six thing. So anyway, to be recommendation. Yes. How about you any extra stuff? You wanna share the world? No, just I'm sort of sick of COVID and fires and all that sort of stuff. Yeah, man. It's, it's looking a little smoky outside. And yeah, the whole West Coast. And my sympathies go out to the folks in Northern California. I just had a meeting with someone and oh my gosh, is it smoky there, it looked like night, in the daytime. In the background, I could see their window. And it was it was dark as if they were in like a different time zone. But they weren't not good. So hopefully, hopefully that all ends soon.

23:44 But before we end this episode, I got a question, Brian, for you. Are you a real programmer? Like a real one like a hardcore? I mean, you do c++? So I put you a little closer. Oh, my God, this posture syndrome players every time I get that question to? I don't think so. Well, let's go to XKCD and answer it. Okay. Okay. XKCD has this cartoon called real programmers. And it starts with two people debating about what kind of editor they're using, right? It says, there's somebody working and they're using nano, the editor and the person. nano for real, real programmers use Emacs. And then Brian walks in. Hey, real programmers use vim. Come on. Oh, yeah, we're real programs use EDD. No, no real programmers use cat and just stick it on the end. Real programmers use a magnetized needle in a steady hand. Excuse me, but real programmers use butterflies. Everyone turns and looks at the person. They open their hands and they let the delicate wings flap once the disturbance ripples outward, changing the flow the eddy currents in the upper atmosphere which acts as a lens that deflects incoming cosmic rays focus in the strike the derive platter and flip the desired bit. Nice. Of course, there's an Emacs command to do that.

23:44 Oh, yeah. Yeah. Command x, command m command butterfly.

23:44 Nice. Yeah. cmca cx. See MC and butterfly. Indeed. Damnit. Emacs. They say?

23:44 Does anybody use Emacs? Yeah. Well, I didn't know. But I'm not a real programmer because I don't use butterflies. I'm sorry. No, I use butterflies. Okay, well, we're in the jokes section. I gotta share a terrible joke that my wife came up with the right okay. 99 bottles of beer on the wall. They ain't bottles of beer. Take one down. pass it around. Now. Everybody's infected. morons.

23:44 That sounds like college here in the US. I think that's what they're doing. With all the news. I'm watching. Oh my gosh. This sorry. Yeah, these jokes are the 2020 jokes. Oh, yeah. So this won't make sense in the future. Yeah, yeah. So in the future, when you speak to a historian, and you say, oh, you're a historian. What do you study? Oh, I actually study the year 2020. Like that won't be enough. You have to say well, Which part do you study like the beginning?

23:44 Do you study the financial crash? You study this crazy election? Do you study the pandemic like you got to be spec. That year is not enough to say what your specialty is? You got to pick one picker?

23:44 Right? I study the fly the fires. Oh boy.

23:44 Well, good talking with you. As always you as well catch you later. Thank you for listening to Python bytes. Follow the show on Twitter via at Python bytes events Python bytes as mb yts and get the full show notes at python If you have a news item you want featured just visit Python by set FM and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Aachen, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page