Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


« Return to show page

Transcript for Episode #260:
It's brutally simple: made just from pickle and zip

Recorded on Friday, Nov 19, 2021.

00:00 Michael Kennedy 00:00

00:00 Hello, and welcome to python bytes where we deliver Python news and headlines directly to your earbuds. This is Episode 260, recorded November 19 2021. I'm Michael Kennedy.

00:00 Unknown Speaker 00:11

00:00 And I'm Brian. I can I'm Chris petty.

00:00 Michael Kennedy 00:13

00:00 Chris, welcome to the podcast. You're a podcast veteran, but the first time here on Python bytes

00:00 Unknown Speaker 00:20

00:00 happy. Thank you very much. It's a pleasure to be here. Good to be back.

00:00 Michael Kennedy 00:25

00:00 We're gonna jump into a bunch of the news. But before we do, you know, tell people a bit about yourself.

00:00 Unknown Speaker 00:29

00:00 Absolutely. I've been in the tech business for about 35 Each us now something like that. And I've been a self self identifying Pythonista for about the last five or six years. I work for these days I work for I just just recently changed jobs within Amazon. I used to work for the web services, elastic file system T. And I now work, I switched gears from you know, living in the clouds to living in the Amazon fulfillment centers. We're now on a network. So

00:00 Michael Kennedy 01:01

00:00 is that mean? We're doing what like programming and COBOL? Are we using now?

00:00 Unknown Speaker 01:05

00:00 Yes, nothing but COBOL and Fortran. They're just all the words all of us possibly No, not at all. We do. My team ops tech it we do network automation in for infrastructure. So it's a lot of fun, actually, it's really kind of neat going from building micro services and things like that being infrastructure oriented at that level to like working with being responsible for automating things like you know, network protocols, DNS, DHCP, fun stuff like that, at scale in the fulfillment centers. It's a really cool gig. Yeah, that

00:00 Michael Kennedy 01:39

00:00 sounds really interesting. And you get an interface with physical things that go physical places, which is Yeah, it really is. That thing. I only ship beds. I don't know. Brian, you also you also work on physical things.

00:00 Brian Okken 01:51

00:00 Yeah. I like working with actual make things. Nice. Yeah,

00:00 Michael Kennedy 01:55

00:00 absolutely. A nice change of pace. Yeah. Out in the audience, that Ned Batchelder Hey, Ned, thanks for being here. This Chris Boston represent.

00:00 Unknown Speaker 02:07

00:00 That is amazing. His he does so much work with Boston, Python. And I can't wait to we can get back to actually like being in a room together with other people again.

00:00 Michael Kennedy 02:18

00:00 I know. Absolutely. So Brian, I used to feel like a cog in the machine. But now I work at a pretty small company where I make a lot of the choices. So I don't have to feel like that. But you know, I think we're going to talk about something that I think is awesome, because Ned is here.

00:00 Brian Okken 02:35

00:00 Yeah, it's got a nice and it's here, we picked this just for unit. But the so net one of the net does a lot of stuff for for Python. And one of the one of the things that he's done is this cool tool called cog, which I've been watching for a while, but I haven't had a really good reason to use. But what cog does, as you point it, you can have these cogs like these, like three brackets and the cog word, and then some code, and then then another magic incantation at the end. It's pretty simple, but it's hard to explain, but you have some code. And when you point cog at it, it will run the code and then put the output of the code somewhere else where you put it to. So seems neat. I wanted to use it. And but I didn't know what a huge use case I would use it for well, Simon Wilson wrote this article called using qog to update help, the help output for a markdown README file. And I think this is brilliant. So this is super awesome use case. So Simon, he even has it going up on is a C SV s to SQL lite project. And the idea is you've got at the end, the end of the project, he actually lists on the readme, he lists the help output. Now I actually have a project that the help output is out of date, because the way I deal with it is I run the help and then copy and paste. But this is Simon's right. This is a great use for qog to just point cog at it so so he wrote a little snippet of code that goes in his readme. And since Kaga ignores comments, you can put it in an HTML comment block in it. So Cago run that file and took the output, stick it out. And then Simon didn't stop there. He said, I want to make sure that this gets run when it needs to get run. So he wrote a test in PI test of course, to to check to see if qog needs to be written. So this is just this is really pretty cool. Like it.

00:00 Unknown Speaker 04:42

00:00 This is the kind of thing that's it's so cool. Pardon me, Mike, this is the kind of thing that's so cool, because anytime you can eliminate a DIY, right anytime you can make one thing, the one true source of whatever piece of the puzzle you're looking to build you wit. And this is a great example of that. Yeah,

00:00 Michael Kennedy 05:02

00:00 out in the audience a little like a real time feedback, Ned says, I also just updated the coverage and cog docs to do the same thing and to get help output into the docs. Nice. Very cool. I love the inception of it. Yeah, this is, this is definitely a cool project, I want to take you all on a journey, come with me into the world of bank, Python, my dream, this, this is an interesting article here, it's one of the things that says I can't exactly tell you precisely where I worked, or what we did. And I have to make up names for the projects that are not the actual names by Cal Patterson. And it says, the strange world of Python as used by big investment banks. So this is quite the read. If you go through it, if you click the Little Reader bit here, it says, you know, 30 minutes of reading time to read it. So it's, it is a non trivial bit of history. But let me see if I can give you some of some of the highlights that I took from this. And you can go read it, if you really feel like you want to dive into it. So one of the things they talk about is the the database. And all these things have interesting names, I'm sure that they have actual interesting names as well that are not these names. So for example, like the database like system is called Barbara. And what I guess the place to start here with this oral history that Cal talks about is that big python implementations are basically forks of the entire Python ecosystem, not just a fork of C, Python, but like the libraries that it uses in everything, which are used at many, but not all of the biggest investment banks. So it says, let's talk about the system called Minerva, Minerva. And it's a global database of Python objects. And Barbara is this way to talk to it. Okay. So a lot of interesting keywords to give a hidden, right. But one of the things that's wild this Barbara thing is, it's like a hierarchical database that stores basically zipped pickle objects. Oh, wait, wait, what? So it's, in when I say hierarchical, I mean, it's in what they call rings. So you would talk to a database. And there might be like a nearby replica that you talk to that might have a record or it might not. And so then there's like a bigger replica, a sort of fallback to go look in this place. And if it's not there, go back and look at another one, they call these rings. And then you would ask for something by its key, and then you get it back. So first of all, what do you guys think of just pickling all of your data objects,

00:00 Unknown Speaker 07:40

00:00 but it's really that that bit made me chuckle because it is exactly this the kind of like, at the time pragmatic engineering trade off that somebody need right? years ago, before we all saw pickles as the the great villain of the Python ecosystem. Before we knew that, there could be downsides to lay all the things and they're like, well, it's perfect pickle idea. What more could you want? And now rolling?

00:00 Michael Kennedy 08:09

00:00 So simple. It's small. Yeah. Yeah. How do you do that

00:00 Unknown Speaker 08:14

00:00 something actual database? Just curious.

00:00 Michael Kennedy 08:17

00:00 Yeah, this is a fine question. They do. If you read it, they do also use databases sometimes. Okay. So but the primary way seems to be this this system called Barbara. So one of the things that comes to mind for me here is, oh, my gosh, what if you upgrade something? What if you change the file the structure encode of something that you put in the database five years ago? Whoops. Yeah. Yeah. That used to be, you know, signed up for the mailing list, yes or no? And now, when you change the order that yes or no became yes or no is admin, Oh, oops, or it just doesn't match anymore. And it crashes. You can't read your data because it's incompatible. So this would happen with Python, the runtime but also the libraries you're using, it seems to me that it's just incredibly unstable. So that how much

00:00 Unknown Speaker 09:10

00:00 would they could that they also have like some kind of really, highly evolved over many years validation system that ensures that the that the pixels that you think are being used, and the way that you think they're being used are actually still used that

00:00 Michael Kennedy 09:24

00:00 way? Yeah, exactly. There's got to be layers. Sam, out in the live audience live stream says, Good luck upgrading a new Python. Yeah, exactly. Yeah, like as endeavor, and that's probably why part of it is like, yeah, we just forked it and froze it, because that's crazy. Let me show you something else. I will try to jump around in the article via search because I took notes. Applications also commonly store their internal state and Barbara writing data classes straight into and out of Barbara, with very simple locking in transactions, if not, and one of the things they decided is there's no file system available to Have these two this platform and the little bits of data that the script picks get picked up and outputs, it all just goes back into this key value store pickled database thing called Barbara. That's pretty wild. Right? Yeah. Okay. Um, they also talked about having these overlay features, which is pretty wild. This has to do with the different rings that I talked about. So when you connect, you can say, I want to connect to the middle office ring, and then the FCC ring, and then the default ring. So then when you do a database query, which by the way, it looks like a dictionary index, just Bracket key, it first is going to look for it in the middle office database. And then it's going to look in the foi CC database, and then the default database. So there's like these layers of settings and these layers of specificity. And they, they even do things that are to some effect of like monkey patching. Like if you if you need to change it, you can have like your dev ring, and then the middle office ring, and then the outer ring, and you could change the value and kind of like for mocking or something. So yeah, to your point, Brian, they say there's some good reason Cal says there's some good reasons not to use Barbara. If you're dating some, well, that makes count the ways. If your data set is large, it may be a good idea to look elsewhere, perhaps in a traditional SQL database, or KTB plus, which I have no idea what that is. There's a limit on the size of Barbara objects, which is compressed at you want to zipped 16 Meg's or smaller, which is pretty interesting. And then paired with this is this direct this thing called dagger, which is a directed acyclic graph that allows you to set up relationships between instruments like okay, I've got a bond, and then I've got a credit default swap on that bond. And then I've got like a fund of default swaps, or you know, like his whole layer of stuff. And all of these things are like linked objects through these pickled things that you can ask questions about and like, get sort of like graph database type things. That's pretty interesting. They talk a lot about how you model these things like clap, like instruments and swaps and stuff in Python. That's not super interesting. But I wanted to highlight this section and see what y'all thought it says, if you understand, you understand Excel, you'll be starting to recognize how a lot of this stuff here works in Excel spreadsheets are cells. And they have dependencies, which is like an a cyclic graph like this dagger hierarchy graph database thing has, and it has a lot. So they basically modeled so much of the code around how Excel works. So you model the data first, and then the behavior rather, usually we write code, and then we have data that comes out of it. So it's, it's pretty interesting. There's also this thing called Walpole, which is a database runner. So when you get to this section, so there's time to drop a bit of a bombshell, let's talk about deployment, versioning, etc. The source code of this entire system itself, even including a Barbara is stored in Barbara and not on disk, and it remains composed. And it's kept in a special Barbara ring called source code. All right, what do y'all think of that?

00:00 Unknown Speaker 13:18

00:00 My release engineer just left the building. Oh, man, that's frightening. Yeah. So

00:00 Michael Kennedy 13:24

00:00 not keeping the source code in the file system breaks some assumptions. Like, how does the program run? Yeah, so this Walpole thing runs, it's built to basically run source code that originates from a database and you it's all one huge database. So like, if you think mono repo, it's like that, because you import from database. And it's just like, whatever you need is there. So versioning, it's like the anti microservice. Basically, the entire whole thing is literally there. And then the way that you deploy it. Let's see there, they talked about vouch to the way that you deploy to it is really interesting in that you, it reminds me of the it works on my machine certification program. So the way you deploy to the system is you just commit it to the database, and boom, it's deployed. It's running, right? Because there's only one place that code lives, it's in the database. So once you do a commit, that's a full on deployment. Now, how does it get into there? There's a voucher system where you can get somebody who can code review what you've done and say, Yeah, I vouch for this, you can commit it unless you have a high reputation for not breaking the system, in which case you can directly commit to it without question.

00:00 Unknown Speaker 14:41

00:00 So I guess a culture of code review is not like a thing there.

00:00 Michael Kennedy 14:46

00:00 I guess. So another interesting knock on that comes from storing your code in a database is how do you edit it right? Like what editor to use? Vi won't work on it pi charm won't work on it VS code won't work. on it, because those work on files. So this Minerva system has its own IDE, because most IDs think you have files, not databases that contain the data. Does that make sense? It is that makes sense is the guy Cal says what I can't understand is why does it contain its own web framework? Investment banks have a one way approach to open source software. Some of it can come in, but none of it can go out.

00:00 Brian Okken 15:30

00:00 Well, I I've seen, I've seen projects that like, like, people building IDs within a company, and I always shake my head and say, Why you this isn't your business? Why are you making an ID within your company?

00:00 Unknown Speaker 15:46

00:00 There are a couple things, there are a couple things that struck me from this article. The first is, as I read it, because I really enjoyed it

00:00 Michael Kennedy 15:52

00:00 didn't really faze me lawn right through it. Right.

00:00 Unknown Speaker 15:58

00:00 It really is, it really is, the thing that struck me is, this is a perfect example of a large organization, a complex code base, an engineering system, and tie, because this really just smells like something that just got like, built up, like, you know, the layers of rings around a tree over time, engineer after engineer year after year and making decisions. And I'm sure like in the modern day, everyone who walks in the door, they probably have to plan for the period of time, where the new engineer just freaks out when they tell him all this stuff as they adjust to the new reality that they're being exposed to. And that in some sense, is a particular wacky case. But it's that way in kind of every large corporate environment in which that's

00:00 Michael Kennedy 16:44

00:00 true. That's totally true. He actually Cal actually discusses this a little bit hearing says, this fact that you can't do very much of open source, you're stuck on some version of Python that's never gonna see never gonna go back to sync with the real Python. Because how do you even do a database migration to that thing? says it could be the biggest disadvantage for you as your profession. Every year you spend in the Minerva monoculture, the skills you need to interact with normal software atrophy. By the time I left, I had pretty much forgotten how to wrestle pip in virtual EMV into shape essential skills for normal Python, when everything is in the same repo. And all the code is just an important way software packaging just does not come up. So yeah, there's a lot of

00:00 Brian Okken 17:29

00:00 that's why that's the way to deal with retention is to have a system where people need to be skilled to use it, and it removes all their other skills, so they can't get another job.

00:00 Unknown Speaker 17:41

00:00 Oh, just about to say, this seems like a retention ad. It matters to me. But you know,

00:00 Michael Kennedy 17:48

00:00 your mileage depends on which side of that leave, stay story you're on. Because if your job is like, I want job security, and I don't want to have to learn new things like welcome in there are a number Nervo whatever this is, right, your your home, that you can stay here forever, they won't replace you. On the other hand, if you want to use like fast API, like I gotta get out here. We can't be on you know, python two, three, this long. I'm out. off.

00:00 Brian Okken 18:16

00:00 So I didn't want to mention that both Minerva and Barbara are things that you can pip install. But those aren't things we're talking about.

00:00 Michael Kennedy 18:24

00:00 No, these are like secret code names. I suspect the Minerva you installed doesn't have its own ID. I'm just guessing. Right? Yeah. There's a lot of a lot of interesting conversation here about this. So quick little, no, I actually ended up I'm like it was late last night. I'm like, I don't want to read this, like 30 minute thing, but I didn't want to talk about it tomorrow. So I actually use this thing called natural reader. You just throw this in here. And it was just like a read it back super nicely. So cool. So I was like, it was like 1130 Last night when I started on this. I'm like, Oh, I'm a little too tired to focus on this much stuff. But uh, yeah, it's pretty nice. You may

00:00 Unknown Speaker 18:59

00:00 have just made my day because I actually, you know, being part yeah, because being partially blind, I, I feel like you know how some people who have like lupus and other kind, they talk about spoons, like, they only have so much energy. My eyes only have so much like ice cream spoons. Yeah, of course of any one day that I can, I can do. And there are times when I'm like, I can not stare at a screen for even a moment longer. And I really love things like audio books, or the rear services like pocket that will actually read you articles. And so this sounds like an awesome tool for those things that

00:00 Unknown Speaker 19:31

00:00 don't have that mutual. Nice. It's

00:00 Michael Kennedy 19:33

00:00 really neat. It has different different modes. I don't know how to make this thing come up without creating an account because I don't know my login is here. But it has like if you're dyslexic it has like a dyslexic font you can turn on and it'll highlight as you write. It's it's pretty neat. So yeah, cool. Not directly related, but I've found it to be useful and doing this research. I was too tired. A little bit of audience interaction here. Brandon says Run Run the source code from a database. That's a one heck of an injection attack. Yeah, my little Bobby Tables will be just all over this thing, right?

00:00 Unknown Speaker 20:07

00:00 It's one big injection risk.

00:00 Michael Kennedy 20:10

00:00 Although on the other hand, I feel there might be some big safety here as well, because it's everything is pickled, you've got to know what version of Python they forked it from. And any variations, they may be changed to the object model to even get it to parse it back. So it might be naturally safe as it's just like an opaque. It's almost like encryption.

00:00 Unknown Speaker 20:32

00:00 Really, really, really weak encryption.

00:00 Michael Kennedy 20:34

00:00 Yeah. And then let's see. Yeah, David says, Absolutely. Everyone in the live audience has a furrowed brow right now. Yes, David, or some like nervous laughter.

00:00 Unknown Speaker 20:47

00:00 You there is one serious series aspect of this, though, we're talking about like retention. And the article talks about like, as he worked on in this in this private ecosystem, he felt his external skills atrophying, that is a common problem that I have heard so many people who work in these large, kind of like pocket universes, whether it's Amazon or Google, or Microsoft, or an IBM or whatever, people, people can kind of feel very disconnected from where the field is currently going at the hot leading edge in the outside world. And, and I think it's a real issue, because you can't always justify pulling in the latest, greatest, sometimes their engineering or earring and patience to that that would be really problematic to your business's bottom line. And so I say and I realize time is, is finite. And it's this is hard. But this is where I say even dabbling in having side projects, even if it's just the kind of thing you do on weekend, every once in a while we have time, it's a great way to sort of like feel unconstrained by the requirements of your business and to go sort of play on your own terms.

00:00 Brian Okken 21:59

00:00 I mean, great helper scripts for yourself, even for your job and in modern Python.

00:00 Michael Kennedy 22:04

00:00 thingo. Yeah, absolutely. Good. Good advice, indeed. Right. Over to you, Chris.

00:00 Unknown Speaker 22:09

00:00 Okay, thank you. So I decided I was thinking about this and thinking about the fact that, you know, most of the people who come in, and our guests here on this podcast are really sort of accomplished hardcore Python programmers, and they're familiar with the latest and greatest tools, there is no way that I'm going to scoop you all. So I decided to go a completely different direction and going, but that's very exactly of the whimsical side of computing, because I think we all need to stay in touch with that. I know, I had an issue a number of months ago, before I changed jobs where I was feeling very burned out. And I felt like I just, I need to sort of get back in touch with like, why am I passionate about about computing and programming in this space. And one of the things that really was fall into my, you know, sooth, by my burnt out soul was these retro consoles have, I used one called tic 80, which was not Python based, it's, it's newer based. But the idea here is you can build like, basically retro style video games very, very, very easily, with very few lines of code. Because these things block, they do have some constraints. But the constraints are completely artificial, right? Like, in this case, this one is fixed 16 color palette, 256 by 256. You know, like, the constraints, the world that you're building in, is uncomplicated, and small. And so as a result, you can simply like step in, and wrap your head around this thing so quickly, and come up to speed. And suddenly, you're like, you know, oh, I need a flying saucer fly across the screen, I built a spray, oh, I'm making music. I'm bad coding, you know, the mean loop might get him. And you can do it in a very comparatively speaking short period of time, with just a few lines of code. And just getting down a little bit here of like, Let's take an example. They have the sort of like, you know, snake eating dots kind of game. And it's comparatively speaking, like when you compare it with like, what are the modern, you know, game, a framework could be like, Unreal Engine, Unreal Engine, or unity where you're talking about, like, here's my 60 pages of coordinate transforms, and you just look at it your way guy. I can't I can't get

00:00 Michael Kennedy 24:35

00:00 my 100 gigs of textures that we're gonna export. Yeah, exactly. So this project is called pixel, I'm guessing Yes. How you p y x CL, and it allows you to create extremely simple 2d animated games in Python, right?

00:00 Unknown Speaker 24:50

00:00 Yes. So I apologize. I should have said that. Yeah, actually. And so back when I was using the other one, I kept thinking the entire time like, wow, you know, I'm really having fun with this. And I'm really, I actually really enjoyed learning Lua. And it's a really fun language. And it has a lot of a lot of, you know, characteristics that are wonderful. But I kept thinking like, wow, I kind of wish I didn't have to, you know, get my brain to switch gears, which is not very good at doing always, I wish I could be coding this in Python. And so this framework gives you the opportunity to do just that. And so you can use these, it almost feels like for those of us who are old enough to remember, it almost feels like coding and basic on the microcomputers that some of us grew up with in the ad, right. Like, it's that sort of kind of like,

00:00 Michael Kennedy 25:38

00:00 back when you could type it in, like I saw it printed, and I typed it in, and I ran it.

00:00 Unknown Speaker 25:43

00:00 Exactly, exactly, exactly.

00:00 Unknown Speaker 25:46

00:00 You know, it's like it sounds great.

00:00 Unknown Speaker 25:48

00:00 Yeah, it really, really is. And so I really, this one, what's what's kind of interesting about this one is, like, this is a great example, the sort of like sounding music section of this snake game here is, um, this one is actually a little bit more Pythonic, which I think is actually really kind of cool for Python folks like us, because the other ones that I've seen kind of, like drifted more back towards the 1980s, like, you know, a little bit lower level of abstraction kind of thing. Whereas this one really takes advantage of some of the beautiful data structures, like sets and dictionaries and the like, that Python has to offer. So you can use this really kind of nice, expressive tax to, to write your games, and I am just super excited. I played with it a bit, I wrote a few lines of it, and I you know, kick the tires, just to see how it worked. But I really have not had the opportunity yet to, to dive deep and kind of rewrite my side project in it. But I totally intend to, and I'm super psyched about it. And I'm really, really glad that it exists.

00:00 Brian Okken 26:52

00:00 Yeah, I'd love to do this. I mean, I got my start typing in games from the back on a trs 80 from the back of magazines, and, and have an example. So one of the things something like some engine like this, one of the things I love to do with somebody is, is take an existing game and say, let's look at the code and see what happens if we modify this this field. Is there any way we can get the character to run faster? Does that make it more fun to play the game or less fun things like that?

00:00 Unknown Speaker 27:18

00:00 Absolutely. And that's and that kind of thing that kind of like, let's take a look at this and tweak it a little bit is so simple with this kind of thing. Because it's like everything is so like uncomplicated and straightforward like chances are like the here is the number of frames that we get as we transition our sprite character across the screen. Oh, it's just tweaked that now you don't you're sort of like, Let's make him run a little faster kind of thing. It's all so satisfyingly in a world where I don't know about you guys, but my day job, especially the old one that I did was like, you know, make a code change or make a configuration change and then wait, you know, eight hours for to deploy through the pipeline, like having something that's immediate instant feedback. Like this was just a delight.

00:00 Brian Okken 28:04

00:00 Sorry, I'm just shocked at eight hour turnaround time. I'm so sorry. That was

00:00 Unknown Speaker 28:09

00:00 it. That was it. That was just a really bad example there that you know, a couple of cases. It's not that's not my day to day life. I actually love my day to day life. I'm very happy. But that that was just nice. You know, to me, we all have these like long build times. Yep.

00:00 Michael Kennedy 28:20

00:00 Yep. Wilma googan. Hey, well says this makes me missed out nostalgic for my ZX Spectrum in the 80s.

00:00 Unknown Speaker 28:29

00:00 Exactly. There's an Atari 800 XL behind my head did i i enjoy as well. So I feel that

00:00 Brian Okken 28:37

00:00 who's not surprised that Will's been coding since the 80s.

00:00 Michael Kennedy 28:43

00:00 Now now, and then also a few more follow up items for your comment on side projects. Sam in the audience says 100% Agree, Chris side projects are a great way to keep yourself sharp and Jose says plus one for side projects to try out the latest indeed. So before we move on, though, right, I want to tell you one about our sponsor. This episode is sponsored by shortcut. Again, thank you shortcut for sponsoring and supporting by the bites. So there's a lot of project manage management tools out there. But they often suffer from either being too simple for engineering teams or too complex. And they end up looking like Bugzilla or something in saying you're like I really don't want to go here. You want something nice and simple, but that actually scales with you something that's good both for managers and for engineers. So that's what shortcut is, it's built to cater to both of these groups. So it's built for software teams based on making workflows easy, big scale objectives, goals. roadmaps can be easily tied into the team or individual work sprints. progress on those are easily follow back to higher level reports. And of course, as all things should be, it's keyboard friendly. You don't have to take your hand off and do a bunch of mousing around. You can use the UI if you like but there's a bunch of hotkeys which is fun static as I think they all should be, so makes you work faster and just keep going. Good version control integration works with Git, GitHub GitLab. So on a lot of the places that you probably already know, you can update your test project just by doing a commit or a PR, which is fantastic. Last thing you want to do is update Git and get up and then go update your system, right? As iteration planning, you can see your team activity when you want to, or just focus on what you're doing, you can even let it schedule your project for you. So nice, burndown charts, cycle time, all that kind of stuff. So finally, you can have a project management tool that's both for engineers, and managers give it a try at Python bytes FM slash shortcut to cool project, and it helps support the show. So thank you for supporting us, Brian. What do you got next? What do we ditch and now

00:00 Unknown Speaker 30:48

00:00 we're ditching code Cove.

00:00 Unknown Speaker 30:51

00:00 So our route

00:00 Brian Okken 30:53

00:00 so Okay, so coke has been great. For a lot of a lot of people have used it. It's a it's a third party third service that you can attach to your repo or something. And it makes it makes sure that you in this has been helpful for merge requests and stuff. So so if somebody has a PR, it makes sure that the coverages and if you've had 100% coverage and make sure that they don't degrade at the the code they add, makes the coverage stay at a high level that but so it's not without problems, however, and Henyk mentions that on some of his projects, it's been sometimes a CI pipeline is blocked, just because it timed out with code coverage. And that's not a good reason to fail a cover, fail build so so anyway, he said, Pinnock has this this project where he said, Wait a second, I'm just trying to make sure everything goes to 100%. Coverage, that pie. Hey, Ned, second second thing here for you, but covers up hype from Ned Batchelder. maintaining it, of course, has a thing that it says fail if under 100, under a percent. So you can do 100%, but you can do 80 or whatever your current percentage is. But anyway, if you want that something like code cuff, you can, you can get that with coverage up high. But there's some tricks around it. So if you've got one of the things coverage does is let's say you've got a project that has some some code that is that is run, like you're trying to take advantage of some of the newer things, but you also newer Python things but you want to run on older python two. So there's some if statements for Python version, for example. But it might be different hardware as well like Windows versus Mac. But there might be code that's not that's run in one test suite and not on the other. So in order to do that, you kind of have to run in parallel mode, and then combine the coverage reports at the end. And it sounds complicated, but mostly we just copy somebody else's work. So in here, Nick has the examples here. So Henyk goes through using GitHub and, and coverage to do everything that you used to be able to you were doing with code. Gov. So this is a nice article with examples. So we've got for each project, we're downloading the coverage report, and then combining it at the end with a report with a fail under 100. Really cool. It's if this is something you want to do, it's basically it's a short article with examples, even shows, he's got in his struck log, he shows his main workflow for GitHub pipelines, that does all of this. So just kind of read his code and make it work. And I'm that I was actually thinking of doing this on a couple projects I've got, I wanted to make sure that they were up 100% and do something like this. But I actually, for some reason, I was dragging my feet. And I didn't know why. But now I don't need to go ahead and use Hynix suggest now you

00:00 Michael Kennedy 33:58

00:00 don't have to invent it just to set up that job on GitHub.

00:00 Unknown Speaker 34:02

00:00 This is actually something I've been I've been kind of struggling with actually struggling with, I should say, this is a journey that I have been on in that testing and code coverage. And that whole class of, of proof of correctness is my weakest suit. And one of the things that I've really been sort of trying to wrap my head around is I see a lot of like, unit test code. That seems like it was it's just there to keep the the coverage percentage up. And as far as I can tell, it doesn't actually test anything. Like we're, you know, creating these mocks and like everything and but they don't actually like enforce any contracts or prove anything out. And I I've tried to figure out how we can use these tools in ways that actually sort of raise the bar and code quality without encouraging this kind of like, I don't know crossing an eye dotting behavior?

00:00 Brian Okken 35:02

00:00 Well, we're kind of getting on a tangent. But I would say that code reviews on tests are important. Yeah. Because that's, that's just I mean, one of the easiest ways to get 100% coverage with no tip failing test is don't put any asserts in your tests. Just saying Good.

00:00 Unknown Speaker 35:20

00:00 That's it. And that makes an excellent point unto itself. Yes. Thank you.

00:00 Michael Kennedy 35:25

00:00 Nice. All right. Well, let's talk about the tippy top, the tip top. So I'm sure you're all familiar with top. It's like this somewhat dreadful way to tell what's happening on a UNIX system on Linux to see like, what's happening is it why is it busy? Is it not busy? I say dreadful, because it's so simple. And there's better tools. One of them being this tip top thing for for a long time I use glances. I didn't know, I still use glasses. I'm not sure if you're all are familiar with the classes. But the UI that it creates is fantastic. So you just type glances. And you get this sort of progress bars that showing the CPU, the memory, all the processes it has like hotkeys, you can sort by CPU, or memory, filter by process name, or ID all kinds of stuff going on here. And this, this is what I use now. But Zack Villars, sent in tip top as a competing thing. And by the way, glances is Python and tip top is Python. So very good candidates for PIP x, because no project needs them. You just want to run a mess commands, but they're Python, right. So if you look at the same picture over on the tip top page, you can see that you get similar information. But you get a little bit better historical understanding of how your CPU is doing how your memory is doing. It even breaks it down per thread or per core. It shows you your memory broken down by what's being used, what's being cached, what's free. There's a graph of that over time. Something that made me sad is it shows the network over time as well that in in an outflow, but what happened to me as one of my servers, I ran this in it, it only would show the output from like a some kind of there were multiple network adapters and would only show the one that's not being used, not the one that's being used. So like Well, that's that part is useless, unfortunately, here, but and then it shows all the programs like the CPU usage, the user is running as the arguments that were passed to it, how much memory it has and stuff. So I don't think I'm going to give up using glances because I really love that. But this feels like another tool to get a view of that sort into it. What do you think I like the

00:00 Unknown Speaker 37:29

00:00 visual design on the face of it, like, especially comparing it's Lance's screenshot showed it feels like a prayer, it's a little prettier in it make it to me, at least it looks a little bit less, like dense and maybe a little bit easier to sort of like, maybe it's just the bounding boxes. But I also feel like it's the way the screen is laid out a way to visually an easier way to visually parse the information that's presenting, that's always something I look for mean to us. Absolutely. Right.

00:00 Brian Okken 38:00

00:00 Why do you need some I don't use anything like this. So

00:00 Michael Kennedy 38:04

00:00 alright. So for example, if you log in to let's say, you logged into a web server that was running a search job, and it said, you're running a little bit low on memory as a over the last few minutes, and you've got five different web apps set up running their web processes with like a bunch of their sub processes running, and you want to answer the question. Okay, which one is it? Which one? Do I have to restart? Okay? Things like that.

00:00 Unknown Speaker 38:30

00:00 It's easy to see where you might not encounter this or have a need for this when you're working in the world of things, right? Like the world of not servers, but embedded devices that might run Python or C or whatever in you know, this is this is like managing infrastructure. So you do this on your desktop?

00:00 Michael Kennedy 38:50

00:00 No, I don't use this on my Mac. But I use this on all of my not necessarily tiptop but glances on all of my Linux servers, including the Python bytes one. So it's, it's nice to go in there and see what's going on in the servers with all I have is SSH. I'm more of a

00:00 Brian Okken 39:03

00:00 high level person, I'd rather pay somebody else to do this for me.

00:00 Michael Kennedy 39:08

00:00 Well, let's like, but it looks pretty. Let's see if I can pull up a picture of top

00:00 Unknown Speaker 39:15

00:00 on behalf of my employer, Brian, thank you. Thank you. Yes, we're happy to take to do that for you no problem.

00:00 Michael Kennedy 39:21

00:00 Like this. This is the thing it's replacing, by the way is it's just a list of processes. And it moves around all the time. And this stuff shoots off the screen. It's like why is it doing this? You know, it's

00:00 Unknown Speaker 39:33

00:00 I mean, it was also written probably in like 1968 or something

00:00 Michael Kennedy 39:37

00:00 Yeah, yeah. I'm not knocking on it. I'm just saying I would rather use you know, something like this. That gives me way more real time like graphs, not just snapshot now it is now it is now it is but like, here's here's what's happening as the things are running and doing their work, right so that that's super helpful to me to see

00:00 Brian Okken 39:59

00:00 a typic to be fair So just just installed Tip Top just for the heck of it on my Mac and really quick install, and it does look pretty. I just don't know what to do with it. But yeah,

00:00 Michael Kennedy 40:08

00:00 and by the way, let's just tie it back to the audience again. So Tip Top uses textual for layout, which is based on which is well McGuigan saying also based on rich, so, it's quite modern.

00:00 Brian Okken 40:20

00:00 I think if we had Ned and will in the audience, we would always touch on something they were doing.

00:00 Unknown Speaker 40:27

00:00 I'll tell you what, Brian, here's a way you could use this totally in your day to day life. Your Mac is running slow, right? And you say, why is my Mac running slow? You could totally use this and then a glance, say, What's worrying all my RAM? What is throttling my disk? What is using up all my CP, et cetera, et cetera?

00:00 Michael Kennedy 40:45

00:00 Okay, yeah, like, for example, you can run Activity Monitor, which will show you the process thing, and you go to the memory thing, I'll show you just a single graph, but like, why is it using all that memory? Is that actually being used? Or is that cash because you loaded a bunch of files, and now it's kind of could be free, but it's not what's happening on the network, you can't see very well on Mac OS. I don't know why. Activity monitors are adverse to showing you network traffic. But you could see like, am I downloading something or not right now?

00:00 Unknown Speaker 41:12

00:00 I can be taught new things. Okay. Thanks.

00:00 Michael Kennedy 41:15

00:00 Yeah, yeah. Anyway, I'm not excited for that use case. I'm excited for using on the servers to understand them better. Yeah. Yeah. All right. Let's bring it home with some more retro Chris.

00:00 Unknown Speaker 41:28

00:00 Back in the world of Livesey. I, I always been into well, you know, always ever since they weren't retro, and they were they would just eat bit machines. I have I grew up started with with eight bit and Geeta, I was an Atari kid, never owned a C 64. But someone has actually written a Commodore 64 emulator called pi c 64. In pure Python. And the reason that I picked this that I it really is, isn't it, I think it is so cool. Because not only do you get this Commodore 64 emulator, it's not 100% coverage, like he is only emulated to display to the point where you can display pecky characters. But the bottom line is, is other than that, a largely full machine and military ones on the PI 65 library, which is a 6502 CPU chip emulator, also written in pure Python. That's all it really is. And and it's so neat, I was able to pull this thing down and what it and I was actually writing like counter 64 Basic that poked bits into screen memory and no drew pictures off the screen. And, and one of the really cool things is, this thing even includes you can sort of like pop out of your emulated Commodore 64 With go by command to get a Python repple step is actually doing that like bollock by the emulated Commodore 64 is memory by tweaking the Python data structures inside the repple. It is just

00:00 Michael Kennedy 43:06

00:00 as is that's pretty deep. Thank you.

00:00 Unknown Speaker 43:09

00:00 It is really cool. And, and it is also from my perspective, one of the reasons I picked it, it's a testament to this programming language that we all use in love for a work and you know, whether it's webby stuff, or testing devices, or in my case, automating network infrastructure can do so much you can reach out into all these other really kind of, you know, full niche areas, whatever you're interested in, and, and do things with Python on your computer. So I just thought this was incredibly cool. I have enjoyed playing with it. And I definitely intend to enjoy playing with it some

00:00 Michael Kennedy 43:46

00:00 more, might be some interesting ways to go back and study operating systems back when they were simple if you're like a computer science student as well, yeah, for sure. I

00:00 Unknown Speaker 43:55

00:00 think there's a lot to learn there. You know, even if you're not necessarily someone who is keenly interested in writing your own operating system, I think there's a lot to learn just in terms of the programming paradigm, right? Like when when you wanted to solve a problem, and you want it to display something on the screen. And a perfectly reasonable answer was, I'd go into both the right bits into screen memory, because that's a more efficient way to display this thing as opposed to using whatever my high level language provides. It is. It's such a different way of thinking the reality that most of us have it day to day. It's just a fun refresh, refreshing place to visit least Martin perspective.

00:00 Michael Kennedy 44:38

00:00 Yeah, very cool. It's a good one. Thank you, Brian. Sign for extra extra extra here all about it. How many extras Do you have?

00:00 Unknown Speaker 44:45

00:00 I have zero today.

00:00 Michael Kennedy 44:48

00:00 You have no extras? You're not adding to the number of extras. Chris? Anything else you want to throw out there? Let people know about?

00:00 Unknown Speaker 44:53

00:00 Sure. Absolutely. My team and Amazon is hiring. Um, hopefully we can put a link in the show notes I work for Optech it, we work out with the fulfillment centers. And I that I cannot say enough how much I love working with this team, people are incredible. And they're sort of like, at least the folks I'm working directly with are people who started out and so filmin centers, and we're interested in technology, right. And they were able to prove that they could actually walk the walk. And now they're being promoted into foreign engineering positions. And I just find that so incredibly refreshing because it's fantastic. Yeah, these people just have a very different perspective. You know, they're not jaded. Like I've been in the tech industry for 20 years types like me, it's, it's really inspiring to work too,

00:00 Michael Kennedy 45:42

00:00 though. Yeah. Not not too many debates of like, why don't we just switch to a functional language? Come on? Practical model, just get stuff done. I love it to a bank before we move. Yeah. Yeah. So we'll put that link in the show notes. Before we move on to my extras, which are not many. Sam says it's so helpful to understand how computer works, when you're writing code, these kinds of things are a good way to learn that talking about the converse 64 emulator. Awesome. Alright, so I just have one. And that's I'm going to be speaking at flask con 2021. And so flask con is the first week of December, which has a remote attendance, I'm going to be talking about HTML and flask once again, because I'm super psyched about all that stuff. And so if you want to see some really cool apps get built using HTML and flask dropping over there. There's a whole bunch of other talks going on as well. So I'll put the link to that in the show notes. All right. Shall we finish it off with a joke?

00:00 Unknown Speaker 46:38

00:00 Yeah, Liberty is the soul of wit. Absolutely. It is.

00:00 Michael Kennedy 46:42

00:00 So we just recently went through the whole Daylight Savings unraveling and went back to normal time. And I don't know what it's like for you all. But when I look at the weather and stuff here, I suspect Chris, it's similar in Boston. I mean, I know it's similar for you, Brian. Yes. It's like sunset is at 4:38pm. And we're headed towards the winter solstice. So it's only getting earlier in the day. It's like, ah, that that doesn't make me it doesn't spark joy in my in my heart that it's already dark at 430.

00:00 Unknown Speaker 47:15

00:00 I like it. Yeah.

00:00 Michael Kennedy 47:18

00:00 I don't mind that it gets dark. I just wish it wouldn't. I wish we could just move it. So there's a little time the end of the day where there's still sun.

00:00 Unknown Speaker 47:24

00:00 I remember. I'm with you, Michael. I'm solar powered. So I enjoy the sunlight. And I don't like it getting dark earlier either.

00:00 Michael Kennedy 47:31

00:00 Yeah, absolutely. I don't hate the winter. I just don't wish it shifted around a little bit. So this is a technical take on that. That our joke as this comes from the New Yorker. I mean, we're pretty highfalutin, right? We're not doing like Dev jokes.io, or whatever. So this is by Matt Reuter. And it's a it's a little cartoon of two people clearly in kind of a darker space, a woman sit in her chair with a laptop. And you could just like the beaming laptop screens, like a lightning rod, right? And the guy here on his tablet, same thing glowing on his face. And underneath that it says I hate how the screens get bright. So early this time of year.

00:00 Unknown Speaker 48:11

00:00 Nice. This is so much about our modern world. It's both it's both hilarious and kind of sad. It only would go

00:00 Michael Kennedy 48:19

00:00 absolutely. Yeah. So welcome to daylight savings or standard time. I guess everyone, the screens do get bright this time of year.

00:00 Unknown Speaker 48:30

00:00 That's funny. Nice.

00:00 Michael Kennedy 48:31

00:00 And, Chris, thanks for being on the show. Brian. Thanks. As always, yeah, thanks a lot.

00:00 Unknown Speaker 48:36

00:00 Thank you so much. And I really appreciate everything that you guys do for the community. It's it's really, guys a great examples of why Python is such an amazing place to live. Thank you. Thanks, Chris. Appreciate it.

00:00 Transcribed by https://otter.ai

Back to show page