Transcript #260: It's brutally simple: made just from pickle and zip

00:00 - Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.

00:05 This is episode 260, recorded November 19th, 2021.

00:10 I'm Michael Kennedy.

00:12 - And I'm Brian Okken.

00:13 - I'm Chris Petty.

00:14 - Chris, welcome to the podcast.

00:16 You're a podcast veteran, but the first time here on Python Bytes.

00:20 Happy to have you here.

00:22 - Thank you very much.

00:22 It's a pleasure to be here.

00:24 It's good to be back.

00:25 - We're gonna jump into a bunch of the news, but before we do, you know, tell people a bit about yourself.

00:29 Absolutely.

00:30 I've been in the tech business for about 35 years now, something like that.

00:35 And, I've been a, and a self, self identifying Pythonista for about the last five or six years.

00:42 I work for these days.

00:44 I work for just, just recently changed jobs within Amazon.

00:49 I used to work for the web services, elastic file system team.

00:52 And I now work, I switched gears from, you know, living in the clouds to living the Amazon fulfillment center is we're now a network.

01:01 So is that mean you're doing what like programming in COBOL or we using now?

01:04 Yeah, nothing but COBOL and Fortran.

01:07 They're just all the worst, all the most possible, you know, not at all.

01:10 We do my team Optitech IT, we do network automation in for infrastructure.

01:17 So it's a lot of fun, actually.

01:19 It's really kind of neat going from building microservices and things like that, being infrastructure oriented at that level to like working with being responsible for automating things like, you know, network protocols, DNS, DHCP, fun stuff like that at scale in the fulfillment sectors. It's a really cool gig. Yeah, that sounds really interesting. And you get an interface with physical things that go physical places, which is interesting.

01:44 Yeah, it really is. Thank you. I only ship bits. I don't know. Brian, You also you also work on physical things.

01:51 Yeah, I like working with actual make things.

01:54 It's nice.

01:55 Yeah, absolutely.

01:56 It's a nice change of pace.

01:58 Yeah, I've known out in the audience.

02:02 Nat Ned batch elder.

02:03 Hey, Ned.

02:03 Thanks for being here.

02:04 This Chris Boston represent.

02:05 That is amazing.

02:08 His he does so much work with Boston Python, and I can't wait till we can get back to actually like being in a room together with other people again.

02:18 - I know, absolutely.

02:20 So Brian, I used to feel like a cog in the machine, but now I work at a pretty small company where I make a lot of the choices, so I don't have to feel like that.

02:29 But I think we're gonna talk about something that I think is awesome, because Ned is here.

02:35 - Yeah, it's kinda nice that Ned's here.

02:36 We picked this just for you, Ned.

02:38 Ned does a lot of stuff for Python, and one of the things that he's done is this cool tool called Cog, which I've been watching for a while, but I haven't had a really good reason to use.

02:55 But what Cog does is you point it, you can have these Cog, like these, it's like three brackets and the Cog word, and then some code.

03:05 And then another magic incantation at the end, it's pretty simple, but it's hard to explain, but you have some code.

03:12 And when you point Cog at it, it will run the code and then put the output of the code somewhere else where you put it to.

03:18 So it seems neat, I wanted to use it, but I didn't know what a use case I would use it for.

03:25 Well, Simon Willison wrote this article called "Using COG to Update Help," the help output for a Markdown readme file, and I think this is brilliant.

03:35 So this is super awesome use case.

03:38 So Simon, he even has it going up on his CSVS website.

03:43 VS to SQLite project.

03:47 The idea is you've got at the end of the project, he actually lists on the read me, he lists the help output.

03:56 Now, I actually have a project that the help output is out of date because the way I deal with it is I run the help and then copy and paste.

04:03 But Simon's right, this is a great use for cog to just point cog at it.

04:10 So he wrote a little snippet of code that goes in his readme.

04:15 And since cog ignores comments, you can put it in an HTML comment block.

04:21 And it so toggle run that file and took the output, stick it out.

04:25 And then Simon didn't stop there.

04:27 He said, I want to make sure that this gets run when it needs to get run.

04:32 So he wrote a test in Pytest, of course, to to check to see if cog needs to be written.

04:39 So this is just, this is really pretty cool.

04:41 I like it.

04:42 - This is the kind of thing that's so cool.

04:44 Pardon me, Mike.

04:45 This is the kind of thing that's so cool because anytime you can eliminate a DIY, right?

04:51 Anytime you can make one thing, the one true source of whatever piece of the puzzle you're looking to build, you win.

05:00 And this is a great example of that.

05:02 - Yeah.

05:03 - Out in the audience, a little live real-time feedback.

05:05 Ned says, "I also just updated the coverage and CogDocs to do the same thing and to get help output into the docs.

05:13 - Nice.

05:14 - Yeah, very cool.

05:15 I love the inception of it.

05:17 Yeah, this is definitely a cool project.

05:20 I want to take you all on a journey.

05:23 Come with me into the world of Bank Python.

05:27 - My drink.

05:28 - This is an interesting article here.

05:30 It's one of these things that says, I can't exactly tell you precisely where I worked or what we did.

05:38 I have to make up names for the projects that are not the actual names.

05:41 It's by Cal Patterson.

05:43 And it says, "The strange world of Python "as used by big investment banks." So this is quite the read.

05:51 If you go through it, if you click the little reader bit here, it says, you know, 30 minutes of reading time to read it.

05:57 So it is a non-trivial bit of history.

06:00 But let me see if I can give you some of the highlights that I took from this.

06:06 And you can go read it if you really feel like you want to dive into it.

06:08 So one of the things they talk about is the database and all these things have interesting names.

06:14 I'm sure that they have actual interesting names as well that are not these names.

06:18 So for example, like the database-like system is called Barbara.

06:22 And I guess the place to start here with this oral history that Cal talks about is that bank Python implementations are basically forks of the entire Python ecosystem, not just a fork of CPython, but like the libraries that it uses and everything, which are used at many, but not all of the biggest investment banks.

06:42 So it says, let's talk about this system called Minerva.

06:46 Minerva.

06:47 And it's a global database of Python objects.

06:52 And Barbara is this way to talk to it.

06:54 Okay.

06:55 So a lot of interesting keywords to keep it hidden, right?

06:57 But one of the things that's wild, this Barbara thing is, it's like a hierarchical database that stores basically zipped pickle objects.

07:08 Wait, wait, what?

07:09 So it's, and when I say hierarchical, I mean, it's in what they call rings.

07:14 So you would talk to a database and there might be like a nearby replica that you talk to that might have a record or it might not.

07:22 And so then there's like a bigger replica of sort of fallback to go look in this place.

07:27 And if it's not there, go back and look at another one.

07:29 They call these rings.

07:31 And then you would ask for something by its key and then you get it back.

07:35 So first of all, what do you guys think of just pickling all of your data objects?

07:40 (laughing)

07:41 - It's really, that bit made me chuckle because it is exactly the kind of like, at the time, pragmatic engineering trade-off that somebody made, right?

07:52 Years ago, before we all saw pickles as the great villain of the Python ecosystem.

08:00 (laughing)

08:01 Before we knew that there could be downsides to pickling all the things, and they're like, well, it's perfect.

08:07 It's a pickle idea.

08:07 What more could you want?

08:08 And now we're all just kind of looking at it.

08:09 - Yeah, it's so simple.

08:10 It's small, it's fast.

08:12 Yeah.

08:13 Yeah, how do you index something?

08:15 - Actual database?

08:16 Just curious.

08:17 - Yeah, this is a fine question.

08:19 They do, if you read it, they do also use databases sometimes.

08:24 Okay.

08:25 So, but the primary way seems to be this, this system called Barbara.

08:29 So one of the things that comes to mind for me here is, oh my gosh, what if you upgrade something?

08:35 What if you change the file, the structure in code of something that you put in the database five years ago?

08:44 Whoops.

08:45 Yeah, that used to be, you know, signed up for the mailing list, yes or no.

08:51 And now when you change the order, that yes or no became yes or no is admin.

08:56 Oh, oops.

08:57 or it just doesn't match anymore and it crashes and you can't read your data 'cause it's incompatible.

09:01 So this would happen with Python the runtime, but also the libraries you're using, it seems to me, and it's just incredibly unstable.

09:09 So that's--

09:10 - How much would it make a bet they also have like some kind of really highly evolved over many years validation system that ensures that the, that's gonna, that the pitfalls that you think are being used in the way that you think they're being used are actually still used that way.

09:25 - Yeah, exactly, there's gotta be layers.

09:28 Sam out in the audience livestream says, "Good luck upgrading your new Python." Yeah, exactly.

09:34 Yeah, like as in never.

09:35 And that's probably why part of it is like, yeah, we just forked it and froze it, because that's crazy.

09:40 Let me show you something else.

09:41 I will try to jump around the article via search 'cause I took notes.

09:45 Applications also commonly store their internal state in Barbara, writing data classes straight into and out of Barbara, with very simple locking and transactions, if not.

09:56 And one of the things they decided is there's no file system available to this platform.

10:02 And the little bits of data that the script picks, gets picked up and outputs, it all just goes back into this key value store pickle database thing called Barbara.

10:12 That's pretty wild, right?

10:13 - Yeah.

10:14 - Okay.

10:15 They also talk about having these overlay features, which is pretty wild.

10:21 This has to do with the different rings that I talked about.

10:24 So when you connect, you can say, I wanna connect to the middle office ring and then the FICC ring and then the default ring.

10:31 So then when you do a database query, which by the way, it looks like a dictionary index, just bracket key, it first is gonna look for it in the middle office database and then it's gonna look in the FICC database and then the default database.

10:45 So there's like these layers of settings and these layers of specificity.

10:49 And they even do things that are to some effect of like monkey patching.

10:54 Like if you need to change it, you can have like your dev ring and then the middle office ring and then the outer ring and you could change the value and kind of like for mocking or something.

11:04 So yeah, to your point, Brian, they say there are some good reasons, Cal says there are some good reasons not to use Barbara.

11:11 If your data, let me count the ways.

11:15 If your data set is large, it may be a good idea to look elsewhere.

11:19 perhaps in a traditional SQL database or KDB plus, which I have no idea what that is.

11:25 There's a limit on the size of Barbara objects, which is compressed at, you want it zipped 16 megs or smaller, which is pretty interesting.

11:33 And then paired with this is this direct, this thing called Dagger, which is a directed acyclic graph that allows you to set up relationships between instruments.

11:42 Like, okay, I've got a bond and then I've got a credit default swap on that bond.

11:46 And then I've got like a fund of default swaps or you know, like it's a whole layer of stuff.

11:52 And all of these things are like linked objects through these pickled things that you can ask questions about and like get sort of like graph database type things.

12:03 That's pretty interesting.

12:05 They talk a lot about how you model these things like instruments and swaps and stuff in Python.

12:12 That's not super interesting, but I wanted to highlight this section and see what you all thought.

12:16 It says, if you understand Excel, you'll be starting to recognize how a lot of this stuff here works.

12:24 In Excel spreadsheets are cells and they have dependencies, which is like an acyclic graph, like this Dagger hierarchy graph database thing has.

12:33 And it has a lot, so they basically modeled so much of the code around how Excel works.

12:39 So you model the data first and then the behavior, rather, usually we write code and then we have data that comes out of it.

12:46 So it's pretty interesting.

12:48 There's also this thing called Wallpole, which is a database runner.

12:52 So when you get to this section, so it's time to drop a bit of a bombshell.

12:56 Let's talk about deployment, versioning, et cetera.

13:00 The source code of this entire system itself, even including a Barbara, is stored in Barbara, not on disk, and it remains composed.

13:08 And it's kept in a special Barbara ring called source code.

13:12 (both laughing)

13:15 - All right, what do you all think of that?

13:17 - My hidden release engineer just left the building.

13:21 (both laughing)

13:22 Oh man, that's frightening.

13:24 - Yeah, so not keeping the source code in the file system breaks some assumptions.

13:28 Like how does the program run?

13:30 So this wallpole thing runs, it's built to basically run source code that originates from a database.

13:35 And it's all one huge database.

13:38 So like if you think mono repo, it's like that because you import from database and it's just like whatever you need is there.

13:46 So versioning, it's like the anti-microservice.

13:51 Basically the entire whole thing is literally there.

13:55 And then the way that you deploy it, let's see there, they talked about vouch.

13:59 So the way that you deploy to it is really interesting in that it reminds me of the, it works on my machine certification program.

14:08 So the way you deploy to the system is you just commit it to the database and boom, it's deployed.

14:14 It's running, right? Because there's only one place the code lives. It's in the database.

14:18 So once you do a commit, that's a full on deployment.

14:21 Now, how does it get into there? There's a vouch system where you can get somebody who can code review what you've done and say, yeah, I vouch for this.

14:30 You can commit it unless you have a high reputation for not breaking the system, in which case you can directly commit to it without question.

14:38 (both laughing)

14:41 - So I guess the culture of code review is not like a thing there.

14:45 - I guess.

14:47 (both laughing)

14:49 So another interesting knock-on that comes from storing your code in a database is how do you edit it, right?

14:54 Like what editor to use?

14:56 VI won't work on it, PyCharm won't work on it, VS Code won't work on it because those work on files.

15:02 So this Minerva system has its own IDE, because most IDEs think you have files, not databases, that contain the data. Does that make sense? It is. That makes sense. The guy, Cal says, what I can't understand is why does it contain its own web framework?

15:20 Investment banks have a one-way approach to open source software. Some of it can come in, but none of it can go out.

15:30 Well, I've seen projects that like people building IDEs within a company, and I always shake my head and say, "Why? This isn't your business.

15:43 Why are you making an IDE within your company?" There are a couple things that strike me from this article.

15:49 The first is, as I read it, because I really enjoyed it.

15:52 Did you read the whole thing?

15:54 I'm most of the way through.

15:56 It's long, right?

15:57 It's crazy, right?

15:58 It really is.

15:59 It really is.

16:00 The thing that struck me is this is a perfect example of large organization, complex code base and engineering system and time.

16:11 Because this really just smells like something that just got like built up, like, you know, the layers of rings around a tree over time.

16:18 Engineer after engineer, year after year, making decisions.

16:22 And I'm sure like in the modern day, everyone who walks through the door, where they probably have to plan for the period of time where the new engineer just freaks out when they tell them all this stuff as they adjust to the new reality that they're being exposed to.

16:36 - Yeah.

16:37 - And that in some sense, this is a particular wacky case, but it's that way in kind of every large corporate environment in a way.

16:44 - That's true.

16:45 Yeah, that's totally true.

16:46 He actually, Cal actually discusses this a little bit here and says, this fact that you can't do very much of open source, you're stuck on some version of Python that's never gonna see, never gonna go back to sync with the real Python, 'cause how do you even do a database migration to that thing?

17:02 Says it could be the biggest disadvantage for you as your profession.

17:07 Every year you spend in the Minerva monoculture, the skills you need to interact with normal software atrophy.

17:15 By the time I left, I had pretty much forgotten how to wrestle pip in virtual EMV into shape.

17:20 Essential skills for normal Python.

17:22 When everything is in the same repo and all the code is just an import away, software packaging just does not come up.

17:27 So yeah, there's a lot of stuff.

17:29 - That's one way to deal with retention is to have a system where people need to be skilled to use it and it removes all their other skills so they can't get another job.

17:40 (laughing)

17:41 - Oh, I was just about to say, this seems like a retention at-int pattern to me, but you know, goodbye let's be married.

17:49 - It depends on which side of that leave-stay story you're on because if your job is like, I want job security and I don't want to have to learn new things like welcome to Minerva, whatever this is, right?

18:02 You're home. You can stay here forever.

18:05 It won't replace you.

18:07 On the other hand, if you want to use like FastAPI, you're like, I got to get out of here.

18:11 We can't be on Python 2.3 this long. I'm out.

18:15 >> I did want to mention that both Minerva and Barbara are things that you can pip install, but those aren't things we're talking about.

18:24 - No, these are like secret code names.

18:26 I suspect the Minerva you installed doesn't have its own ID.

18:29 I'm just guessing, right?

18:31 - Yeah.

18:32 - Though there's a lot of interesting conversation here about this.

18:36 Just a quick little note.

18:38 I actually ended up, I'm like, it was late last night.

18:40 I'm like, I don't want to read this like 30 minute thing, but I do want to talk about it tomorrow.

18:44 So I actually use this thing called natural reader.

18:45 You can just throw this in here and it'll just like read it back.

18:48 Super nice view. - Oh, that's so cool.

18:49 So I was, it was like 1130 last night when I started on this, I'm like, Oh, I'm a little too tired to focus on this much stuff.

18:56 But, yeah, it's pretty nice.

18:59 You may have just made my day because I actually, you know, being part, yeah, because being partially blind, I, I, I feel like, you know how some people who have like lupus and other kind of, they talk about spoons, like they only have so much energy.

19:10 My eyes only have so much like ice cream spoons.

19:13 Wait, of course, of any one day that I can, I can do it.

19:17 There are times when I'm like, I cannot stare at a screen for even a moment longer.

19:22 And I really love things like audio books or the rare services like pocket that will actually read you articles.

19:28 And so this sounds like an awesome tool for those things that don't have that feature already.

19:33 It's really neat.

19:34 It has different, different modes.

19:36 I don't know how to make this thing come up without creating an account because I don't know my login is here, but it has like, if, if you're dyslexic, it has like a dyslexic font you can turn on and it'll highlight as you write.

19:47 It's pretty neat.

19:48 - That's awesome. - So yeah.

19:49 - Cool. - Not directly related, but I've found it to be useful in doing this research.

19:54 I was too tired.

19:55 A little bit of audience interaction here.

19:58 Brandon says, "Run the source code from a database." That's a one heck of an injection attack.

20:03 Yeah, I mean, little Bobby Tables would be just all over this thing, right?

20:06 (both laughing)

20:08 - It's one big injection risk.

20:10 - Although on the other hand, I feel there might be some big safety here as well, because everything is pickled.

20:16 you've got to know what version of Python they forked it from and any variations they may be changed to the object model to even get it to parse it back.

20:25 So it might be naturally safe as it's just like an opaque, it's almost like encryption.

20:29 But it's weird.

20:31 Really, really, really weak encryption.

20:34 Yeah. And then let's see.

20:37 Yeah, David says, absolutely everyone in the live audience has a furrowed brow right now.

20:42 [laughter]

20:44 Yes, David, or some like nervous laughter.

20:47 There is one serious aspect to this though.

20:50 We're talking about retention and the article talks about like, as he worked up in this private ecosystem, he felt his external skills atrophying.

21:01 That is a common problem that I have heard so many people who work in these large pocket universes whether it's an Amazon or a Google or a Microsoft or an IBM or whatever.

21:14 People can kind of feel very disconnected from where the field is currently going at the hot leading edge in the outside world.

21:23 And I think it's a real issue because you can't always justify pulling in the latest greatest.

21:28 Sometimes there are engineering and patients to that that would be really problematic to your businesses bottom line.

21:35 And so I say, and I realize time is finite and this is hard, but this is where I say even dabbling in having side projects, even if it's just the kind of thing you do on weekend every once in a while when you have time, it's a great way to sort of like feel unconstrained by the requirements of your business and to go sort of play on your own terms.

21:59 I mean, write helper scripts for yourself, even for your job in modern Python.

22:04 Bingo. Yeah, absolutely. Good, good advice. Indeed. All right. Over to you, Chris.

22:09 Okay, thank you. So I decided I was thinking about this and thinking about the fact that, you know, most of the people who come in and our guests here on this podcast are really sort of accomplished hardcore Python programmers, and they're familiar with the latest and greatest tools. There is no way that I'm going to scoop you all. So I decided to go a completely different direction and go in the direction. That's right, exactly, of the whimsical side of computing, because I think we all need to stay in touch with that. I know I had an issue a number of months ago before I changed jobs where I was feeling very burned out and I felt like I just I need to sort of get back in touch with like why am I passionate about computing and programming in this space.

22:55 And one of the things that really was ball into my, you know, soothed by my burnt out soul was These retro consoles up. I used one called tick 80, which was not python based. It's it's lua based But the idea here is you can build like basically retro style video games Very very very easily with very few lines of code because these things They do have some constraints, but the constraints are completely artificial, right?

23:26 Like in this case, this one is fix 16 color palette 256 by 256 you know like The constraints the world that you're building in is uncomplicated And small and so as a result you can sort of like step in and wrap your head around this thing So quickly and come up to speed and suddenly you're like, you know Oh, I I need a flying saucer fly across the screen. I built a spray. Oh i'm making music I'm that's coding, you know the mean loop of my game and you can do it in a very Comparatively speaking short period of time with just a few lines of code. I'm just skipping down a little bit here like let's take an example. They have the sort of like, you know, um sneak eating dots kind of game and And it's comparatively speaking, like when you compare it with like one of the modern, you know, game framework code bases.

24:24 Like Unreal Engine or something.

24:26 Yeah, Unreal Engine or Duty, where you're talking about like, here's my 60 pages of coordinate transforms and you just look at it, you're like, I can't, I can't even.

24:35 And my 100 gigs of textures that we're going to export.

24:39 Yeah.

24:40 Exactly.

24:41 So this project is called Pixel.

24:42 I'm guessing that's how you, P-Y-X-E-L.

24:44 And it allows you to create extremely simple 2D animated games in Python, right?

24:50 Yes, I apologize.

24:51 I shouldn't have said that.

24:52 Yes, exactly.

24:53 Yeah, yeah, of course.

24:54 And so back when I was using the other one, I kept thinking the entire time, like, wow, you know, I'm really having fun with this.

25:02 And I really, I actually really enjoyed learning Lua.

25:05 And it's a really fun language.

25:06 And it has a lot of, you know, characteristics that are wonderful.

25:12 But I kept thinking like, wow, I kind of wish I didn't have to, you know, get my brain to switch gears, which is not very good at doing always.

25:19 I wish I could be coding this in Python.

25:21 And so this framework gives you the opportunity to do just that.

25:26 And so you can use these, it almost feels like for those of us who are old enough to remember, it almost feels like coding in basic on the microcomputers that some of us grew up with in the 80s, right?

25:37 Like it's that sort of kind of like back, back when you could type it in, you're like, I saw it printed and I typed it in and I ran it.

25:43 Exactly.

25:44 Exactly.

25:45 Exactly.

25:46 You know, it's like, yeah, it really, really is.

25:49 And so I really, this one, what's, what's kind of interesting about this one is like, this is a great example, the, the sort of like sound and music section of this snake game here is, this one is actually a little bit more pythonic, which I think is actually really kind of cool for Python folks like us, because The other ones that I've seen kind of like drifted more back towards the 1980s Like, you know a little bit lower level of abstraction kind of thing Whereas this one really takes advantage of some of the beautiful data structures like stats and dictionaries and the like That python has to offer so you can use this really kind of nice expression Text to to write your games and I am just super excited. I I played with it a bit I wrote a few lines of it and I kicked the tires just to see how it worked.

26:39 But I really have not had the opportunity yet to dive deep and kind of rewrite my side project in it.

26:46 But I totally intend to and I'm super psyched about it.

26:49 And I'm really, really glad that it exists.

26:52 - Yeah, I'd love to do this.

26:53 I mean, I got my start typing in games from the back, on a TRS-80 from the back of magazines and have an example.

27:01 So one of the things, some engine like this, one of the things I love to do with somebody is is take an existing game and say, let's look at the code and see what happens if we modify this this field. Is there any way we can get the character to run faster? Does that make it more fun to play the game or less fun? Things like that.

27:18 Absolutely. And that's and that kind of thing, that kind of like, well, let's take a look at this and tweak it a little bit is so simple with this kind of thing, because it's like Everything is so like uncomplicated and straightforward.

27:32 Like chances are like the, here is the number of frames that we skip as we transition or a sprite character across the screen.

27:39 Oh, let's just tweak that.

27:40 Now you've done your sort of like, let's make him run a little faster kind of thing.

27:45 It's all so satisfyingly in a world where I don't know about you guys, but my day job, especially the old one that I did was like, you know, make a code change or make a configuration change and then wait, you know, eight hours for to deploy through the pipeline, like having something that's immediate, instant feedback like this was just a delight.

28:04 Sorry.

28:05 I'm just shocked at the eight hour turnaround time.

28:08 I'm so sorry.

28:09 That was it.

28:09 That was it.

28:10 That was just a really bad example.

28:11 There are that, you know, a couple of cases.

28:13 It's not, that's not my day to day life.

28:15 I actually love my day to day life.

28:16 I'm very happy, but that was just, you know what I mean?

28:19 We all have these like long build times.

28:20 Yep.

28:21 Yep.

28:22 Will McGugan, hey Will, says, "This makes me nostalgic for my ZX Spectrum in the 80s." (Will laughing)

28:30 - Exactly.

28:30 There's an Atari 800XL behind my head that I enjoy as well.

28:35 So I feel that.

28:37 - Who's not surprised that Will's been coding since the 80s?

28:40 So anyway.

28:41 (Will laughing)

28:43 - Now, now.

28:44 - And then also a few more follow-up items for your comment on side projects.

28:48 Sam in the audience says, "100% agree, Chris.

28:50 side projects are a great way to keep yourself sharp.

28:53 And Jose says, plus one for the side projects to try out the latest indeed.

28:57 So before we move on though, right, I wanna tell everyone about our sponsor.

29:02 This episode is sponsored by Shortcut again.

29:05 Thank you Shortcut for sponsoring and supporting Python Bytes.

29:09 So there's a lot of project management tools out there, but they often suffer from either being too simple for the engineering teams or too complex.

29:18 and they end up looking like Bugzilla or something insane.

29:22 And you're like, I really don't want to go here.

29:24 You want something nice and simple, but that actually scales with you.

29:27 Something that's good both for managers and for engineers.

29:30 So that's what Shortcut is.

29:32 It's built to cater to both of these groups.

29:35 So it's built for software teams based on making workflows easy.

29:39 Big scale objectives, goals, roadmaps can be easily tied into the team or individual work sprints.

29:45 Progress on those are easily flow back at higher level reports.

29:48 And of course, as all things should be, it's keyboard friendly.

29:52 You don't have to take your hand off and do a bunch of mousing around.

29:56 You can use the UI if you like, but there's a bunch of hotkeys, which is fantastic as I think they all should be.

30:01 So it makes you work faster and just keep going.

30:04 Good version control integration, works with Git, GitHub, GitLab, and so on.

30:09 A lot of the places that you probably already know.

30:11 You can update your task project just by doing a commit or a PR, which is fantastic.

30:16 The last thing you want to do is update Git and GitHub and then go update your system, right?

30:21 Has iteration planning.

30:22 You can see your team activity when you want to or just focus on what you're doing.

30:26 And you can even let it schedule your project for you.

30:29 So nice burn down charge, cycle time, all that kind of stuff.

30:32 So finally, you can have a project management tool that's both for engineers and managers.

30:37 Give it a try at pythonbytes.fm/shortcut.

30:40 It's a cool project and it helps support the show.

30:42 So thank you for supporting us.

30:44 Brian, what do you got next?

30:46 What are we ditching now?

30:47 We're ditching CodeCov.

30:50 Harsh crowd.

30:52 So, okay. So, CodeCov has been great for a lot of people who have used it.

30:58 It's a third-party service that you can attach to your repo or something.

31:05 And it makes sure that you, and this has been helpful for merge requests and stuff.

31:10 So, if somebody does a PR, it makes sure that the coverage is 100.

31:15 if you've had 100% coverage and make sure that they don't degrade it.

31:18 The code they add makes the coverage stay at a high level.

31:21 But so it's not without problems, however.

31:27 And Hinnick mentions that on some of his projects, it's been sometimes a CI pipeline is blocked just because it timed out with CodeCov.

31:39 And that's not a good reason to fail a build.

31:43 So, so anyway, he, so Henek has this, this project where he said, wait a second, I'm just trying to make sure everything goes to 100%.

31:54 Coverage.py, hey Ned, second, second thing here for you.

31:58 But Coverage.py from Ned Batchelder, maintaining it of course, has a thing that it says fail if under 100, under a percent.

32:07 So you can do 100%, but you can do 80 or whatever your current percentage is.

32:12 But anyway, if you want that, something like CodeCov, you can get that with coverage.py, but there's some tricks around it.

32:19 So if you've got, one of the things coverage does is let's say you've got a project that has some code that is run, like you're trying to take advantage of some of the newer things, but you also, newer Python things, but you want to run on older Python too so there's some if statements for Python version, for example, but it might be a different hardware as well, like Windows versus Mac.

32:41 but there might be code that's run in one test suite and not on the other.

32:45 So in order to do that, you kind of have to run in parallel mode and then combine the coverage reports at the end.

32:53 And it sounds complicated, but mostly we just copy somebody else's work.

32:58 So, and Hinnick has the examples here.

33:01 So Hinnick goes through using GitHub and coverage to do everything that you used to be able, you were doing with CodeGov.

33:10 So this is a nice article with examples.

33:14 So we've got, for each project, we're downloading the coverage report and then combining it at the end with a report with a fail under 100.

33:24 Really cool.

33:25 If this is something you want to do, it's basically, it's a short article with examples.

33:29 Even shows, he's got in his struct log, he shows his main workflow for GitHub pipelines that does all of this.

33:38 So just kind of read his code and make it work.

33:41 And I was actually thinking of doing this on a couple of projects I've got.

33:46 I wanted to make sure that they were up at 100% and do something like this.

33:50 But I actually, for some reason, I was dragging my feet and I didn't know why, but now I don't need to.

33:55 I'll go ahead and use Hennig's suggestion.

33:58 - No, you don't have to invent it.

33:59 Just set up that job on GitHub.

34:02 - This is actually something I've been kind of struggling with, I shouldn't say struggling with.

34:07 I should say this is a journey that I have been on in that testing and code coverage and that whole class of proof of correctness is my weakest suit.

34:18 And one of the things that I've really been sort of trying to wrap my head around is I see a lot of like unit test code that seems like it's just there to keep the coverage percentage up.

34:33 And as far as I can tell, it doesn't actually test anything.

34:36 Like we're, you know, creating these mocks and like in everything and, but they don't actually like enforce any contracts or prove anything out.

34:47 And I, I, I've trying to figure out how we can use these tools in ways that actually sort of raise the bar on code quality without encouraging this kind of like, I don't know, T-crossing and I-dotting behavior.

35:02 >> Well, we're getting on a tangent, but I would say that code reviews on tests are important.

35:07 >> Yeah.

35:07 >> Because that's just, I mean, one of the easiest ways to get 100 percent coverage with no tip failing test is don't put any asserts in your tests.

35:16 Just saying.

35:20 >> Good. That makes an excellent point unto itself.

35:24 Yes, thank you.

35:24 >> Yes.

35:25 >> Nice. All right.

35:26 Well, let's talk about the tippy top, the tip top.

35:30 So I'm sure you all are familiar with top.

35:32 It's like this somewhat dreadful way to tell what's happening on a Unix system.

35:37 On Linux to see like what's happening, is it busy, is it not busy?

35:41 I say dreadful 'cause it's so simple and there's better tools.

35:44 One of them being this tip top thing.

35:46 So for a long time I used glances.

35:49 I didn't know, I still do use glances.

35:50 I'm not sure if you all are familiar with glances, but the UI that it creates is fantastic.

35:56 So you just type glances and you get this sort of progress bars that show you the CPU, the memory, all the processes.

36:03 It has like hotkeys you can sort by CPU or memory, filter by process name or ID, all kinds of stuff going on here.

36:10 And this is what I use now, but Zack Viller sent in TipTop as a competing thing.

36:16 And by the way, Glances is Python and TipTop is Python.

36:19 So very good candidates for pipx because no project needs them.

36:23 You just want to run them as commands, but they're Python, right?

36:26 So if you look at the same picture over on the tip top page, you can see that you get similar information, but you get a little bit better historical understanding of how your CPU is doing, how your memory is doing.

36:40 It even breaks it down per thread or per core.

36:44 It shows you your memory broken down by what's being used, what's being cached, what's free.

36:49 There's a graph of that over time.

36:51 Something that made me sad is it shows the network over time as well, the in and out flow, But what happened to me is one of my servers, I ran this and it only would show the output from like some kind of, there were multiple network adapters and would only show the one that's not being used, not the one that's being used.

37:08 So I'm like, well, that part is useless, unfortunately here.

37:11 And then it shows all the programs, like the CPU usage, the user it's running as, the arguments that were passed to it, how much memory it has and stuff.

37:20 So I don't think I'm gonna give up using Glances 'cause I really love that.

37:24 But this feels like another tool to get a view of that sort into it.

37:28 What do you all think?

37:29 I like the visual design on the face of it.

37:31 It, it like, especially comparing it to Lance's screenshot that you showed, it feels like it's a little prettier and it may, to me, at least it looks a little bit less like dense and maybe a little bit easier to sort of like, maybe it's just the bounding boxes, but I also feel like it's the way the screen is laid out.

37:48 A way to visually, an easier way to visually parse the information it's presenting, that's always something I look for a new tool.

37:57 - Absolutely. Brian?

37:59 - Why do you need to, I don't use anything like this.

38:03 - All right. So for example, if you log in to, let's say you logged into a web server that was running a search job and it said, you're running a little bit low on memory as a, over the last few minutes, and you've got five different web apps set up running there web processes with like a bunch of their sub processes running and you wanted to answer the question. Okay, which one is it? Which one do I have to restart? Okay, things like that.

38:30 - It's easy to see where you might not encounter this or have a need for this when you're working in the world of things, right? Like the world of not servers, but embedded devices that might run Python or C or whatever. And you know, this is like managing infrastructure.

38:47 - So you don't use this on your desktop?

38:49 - No, I don't use this on my Mac, but I use this on all of my, not necessarily tip top, but glances on all of my Linux servers, including the Python bytes one.

38:58 So it's nice to go in there and see what's going on in the servers with all I have is SSH.

39:02 - I'm more of a high level person.

39:04 I'd rather pay somebody else to do this for me.

39:06 (laughing)

39:08 - Well, let's like--

39:10 - But it looks pretty.

39:12 - Let's see if I can pull up a picture of top.

39:16 - On behalf of my employer, Brian, thank you.

39:18 Thank you. Yes, we're happy to take me to do that for you. No problem.

39:21 Like this, this is the thing it's replacing by the way, is it's just a list of processes.

39:27 And it moves around all the time and the stuff shoots off the screen. It's like, why is it doing this?

39:32 You know, it's I mean, so here was also written probably in like 1968 or something. Yeah, yeah, absolutely. Yeah.

39:39 I'm not knocking on it. I'm just saying I would rather use, you know, something like this.

39:45 that gives me way more real time, like graphs, not just snapshot, now it is, now it is, now it is, but like here's what's happening as the things are running and doing their work, right?

39:57 So that's super helpful to me to see.

39:59 - To be fair, I also just installed TipTop just for the heck of it on my Mac, and a really quick install, and it does look pretty.

40:06 I just don't know what to do with it, but.

40:08 - Yeah, and by the way, let's just tie it back to the audience again.

40:11 So TipTop uses textual for layout, which is based on, which is Will McGugan's thing, also based on Rich.

40:18 So it's quite modern.

40:20 - I think if we had Ned and Will in the audience, we would always touch on something they were doing.

40:26 (laughing)

40:27 - I'll set you up, Brian.

40:28 Here's a way you could use this totally in your day-to-day life.

40:30 Your Mac is running slow, right?

40:33 And you say, why is my Mac running slow?

40:35 You could totally use this and at a glance say, what's hoovering on my RAM?

40:40 What is throttling my disk?

40:42 what is using up all my CP, et cetera, et cetera.

40:45 - Okay.

40:46 - Yeah, like for example, you can run activity monitor, which will show you the process thing.

40:52 And you go to the memory thing, which will show you just a single graph.

40:54 But like, why is it using all that memory?

40:56 Is that actually being used or is that cache?

40:58 Because you loaded a bunch of files and now it's kind of could be free, but it's not.

41:01 What's happening on the network, you can't see very well on macOS.

41:05 I don't know why activity monitors are adverse to showing you network traffic.

41:09 but you could see like, am I downloading something or not right now?

41:12 - I can be taught new things.

41:14 Okay, thanks.

41:15 - Yeah, yeah, yeah.

41:17 Anyway, I'm not excited for that use case.

41:19 I'm excited for using it on the servers to understand them better.

41:21 - Yeah.

41:22 - Yeah, all right.

41:23 Let's bring it home with some more retro, Chris.

41:27 - Back in the world of Wibzy, I've always been into, well, you know, always ever since they weren't retro and they were just 8-bit machines, I have, I grew up, started with, with eight bit computing.

41:42 I was an Atari kid, never owned a C64, but someone has actually written a Commodore 64 emulator called Pis C64 in pure Python.

41:53 And the reason that I picked this, it really is, isn't it?

41:58 I think it is so cool because not only do you get this Commodore 64 emulator, It's not 100% coverage.

42:06 Like, he has openly emulated the display to the point where you can, display pets characters.

42:14 But the bottom line is, is other than that, a largely full machine emulator runs on the high 65 library, which is a six five Oh two.

42:24 A CPU chip emulator also written in pure Python.

42:28 It really is.

42:30 And, and it's so neat.

42:31 I was able to pull this thing down and run it.

42:34 And I was actually writing like Commodore 64 basic that votes bits.

42:39 It's a screen memory and, you know, drew pictures off the screen.

42:42 And, and one of the really cool things is this thing even concludes, you can sort of like pop out of your emulated Commodore 64 with the go by command to get a Python rapport is actually doing that.

42:56 Like modifying the emulated Commodore 64 is memory by tweaking the Python data infrastructures inside the REPL.

43:05 It is just, it is.

43:07 That's pretty deep.

43:08 Thank you.

43:09 It is, it is really cool.

43:10 And, and it is also from my perspective, one of the reasons I picked it, it's a testament to this programming language that we all use and love for our work.

43:20 And, you know, whether it's webby stuff or testing devices or in my case, automating network infrastructure, it can do so much.

43:27 You can reach out into all of these other really kind of, you know, full niche areas, whatever you're interested in, and do neat things with Python on your computer.

43:39 So I just thought this was incredibly cool.

43:41 I had enjoyed playing with it and I definitely intend to enjoy playing with it some more.

43:46 It might be some interesting ways to go back and study operating systems back when they were simple if you were like a computer science student as well.

43:54 Yeah, for sure.

43:55 I think there's a lot to learn there.

43:57 You know, even if you're not necessarily someone who is keenly interested in writing your own operating system, I think there's a lot to learn just in terms of the programming paradigm, right?

44:09 Like when you wanted to solve a problem and you wanted to display something on the screen and a perfectly reasonable answer was, "I'm going to poke the right bits into screen memory because that's a more efficient way to display this thing as opposed to using whatever my my high level language provides it is, it's such a different way of thinking the reality that most of us have it day to day.

44:34 It's just a fun, refresh, refreshing place to visit.

44:37 At least from my perspective.

44:38 Yeah.

44:39 Very cool.

44:40 It's a good one.

44:40 Thank you.

44:41 Brian.

44:41 It's time for extra, extra, extra here all about it.

44:43 How many extras do you have?

44:45 I have zero today.

44:47 You have no extras.

44:49 You're not adding to the number of extras.

44:50 Chris, anything else you want to throw out there that people know about?

44:53 Sure.

44:54 Absolutely.

44:54 My team at Amazon is hiring.

44:57 Hopefully we can put a link in the show notes.

44:59 I work for OpsTech IT.

45:01 We work with the fulfillment centers.

45:04 And I cannot say enough how much I love working with this team.

45:09 People are incredible.

45:10 And they're sort of like, at least the folks I'm working directly with are people who started out in the fulfillment centers and were interested in technology, were bright, and they were able to prove that they could actually walk the walk and now they're being promoted into full-on engineering positions.

45:28 And I just find that so incredibly refreshing because--

45:32 - Yeah, that's fantastic.

45:33 - Yeah, these people just have a very different perspective.

45:36 You know, they're not the jaded, like I've been in the tech industry for 20 years, types like me.

45:40 It's really inspiring to work with them.

45:43 - Yeah, not too many debates of like, why don't we just switch to a functional language?

45:46 Come on.

45:47 (both laughing)

45:49 - Exactly. - More practical and we'll just get stuff done.

45:51 I love it.

45:52 - Exactly.

45:52 Before we move, yeah.

45:54 Yeah, so we'll put that link in the show notes.

45:55 Before we move on to my extras, which are not many, Sam says, "It's so helpful to understand "how a computer works when you're writing code.

46:02 "These kinds of things are a good way to learn that." Talking about the Commvaar64 emulator.

46:07 Awesome, all right.

46:08 So I just have one, and that's, I'm going to be speaking at FlaskCon 2021.

46:15 And so FlaskCon is the first week of December, which has remote attendance.

46:20 I'm gonna be talking about HTMX and Flask once again, because I'm super psyched about all that stuff.

46:25 And so if you wanna see some really cool apps get built using HTMX and Flask, drop in over there.

46:31 There's a whole bunch of other talks going on as well.

46:33 So I'll put the link to that in the show notes.

46:35 All right, shall we finish it off with a joke?

46:38 - Yeah.

46:39 - Levity is the soul of wit, absolutely.

46:41 - It is.

46:43 So we just recently went through the whole daylight savings unraveling and went back to normal time.

46:50 And I don't know what it's like for you all, but when I look at the weather and stuff here, I suspect Chris, it's similar in Boston.

46:59 I mean, I know it's similar for you, Brian.

47:01 - Yes.

47:02 - It's like sunset is at 4.38 PM and we're headed towards the winter solstice.

47:06 So it's only getting earlier in the day.

47:08 And it's like, ah, that doesn't make me, it doesn't spark joy in my heart that it's already dark at 4.30.

47:15 - I like it.

47:16 - Yeah.

47:17 I don't mind that it gets dark.

47:20 I just wish it wouldn't.

47:21 I wish we could just move it.

47:22 So there's a little time the end of the day where there's still sun.

47:25 I remember.

47:26 I'm with you, Michael.

47:27 I'm so low powered.

47:28 So I enjoy the sunlight and I don't like it getting darker earlier either.

47:31 Yeah, absolutely.

47:33 I don't hate the winter.

47:34 I just don't wish it shifted around a little bit.

47:36 So this is a technical take on that.

47:39 Our joke is this comes from the New Yorker.

47:41 I mean, we're pretty highfalutin, right?

47:43 We're not doing like dev jokes dot IO or whatever it was anymore.

47:47 (laughing)

47:48 - So this is by Matt Reuter, and it's a little cartoon of two people, clearly in kind of a darker space.

47:55 A woman sitting in her chair with a laptop, and you could just, like the beaming laptop screen is like lighting her up, right?

48:01 And the guy here on his tablet, same thing, glowing on his face.

48:05 And underneath it, it says, "I hate how the screens get bright so early this time of year." (laughing)

48:11 - Nice.

48:12 - This is so much about our modern world.

48:14 It's both hilarious and kind of sad.

48:17 It's all it would go.

48:18 (laughing)

48:19 - Absolutely.

48:20 Yeah, so welcome to Daylight Savings or Standard Time, I guess, everyone.

48:27 The screens do get bright this time of year.

48:30 - That's funny.

48:31 - Nice.

48:31 - And Chris, thanks for being on the show.

48:34 Brian, thanks as always.

48:35 - Yeah, thanks a lot.

48:36 - Thank you so much.

48:37 You know, I really appreciate everything you guys do for the community.

48:40 It's really, you guys are great examples of why Python is such an amazing place to live.

48:46 Thank you.

48:47 - Thanks, Chris.