Transcript #269: Get Rich and replace your cat

00:00 Hello and welcome to Python bites where we deliver Python news and headlines directly to your earbuds. This is episode 269 recorded February 2nd 2022 and I'm Brian Okken And I'm Luciana and I'm Brett Cannon and we have Brett. We so we invited Luciana on and we decided to have Brett on also so we got have some fun Michael's actually, gone this week, but hope he enjoys listening. So hi, Michael. And, and we'll talk to him next week.

00:33 So before we get into the topics, though, I think people hopefully know who Brett is.

00:39 Brett Cannon, you've been on the show before.

00:42 A couple times.

00:44 Yeah.

00:44 And then...

00:44 Thank you very much for having me back spontaneously.

00:47 Spontaneously. Like minutes ago, we said, Hey, we should get Brett on. So Luciana, Welcome to the show. Can you tell people a little bit about who you are?

00:56 Thank you. Yes, I am. Well, let's see. I work with Brad on a Python extension for Visual Studio Code, but I'm a program manager. And yeah, I'm very excited to be here in the show. I was telling Brian that I used to listen to Python Bytes going to work back when we worked on offices.

01:20 So it's really, really cool to be here.

01:22 - Yeah, nice.

01:24 Well, thanks.

01:26 I'm glad to have you here.

01:27 So let's jump into the topics.

01:29 We have an abbreviated list of topics, but we'll make it fun anyway.

01:35 I'm gonna start us off with Rich CLI.

01:38 So Will McGugan, it seems like we talk about him every show, but this is cool.

01:48 So I was like, Oh, I'll try it out first.

01:50 And I'm already hooked.

01:52 So RichCLI is, I don't know, it seems lame if I describe it.

01:57 So you install it.

01:59 I installed it with PipX because I just want to use it everywhere.

02:03 And then what it does is it does like pretty printing of stuff, like you can give it a file and basically that's what I use it for.

02:12 Maybe it does other stuff, but you can give it a file like example.py or example.markdown or something.

02:18 And it will do the rich printing, pretty printing right on your screen in the terminal.

02:25 That seems like not that big of a deal, right?

02:28 But it is so cool 'cause it's all like nice colors and you can have themes.

02:32 And I used, what?

02:34 You can do pretty printing of JSON.

02:37 So it'll format your JSON for you without having to reformat it.

02:41 You can do a dash M flag for Markdown.

02:44 I've been doing that to see my Markdown and it does some cool stuff with Markdown.

02:49 Although supposedly we're like with Python files and C++ files and stuff, it's gonna do syntax highlighting based on extension.

02:57 I'm wondering why I gotta pass in a dash M for Markdown.

03:01 It seems like it's dash MD should work fine, but anyway.

03:05 - Could be my immediate question was why do you have to specify the dash M just for Markdown?

03:11 And then like, and then there's some flags that's cool.

03:14 Like, like whether, whether or not you want to use it with pipes so you can, for if it doesn't, if it's piped to something, it will, it'll strip out the formatting stuff like the color codes and whatever, but you can force that to be on.

03:28 Well, I wanted to, I tried to use it with a big file and it just flew past me.

03:34 And, and then, so I tried to pipe it to more or less and it, it stripped out the formatting.

03:40 And so, well, I tried to fix that and then more or less don't handle that.

03:45 And so, and this was just yesterday.

03:47 I'm like, kind of really be cool if RichCLI has supported paging within it.

03:53 And so today, Will said, "Hey, I added paging to RichCLI." So RichCLI now has paging, so.

04:00 - Of course he did.

04:01 (laughing)

04:03 - Like within a day.

04:04 So anyway, pretty cool projects.

04:07 Have either of you checked this out yet?

04:10 - I did the exact same thing you did.

04:12 I used PIPX and sold Rich yesterday, Rich CLI yesterday.

04:15 One thing is I will say I use a, I've actually been using a replacement.

04:22 So this is kind of like a replacement for the cat tool on Unix, right?

04:26 And I've been using an alternative called bat, B-A-T, that's implemented in Rust and has a lot of this as well in terms of color syntax and stuff.

04:37 But I'm willing to bet Rich is, which I'm willing to bet, Will's gonna move a bit faster.

04:44 So I'm just gonna probably switch over to using Rich CLI over bat and just see where this goes.

04:49 But I mean, come on, Will even made the output look great.

04:53 So.

04:54 - Yeah, so what-

04:55 - Sorry for the output for the help, it looks phenomenal.

04:58 - One of the things that I noticed was that the default theme, did the coloring was, didn't look quite right on the black background that I was using.

05:09 So I just, I exactly, I was using it for cat.

05:14 So I just aliased my cat to use rich with the Monokai format turned on.

05:21 So Luciana, would you use this for anything?

05:25 - Yeah, I was gonna say I'm behind, I haven't tried it yet.

05:28 But once I saw the amazing things they have there, it's quite a few.

05:33 I was like excited to try it out.

05:35 So we'll do it today, hopefully.

05:38 - Yeah, cool.

05:39 Well, let's switch over to debugging.

05:45 - Yeah, so I guess most folks, I don't know, at least I, when I started learning Python, I used to debug using print statements, which is like, no shame in that, it works very well.

05:55 I still do that when it's like for real quick things, but I wanted to bring up this awesome tool that we have, DebugPy.

06:04 It's a debugger for Python, and you can use it like as a CLI tool.

06:08 You can also just import it and set breakpoints normally.

06:10 I am a big fan of UI debugging, so I actually prefer to use it built into the Python extension for VS Code, which is the thing that we work on every single day.

06:21 So, yeah, I wanted to bring up a few features.

06:25 They're really, really cool, and I find very helpful, and it helps me save up a lot of time when I'm debugging code.

06:33 So the first one of them is kind of like conditional breakpoints.

06:37 Yeah, sorry, go ahead.

06:40 - No, no, go ahead, neat.

06:42 - Yeah, it's super neat.

06:43 And this conditional breakpoints is something that I bring up because I guess when you're trying out debugging for the first time using the UI, you set up a breakpoint and then you hit it there and then you always reload the window or reload the debugger, sorry, to hit the breakpoint again.

06:59 And then you keep stepping over until it hits a condition that you want.

07:02 And in this GIF there, they have an example in TypeScript, but it works on Python files as well once you install the Python extension.

07:08 But basically, you added the breakpoint to insert a certain condition so that the debugger will only stop when that expression is met or the condition is met.

07:18 So one example there is hitCount.

07:20 So you can imagine if you have a for loop, and you really want to stop when, for example, after 10 iterations of that loop.

07:28 So you don't want to stop on that breakpoint and hit the step over 10 times.

07:32 You just want it to stop when it hits 10 times.

07:36 So you can do that with the hit point.

07:38 You can also add expressions.

07:40 So if you know, like, oh, I want the debugger to stop only when this variable gets the value of, I don't know, 5, then you can also do that.

07:51 And it's pretty cool.

07:52 Saves up a lot of time.

07:54 And you can also do log points instead.

07:56 If you're not interested in breaking the execution, just want to make sure that line was executed.

08:01 So it prints out on the output that, hey, this is, we hit that point.

08:06 This is like a log point, so it doesn't really break anything, but it's very informative.

08:12 - Cool.

08:14 So this is an extension for VS Code?

08:19 - Yes, I believe most IDEs support that same feature, but it's a feature that I always recommend folks to look for in their IDEs or editors, 'cause it does save us a lot of time.

08:31 - So to be clear about the tech stack on this, so there's a couple of layers to this.

08:36 So at the bottom layer is PyDevD by Fabio Sdrazny.

08:41 We have him on contract as a consultant to do anything we need to have get fixed and stuff and just support the project.

08:50 Because on top of this, we have another project that we have called DebugPy, which implements what's called the Debug Adapter Protocol or DAP.

08:59 It's a meant to be a language agnostic protocol for debuggers to talk to editors comes out of VS Code.

09:06 It's used by Visual Studio as well, and actually it's now used by Jupyter.

09:10 This is how Jupyter was able to launch recently with debugger support.

09:14 'cause actually run debug pie.

09:17 And on top of that we have support because we're VS Code extension.

09:21 We support the debug adapter, so we ship debug pie in box right now for supporting this.

09:27 Although we are talking about breaking the extension, breaking debugging out to its own extension so that we can actually update it at a better cadence with debug pi 'cause we can do platform specific V6s now, extensions for VS Code.

09:40 So we'll shrink that down a bit for our users.

09:44 - So is there like a tutorial somewhere that has like all of the cool Python extensions that I need for VS Code to get VS Code with Python to work really great?

09:56 Yeah, we have our get started tutorial.

09:59 It has like it doesn't have a lot of extensions, so there's definitely more extensions that I find to be personally cool.

10:05 Maybe we could have a blog post about this and like cool extensions for Python specifically in VS Code 'cause there are some of them.

10:11 I think Don Giovanni from, he's the one who created the Python extension before it was Microsoft thing and Microsoft hiring him and everything.

10:20 So he has a one that I always talk about, which is the Python environments manager extension.

10:26 It just displays the environments that it can find on your machine and what packages are installed there.

10:31 It's kind of like a environment viewer and it's pretty cool.

10:34 It's one that I always tell people about.

10:37 So.

10:38 - You know, a deep dive into all the useful extensions to add would be a great testing code episode.

10:43 Just saying.

10:44 - I love that idea.

10:46 - I like it.

10:48 Okay.

10:49 Well, speaking of plugging stuff, I want to take a moment to thank our sponsor.

10:56 So this episode is sponsored by Datadog.

11:00 Datadog is a real-time monitoring platform that unifies metrics, traces, and logs into one tightly integrated platform.

11:07 Datadog APM empowers developer teams to identify anomalies, resolve issues, and improve application performance.

11:14 Begin collecting stack traces, visualizing them as flame graphs, and organizing them into profile types such as CPU, IO and more.

11:23 Plus Datadog APMs live search.

11:27 So you can perform searches across full streams of ingested traces generated by your application over the last 15 minutes.

11:35 Try Datadog APM free for a 14 day free trial and Datadog will send you a free T-shirt.

11:43 And actually I have one of these purple T-shirts and I love it.

11:46 So visit pythonbytes.fm/datadog or just click on the link in your podcast player show notes to get started.

11:54 Awesome.

11:55 I gotta pay the bills.

11:59 So where are we at now?

12:02 We're back on me again.

12:04 Unless Brett, you got something right out of the blue you wanna talk about?

12:09 Put you on the spot.

12:13 - That's okay.

12:14 I believe you've talked about exception groups on the podcast already.

12:19 I don't, oh yeah, I think so.

12:21 - Yeah, so just 'cause DataDog, the sponsor, obviously cares about tracebacks and stuff.

12:25 One interesting thing that's coming up in terms of a PEP, it has not been given to the steering council yet, but there's actually a PEP coming up that's gonna add a Dunder Notes field, potentially, to exceptions.

12:40 This actually came out of the exception group work because Zach from Hypothesis was working with it over on that side and realized that it would be convenient to be able to kind of just basically literally, the name says, add a note to extensions to kind of help explain why this came about.

12:58 And so there's a PEP coming that's going to basically just allow libraries and stuff that work with exceptions to just attach a little note saying, this is why this happened, more than just a little thing, especially for exception groups, you can kind of get kind of big and murky by having a single note added to it to help explain why this came about.

13:19 It's hopefully gonna help make it easier for users to understand a bit more why something happened.

13:25 So.

13:26 - That's pretty neat.

13:27 I like it.

13:28 Nice.

13:29 I got some feedback recently that I'm not covering enough test stuff on Python bytes.

13:37 So I'm gonna try to, I'm gonna try to solve that.

13:42 I'm not gonna cover test stuff every week, but I've got a test thing this week that I'm pretty excited about.

13:48 So Simon Willison, great guy, he wrote an article called "Documenting Unit Tests." And this, wow, it's old.

14:00 It was came out in 2018.

14:01 I just noticed it.

14:02 So yeah, there's the internet for you.

14:04 But it's like, it's a super cool idea.

14:08 So the idea is we've got tests like pytest and other test runners, you can do anything.

14:15 he's using pytest to test your source code, but why not?

14:19 And Python has all these introspection things you can do.

14:22 So he's using introspection to look at the code and then introspection to look at the documents with his test code, and then make sure that there's some items that are definitely in the docs.

14:37 As an example, so in this, his example in the blog post is mentoring data, his project dataset.

14:47 And so the test for this is right there.

14:49 So you can go take a look at what he's doing, but he takes a look at the repo.

14:54 And for instance, for all, if he adds a config option, he's got to test to make sure that the config option gets documented.

15:00 That's cool.

15:02 So it's just tests or fail.

15:03 If he forgets to document a new option, also plugin hooks and different views within dataset, they all have to make sure that he has special, The views, like he wants special pages for each view.

15:16 So it makes sure that there's a page there.

15:18 So I just, there's a couple neat things here.

15:21 Just the concept of using tests to make sure that there's things documented.

15:26 It doesn't tell, he makes a comment that it's not telling, like a lot of tests, it's not telling you that the documents are good or documented well, but that they're documented at all, which is good.

15:38 Same with our code.

15:40 Tests don't necessarily tell you that your code is perfect, it's just that they're tested.

15:46 The other thing that I wanted to point out about this is there's some nice use of features of pytest.

15:51 So there's a nice use of fixtures and parameterization.

15:54 So he's got like the views, for instance, he goes out and looks at generate, like goes and gets a list of views from the code and then use that list to have a generated test to make sure that each of those views is tested or is documented.

16:11 This is a nice idea.

16:12 I've been trying to do that sort of generated parameter lists.

16:17 I'm trying to do that more in my code because it's just kind of a cool idea to generate your own test cases.

16:24 So anyway, good job, Simon.

16:26 - Yeah, as someone who spends more time reading code and writing code, I definitely appreciate this initiative.

16:34 It's very much better to have documented code out there.

16:37 - Well, a combination of like making sure that the test, the documents are tested or things are covered.

16:45 And then also I think a few episodes ago, we talked about a project, projects using generating documents themselves.

16:57 So like for instance, instead, if you're gonna plop your help text in your document, don't copy and paste it, use code to generate that.

17:07 And then it's always gonna be correct if you got it in your build system.

17:10 So nice ideas.

17:12 - And it was about using cog, right?

17:16 - Yeah, that was it.

17:17 Cog, I forgot.

17:18 - Yeah.

17:20 - And what I really love about that is when I came across cog a long time ago, I thought I gotta have a use for this, but I can't think of it.

17:30 But this is the perfect use because we always often do that within our like little readme or something is just to plop a little example of code using your tool, whatever library you have.

17:43 So having cog generate that for you is great.

17:45 So I think I need to go back and do it.

17:48 So I've got a little toy project called cards that I use for teaching testing.

17:53 And I changed the interface for it for layout reasons.

17:58 The tables that I was using didn't show up in eBooks very well.

18:03 So I switched the table format, but then I forgot to change the readme.

18:08 But if I had had cog in place, I would already be doing it.

18:13 So it's a neat, my first thought was to write tests to make sure that the output in my readme was the same as the code.

18:21 But I think a better idea is to just have the readme generated.

18:24 So I mean, either one works, but both are good ideas.

18:27 - Yeah, and to tie this back to this point, Simon's the one that actually started the trend on Twitter of everyone using cog to use this on their readme.

18:37 At least that's how I first came across it.

18:38 And then I subsequently saw multiple people do it.

18:40 And then I finally had the same reaction you had, Ryan.

18:43 I was like, oh yeah, I've been looking for a reason to use cog for ages.

18:45 I never quite had one.

18:46 I was like, this is a great one.

18:47 So I went and updated the readme for the Python launcher to use this to generate all the URLs for downloading the built binaries for it.

18:55 So it's great.

18:56 - Gulf binaries?

18:59 - Built.

19:00 - Built binaries, okay.

19:02 - 'Cause I tried to build the launcher for like a bazillion different platforms.

19:06 - And I'm grateful for that 'cause I'm using the launcher like almost every day now.

19:13 - Oh, fantastic.

19:14 - We are, thank you, Brett.

19:16 - Well, so let's pop over to talking about ourselves.

19:23 - Oh, that was such a good, wow, I'm impressed.

19:28 Yes, pep673 self-type.

19:31 I heard from, well, Brett Canundar that it has been recently accepted.

19:37 And it's interesting for me, because I'm now learning more about types in Python.

19:41 I'm very new to it, admittedly.

19:44 And I thought it was interesting, because, well, it adds this away to another methods that return an instance of their class.

19:52 And they give a very cool example there of a class.

19:55 There's shape, and then we have circle, and here it's from shape.

19:59 And then shape has this method called setScale, And then the type that returns is shape, because it returns self.

20:08 And for circle, it returns self as well.

20:11 So it's a type circle.

20:12 And then it shows there are some, for example, editors that may throw in error when you call circle setScale, which is a method from shape, and then setRadius, which is something from circle.

20:27 But once you call them, it will say error, because shape doesn't have this method that is specific to circle.

20:37 So to have this better way to frame that would be to return self.

20:44 And it was accepted.

20:45 And it's very interesting.

20:47 I feel like it seems very clean to me and a very good solution.

20:51 So I'm personally excited that it was accepted.

20:54 It was very well explained as well.

20:55 And that was real.

20:58 For people on the podcast who can't see what Brian's scrolling through, basically, think of capital S self as the return type for all those class methods you write that construct an instance of your object.

21:13 This is what this is built for, right?

21:16 It's fantastic.

21:17 If you have the from whatever method, class method on your class to construct that instance of that class from something else, you'd now have a return self with a capital S versus specifying the exact class it would have returned.

21:30 So you don't have to worry about what that's going to look like and having to worry about the spelling or any of that forward declaration problems you can have if you don't do from Dunder Future import annotations, right?

21:43 Where it's like, oh, I'm referring to a name that doesn't exist yet and Python throws a fit because that object doesn't exist yet by that name.

21:50 So it solves a couple problems like that, but it's really basically if you write class methods that construct instances of that class, it's great.

21:58 - Well, I don't, but I probably will now when this comes out.

22:03 So is this part of 3.11, do we know?

22:07 - 3.11, right, yes. - 3.11.

22:09 - Nice.

22:10 - Yeah, so you can probably get that probably today, if not in the near future, from the typing extensions package, which kind of tries to do a back port and forward port of anything in the typing world that's experimental has been landed but not in a release yet.

22:26 - Okay.

22:27 - And I believe the type checkers, I don't know if they all support it yet, but I'm sure very shortly they will all support this.

22:32 So you can get to it today.

22:35 - And then we get, we keep people that are writing tools like Flake and Black on their toes, catching up with all these changes.

22:45 - Well, luckily it won't hit them 'cause it's all typing.

22:46 So just strings to Black and I don't think Flake but like mypy, pyright/pylance, pytype, pyre, all those type checkers are the ones that are gonna have to get updated for this.

23:02 But they were all in on the discussion of even proposing this.

23:05 - Nice.

23:07 But is that, will you repeat that?

23:10 So all these people writing these tools were involved with these sorts of type processes?

23:17 - Yeah, so the way typically decisions around typing happen, especially from a static typing perspective, I don't want to make statements on Pydantic's part in the dynamic typing world, 'cause there's kind of two parts to typing, right?

23:31 There's the general typing as in, oh, there's the concept of typing now in Python, but there's the people who do static typing, right?

23:37 And that's the kind of thing where you add your type annotations, your type hints, and you run a tool later that somehow reads them offline, as it were, like mypy or PyWrite or whatever.

23:48 And then there's the dynamic typing folks, that's like Pydantic where during runtime, they actually look at the annotations.

23:53 From the static typing perspective, there is something called the typing SIG.

23:58 It's a mailing list and all the major static type checkers all work there together to try to figure out where they wanna take things and kind of what gaps there are in terms of typing.

24:11 Basically, how can they move typing forward in Python?

24:14 It's also a place people can go to ask for help if they have a generic typing question it isn't too tool specific.

24:19 But to answer your direct question, Brian, the tools that are involved there, which are basically all the type checkers, static type checkers that I know of.

24:28 So there's PyWrite, which underlines PyLance, which is what we use for our completions in VS Code.

24:36 There's mypy, there is PyType that comes from Google, and then there's Py that comes from Meta.

24:43 - Okay.

24:46 - Wow, dizzying.

24:49 - Yeah, although, I mean, one of the nice things about all the peps, right, is this is all backed by specs.

24:55 So there's not a lot of crazy deviation, as it were, from the semantics.

24:59 It's more of a bug if someone disagrees.

25:02 So they're all pretty consistent, but some of them do a bit more like inferencing, like PyType does more like runtime inferencing and other inferencing to try to kind of fill in the gaps.

25:12 That's actually why Google wrote it, I think, to help them move over to Python 3, start introducing types to their code base.

25:18 - Yeah.

25:19 - So they all have slightly different strengths and weaknesses, but they should at least type wise more or less, once they've all implemented the same specs, lead to the same type outcomes as it were.

25:30 - Yeah, okay, cool.

25:31 - See, I learned so much about typing just by listening right now.

25:36 - Yeah, well, we're to the end of our normal topics.

25:43 - Yeah, so it was announced in my team that Black is no longer in beta version.

25:52 So version 2020.1.0 has come out.

25:56 So it's their first stable release, I believe.

25:59 So that's very exciting.

26:01 I really like the highlights that they have there that Python 2 is no longer supported.

26:08 (laughs)

26:09 - Yeah, I think that's a good thing.

26:10 Python 2 is no longer supported.

26:12 (laughing)

26:14 - Nice.

26:15 (laughing)

26:16 - So that's a cool announcement for the week.

26:20 And the other one that I have is to remind folks about PyCascades 2022.

26:26 It's happening this weekend.

26:28 It's remote.

26:29 Tickets are on sale.

26:31 So if you're interested in joining the event, it's really awesome.

26:34 It was my first Python conference and the community there is just really amazing.

26:39 So again, remote, join us this weekend.

26:43 - Pike Cascades was your first conference?

26:45 - It was.

26:46 It was back in 2019.

26:49 - Was that the Portland one?

26:53 - It was in Seattle.

26:55 - Seattle, okay.

26:56 Yeah, I guess it was the 2021, it was in Portland right before everything shut down.

27:02 - Yes, 2020, it was right before it, yeah.

27:05 - Yeah, 'cause that was the one I had to miss 'cause I was out of the country on vacation when it happened.

27:10 - Yeah.

27:11 - Not in China, I hope.

27:14 - No, I was in Thailand for a friend's wedding.

27:17 - Interesting, nice.

27:19 - That was back in the day when we kept track of what countries do have COVID cases, not how many cases in every country.

27:26 So it was a different time.

27:29 - Yeah.

27:30 - A while ago.

27:32 - So I only have like, I guess one, I don't have anything to show for this, but I did a couple extra things.

27:38 It wasn't gonna talk about it too much, but I did notice that pip is now at 22, version 22 dot something.

27:46 It's what, we're at 2202 now.

27:49 But one of the neat things about it is there's some cool stuff around having pip vendor.

27:57 There's a couple neat things.

27:59 There's a one of the, they're vendored some of rich.

28:02 So some of the, especially the error output.

28:06 If everything goes great, you probably won't notice much of a difference, except for maybe the progress bar if things are slow.

28:13 But the error report if something goes wrong, like if it's building something, that's where things apparently show up really neat, some better error outputs.

28:23 And better error outputs are always great.

28:26 Also, interesting that they replaced talks in their development workflow with NOX.

28:33 And I don't know if that's worth noting or not, but it's interesting nonetheless.

28:38 - Yeah, I know Pradyun, one of the core developers of pip is a big Knox fan.

28:43 I'm also a Knox fan.

28:44 So I think part of that's just kind of the difference.

28:48 For those of you who don't know the difference between TOX and Knox, other than the letter, Knox is a bit more of a, I would say, general task runner.

28:58 So because you can write just straight, it's writing just straight Python code.

29:02 So I think for projects that like to use it for like release process stuff, like on the packaging project, Pragen wrote a Knox file that has like literally a release command that will do everything you need from within Knox.

29:16 So you don't have to have a separate shell script or something else.

29:20 So it can all be in the same file that you also specify how to run your linter and how to run your tests and stuff.

29:24 So.

29:25 - That's pretty neat.

29:25 One of the things that I've, and I don't remember if there's a way to get around it or not, TOX operates in a virtual environment.

29:34 So it creates, so one of the things, the default behavior at least, creates a virtual environment and then runs, runs whatever you're gonna have it do within that virtual environment.

29:46 So like the idea being, you're gonna have multiple Python versions.

29:50 So it creates a virtual environment for each Python version that you're running.

29:54 There's some stuff like this extra stuff, like I just want to publish now.

29:58 I'm already like, I'm already in a virtual environment 'cause I'm always in a virtual environment.

30:03 I don't really need a Tox to generate another one just to run a command.

30:08 Is that sort, do you know if that sort of thing is easier in Knox or does Knox also create a virtual environment before it does stuff?

30:15 - So Knox can create a virtual environment.

30:20 I can't remember if it does automatically for every command.

30:26 - Putting you on the spot.

30:27 But it's something I'll look up there.

30:29 Ask Thea, she can tell you.

30:31 - Yeah, yeah, definitely.

30:34 So, go ahead.

30:35 - Sorry, I was gonna say that, speaking of pip being updated, I was trying to use it with pip Tools today, and it seems that pip Tools is not working with the latest version of PIP, just FYI.

30:46 There's an open issue, I'm sure the team's investigating, but for all pip Tools users, they're not alone.

30:51 - Yeah, so the problem, yeah, so I can, yeah.

30:57 So what happened was is pip-tools is using the private API of pip and it changed.

31:03 And so pip-tools broke.

31:04 So pip-tools, if you use it, you will need to pin to a version of pip 2021 or older to work with pip-tools until they figure out what they need to do to use the changes that pip made to their internal API.

31:23 - And to be fair to the pip-tools team, - pip 22 just came out like a handful of days ago.

31:29 So it's really new.

31:31 - Yeah.

31:32 - I'm a huge fan of pip-tools, definitely love it.

31:35 - Well, one other thing actually.

31:40 - Yeah, just one other thing on the new pip release for people to be aware of.

31:44 So one thing that happened here is to try to cut down on the vendoring, pip stopped vendoring HTML5 lib and now use the HTML parser in the standard library.

31:58 And that actually broke a bunch of simple repo API indexes.

32:06 For those of you who don't know the simple repo API, if you go to any URL in PyPI for a project and replace the project part of the URL with simple, you'll see what I'm talking about.

32:16 But basically there's a spec that just says, There's a PEP that says, hey, write this HTML and then packaging installers can read it and figure out where to find all the files and the projects for something.

32:30 It turns out a lot of people weren't following the spec.

32:34 And so a lot of these indexes broke because pip wasn't able to read them anymore because it couldn't be parsed as pure HTML.

32:46 They were doing funky things that broke.

32:48 So they, I believe, either have or are planning to roll it back with a flag to let you alternatively use HTML5 Lib for now with the goal of getting all these projects that have broken output, they don't follow the spec, to fix themselves so that they can go back to not having HTML5 Lib in the box.

33:08 So just a heads up, if you use an alternative package index and it suddenly broke, that's why they're working on a fix.

33:16 But I would talk to the project that creates that index for you and have them work on making sure that that project creates valid HTML5 that can be parsed by the HTML parser in the standard library.

33:29 So that--

33:29 >> OK.

33:30 >> --because that'll probably go away in pip 23.

33:33 >> But it's not broken now, right?

33:37 It's just-- it's flagged?

33:39 >> It was broken.

33:40 They didn't realize how much of the world was doing invalid HTML for these indexes.

33:44 So I don't know if it's in 22.0.2 or if it's going to be in the next release.

33:49 But there is an open issue where they're--

33:52 I at least was looking at where they were fixing it.

33:54 Because I got looped in because I have one of my side projects, Mouse Bender, that actually wrote some code around this originally.

34:02 There's talk about moving up to packaging, about a lot of behind-the-scenes stuff that people don't need to care about.

34:08 But the--

34:09 [INTERPOSING VOICES]

34:10 Yeah.

34:10 >>No, go ahead.

34:11 >>Well, I was just going to say, so the plan is, I think, to add a flag to say use old HTML parser or something along those lines for now.

34:18 So you can say, hey, use the HTML5 lib version of parsing HTML, but it's just to be clear here, there will be a fix, but it's a temporary fix 'cause people should be getting after their projects that create these indexes to do the right thing and actually follow the standard.

34:34 - Okay, so as someone that got bit by this, because we have an internal PyPI caching server that also we store our own stuff on and we don't own it.

34:46 We like, it's a third party thing that we're licensed.

34:49 So I don't know how long it's going to be.

34:52 So hopefully that, hopefully they move quickly, but at the same time, hopefully this extra, use the old one flag or whatever will work for a while.

35:01 I was a little surprised when I saw the warning pop up of like, oh, you're not using a good, you know, it's invalid HTML.

35:08 And I'm like, what am I supposed to do about it?

35:10 (laughing)

35:12 - Well, and I don't think the developers were expecting it to be so widely broken either, right?

35:18 It's like, what are these projects doing that are creating invalid HTML?

35:23 But apparently--

35:23 - Oh, I did it all the time though.

35:25 Like, okay, just to be fair, if it works, it works.

35:29 It's HTML.

35:30 - Yeah, well, and that was the problem, right?

35:31 Is the spec, the PEP that defines it very specifically says it must be valid HTML5.

35:38 And I think it was specifically so that it could be parsed by the standard library.

35:41 So it also means none of these pages could have even been parsed by the standard library.

35:45 So I think that's what happened, right?

35:46 Is people, because people, because pip had been using HTML5 lib, which is obviously way more lax in terms of what it can parse, everyone just like, "Oh, okay, it's good enough." And didn't realize that the spec specifically mandated valid HTML5.

36:00 And so hopefully all these projects that got caught up can get a fix out over the next year.

36:05 'Cause as I said, I suspect this will be in until pip 23, which will come out in 2023.

36:11 So it's not like it's gonna disappear in a month, this flag.

36:14 So we'll let the fix out, but hopefully everyone can kind of get caught up so that they can get back to not vendoring HTML5.

36:21 - Okay.

36:23 Before we go jump to a joke, I want one last extra is for the first time in like 18, 20 months, I'm not writing a book.

36:33 (laughing)

36:35 I'm no longer writing this book.

36:38 The writing is done.

36:39 The editing is done.

36:41 So it's off to all the rest of the cogs that have to go through the process to get it.

36:46 But the ebook right now is really solid.

36:48 So anybody that grabs the ebook now, if you've grabbed it before, all the updates are there, but this has got all the copy edit changes and the final edits, the rest of its layout and stuff to try to get the physical book for him.

37:00 So that's coming out sometime this month, maybe, hopefully.

37:04 So that's my hope is by the end of February, I'll have a physical copy in my hands.

37:10 - Well, congratulations.

37:11 Everyone who listens to this podcast and Testing Code and anywhere else for you, know you put a lot of blood, sweat, tears and love into that book.

37:20 So congratulations for reaching this point.

37:22 - Well, I just was realizing that my youngest is 12.

37:24 My youngest daughter is 12.

37:26 I spent a year and a half on this book, a year and a half on the first edition.

37:29 So I've been writing about Pytest in book form for a quarter of her life.

37:35 It's pretty weird.

37:36 Are you ready for a joke?

37:39 - I should thank you for your work, Brian.

37:40 (laughing)

37:42 I'm gonna be ready to physical hobby, hopefully.

37:44 - So one of the goals of the second edition was to try to put it in a format that would be easier to teach and learn.

37:51 So the next step is I'm gonna work on trying to teach people in person and through video courses and stuff like that, turn this into, 'cause I know a lot of people don't read, but you're coders, why don't you read?

38:04 You have to read your code, but still, some people don't read books.

38:08 How about something funny?

38:10 - Yes.

38:11 - Today's joke brought to you by Luciana.

38:13 Thank you for finding something funny.

38:15 And we'll add this to the thing.

38:19 So I love this.

38:21 So for people listening, there's a picture of an ocean and you've got a really big ship.

38:28 It's called, I don't know what's on the ship.

38:30 It's a big ship.

38:32 - I don't know what you're talking about.

38:33 I don't know, but it says, but it looks all official like and everything.

38:38 And it says proper software design.

38:40 And then there's a guy on a surfboard standing up.

38:44 So it's like a wakeboard thing.

38:46 And he's holding an umbrella, getting pulled by an umbrella on a surfboard and says, whatever I'm doing.

38:51 So it's funny.

38:54 - That's how I feel whenever I'm coding.

38:57 - And sometimes it feels like this going from, from the old days of like these huge design documents to now, oh, we're doing agile.

39:09 So let's just write some ideas on some Post-it notes and stick them on board and run from it with there.

39:14 - If it works, it works, right?

39:17 - Yeah, it's got tests, right?

39:20 Well, I think so.

39:23 It's unit tested, it's 100% coverage.

39:25 It's good, let's ship it.

39:26 So perfect.

39:30 Nice.

39:30 First of all, thank you, Luciana, for joining the show.

39:34 I think it's your first time, right?

39:36 - Yes, it was my first podcast.

39:39 - Awesome. - Participation.

39:40 Thank you for having me, and thanks, Brett, for joining last minute.

39:44 (both laughing)

39:45 - Yeah, thanks, Brett.

39:46 It's always good to see you.

39:47 - I'm happy to play the fake Michael for this episode.

39:50 (both laughing)

39:51 - Thanks, everybody, on the stream for showing up, and we'll see you all next week.

39:56 Thanks for listening to Python Bytes.

39:59 Follow the show on Twitter via @pythonbytes.

40:02 That's Python Bytes as in B-Y-T-E-S.

40:05 Get the full show notes over at pythonbytes.fm.

40:08 If you have a news item we should cover, just visit pythonbytes.fm and click Submit in the nav bar. We're always on the lookout for sharing something cool.

40:17 If you want to join us for the live recording, just visit the website and click Live Stream to get notified of when our next episode goes live.

40:24 That's usually happening at noon Pacific on Wednesdays on YouTube.

40:28 This is Brian Okken. On behalf of myself and Michael Kennedy, thank you for listening and sharing this podcast with your friends and colleagues.