Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


Transcript #277: It's a Python package showdown!

Return to episode page view on github
Recorded on Monday, Mar 28, 2022.

00:00 - Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.

00:05 This is episode 277, recorded March 28th, 2022.

00:10 And I am Brian Okken.

00:12 - I am Michael Kennedy.

00:14 - And I'm Thomas Geiger.

00:15 - Welcome Thomas, welcome to the show.

00:17 Thanks for coming on and being a guest.

00:19 Can you tell us a little bit about you?

00:22 - Thanks Brian and thanks Michael.

00:23 Big fan, so it's an honor being here.

00:26 I'm the creator and maintainer of the PyProtask runner, which it so happens you discussed last week.

00:31 So I come in riding on that wave.

00:34 - Yeah.

00:37 - Yeah, very cool projects.

00:38 Congrats on it.

00:39 - Thank you very much.

00:40 - Well, so Michael, it's March.

00:43 - It is March.

00:44 It's like March madness, right?

00:46 - Yeah.

00:47 - So Chris May sent in this thing that says, "Hey, Python Bytes people, here's a fun thing to cover, March madness." But for Python, and for those of you who are not college basketball fans and follow it carefully.

01:01 March Madness is basically the playoffs for the college basketball and it's single elimination.

01:07 You start with 16, I think, then every team plays another one, then it's down to eight, then down to four and so on.

01:13 So that's the idea, but for Python.

01:15 - Oh.

01:16 - And check it here, we have round one.

01:18 I guess it starts with 32 and then 16 and so on.

01:21 So we've got these different rounds and some of the rounds have already occurred, but the winner, the champion is still yet to be crowned.

01:28 So you all need to get out there and vote.

01:31 I'll tell you how in a second.

01:32 - I'm a bit amazed NumPy is outdoing pytest there.

01:35 - It's outdoing it pretty handily.

01:39 I mean, it did outdo it, right?

01:40 So if you go here, what you see is this, this tournament bracket.

01:43 And the first ones were like NumPy versus Redis and NumPy won.

01:49 And then pytest versus LXML, Parser, and pytest won that one handily.

01:54 And then NumPy and pytest had to face off.

01:58 And as Thomas says, surprisingly, NumPy pretty badly beat up on pytest.

02:04 Brian, are you okay with this?

02:06 How are you feeling?

02:07 - I didn't get to vote, so I'm not sure how this was done.

02:10 - Yeah.

02:11 - This is gonna be the start of a long blood feud between the NumPy community and pytest.

02:15 - Well, and the other part of this story I'm telling, the other side of the bracket was Scikit-learn versus Beautiful Soup.

02:23 And Beautiful Soup, oh my gosh, I think it was a buzzer beater.

02:26 Came in at the last second and it's like 52% to 48%, Beautiful Soup won.

02:32 And so now this week we're in the elite eight.

02:36 And so you can come and vote.

02:37 I'm gonna vote.

02:38 My metric here is sort of how useful and how impactful is this thing.

02:43 Not necessarily do I like it better.

02:45 So I'm gonna vote over here.

02:46 I'm gonna say for NumPy versus Beautiful Soup, NumPy.

02:49 I actually would use Beautiful Soup probably more, but I think NumPy is more impactful.

02:53 pip versus Matplotlib, I'll pip all day long, same reason.

02:57 Pandas versus Docker.

02:59 Ooh, I do like me some Docker.

03:01 I'm gonna go with pandas.

03:03 And then wheel versus requests, I'm gonna go with requests.

03:05 I know wheel is important under the covers, but I don't see it, so I don't wanna think about it.

03:09 So requests, top of mind, I use that all the time.

03:11 So here you can see I voted, and everyone else who would like to, you can just click the link in the show notes, and you can vote too.

03:17 And these are basically open for a week, and then the elimination happens and it moves on.

03:22 We're going to see what happens in the final four, coming real soon actually.

03:26 >> We're going to have to highlight this earlier in the month next year so that people can vote.

03:34 >> You want to create some voting blocks like in the reality TV shows.

03:38 The one on the island, Survivor.

03:40 >> Yes, Survivor, exactly.

03:43 I'm sad to say Scikit-learn's torch has been extinguished.

03:48 >> You're going to have to leave the island. Yes, that's right.

03:51 Anyway, thank you, Chris, for sending this in.

03:53 This is fun and it's very low stakes.

03:57 It's just sort of, you know, people just enjoy this for what it is.

04:00 Yeah, bragging rights and whatnot.

04:02 So we'll send out a tweet or something about it.

04:06 You can get in there and check this out.

04:07 >> Definitely.

04:09 >> Yeah. How about you, Brian? What's your next one?

04:11 >> I'd like to talk about NB preview, which actually I thought we covered, but I couldn't find it anywhere.

04:16 So NB preview is a notebook previewer.

04:20 So IPython or Jupyter Notebook.

04:23 It's neat, it's a command line thing and I like to spend a lot of time on the command line.

04:31 Once you pip install it or since it's not really part of your project, I used pipx installs of this.

04:39 >> Yeah.

04:40 >> But you say NB preview and then you can give it some options, but then a notebook file name and it will, it just previews your notebook in ASCII, which is awesome.

04:55 But it's not just ASCII, it's rich.

05:00 So we've got colors and nice colors and tables and stuff.

05:04 There's actually quite a few features that I wanna run down.

05:07 One of the things I loved right away was, it's not just a file.

05:13 I said, I tried it out on some local files, but you can give it like a URL or something.

05:19 There's a great way to, you can get a whole bunch of stuff.

05:22 You don't have to have local notebook files to put it in.

05:25 - Oh, that's cool.

05:26 - Yeah, here it's showing even you can curl something and pipe it to it.

05:32 So it'll take inputs as pipes.

05:34 And the fact that it's a command line tool and it deals with pipes correctly is what I really like about it.

05:40 So you can pipe a notebook to it.

05:42 I don't know if you do that or not, but you might want to pipe output.

05:45 So by default, you get these nice colors, but if you pipe it to an output, you can pipe it to grep or something, and you can grep for things.

05:54 So this is great.

05:56 I don't know if you've ever tried to grep for something in a notebook, but there's a lot of junk around it.

06:02 There's a lot of formatting stuff.

06:04 If that's not really what you're looking for, it's not helpful. So having this tool to strip that out, it's pretty nice.

06:09 >> Oh, yeah. That's really nice.

06:11 I love the ability to just pull this up and view them.

06:13 Given that it's based on Rich, like it has formatting for all the cells.

06:18 I mean, Jupyter is like Markdown plus code and rich highlighting for both of those.

06:24 So that's cool.

06:25 - Yeah, it looks like it's got some pigments under the hood also, which just so happens Ian wrote up last week, I think.

06:31 - Yeah, exactly.

06:32 So a lot of continuations said this week.

06:35 - So a lot of cool stuff that you would expect like code highlighting and stuff.

06:37 But the thing that like really stood out to me is what does it do with images, like graphs and stuff?

06:44 and the images are kind of amazing.

06:46 They're like these, by default, these block things, which not that clear to use for utilities, but it kind of shows you what it's gonna do.

06:58 And there's a few options.

07:00 You can do this block level thing.

07:05 And I like the characters, so it does like--

07:07 - The ASCII art.

07:08 - ASCII art stuff of your images.

07:11 or it uses the braille stuff.

07:14 I don't know if there's an example here, but you can do braille for all the dots to show up, which is kind of neat.

07:21 It even does like cool data frame rendering.

07:23 So if you've got a data frame printed out there in your notebook, it'll format it nicely.

07:31 So even late latech is a formatted, which is kind of a surprise.

07:36 I didn't expect that.

07:37 So that's kind of neat.

07:39 Anyway, I specifically, oh, cool, hyperlinks too.

07:44 So you can click on HTML that's in there.

07:46 That's kind of neat.

07:47 The thing that I really liked that is the simple part though is to be able to strip stuff and pipe it to grep and things like that.

07:54 So this is handy.

07:55 - Nice.

07:57 Thomas, what do you think?

07:59 - Oh, this is great.

08:00 I don't really use notebooks all that much to be honest with you.

08:03 So it's a little bit lost on me, but more command line is absolutely good.

08:08 and it looks delicious.

08:11 - Yeah, it does.

08:12 It's the terminal, the TUIs, the terminal user interfaces are definitely coming on strong these days.

08:18 We forgot to ask you, what kind of Python do you do?

08:21 What's your flavor of Python?

08:23 Are you building APIs?

08:24 Are you doing data science?

08:25 Like what kind?

08:27 - Well, the Piper project is what consumes most of my hours.

08:30 So I guess that's normal-ish Python as opposed to notebook-ish Python.

08:37 Data science, I don't really do too much either.

08:40 So it's mostly traditional style Python programming.

08:44 - Yeah, got it.

08:45 All right, well, your topic is up next.

08:47 Tell us about it.

08:48 - Well, funnily enough, this is very traditional programming.

08:50 What I bring for you for your delectation is PyFakeFS, which I think is a sadly, relatively unknown open source library.

09:02 And I'd like to give them some props and recognition because I think it's amazing.

09:06 and it's made a huge difference to me and my own code and the Piper project.

09:10 So hopefully this helps out some other people.

09:13 Now, what it is, is a fake file system.

09:16 So in a nutshell, it intercepts all calls from Python to the actual file system.

09:23 So if you think of the open function, the built-in open that is, or shutil or pathlib, all of those that might have real-world side effects in terms of the disk, the fake file system will intercept these.

09:36 And this is completely transparent.

09:38 And which is to say that your functional code doesn't need to know about this.

09:42 So the patching happens without you needing to inject something, or without you needing to go and alter your actual code to take countenance of the system.

09:53 Now, what's great about this is the moment you start talking about testing a file system, you're almost by definition in integration testing or functional testing terrain.

10:02 like it's not a unit test anymore, which comes with its own disadvantages.

10:06 So if you do want a unit test, then let's consider a simplistic example, right?

10:12 If you want to, if you're the code under test writes an output file.

10:16 So first of all, you need to patch out that if you're in your unit testing framework with something like mock open.

10:22 But secondly, you probably have a path that have been there somewhere where you're either creating the parent directories for the path to check that they exist before you try and write to that location.

10:32 So now we already have two things we have to patch out.

10:35 And then on top of that, you might be doing it in a loop, you might be writing more than one file, and the testing becomes very clumsy very quickly.

10:44 Whereas once you use the PyFakeFS library, you can just write as normal, validate against that file system using the standard Python inputs.

10:54 And what you end up with is, and once the test finishes, it all just goes out of scope and you don't even need to bother cleaning it up.

11:01 What's--

11:02 - Yeah, that's cool.

11:02 And you can specify the string that is the content to the file.

11:05 So when the thing reads it, you can control the what it sees, right?

11:08 - So it comes with a, and Brian, you're gonna love this.

11:11 It comes with a super handy pytest fixture.

11:15 So if you are using pytest, which you should, you can just add the FS fixture to your unit test.

11:21 And now everything in your unit test will be going to the fake file system rather than the real underlying file system.

11:30 - That's pretty cool.

11:31 - Yeah, and the helper functions allows you, like you were hinting at, Mike, you can specify encodings, you can write in binary, it's super useful.

11:40 Something else that I use quite a lot is the ability to switch between Linux, Mac, and Windows file systems, which again, for Piper, is such a boon to be able to test the cross-platform compatibility.

11:52 - Oh, interesting.

11:53 So if it asks for the representation from a pathlib thing, it'll do C colon backslash instead of forward slash.

11:59 - Yeah, exactly right.

12:01 So all of these things are, you know, I'm relatively conservative when it comes to pulling in new libraries because I'm, especially if the library feels heavy and I feel I can do it just using standard lib functionality and also with some libraries, I'm a little bit worried that they might stop being maintained or something like that.

12:20 But PyFakeFS has been around since 2006, developed by Google.

12:26 It was open sourced in 2011.

12:29 The maintainers are really on it.

12:30 I submitted and had a PR merged earlier this year within an afternoon on a Saturday, which for open source is very quick.

12:41 So they're on top of it.

12:43 Great project, check it out on GitHub.

12:45 Check out the documentation too.

12:46 It's well-documented and it's super useful.

12:49 - And I was looking at the toxini.

12:51 It looks like it's tested to be compatible with PyPI also, which is kind of nice.

12:56 - Yeah, yeah, absolutely.

12:58 Especially for what I'm doing in Piper, where wrangling configuration files is a lot of the functionality as a task runner.

13:05 You're forever reading JSON, writing out YAML, converting between formats, converting between encodings, swapping out values inside configuration files, merging configuration files.

13:16 And I'm now able to test all of this stuff without having to write integration tests for each and every permutation, which has been such a boon.

13:26 - This actually does way more than I thought it did.

13:28 This is, I'm gonna check this out.

13:30 This is neat.

13:31 - Yeah, there's a lot of cool stuff there, absolutely.

13:34 - Yeah, and also if you've--

13:35 - Chris and Alvaro both think pretty, pretty neat out there.

13:39 They're digging it.

13:40 - Yeah, and I see the comment there.

13:42 It is like Tempath with the difference that it's not actually writing to the disk itself, of course.

13:49 And what's also a little bit difficult when you're using the temp directory and the temp file modules is, depending on how you're testing, it doesn't always help you very much.

13:58 Because the thing that might be generating the file might be the code under test.

14:02 So you're effectively gonna have to intercept that and create a temp file to attach to it, and then the temp file will clean itself up.

14:10 But that starts interrupting the flow of the functional code so much that I start questioning whether it's even a useful unit test anymore.

14:18 - Yeah, absolutely.

14:20 - Well, very, very cool.

14:21 So, Brian, before we move on, let me tell you about our sponsor, all right?

14:24 - All right.

14:25 - This episode of Python Bytes is brought to you by Microsoft for Startups Founders Hub.

14:31 Starting a business is hard.

14:32 By some estimates, over 90% of startups will go out of business in just their first year.

14:38 With that in mind, Microsoft for Startups set out to understand what startups need to be successful and to create a digital platform to help them overcome those challenges.

14:47 Microsoft for Startups Founders Hub was born.

14:50 Founders Hub provides all founders at any stage with free resources to solve their startup challenges.

14:56 The platform provides technology benefits, access to expert guidance and skilled resources, mentorship and networking connections, and much more.

15:05 Unlike others in the industry, Microsoft for Startups Founders Hub doesn't require startups to be investor-backed or third-party validated to participate.

15:14 Founders Hub is truly open to all.

15:17 So what do you get if you join them?

15:18 You speed up your development with free access to GitHub and Microsoft Cloud computing resources and the ability to unlock more credits over time.

15:27 To help your startup innovate, Founders Hub is partnering with innovative companies like OpenAI, a global leader in AI research and development to provide exclusive benefits and discounts.

15:37 Through Microsoft for Startups Founders Hub, becoming a founder is no longer about who you know.

15:42 You'll have access to their mentorship network, giving you a pool of hundreds of mentors across a range of disciplines and areas like idea validation, fundraising, management and coaching, sales and marketing, as well as specific technical stress points.

15:55 You'll be able to book a one-on-one meeting with the mentors, many of whom are former founders themselves.

16:01 Make your idea a reality today with the critical support you'll get from Founders Hub.

16:06 To join the program, just visit pythonbytes.fm/foundershub, all one word, the link's in your show notes.

16:12 Thank you to Microsoft for supporting the show.

16:14 - Awesome, thank you Microsoft.

16:16 Now, let me tell you about something that sounds incredibly simple, but as you kind of unwind it, you're like, wait, it does that too?

16:24 Oh, it does that too?

16:25 Oh, that's kind of cool.

16:27 Pretty similar to the fake file system that Thomas was just telling us about.

16:30 This thing called Sternum.

16:32 Sternum is a fantastic name.

16:36 It's short for string enum, right?

16:38 Enums, when were enums added?

16:40 Was that three, four?

16:41 something like that a little while ago.

16:44 So enums have been in Python for a while, pretty much pre-history now that those are no longer supported.

16:52 And with enums, you can write cool code that says this class, its fields are enumerations, and then you can say, you know, enum type dot enum value, and you can use that instead of magic words.

17:03 So for example, you might have HTTP method or something like that, or let's say HTTP status, Let's start with that one because that's like a built-in type thing you could do easily.

17:13 You could have a 200, a 201, a 400, a 500, a 404, those kinds of things.

17:20 You could have like HTTP statuses dot and then those types with those numbers, right?

17:25 But there's a couple of challenges to working with those.

17:28 Their natural representation is a number, not a string.

17:33 And I know you can derive from enum and then also derive from string.

17:37 But like I said, more stuff happening than just that.

17:40 So this sternum allows you to create enums like that and use the enum auto, enum.auto field.

17:48 So I can say, here's an HTTP method, but like verbs is really probably what it should be.

17:52 So you can have a get, you can have a head and a post and a put, and you just say auto, auto, auto, auto.

17:57 But the actual representation is that the get is a string get.

18:03 And the like, put, want, or post is, you know, put or post.

18:09 Yeah, and Alvaro is out there pointing out, thank you, that sternum was temporarily part of 3.10, but that it was dropped.

18:15 I saw a note that it might be included in 3.11 again.

18:19 Okay, that'd be fantastic.

18:21 Yeah, so there's some really neat stuff in here.

18:23 For example, one of the things that's nice is because this thing basically has the value string, where you're using it, you can actually use it where a string would be accepted.

18:33 So here if you're doing a request to a URL and you've got to say method equals, Here you can say method equals HTTP method dot head or whatever from the enum, and it directly passes just the string head to the method.

18:45 It's a really nice way to gather up string values that are part of a group like HTTP verbs or something like that.

18:53 >> Wow.

18:54 >> That's pretty neat.

18:55 >> Okay. The side question is, I don't really use auto much.

18:59 Is auto used anywhere else or is auto just an enum?

19:03 >> It comes out of the enum module.

19:04 >> Okay. It's part of this.

19:06 So it's part of the enum thing.

19:08 - Right.

19:09 - And one of the things I really like about this that is super tricky with enums is databases.

19:14 So for example, imagine we had, we had like get, head and post.

19:20 So we just had auto, but it was an integer based one.

19:22 So it was like one, two, three.

19:24 And we store it in the database, right?

19:26 As a one or two or three, and then you parse it back fine.

19:29 But then somebody adds another auto thing in there and they don't put it at the end.

19:34 They're like, oh, this one starts with a D.

19:35 so it goes after delete.

19:37 >> Yeah.

19:37 >> Well, all the stuff after that one is now off by one in the database.

19:43 This if it goes into the database, it goes in as a string and it'll parse back as the string.

19:47 It also has cool stuff like lowercase sternum, and uppercase stringenum.

19:53 You can derive from that instead and then no matter how you define your fields you get a lowercase string version or an uppercase string version.

20:01 >> Okay.

20:02 >> There's other cases as well.

20:04 There's Pascal case, snake case, kebab case, macro case, and camel case.

20:10 Go crazy on them people.

20:12 You can have the same code, but then the string representation varies.

20:17 That's pretty awesome.

20:19 >> I think I'm going to go with kebab case just because that's fun to say.

20:22 >> It's so fun, I know.

20:24 You can also directly assign the value.

20:29 Enum value equals some string and then you don't have to worry about a casing that's exactly the string that you put.

20:36 >> Yeah.

20:36 >> Right? So there it is.

20:39 It's like regular enum but strings.

20:41 As people pointed out that it's not that different from what people have been considering for CPython.

20:47 I'm pretty sure I'd heard about it as well in being in there, but the fact that it's not there, maybe it'll be there, maybe not.

20:53 We'll see. It's interesting.

20:54 But this has a lot of cool features and if you're not using 3.11 or want to depend upon it, this is a small little project.

21:01 >> Yeah. It's nice. Cool.

21:03 - Yeah, Thomas, what do you think?

21:05 - This is great.

21:06 I especially like how it's smart enough to auto-cast so that we use the enum, it will end up translating to a string when you're actually hitting the database or your underlying API.

21:19 - Yeah, it makes it actually usable in those situations, just directly, yeah, which I think is great.

21:24 - And funnily enough, the example they chose is so great by way of great documentation because HTTP verbs are just almost the example of magic strings, right?

21:34 - Yeah, exactly, exactly.

21:37 Quite cool.

21:38 All right, Brian, over to you.

21:40 - I'd like to review your code a little bit.

21:42 No, I don't know.

21:44 I was trying to do a transition thing going.

21:46 But so Tim Hopper wrote this article, which I absolutely love, and it's called "Code Review Guidelines for Data Science Teams." And I just recommend everybody go read it.

21:57 It's short, it's good.

21:59 But one of the things I really like that he highlighted is before he got into the code review or the code review guidelines, he started with why are we doing a code review?

22:13 What is a code review for?

22:15 And this is something I think that is important just to talk with whoever, whatever team is going on and talking, maybe even sticking it in a participation guideline in a project, open source project even, is that it's not just, it's not just so that we can look at the code or check it, merge it.

22:35 So his reasons for a code review are first code correctness, and that's what we think about is making sure the code's correct, but also code familiarity.

22:47 So you might be the expert on a project and everybody else is only kind of new on it.

22:52 You still should have code reviews for your code changes so that everybody else can watch also and get familiar with the changes going on.

23:02 So that's nice.

23:03 Design feedback, of course, and mutual learning and regression protection are all the reasons why he did a code review.

23:11 And the other thing I also love is what to leave out of code review.

23:17 So code reviews are not about trying to impose your guidelines on somebody else.

23:22 And they're also not a reason to push off responsibility So as long as your code is getting reviewed, it does not be correct, right?

23:32 Because somebody will catch any problems.

23:34 It's a bad thing to do in a code review.

23:37 So make sure your code is correct, that as far as it's all cleaned up as soon as you – what you think is ready, and then submit it.

23:45 But then also be nice.

23:47 So being nice is important.

23:49 >> Yeah, very cool.

23:51 >> So then he goes through – I'm not going to go through all these here, but he goes through different things about what to think about before you do a create a pull request and then what to do if you're reviewing a pull request. And a lot of these are just they're just around being a kind human to the person on the other end. So that's really kind of what it's about.

24:14 So I saw a mention in there somewhere that I really liked, which is, I mean, by nature, a code review is sort of nitpicky, right? You're paying attention to flaws, but it's - It's nice to compliment also.

24:25 Like if there's something nifty or cool or cute, acknowledge, compliment, call attention to it.

24:30 - Oh, that's a good point.

24:32 And I really like that.

24:33 I also think, so one of the things that you don't wanna do in a code review is like one of the guidelines is we're not looking for perfection.

24:42 That isn't one of the things we're looking for.

24:48 But so what happens if you notice something and you're like, it's a little weird.

24:53 I'd like to say something about it, but I don't know how to say that.

24:57 His comment is to have, if you've got a minor thing that you want to comment on, go ahead and sort of tag it.

25:03 He recommends tagging it with knit in IT, or a nitpick or something.

25:08 Just to be clear that I'm, I don't know if I like the word knit, but to be clear, hey, I noticed this, maybe we want to change this in the future.

25:17 Somehow indicate to the person that they don't need to fix this before the PR gets merged.

25:22 you're just noticed it.

25:24 So, and it might be something that the person that's submitting the PR didn't realize in the first place and went, oh yeah, I don't like that either, I'm gonna fix it.

25:32 Or yes, I do know about that and I do plan on fixing it later or whatever.

25:37 So just an interesting guideline.

25:40 And I think it can just, I'm kind of, I've been on a kick lately of reading things about community and creating cohesive teams.

25:50 And the review process is definitely somewhere that you need to have attention to for most teams.

25:57 So anyway, that's it.

25:58 - Yeah, I like it.

25:59 This is really handy.

26:01 I love the idea of having as much as possible have the automation make the complaints.

26:07 - Definitely.

26:08 - And like Thomas said, have the people give the compliments and the sort of interesting discussion, right?

26:11 But like if black can just take care of the formatting, like you shouldn't have to debate the formatting.

26:15 - Yeah.

26:16 - And if a linter can tell you, you know what, there's something wrong with this, just let the linter be the bad guy.

26:21 - Yeah, it was one of the guidelines that he brought up, which is interesting, is especially with CI, and we're pushing a lot of things on Black or linters, that to wait, so wait a little bit.

26:33 So don't review a code review right away, especially not if the CI hasn't finished.

26:40 Let the CI finish and let the person creating it fix anything before you jump in.

26:46 I also, pet peeve of mine, don't comment on it right away.

26:50 I might, one of the things I do frequently is I'll create a PR, especially in a work setting.

26:58 I'll create a PR, and then there's some complicated things.

27:01 So I plan on going through and writing some comments around some of the complicated bits, like why did I do certain things?

27:09 And so if you see a PR right away, especially from me, wait 10 minutes or so before commenting on it, 'cause I might have answered your question before you get a chance to ask it.

27:21 - An exclamation might be coming, yeah, indeed.

27:24 - Anyway.

27:25 - Awesome, all right, Thomas, over to you.

27:28 - We're about to head into controversy, because there's been some discord.

27:33 - Are you gonna bash on something?

27:34 Come on, come on.

27:35 - I'm gonna bash it over the head with a, like a caveman.

27:39 - Bash it with Python.

27:40 - So partly inspired on the continuation last week's discussion you had about running subprocesses from Python. And Itamar Turner Trauring wrote an article this week called "Please, please, emphasis mine, stop writing shell scripts." Now, this, as you might imagine, raised a bit of questions on the usual places like Reddit and Twitter. But if nothing else, controversy aside, the article is a very good and succinct summary of the most common gotchas and problems with Bash, which we can almost all summarize as that error handling is strange if you're used to other programming languages, like Bash is a kingdom unto its own when it comes to programming languages. So he also gives a great recommendation for if you really, really have to write in Bash, what you might want to do, and that would be to use the unofficial Bash strict mode, which basically involves setting that boilerplate on top of your Bash. I'm not going to cover all the details, but basically the E and the U option will fail immediately on error. It will fail on unset variables, and if you add the pipe fail option, errors won't pass between pipes. A pipe will actually fail immediately if there's an error process.

29:05 - Like it should. - Like it should, indeed. But the point is, Bash is an old technology and there's a lot of problems here. And let me add, although this article mostly aims at Bash, I am very happy, including Bourne and SSH and Fish and TakeYourPick underneath the same dictum. Now, he goes on to talk about the typical reasons we hear of why we should be using Bash, of which the top one is, well, it's the most common -- you're guaranteed to have an SH runtime, at least, on any given machine that you're going to be using. But the point is, not really. Because when we're doing code automation, almost by definition, the programming language you're coding in, its runtime is on the server. So this argument that somehow it's good to go to the lowest common denominator, aka sh or bash, when you already have Python on the machine is sort of, well, why? And especially when we're talking about Python, which is so great at automation, it just baffles the mind.

30:12 That's a good point. You don't have to set up a compiler or any of that kind of business.

30:16 I'd say the same thing about Golang. I mean, by definition, when you're compiling Go, the Go compiler is right there. You might as well be writing a Go script, or whichever your programming language is. Maybe if you're starting to talk about C, or C++, there's maybe a different argument that we can have there. The second point it brings up is what I'm going to paraphrase as "get good", which is this Bash guru response which we saw a bit of in the last week, that you're just bad at Bash. If you were better at Bash, you wouldn't be complaining about these things.

30:52 Which, you know, is not a great reason.

30:54 It's, you know, just because, it's not better because it's hard, right?

30:59 We have better tools available.

31:02 We have tools that behave more responsibly.

31:04 And something that I think is very important, in line with what you've been talking about, Brian, about building teams, is very often that your automation activities start becoming this specialized zone that only two or three people on the team can even look at because they're the Bash gurus and everyone else is too afraid to touch it.

31:22 Whereas if you keep your automation activities within the language you're coding in, suddenly everyone on the team can start carrying their weight, right?

31:30 - Yeah, I kind of relate to this a lot.

31:33 I've been on projects where we've had a lot of our automation in Bash and others that have been other languages.

31:40 Right now, it was one of those things of, especially if you're not, If you're not looking on a Windows environment, Bash isn't there automatically.

31:51 A lot of the team members might not be familiar with it.

31:56 The thing that I don't know if he addresses this, the thing that I was thinking about was, we all know Python if we're programming Python, but we might not all know the automation parts of it, the way to do file manipulation or.

32:13 >> Right. SHUtil and that kind of stuff.

32:17 >> Stuff that we might be familiar with with Bash because if we're using it all the time on the command line, I already know how to do it. But I might not know how to do that sort of stuff in Python because I'm not using Python like that. But anyway.

32:31 >> Well, my response to that would be that whatever the thing is that you don't know how to do in Python, your chances of running into trouble with Bash are, to my mind, a lot higher than they are with Python. Or at least when things misbehave in Python, your control of flow is better so that you probably will have a -- especially as the scripts start getting bigger, you will have better control over where the issues might be. Or you would be better able to isolate those areas that you're not exactly sure of. I saw someone in chat last week raise the specter of make files that call shell scripts that call make and I mean, this is not uncommon.

33:13 I'm sure we've all seen these things.

33:16 And I'm actually very interested in the psychology around this because we're all coders, right?

33:21 I assume we're here because we enjoy automating things.

33:24 We enjoy solving problems.

33:26 We probably, you know, have a certain problem solving sort of mindset that got us into this to begin with.

33:33 Yet, it seems like we spend so much time automating our customers' business processes that we forget to automate our own coding processes.

33:41 Or when we do, we deallocate the priority, we debudget it, we end up focusing on all sorts of other things other than this essential housekeeping.

33:50 - Yeah, or treat it like throwaway code instead of code that needs to be carefully factored and put together. - Exactly right.

33:56 And I would argue it's a bit like housekeeping.

33:58 You know, no one likes doing it, but if you don't want to live in a pigsty, you gotta do it, you know, instead of--

34:03 - Yeah, well, also, to be honest, I was there once of like, I don't know how to do this automation stuff in Python, but it bugged me that I didn't know how.

34:13 So I'm like, okay, well, what do I need to learn?

34:17 Like the few things like searching for stuff, like I normally would have used Perl for Regex or something like that, or said.

34:24 All that stuff you can do with Python, and actually there's tons of articles on it.

34:27 It's really not that hard to go, okay, the pieces I'm missing, how do I do that?

34:32 And just go learn it.

34:34 And then it's not that hard to switch a lot of automation to Python.

34:38 - Yeah, definitely not.

34:39 And I mean, so much other automation happens in Python anyway.

34:43 I mean, in fact, kind of compiled programming languages will often use Python as an automation language.

34:49 It's so handy for the automation process.

34:51 There is another psychological thing, which I find, or I think psychological thing, that I find quite curious here, which is this dealing with complex shell scripts almost becomes this technocratic rite of passage, where when you couple that with imposter syndrome, it's very easy to be intimidated by the bash bros when they do these really clever one-liner bashisms that you can't make head or tail of.

35:16 And it's like, yeah, look how clever this is.

35:18 But it's very hard to maintain, and it's almost hard to call that to account unless you're very sure of yourself, because you almost have to justify yourself as to why you dislike it.

35:28 Like you first have to prove your bona fides.

35:31 I think it's sort of the tech equivalent of, you know, back in my day, like, we didn't have X, you know, like, whatever X is, shoes or toilet paper, or, like, whatever.

35:43 Just because something used to be difficult doesn't mean it needs to be difficult forevermore, right?

35:49 Like, the extra difficulty doesn't make it better. It's not a video game like Elden Ring, you know?

35:54 The easier this is, the more quickly and effectively you can do the housekeeping, the more you can get up with the features that actually pay the bills, which is to say the shiny functional stuff that you can demo and put in front of customers.

36:07 >> Yeah, absolutely.

36:09 I have some real-time feedback and also a comment for you.

36:12 Alvaro says, "There's a VS Code plugin called Shellshock." If he's remembering it correctly, tells me when I'm doing something wrong or might blow up.

36:20 There's also a plugin for PyCharm.

36:21 If you're going to do it, have those things for sure. Yeah, funnily enough, we've got immediate feedback to that, which is the author of the original article mentions shell check, which is effectively like, like the comments are mentioned a lender for bash, but the reviewer, but the article also mentions that it doesn't actually catch all things either. So like all lenders, it can very easily lull you into a false sense of security while it's not really necessarily addressing the underlying problems.

36:49 And I almost feel like I don't even need to say this because anyone who's ever tried to debug a long bash script should know this.

36:58 They're tricky.

36:59 They fail in mysterious places and it's very hard to figure out why and how.

37:04 And -- >> Yeah.

37:05 But I do like the -- this article pointing out how if you have to, to set up those flags to make it, you know, fail quicker.

37:12 Because that helps a lot.

37:14 So that's nice.

37:15 Yeah.

37:16 >> Yeah, for sure.

37:17 - Also just to give the author massive amounts of credit, this isn't clickbait.

37:21 He didn't position this as never ever use bash.

37:25 In fact, he explicitly says that, and okay, if you're doing something super simplistic, like the typical sort of things that goes into a get hook, a pre-commit hook, where you're just running a command or two, then yeah, sure, of course, shell script's fine.

37:37 But I would say as soon as you're running loops, as soon as you're doing conditional branching, as soon as you're worried about retries, as soon as you're doing--

37:46 - Oh yeah, definitely.

37:47 Switch to Python.

37:48 - Absolutely.

37:49 - Yeah.

37:50 - Yeah.

37:51 And then another quick question, just a quick follow up.

37:53 Have you considered Conch?

37:56 - I have not even heard of Conch.

37:57 Nevermind considered it.

37:59 - So it's, I haven't done much with, I've sort of looked at it.

38:03 It is a shell, like a competitor to Bash or Zshell or something like that, where it's a proper Python environment directly in the shell.

38:14 That's almost PowerShell-esque.

38:16 >> Yeah, it's a little bit like PowerShell, where PowerShell is like .NET C#, like, but not really.

38:22 I suspect it's similar here.

38:24 >> I know it's supposed to be pronounced conch, but my brain says zonch, because it's funner to say.

38:30 >> Zonch.

38:31 >> I know, but it has the shell, it has the conch shell, so you know that's how you got to say it.

38:35 >> Even have the ASCII art going on for the logo.

38:38 >> They do indeed.

38:41 - Alright, well, cool Thomas, that was a good conversation. - It was good.

38:44 So, do we have any extras?

38:46 Michael, do you have any extras?

38:48 You know I got extras, right?

38:50 I also, first, real quick follow up, real time follow up from Henry Scheiner in the audience, that pep 663 was the pep around string enum.

38:59 Oh, okay.

39:00 And he's not sure if removing the support for that pep means removing string enum from the standard lib or not though.

39:05 Doesn't do all the other stuff like the casing and the various other things that that cool package I talked about does.

39:10 So maybe that package is no matter what relevant still or inspiration for the next one or whatever.

39:16 In terms of extras, I do have some extras. Let me see what order I wanted to cover them in.

39:22 I had two, but then one got rescheduled. This is supposed to be the transformation from bugs.python.org over to GitHub, but that got pushed back a week. So I'm not going to talk about that. >> You just did.

39:34 >> Well, I was going to say it's happening. It should have happened by the time you hear this.

39:39 Let's go check it out. No, it's not true anymore.

39:41 >> No. Okay. Just if you're curious, supposedly it's moved to April 1st, but it's April Fool's Day, so I'm not sure if it's really going to happen or not.

39:49 >> Maybe it's a long con where the joke is being set up far in advance.

39:54 >> Exactly. We're actually never doing this.

39:57 I'm looking forward to that happening. That's great.

39:59 All right. I just have a general theme of stuff that's like, they're all together, a changing of the guard, if you will.

40:07 Let's see here.

40:08 So I have been switching so much of my software stuff around.

40:12 I've started using Vivaldi.

40:13 Now I've been using Firefox for a long time.

40:15 But I started using Vivaldi, which I think is a really neat take on a browser.

40:19 So switched over to Vivaldi and started using that.

40:23 There's a bunch of different things like Mozilla laid off 250 people recently.

40:28 >> They're axing the developer tools team too, which is just tragic.

40:31 >> Exactly. Cut the developer tools team.

40:33 They cut the threat team, a team that looks for like a tactic.

40:38 It's like, I don't know, it's starting to make me a little nervous.

40:43 I'm trying out Vivaldi.

40:44 I've been doing that for like a month or so, and I'm enjoying that.

40:47 >> Mike, you said it's a different take on a browser.

40:50 It sounds like there's something conceptually different about it.

40:54 >> It's just super customizable.

40:56 I think that's the thing. It's like there's just all sorts of stuff.

40:59 It comes with a built-in ad blockers and tracker blockers.

41:02 I know some of them do tracker blockers, but built-in ad blockers, nice.

41:04 I mean, Brave is the other one that kind of does that, but Brave's like, well, let's just trade those ads for our cryptocurrency ads that we'll put in there for you.

41:11 And you get a little bit of cryptocurrency, this is like, no, we'll just block the ads.

41:15 So anyway, I switched over to that, partly motivated by just concern around this, but also just wanted to try some stuff out from Google Docs over to Zoho for other stuff and for like business email.

41:27 There's so interesting stuff going on there.

41:29 And then like also DuckDuckGo, I've been using that for a while.

41:32 And I tried that a while ago.

41:34 I didn't feel like you switched.

41:36 To me now, there's just almost no difference in the quality compared to Google these days.

41:42 Where it used to be, I'd try and like, I might have to go to Google for that.

41:45 Several times a day, now I don't really.

41:47 If I get stuck here, usually I try to go to Google and get it and I get still stuck.

41:51 So just got to deal with it.

41:53 So that's it for all my items.

41:55 I'm just down to telling a joke.

41:57 Thomas, you got anything extra you want to share throughout the rest of the world?

42:00 >> Not particularly. I'm looking forward to your joke.

42:02 >> Give a quick shout out to Piper real quick.

42:03 - Oh yeah, check out last week's episode.

42:06 - I know we covered it last week, but yeah.

42:07 - Michael actually did as good an introduction to Piper as I could give, so congratulations and well done.

42:14 - Thank you.

42:14 - If you do want to check it out, support open source software, do the usual, share, like, subscribe, all the rest of it.

42:20 You can check it out on GitHub.

42:21 It is the Piper Task Runner, P-Y-P-Y-R.

42:26 And incidentally, if you don't want to run bash scripts, then a task runner might be a, Might be a good way of not doing so.

42:36 - Yeah, I was thinking about your project while you were talking about this.

42:40 - I don't want to shill too horribly, so I try to keep that to the end.

42:45 - That's our job.

42:46 We only shill, we basically just shill cool stuff all week.

42:49 That's our podcast.

42:50 Ryan, how about you?

42:51 Got anything extra you wanna shout out there?

42:53 - I've got some stuff, but there's nothing I can share right now, so yeah.

42:57 - All right, well, we'll be waiting.

42:58 How about we share a joke then and wrap it up there?

43:00 - Sounds good.

43:01 - So I feel like this is a missed opportunity, 'Cause we had Ian on last week and he was all about cybersecurity and using notebooks to track threats and stuff.

43:11 Well, has he considered this?

43:13 (laughing)

43:15 - That was in a James Bond movie, right?

43:19 It's been in several, I think.

43:21 - It could have been.

43:23 So here is like a big server rack with just like a hundred ethernet cables and in a big printed sign on it says, in case of cyber attack, break glass, pull cables.

43:32 I'll also say what surprises me the the internet is going soft in its old age because back in my day Haha, the first comments would have been complaining that the cables aren't tidy enough Wow, you got to get a good grip on exactly One zippy move with your arm and you give it a yank There's a lot of cables they should put like orange tags on the ones that are important to pull or something. Yeah >> Exactly. This is the sort of criticism that I would have expected.

44:03 >> Actually, the entire thing has a power switch, just so power off the whole thing.

44:09 >> You don't want to lose data. I mean, come on. No, just kidding.

44:12 >> Also, where's the axe?

44:14 How do you break the glass?

44:15 >> Exactly.

44:16 >> Exactly.

44:17 >> Oh, or just open the door handle.

44:19 >> Very not very thought through.

44:21 It reminds me a little bit of that in case fire, get commit, get push, run.

44:26 >> Also, we're talking about IT people who generally probably aren't that much into the pushing regime.

44:33 So--

44:35 >> Or lifting axes.

44:36 That might be a strain.

44:40 Sorry, I'm going to get hate mail for that.

44:42 >> We are.

44:43 >> Yes, indeed.

44:45 Wow.

44:46 I thought it was fun, Brian.

44:48 >> Well, thanks everybody for having a fun episode again.

44:52 Thank you, Thomas, for showing up.

44:54 Thanks, Michael.

44:55 And thank you, everybody in the chat, for showing up.

44:57 So we'll see you all next week.

44:59 Bye everyone.

Back to show page