Transcript #322: Python Packages, Let Me Count The Ways

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.

00:04 This is episode 322, recorded February 7th, 2023. I'm Michael Kennedy.

00:10 And I'm Brian Okken.

00:12 I'm Calvin Hendricks-Parker.

00:13 Welcome, Calvin. Awesome to have you here.

00:15 I'm excited to be back. I love it's kind of my annual retreat.

00:18 That's right. That's right. Your annual drop-in to share all your ideas with us.

00:23 You got some good ones to talk about, I know.

00:25 Also want to say thank you to Microsoft for Startups Founders Hub for sponsoring this episode.

00:30 If you want to connect with any of us, me, Brian, Calvin, or even just the show, you can find us over on Fastedon.

00:37 We're all over there, actually, on Mastodon, so you can find the links in the show notes.

00:41 And if you're listening but not watching, just go to pythonby.fm.com slash stream slash live,

00:46 where we will be streaming live every Tuesday, usually.

00:50 But you can definitely subscribe to get notified there.

00:53 Calvin, I know you've been here before, multiple times.

00:56 You always come to put us to shame with your fancy camera that you got going.

01:01 We love it.

01:02 Tell people who maybe haven't heard about you or met you before who you are.

01:06 Sure. So I'm Calvin Hendricks-Parker.

01:08 I'm co-founder and CTO of Six Feet Up.

01:10 We are a software development company that helps digital leaders transform their impact.

01:15 We do a lot of interesting work now around impactful projects.

01:19 So it's like climate change, making the world a better place.

01:22 In addition, I've started a couple of communities.

01:24 So the IndiePy community here, local Indianapolis.

01:27 And then we kind of branched out from that into some little local conferences into now Python Web Conference.

01:33 So that's probably the big thing you'll hear from me at least this month and next month.

01:38 Yeah, we'll talk about that at the end of the show.

01:39 But that's an amazing conference.

01:41 You have, I would say, potentially more talks or on par as many talks as US PyCon.

01:47 It's really a lot going on there.

01:49 Last year, there were almost too many.

01:51 Like, it was so good.

01:52 I just kept saying yes to everybody.

01:54 This year, we pulled back a little bit.

01:56 We had more submissions.

01:58 We'll talk about this later.

01:59 But it was really hard to choose.

02:00 Yeah, it's tough to say no to deserving people.

02:03 But it's so much talent.

02:05 So many great people.

02:06 Indeed.

02:07 All right, Brian, kick us off.

02:09 I just wanted to talk about packaging today a little bit.

02:12 I've got, I'm splitting it up into two segments.

02:14 But I really want to focus on right now on the PyPA has a packaging Python projects tutorial page.

02:22 It's linked in the show notes, of course.

02:25 But I think this is a good, it's really pretty good, well written.

02:29 And so I want to point people in this direction.

02:31 If you want to share your Python code with somebody else, packaging it is the right way to go, I think.

02:37 And this tutorial walks through, it's not necessarily trivial to do it.

02:43 So, and it, I wish it was easier, but it, it's not bad.

02:47 So it walks through what you, you know, walks through setting up your directory structure.

02:52 It's, and then you have to have some package files with it.

02:56 You've got to have a license file and a pyproject.toml now and a readme.

03:01 And then it talks about your source directory and your test directory.

03:05 So you've got the directory set up.

03:07 I know you're a fan of the source directory for packages.

03:10 It looks like.

03:11 Yeah.

03:12 Exactly.

03:13 Your philosophy is winning out here is the official recommendation.

03:17 Yeah.

03:17 And I, I mostly, I used to have good reasons behind it.

03:21 Now I just like having the top level stuff, just be packaging and having the actual source

03:27 code be somewhere else in the source directory.

03:29 So it just, it's a, it's a, while you're developing, it's, it's nicer.

03:33 If you have a single that there is like caveats though.

03:36 If you have a single file that you're just sharing, that might seem like overkill.

03:41 And I, I, I, I'm fine with, of course, I have packages as well that are single file that

03:46 I don't do this with.

03:47 So you don't have to, but most things, most projects are going to have more than one file

03:51 if you're sharing it.

03:53 So there's this.

03:54 So, and then what do you put in it?

03:56 So it talks about what you put in the different things in the PyProject.toml and a little

04:01 bit into the readme and some of the metadata.

04:03 The, one of the things we're going to jump into a little bit later in the show is that

04:10 there are lots of options for what build backend you use.

04:14 This, this project itself, or this document talks about hatchling, setup tools, FLIT and

04:20 PDM.

04:21 And they, neither of them look that different and they kind of do the same thing, but you

04:26 can have different metadata that goes with it.

04:28 So that's, I think both a good thing that we have options and a bad thing that some people

04:33 don't know how to choose.

04:34 So is it really tough to like, know why should you do it?

04:39 Maybe setup tools probably shouldn't be the choice.

04:42 I don't feel like that you're kind of moving a little bit away from that, but yeah.

04:46 The history here is so deep.

04:47 Like, I feel like there's like a whole day, a whole college course you could give on why

04:52 these things are the way they are now.

04:53 Yeah.

04:54 Yeah.

04:55 So the, in it, the project, the tutorial goes down into even distributing.

05:00 So all the way down to recommending that you use Twine to upload to the test repository,

05:09 test by PI.

05:10 And you have to get a token and it's not trivial and all this sort of stuff.

05:15 So it's a little, it's still a little daunting, but this will walk through, walk you through

05:19 the, all the steps.

05:20 I have a personal opinion, of course.

05:22 And my opinion is read this and then check out Flit.

05:26 Because for simple projects, Flit is, I think, still the easiest.

05:31 It replaces the build step with Flit build.

05:35 And it replaces the creation of some of these files with Flit init.

05:40 And it replaces the Twine upload with just Flit publish.

05:44 And I use it on a few projects and I haven't had reason to switch yet.

05:50 There's, it's not perfect, but it's pretty easy for small projects.

05:54 So I would say read both of these if you want to share some Python code.

05:57 So.

05:58 Excellent.

05:59 Yeah.

05:59 Very good recommendation.

06:00 And I think people need guidance here because there are so many choices.

06:03 There's going to be a lot of conversation in the Python space around this actually.

06:07 For sure.

06:07 Yep.

06:09 All right.

06:10 Let's move on though for right now.

06:11 Real time follow up here.

06:14 As of about 30 minutes ago, OFEC, the creator of Hatch, and I are going to have a conversation

06:20 March 1st on Talk Python.

06:22 So if people want to hear more about Hatch, that's coming up in about three weeks or so.

06:26 But what do I want to talk about?

06:28 Something fun.

06:30 Last time we had the Unreple.

06:32 This time we have the Untangle.

06:33 So Untangle by ST Chris on GitHub.

06:39 Converts XML to dynamic Python objects.

06:43 So thankfully we've largely moved past XML.

06:47 And really thankfully we've largely moved past XML namespaces.

06:51 Because that just is a complication that never needed to exist.

06:54 But there's still plenty of XML stuff out there that people have to work with.

06:58 So here's a cool little library that people can use.

07:01 So we already have XML support in Python through the element tree API, which is pretty nice.

07:09 It works well.

07:09 You give it an XML document.

07:11 But then you do sort of navigate it through like child parent or dot attribute.

07:18 There's like some, you know, it's not totally the cleanest, most straightforward API.

07:24 So with this Untangle thing, you can treat it like a Python object that just dynamically adopts the nodes as fields of those objects as a hierarchy.

07:36 So super simple.

07:37 So super simple.

07:37 You just get it to parse either a URL, a file, or from a string, just an XML string.

07:43 There's an example that is just real simple.

07:45 It has a root and it has a child node.

07:48 And on the child node, it has an attribute where it says name equals child one.

07:53 Okay.

07:53 So if you were to load that up with this Untangle thing, you just, whatever you get back, you say dot root, because that's the first element.

08:01 You wanted to say, for example, that name attribute, say dot root, because that's the name of the root is whatever the first element.

08:07 Then the next element is dot child.

08:09 And then to get the attribute, you just treat it like a dictionary and say bracket name and outcomes, outcomes have value.

08:14 So that seems really, really clean and nice.

08:16 It does look clean.

08:17 Other than the fact you're still dealing with XML.

08:19 Yeah, well.

08:20 I was just dealing with XML this morning.

08:24 Yuck.

08:24 Yeah, absolutely.

08:26 So I suppose there's an API for enumerating things where you have multiple ones, right?

08:32 You could have child, child, child, child with name one, child with name two, right?

08:36 As a list of things in this XML.

08:39 So it's, there's probably, you know, maybe if there's multiple ones, I don't know, maybe it's a list or something.

08:44 I have to look, actually realize I didn't totally look into that, but it's a pretty cool library.

08:49 If people are doing things like that, they can definitely check it out.

08:54 Here we go.

08:54 Maybe.

08:54 Yeah.

08:55 I think I will try this out because I have used elementary and it does the job, but I always thought maybe it could be easier.

09:04 What's the speed like?

09:06 Do you know anything on if it's fast, slow?

09:08 Honestly, I would imagine what it's doing is it's probably using element tree.

09:13 Yeah.

09:13 It's just like a thin wrapper over top of it, but I don't, I don't know.

09:18 Let's see.

09:18 Oh, it's using defused XML.

09:23 Yeah.

09:23 Sax.

09:24 Okay.

09:25 I have no, I don't know anything about Sax XML, but yeah.

09:29 Okay.

09:29 Pretty interesting.

09:31 No, I don't know anything.

09:31 It doesn't.

09:32 Unfortunately, the, the, the read me doesn't have a lot of information about, you know, like its performance and other things.

09:38 But yeah.

09:39 They also have a blog post.

09:40 You can check out about why they built it, which is kind of neat.

09:42 So also sweet, short and sweet, but check it out.

09:46 If that sounds like something that would benefit you.

09:48 That's very cool.

09:49 Indeed.

09:49 All right.

09:50 Over to you, Calvin.

09:50 Well, something else that could benefit you is going to be, my pie is official one.

09:56 Oh, released, I think this was as of about a week and a half ago.

10:00 So it was pretty, pretty new news.

10:02 A ton of stuff here.

10:03 So for those, you don't know, my pie is the static type checker for Python, basically a Python linter on steroids.

10:10 so if you are using type hints, this is going to be a great way for you to make sure your code is, basically ready to go before it even runs.

10:19 something a lot of developers coming to Python from other statically or strongly type languages, really appreciate.

10:27 And I think it's something that a lot of Python developers are moving to, especially with like the surge in populative languages like TypeScript, because the IDEs are really supporting this like so well.

10:38 So it, you, you get a significant improvement in your quality of life.

10:41 If you move to a tool like my pie and start using my Python and Python hints.

10:45 So, type hints in Python.

10:48 there's some things to watch out for.

10:49 So as this new release rolls out, they are changing their numbering scheme.

10:53 It used to be like 0.999, you know, they kept kind of extending this like 0.9, beta release window, but it's actually now 1.0.

11:03 It is not Simver.

11:04 so do not count on minor releases, not having backward incompatibilities.

11:09 although they should be very minimal is what they claim.

11:12 But something you may want to be aware of if you are upgrading to this.

11:16 And when you go in and pin your, you know, my pie version, you want to be really explicit about it because they're going to make sure they announce those significant backward incompatibilities in any kind of blog posts in the release before that feature releases.

11:28 So the idea is...

11:29 Do you say it has a, like a gut feel ver?

11:31 Yeah, yeah, right.

11:32 A season dance ver?

11:33 Like what's the official term?

11:35 Like, yeah, it feels like a big release.

11:36 Let's go ahead and change this part of the version.

11:38 But on that same note, they do have feature flags for this.

11:42 So if there is a new big feature coming, there's going to be a backward incompatible feature.

11:46 The previous release notes will note it.

11:48 The release will have it.

11:49 There'll be a flag to turn it on.

11:50 So you can try it out easily in your workflow.

11:53 See what breaks without having to, you know, downgrade, re-upgrade, downgrade, re-upgrade.

11:57 So you can actually turn it on and off and make sure it works inside your workflow and your process.

12:01 Lots of performance improvements.

12:03 So 1.0 is 40% faster than 0.991.

12:06 There's about 20 optimizations in this list.

12:10 And this is, you can see by my scroll bar, there's a ton of stuff that happened when they went to release 1.0.

12:16 But some more things in here that are nice features, for example, warning about variables before definition

12:21 kind of goes along with this possibly undefined variables.

12:24 So for example, B is never true.

12:26 X never happens.

12:27 You'll never be able to print X.

12:29 mypy will tell you about that before you ever run your code.

12:31 So it's kind of like just a lot of those belts and suspenders, nice safety checks.

12:35 Keep your code nice and clean.

12:37 Combine this with, you know, tools like black and things like that to just have super tight code that is easy to read for everybody.

12:43 Some new features that are supported as of Python 3.11.

12:47 The new self type is supported.

12:49 So if you are having a method that returns an instance of the class, you can use the self type.

12:57 So if you have a sub class, you get the correct sub class return instead of having to explicitly, you know,

13:02 redefine that for every sub class you are doing.

13:05 So more convenience methods for those who are diving deep into the typing system here.

13:10 The self type is so good.

13:12 Yeah.

13:12 It's really, it cleans up a lot of problems.

13:14 It was really janky to, you know, if you're inside the class, you put in quotes, the name of the class.

13:20 Well, how's it supposed to find, you know, that one, or if it's going to create a circular dependency, you've got to do weird things.

13:26 So yeah, that's really, really nice.

13:28 I do wish that Python had like a two stage parser aspect where it would say, go through, here's all the stuff that's defined in the class.

13:37 Oh, here's some type things and other reference.

13:40 We might not know what that is.

13:41 Let's go all through and then go back again and see if we can figure out what that is, you know.

13:45 But the way it works now is when you're in the middle of a class, that class doesn't exist yet because it's not done defining until you're at the end.

13:52 But if you've got to say, I'm going to compare this against another one of me, well, how do you say that in types?

13:57 Self.

13:57 Right.

13:58 That's what we got.

13:59 And I can see this like being rolled in to proper Python.

14:04 It seems like at some point.

14:05 I just feel like types went on from being this pie in the sky idea to bolted on type feature to it's really getting ironed out and really well supported by the community and really well supported by all the editors.

14:18 There's some new stuff in here that I don't know too much about.

14:22 We're not heavy into types yet.

14:26 We're doing a lot with Django and doing some type ins there to start making some APIs.

14:30 But that's where we're still diving in.

14:32 A bunch of new like miscellaneous features.

14:34 So if you are looking for features galore that has happened in mypy 1.0, there's a bunch of fixes to crashes.

14:40 So this seems like it's a really robust release.

14:43 Another important one here is going to be the Python 310 match statement support.

14:48 So it can compile those as well.

14:50 So they're staying on top of all the recent language features, which is obviously what's needed to get momentum behind a project like this.

14:58 I think anybody who's using type ins better be using the mypy stuff.

15:02 For those who don't know, it's also supported by Dropbox.

15:06 So when I said 40% or no, yeah, the 40% faster, that was actually running against the Dropbox main code base.

15:13 So they're actually, that's the benchmark against Dropbox itself.

15:17 Interesting.

15:18 Yeah.

15:18 Yeah.

15:19 mypyC is pretty an interesting thing that opens up once you start to adopt types.

15:22 For me, the main reason I use types in my world, and I realize that I live in a different world than a lot of folks.

15:29 But for me, it's really about making the editor tell me what it should know already and not, do I need to go look in that method?

15:38 Does this actually take, when it says user ID, is that a string or is that a BSON from MongoDB?

15:44 I can't remember what kind of thing this stores for this day, it says, you know?

15:48 Yeah.

15:48 And then, oh, okay, that's obvious, right?

15:49 Just so you don't have to keep bouncing around.

15:51 You just stay in the way you're trying to focus.

15:54 And I love it.

15:54 Yeah.

15:54 It's that extra smart.

15:55 I mean, if you're using a PyCharm or VS Code and you thought it was smart before, once you introduce these types in there, it gets even smarter.

16:03 Absolutely.

16:04 All right.

16:05 Awesome.

16:06 Awesome one.

16:07 Now, two pieces of real-time feedback.

16:08 One, I found out with the untangle that if you have multiple nodes of the same thing, like you would in an RSS feed, you have multiple posts or a blog post, same thing for the posts.

16:20 You just still say root dot, you know, main thing dot child or whatever the name of that node is.

16:26 That becomes a list instead of a single thing if there's more than one.

16:30 Okay.

16:31 So that's pretty interesting.

16:33 And then the other one here is over to you, Brian.

16:37 Jeff asked, can an old project be converted to work with Flit?

16:41 And yes, probably, maybe.

16:43 It really depends on the project.

16:45 Some of it can be re...

16:48 There's not that much to it, right?

16:50 There's PyProject.toml and a few other things, and it's probably not that much work to migrate it, I guess.

16:56 Yeah, but I mean, an old...

16:59 I'm assuming by old project, it's an old setup.py setup tools project, and you could do anything in that.

17:07 So if you were going crazy, you can't go crazy anymore.

17:11 But if it was a fairly simple, straightforward, then yeah, probably.

17:16 If you were going crazy, you get what you deserve.

17:18 How about that?

17:18 So, all right, before we move on to our next topic, let me tell you about our sponsor for this week, Microsoft First Startups Founders Hub.

17:30 Really great program.

17:32 If you have a startup and you want to get tons of support, not just technical support, but also networking with people who have founded their own startups, then you should check out Microsoft First Startup Founders Hub.

17:44 As Brian mentioned last time, we heard back from one of our listeners who signed up and had done several startups and said,

17:50 I was a little skeptical, but this is a really great program.

17:53 It's fantastic for us, and I just wanted to let you know how well it's working out.

17:57 So definitely a good program.

17:58 People will check it out.

17:59 So what it is Microsoft created this program to help people become successful starting a business.

18:06 By some estimates, 90% of startups go out of business within the first year.

18:10 That's not what you want for your startup.

18:12 You want it to succeed.

18:14 So with Microsoft First Startup Founders Hub, you get access to free resources.

18:18 So this includes platform technologies like Azure, GitHub.

18:23 They partner with OpenAI.

18:25 I've heard ChatGPT is catching on, so that's kind of cool.

18:28 So you get access to some of the resources from them, which is fantastic.

18:32 But you also get access to the mentorship network, which I think maybe is even the most important part.

18:38 You get access to a pool of hundreds of mentors across ranges of disciplines, across areas like idea validation, fundraising, management and coaching, sales and marketing, and others.

18:48 So you can go and book one-on-one meetings with these folks, and many of them are former founders themselves.

18:53 So you can make your idea a reality today with the critical support you get from Microsoft for startups, Founders Hub.

19:00 Just visit pythonbytes.fm/foundershub 2022 to go apply.

19:07 The link's in your show notes.

19:08 You don't have to be VC-backed.

19:10 You don't have to be third-party validated.

19:12 You just apply.

19:13 If they like what you want, it's free, and they'll take you on to the program, and you can get started.

19:18 So thank you to Microsoft for supporting our show.

19:22 And with that, Brian, over to you.

19:24 Yeah.

19:25 So go back to the same article for a second.

19:29 Just scroll a little further down the same page.

19:31 Okay, got it.

19:31 So within the same packaging Python projects thing, there was...

19:38 I have a reason to bring this up.

19:39 But anyway, it lists a whole bunch of tools you can use for your build a backend.

19:44 It's got PDM, FLIT, Subtools, and HatchLink.

19:47 There's another page on the same in PyPI or PyPA that is the packaging flow.

19:54 And this one talks about it as more of a high level.

19:57 It's not a tutorial that talks about the high level thing.

20:00 And there, it lists all of those, FLIT, Hatch, PDM, Poetry, Subtools, and then also Trampolin and Way, which I've not heard of.

20:08 So this can cause confusion.

20:11 And that's really what we're talking about.

20:13 Is I have...

20:14 And I'm bringing this up because just independent of my involvement with Python bytes, I've been involved with people having trouble figuring out how to package their Python code independently.

20:27 I've got a web developer doing it, a tools backend person, and then somebody that's just trying...

20:32 He's a C# person that just was trying to package a front end to his tool, a Python front end.

20:38 And he was like, I don't know how to do all this.

20:40 And it is confusing.

20:43 And it even is confusing for seasoned Python developers.

20:47 And here, we've got...

20:48 Oh, I didn't link it right now.

20:50 But there was a Doug Hellman from Module of the Week posted that he's published lots of packages.

20:59 And he went to do it again and was a little lost and wanted some advice.

21:03 So new people or people that are outside of the scope are confused.

21:09 And I think it's fair to recognize that.

21:12 And so there's a lot of discussion around it.

21:15 And Pradyan put together a blog post, was talking about it on Disqus.

21:22 And he put together an article called Thoughts on the Python Packaging Ecosystem.

21:27 And I think it's a kind of a fair discussion around it.

21:30 And one of the points is that a lot of Python users are not software engineers.

21:37 But we also have to recognize that a lot are.

21:40 So it's a diverse ecosystem.

21:44 And I think it's going to be a mess.

21:47 It's going to be a mess for a while.

21:48 And I think it's going to be...

21:50 I think it's good that it's a mess in some cases.

21:53 It's good that we have...

21:54 We don't just have setup tools now.

21:56 We have a lot of ways that are working on different pieces of the problem of, you know, how to make it easier around PyProject.atomla.

22:07 And you can have a different build backend.

22:09 He breaks down the points.

22:12 It's a big article.

22:13 But one of the things is it kind of unintentionally, the packaging ecosystem unintentionally became a competitive space that it is today.

22:22 And maybe we need to make an...

22:25 The community needs to make an explicit decision if we should continue operating under that model that led to what we have now.

22:32 We could pick from in different tools to do in different things.

22:37 That's a good model, according to this.

22:40 Or in approximately the same choices to get a really bad user experience.

22:46 Bad for some people, good for others.

22:48 And then, you know, also maybe we should pick a default because picking a default doesn't make the other approaches illegal.

22:55 They just aren't the default.

22:57 And I'm kind of leaning towards that.

22:59 I think it would be good to have something, but I'm not sure what.

23:03 Longer down the line, pardon me, of the scroll, is PIP.

23:09 One of the top ideas is maybe we should have PIP.

23:12 Well, okay, I'll back up a little bit.

23:14 Some of these tools like Poetry and PDM are workflow tools.

23:18 And other tools like Flit are more build.

23:23 Flit's more like I want to build, just build a wheel.

23:26 It also does publish, but mostly the workload is around building the wheel.

23:31 Whereas PDM and Poetry are like this whole workflow thing, control your virtual environment, do a whole bunch of stuff.

23:39 And I know a lot of people from other languages like JavaScript, maybe Go, I don't know, are used to these high-end workflow things that do everything for you.

23:51 But we're already used to a multi-tool model now, so I'm not sure how to get from here to there.

23:56 It would be interesting to have pip do it.

23:59 I mean, we already all use PIP.

24:01 So could pip expand to do both the focus thing it's doing now, installing a package, and then also do some workflow stuff?

24:10 And I guess I'd be kind of open to explore that idea, but I was curious what you guys thought.

24:16 I'd be open to explore it.

24:17 Not to disparage anything, but I personally dislike the things that try to put their workflow onto you.

24:22 I'd much rather have something simpler.

24:25 And as someone who teaches Python, this is a huge problem for people getting into language.

24:29 They feel like it's a big challenge.

24:31 Why is it so hard?

24:33 Why do I have to learn all these things?

24:34 I picked the wrong thing, and now am I screwed?

24:36 No, you're not out of luck.

24:38 You just delete your virtual environment and start over.

24:41 But yes, things are a little...

24:42 I think having many of them, one, is a problem.

24:47 And I think something that forces workflows on you, I'm not a huge fan of.

24:53 I feel like we've already got lots of tools that solve it pretty well.

24:57 I'm kind of not against pip doing a little bit more.

25:00 I use pip-tools for managing my dependencies, and it is so good.

25:04 It is so good.

25:06 You state just your top-level dependencies, and when it's time to...

25:09 Let's see what new.

25:10 Let's make sure we upgrade to the new things.

25:11 You just run pip-tools upgrade, and it'll regenerate.

25:14 You can evaluate what the changes are.

25:15 I really, really like pip-tools.

25:17 I think it's a much simpler scenario than a lot of them.

25:21 And I know other people really like it.

25:23 Anyway, this is what I like.

25:25 We could possibly leave pip alone and expand pip-tools to do the rest of the workflow stuff.

25:29 Yes, potentially.

25:30 Yeah, we could, but I kind of agree with that.

25:33 Like, Unix small tools do one thing really well, which is why I've kind of gone with the

25:38 pip-tools approach as well, because I really feel like pip-tools does such an incredible

25:42 job at that dependency resolution and getting exactly what you need way better and way faster

25:48 than pip itself does.

25:49 I think that approach that, like, Flit takes, it seems like...

25:52 I've never used Flit, but the fact that it can just wrap around pre-existing tools to

25:56 provide a common interface for users to be able to build, package, push the whole deal

26:03 seems, like, really nice because it guides them through step-by-step that process until

26:08 maybe at some point they don't need it or they just enjoy it, and it's using some of those

26:12 standard tools behind the scenes.

26:13 Yeah.

26:13 But also with Michael, I don't like the predefined workflows that, like, I'm not a huge fan of

26:18 poetry necessarily because it just doesn't fit the way my brain wants.

26:22 the work down a project.

26:23 And I'm not surprised.

26:24 So I was thinking an analogy of, like, if you're going to have a pocket knife, which tools should

26:31 be in the pocket knife?

26:32 And if you go into, like, the medium pocket knives on the Victorinox site...

26:36 Oh, no, it won't load.

26:37 You get, like, tons...

26:38 Even the small ones.

26:39 So the medium ones, you get, like, tons of different tools.

26:42 Which tools should be there?

26:43 Nobody's going to agree.

26:45 And I thought, well, at least in the small ones, maybe people can agree.

26:49 Because, oh, no, not even in the small ones, people can't agree.

26:52 Which should go together.

26:54 So, yeah.

26:55 I want my Swiss Army knife to also have drywall tools and something from my car.

27:01 But it seems like the combination, though, of really small tools that do one thing really

27:06 well with some tools to layer on top of it, whether it's your IDE, whether it's FLIT.

27:11 And you can have the opinionated workflows layered on top of it.

27:15 They rest on top of a solid set of identical tools that produce the identical thing.

27:20 Yeah.

27:21 I'm in that camp as well.

27:24 That's why.

27:24 But I totally get that that's confusing for people.

27:27 Oh, totally confusing.

27:29 The first time I go to explain to someone how they're going to manage, like, their

27:32 dependencies on a Django project, I'm like, oh, my gosh, I have to step back and think about

27:36 the 20 steps that kind of go into how I learned to do this that I'm now explaining to some new

27:41 person.

27:42 I'm like, I'm sorry, this doesn't sound easy at all.

27:44 There's got to be a better way.

27:46 But I don't know if there is.

27:47 Yeah.

27:48 Anyway.

27:49 Okay.

27:49 Rabbit hole.

27:51 But.

27:51 Oh, totally.

27:52 I mean, Python packaging is the ultimate rabbit hole in the Python community.

27:56 Yeah.

27:56 Speaking of packages.

27:58 Packages.

27:58 Michael, do you want to jump into yours?

28:01 I do want to talk about more packages.

28:03 This is the simplified version, but also a bit of real time follow up as well.

28:07 Again, I'm having many of the folks that were on that discuss thread that you mentioned,

28:13 Brian, about this topic on Talk Python probably in a couple of weeks as a big panel discussion

28:18 about it.

28:19 So if you all want to hear more of this, but for a long time with the folks who are participating

28:24 and dive on in.

28:25 Okay.

28:26 Oh, also, apparently, I said chat GTP, right?

28:32 And I meant ChatGPT, as Sharaf pointed out.

28:35 So thanks for that.

28:36 That's the one I meant.

28:38 You know the one.

28:38 All right.

28:39 So there's this cool GitHub page, I guess, called Top PyPI Packages.

28:44 And so this is a pretty neat place.

28:47 It shows you the top 5,000 most downloaded PyPI packages updated monthly.

28:53 So you can scroll down here and you can say, show me the top 100, show me the top 1,000 or

28:59 whatever.

29:00 I'll pull up the 1,000 for you, Brian.

29:02 There you go.

29:03 So these are updated over the last 30 days.

29:05 And it actually says when it was last updated.

29:07 And that was February 1st, which is a week ago as of this recording.

29:11 So that's pretty fresh, really.

29:13 And so you can find things like pytestCheck.

29:16 Is that in here somewhere, Brian?

29:18 It's in the top 5,000, but not the top 1,000.

29:21 Oh, okay.

29:22 Did I get the wrong one?

29:23 Sorry.

29:23 I tried to pull you up there.

29:24 But you see things like Boto3 and URLlib are the most popular.

29:29 So Boto3 is downloaded 522 million times every month, every 30 days or so, which is just insane.

29:38 You got to be careful because some of these are really popular dependencies rather than directly really important libraries.

29:44 Like I use Boto3 all the time as a dependency, less often directly.

29:49 Definitely for URLlib3, I don't think I've ever used it directly, but obviously it gets installed.

29:53 But I'm going to use many things.

29:55 So this is pretty cool if people want to check it out.

29:58 Calvin, what do you think?

29:59 Oh, I'm excited about this too because just thinking about the number of times the Amazon cloud must go download Boto3 from PyPy.

30:08 Mind-blowing.

30:10 Like there's all the serverless that's on server someplace downloading packages in the background.

30:15 Now you get a feel for it with a big number.

30:18 Yeah, it's really nuts, which is also a little bit ironic because it comes from AWS and they send it over to PyPI and then it comes back to AWS.

30:24 It's a weird world.

30:28 Don't they use Fastly for their CDN?

30:31 I think they use Fastly.

30:32 PyPI?

30:34 Yeah.

30:34 I believe so.

30:36 Yeah.

30:36 It's like they have to go to a competitor to get like...

30:39 Yes, exactly.

30:39 Yeah.

30:41 How interesting.

30:42 So this is interesting in and of itself.

30:44 And it tells you which ones are there.

30:45 And there's an API which you can hit.

30:46 Let's not hit the minified one.

30:48 Let's hit the non-minified one.

30:50 And it'll tell you actually returns when it was updated, what the query was, et cetera, et cetera.

30:55 And so you actually can get this as a JSON endpoint, which is pretty cool.

30:58 It's just a static file that gets updated every couple of days on this GitHub page setup.

31:04 So that's also an interesting way to think of an API.

31:06 My API is a static file that periodically gets refreshed.

31:10 But there it is.

31:11 So you don't have to go hard on much there.

31:13 I kind of love it, though.

31:14 I kind of do, too.

31:15 Just a static file.

31:16 Yeah.

31:17 Yeah.

31:18 Think how scalable that bad boy is.

31:20 Okay.

31:20 Especially if you put a CD in behind it.

31:23 And then related to that by the same person, there is Hugovic top PyPI packages on GitHub,

31:31 which is a regular dump of the most downloaded packages from PyPI.

31:34 So that sounds like, yeah, Michael, you told us about that.

31:38 Thanks a bunch.

31:38 No, it's not the names.

31:41 It's all of it.

31:42 It's all of the data that is here.

31:46 Where do they have it?

31:47 This one?

31:48 I don't know if I want to open this up.

31:49 But it's really big.

31:50 So it's all the XML.

31:52 I'm not sure where it is in here.

31:54 But it's all the XML files of, or rather, XML.

31:58 The JSON files that Untangle got me thinking XML.

32:01 So it's all the JSON files of what comes out of the PyPI API.

32:06 But all of them.

32:07 It's when I checked it out and unzipped it, and it was like 22 gigs on disk.

32:12 So this is a big beast right here if you download it.

32:16 Actually, no, this is not the right link.

32:18 I'll have to find it.

32:19 I'll put it in the show notes.

32:20 But there's this guy.

32:23 Darn it.

32:23 This is the wrong one.

32:24 But there's also, linked off of this, there's one where you get all of the packages, which

32:29 is all the JSON for every package.

32:31 I'll have to figure out where I got that from, unfortunately.

32:33 It's the wrong link I pulled up.

32:35 But anyway, these are both really cool resources.

32:38 And you'll have to check the links for that other one.

32:40 Cool.

32:40 Yeah, definitely.

32:41 Yeah.

32:41 All right.

32:42 Calum?

32:43 Coming in at 57 on that list for the last 30 days is SQLAlchemy.

32:49 So for those of you who aren't familiar and do like Swiss Army Knives, this is the Swiss

32:53 Army Knife of database tools.

32:55 It's not only an ORM, but it has a whole SQL toolkit built into it.

32:59 So if you've never touched SQLAlchemy, but do deal with databases, you may want to check

33:03 this out.

33:03 And more importantly, and this kind of ties in with my previous announcement on the mypy

33:07 1.0 release.

33:08 There's a lot more support in this release for types and typing.

33:13 So there's kind of native out of the box mypy experience has been improved.

33:18 Now, the author of SQLAlchemy, Mike Baer, is actually going to be joining us in a few weeks

33:24 here at Python Web Conference and giving a tutorial length session on SQLAlchemy.

33:29 He did this last time two years ago when he released 1.4.

33:32 So for those of you who've kind of been following along in the background, the 1.4 release was

33:36 kind of the first half of this major revamp of SQLAlchemy into this new world.

33:41 And so 2.0 is the second half of that revamp.

33:44 So it's really been like four years coming for this major release of SQLAlchemy.

33:49 So if you want to know what's new and hear it from the horse's mouth, Mike Baer is going

33:53 to be joining us and he'll be happy to answer all of your questions.

33:57 Let's see.

33:58 There's a ton of new things released in here.

34:01 I think if I scroll down.

34:03 Yeah.

34:03 So plugin free, 484 compatible ORM syntaxes.

34:08 So again, kind of ripping out the old stuff.

34:10 All the typing support in here is all fresh.

34:14 If you are using the mypy plugins from before, you're going to want to take all those out.

34:20 They're considered deprecated at this point.

34:22 So that's something that's very important for you to consider.

34:25 But now SQLAlchemy 2.0 is using all the new Python 3.0 wonderfulness of like data classes,

34:31 enums, inline annotations.

34:33 It's just really a nice modernization of a really powerful package.

34:38 See what else was in here?

34:39 There was a super importance.

34:41 Oh yeah, some performance stuff.

34:42 So this all new fully integrated ORM approach to bulk inserts is quite interesting.

34:48 You will get a major speed increase, except if you're using MySQL.

34:52 The MySQL folks have not implemented the insert returning syntax.

34:57 But if you're on MariaDB, you're good to go.

34:59 So that's something to pay attention to.

35:01 In addition to that performance, there's also the bulk optimized schema reflection.

35:06 So if you were using SQLAlchemy, it was basically doing a table by table reflection.

35:12 If you were asking it for metadata about the tables.

35:15 Now in all of the backends, there is basically a...

35:18 Oh, sorry.

35:20 I think it's only for Postgres and Oracle right now.

35:22 So Postgres and Oracle support the bulk schema upgrades.

35:26 You can see right here for Postgres, a 250% improvement.

35:29 And for Oracle, a 900% improvement.

35:32 I'm not sure what it was doing before, but it's a heck of a lot faster now.

35:36 At 900% improvement.

35:37 Always a benefit there.

35:39 And in addition, I'll kind of point out some of the things that were in the migration guide

35:44 and the what's new in 2.0 guide.

35:45 You're going to want to make sure you check those out here.

35:48 There's like the migration process linked from this article or blog post.

35:52 You're going to want to make sure you read through those because there's major incompatibilities

35:55 from 1.4 into 2.0.

35:57 But the migration guide is really similar to like the 2 to 3 guide Python had.

36:03 Tells you how to mechanically go through and make sure you're going to be compatible for

36:07 the latest version of SQLAlchemy.

36:08 Another speed improvement here is that all the native C extensions have been ported to Cython.

36:14 Some of them are...

36:16 They're all as fast.

36:17 Some of them are slightly faster than their previous C extensions.

36:20 But this also removes risks of memory corruption and stomping on each other and things that

36:26 C does really well that you don't want.

36:28 So that's also a nice benefit.

36:29 And then the kind of tying into that same benefit is the SQLAlchemy is now PEP 5.17 enabled.

36:36 So that means the PyProject.toml file that we just saw in the last couple of announcements

36:41 is supported, which means when you're on your local machine and doing a source build of SQL

36:46 Alchemy, it can automatically go grab some of the dependencies such as Cython, which can

36:51 be tricky to get installed for some people.

36:52 So that is simplifying a lot of people's lives to go to SQLAlchemy 2.0.

36:57 Yeah, SQLAlchemy 2.0 is a huge release.

36:59 Yeah, it's a big deal.

37:01 It's been years in the coming.

37:02 And then it was in this intermediate 1.4 mode where you could choose either API and

37:08 they finally removed some things.

37:09 But to me, the biggest deal is the async.

37:12 Yeah.

37:12 Right.

37:13 It's in the prior versions, there was just no way to do any async things, which meant

37:17 on all the scalable areas, you were kind of stuck trying to find somewhere else to go.

37:22 Now, if you really like SQLAlchemy, you can just use the async API.

37:25 Yeah.

37:25 I think a lot of people are going to benefit from those types, all the type support in mypy

37:29 native support.

37:30 Yeah, absolutely.

37:31 And game back to your editor.

37:32 The Vulcan search have always been a hassle in ORMs.

37:36 So it sounds like that's really gotten a lot of love too.

37:39 Yeah, definitely.

37:40 I look forward to it.

37:41 I do as well.

37:43 All right.

37:44 Well, that's all of our topics.

37:46 Brian, what do you got for us as an extra?

37:49 I guess just one quick extra is like one of the things I've been working on on the side

37:55 is I'm going to create a new build backend for Python and a new workflow tool.

38:00 No.

38:01 I'm not.

38:04 But what will you name it?

38:08 Well, I think flick because I always thought flit should have been flick.

38:12 So like a little football.

38:14 So yeah.

38:15 There you go.

38:16 So how about let's go to Calvin's extras?

38:19 Yeah.

38:20 Sure.

38:20 So I kind of mentioned this in the intro, but we are putting on the Python Web Conference.

38:24 This will be our fifth annual.

38:26 So big five years.

38:28 I'm wearing one of the jerseys from last year.

38:31 You can't see it, but it was really awesome last year.

38:34 We had 90 speakers.

38:35 We had, I think it was over 400 attendees, something like 23 or 22 time zones.

38:42 Like it was really a global event.

38:44 That's one of the things I'm really proud of.

38:46 This event was meant to fill a hole, I felt like, in the Python community for folks who couldn't

38:51 travel, couldn't afford to go to an in-person conference or wasn't a regional conference near

38:55 them.

38:56 I know the PSF is doing a lot of work right now and getting some regional conferences into

38:59 other areas, but this is still, and will always be virtual and always be accessible to folks.

39:05 So if you go and register and you can't afford to pay for this, we do have a grant program.

39:09 We want to make sure that if you have a need to be in this room, you should be in this room

39:13 and everyone is included.

39:15 This year is going to be no exception.

39:16 There's a lot of amazing speakers who have joined us.

39:19 It'll be five half days from the 13th to the 17th.

39:22 There'll be a keynote speaker kickoff every day, a keynote speaker at the end of every day.

39:26 And there's some awesome speakers coming up.

39:28 We've got some people from GitLab.

39:29 We've been focusing on finding some folks from climate initiatives to actually come and give

39:34 some of our talks where they're using technology for good in these areas.

39:38 There'll be a set of tutorials as well.

39:40 So I mentioned Mike Baer's tutorial already, but there'll be three or four other tutorials

39:45 as well.

39:46 I know Matt Harrison is going to be giving one of the tutorial, not tutorial, he'll be

39:49 giving a keynote this year.

39:51 So he's back again from last year.

39:53 He gave a keynote and a tutorial.

39:55 Man's the hardest working man in Python.

39:58 But we really appreciate him joining us as well.

40:01 And if you've got questions, make sure you just go to pythonwebconf.com.

40:05 You can also find me on CalvinHP on Twitter.

40:08 There's also a Twitter account for Python Web Conf.

40:10 So you can follow Python Web Conf on all the socials and join us.

40:14 There'll be socials actually during the event as well.

40:16 So it should be tons of fun.

40:17 We'll have just everything for the community.

40:21 So I have a question right off the bat.

40:22 It says webconf.

40:24 If I'm not a web developer, am I still going to be interested in this?

40:27 Or is it primarily web development?

40:28 That's an excellent question, Brian.

40:30 We originally, the webconf name meant a conference for web people.

40:35 But we switched it around, you know, because we felt like this was an opportunity for the

40:40 conference for Python to be on the web.

40:42 And now we have app dev track, which is like not more than just web.

40:45 We'll have a cloud track.

40:47 We will have a PyData track.

40:49 It's actually officially sanctioned PyData event.

40:53 And we have a culture track in addition to the tutorial.

40:56 So there is actually something for everybody in the Python world here.

41:00 And we're really trying to, again, I mentioned filling the gap for people who can't make it

41:04 to specific conferences.

41:05 We're also trying to fill the gap for talks that I feel are a little more on the intermediate

41:09 to advanced level.

41:11 If you're looking for very entry-level tutorials, this may not be the conference for you.

41:16 I mean, there'll be great community, lots of questions you can ask.

41:19 There'll be tutorials.

41:19 But I really wanted to make sure that the talks kind of scaled up that next notch.

41:24 When I go to a lot of regional conferences or even PyCon, there's some entry-level stuff

41:30 there that's great for people who are new to Python.

41:31 This is that next step.

41:33 Cool.

41:34 I don't need to cut you off, but can we see if Michael's got any extras?

41:39 Yeah.

41:39 Yeah.

41:39 I do.

41:40 One, while Calvin's thing is up, I'm going to be speaking at that there conference about

41:46 making your Python web apps fly with CDNs.

41:49 Speaking of CDNs earlier.

41:50 So I'm really excited about that one.

41:51 That's a fun topic.

41:53 But I want to tell you a couple tips and tricks.

41:56 I'll keep these short here, but they're kind of fun.

41:58 So I recently discovered that with many of the browsers like Vivaldi or Chrome, sadly not

42:06 Firefox.

42:06 Honestly, that's one of the reasons I quit using Firefox.

42:09 They support progressive web apps, which many people may know.

42:12 But even if you have a thing that is not a progressive web app, you can install it as a standalone,

42:18 from your perspective, application that lives in the dock.

42:21 So the thing I got on my screen and I linked to in the show notes is here's my Vivaldi, but

42:26 then I have my different mail clients.

42:28 I got ProtonMail and Gmail and I got the Google Calendar.

42:31 All of those are just regular web apps.

42:33 I don't think any of them are progressive web apps.

42:35 But if you right click on the tab, you say create shortcut and say open in a window, they

42:39 become their own little app that you put in your dock or your taskbar or whatever.

42:43 So for example, like here's ProtonMail running.

42:47 You can see, again, link in the show notes to the picture.

42:49 And it just looks like an app.

42:51 Like you would know it was an app.

42:52 You'd think it was an ElectronJS app, basically.

42:54 So if you want to have those as separate things, you're going to have running an Alt-Tab or Control-Tab

42:59 over to you.

43:00 There you go.

43:00 Oh, no, I closed my browser and my mail's gone again.

43:03 I got to open it back up.

43:04 If that ever frustrates you, here's a cool fix for it.

43:06 All right.

43:08 Also, speaking of Proton, I started using SimpleLogin.

43:10 Have either of you all used this?

43:11 No.

43:12 So I have a pro account at Proton, which is like, I don't know, a couple dollars a month

43:17 for a bunch of years if you pay away in advance, I think.

43:20 It's not very much.

43:21 But along with that comes this thing called SimpleLogin, which every time you go to a

43:26 website, I don't think I have the extension in this profile in my browser.

43:30 But if you go to a website, it suggests, hey, we'll create an automatic fake one-off email

43:35 address that gets sent to wherever you decide it gets sent to.

43:38 And you can reply to it and it goes back through there and things like that.

43:42 It's really cool.

43:42 So if I went to arstechnica.com and I created it, it would be like some random thing at

43:48 arstechnica-simple-logon-ish.io or whatever.

43:53 So you kind of know where it comes from.

43:54 But if you want to just delete that and make it mail stop or you can figure out if it's sold

43:59 or whatever.

44:00 So anyway, recommendation on checking that as well.

44:02 It's also open source.

44:04 You can host it yourself if you like hosting email.

44:07 I don't really feel like doing that.

44:11 If you've heard about a new topic, if you've heard about all the crazy stuff with chips,

44:17 the Chips Act in the US, similar stuff in Europe.

44:20 If you've heard about Biden banning chips or the US government banning chips to be used in

44:26 certain things, primarily in China.

44:29 But there's a bunch of angles in which this kind of stuff is heating up.

44:34 Not just with the US, but primarily around the US.

44:37 There's an incredibly good documentary on YouTube called US vs. China, The War You Can't See

44:44 that talks about the history of this.

44:46 And it just reminds me of how much good stuff there is on YouTube.

44:49 I mean, I know there's a lot of stupid cats falling off of chairs and like idiots.

44:54 But hidden in there, there are some...

44:56 This is a better documentary than you would find on professional top tier television networks.

45:02 It's really good.

45:04 Anyway, if that's interesting to you.

45:06 I will second that documentary note.

45:09 I have a 15-year-old here who's obsessed with long form YouTube.

45:13 And some of the things he has shown me has just blown my mind.

45:16 Like why the superconducting super collider didn't succeed.

45:20 Eight hours of like documentary on that.

45:22 I was like, what?

45:23 Yeah, there is no more Discovery Channel.

45:25 There's no more History Channel.

45:27 Yeah.

45:27 I know they exist in name, but not in what they were, where they actually have real science and

45:31 real information.

45:32 They're just reality TV and other silly stuff.

45:34 So, all right.

45:36 I know we'll get short on time, Brian.

45:37 So, I'll go quick.

45:38 Talk Python's hiring.

45:41 So, I'm looking to hire somebody that does Flutter and Dart, not Python.

45:46 Because if it's Python, I'd be doing it myself.

45:48 But if you're out there and you have experience with publishing apps and Flutter apps in particular

45:52 to the various app stores, if you're used to working on like Android and iOS and others with Flutter,

45:59 and you can show me an app in the app store, I would love to work with you.

46:02 I'm going to post this somewhere pretty soon, but I want to put it out to the community first

46:07 because I'd rather hire somebody from our community than just some random person off the internet.

46:11 And it's primarily to do a complete rework of our mobile apps for Talk Python training.

46:15 So, that'll be a lot of fun.

46:17 If people are interested, just email me, michael at talkpython.fm.

46:22 That's it.

46:23 Those are all my extras.

46:24 Yeah.

46:25 Nice.

46:25 Calvin, thanks for being here.

46:28 Thanks for having me.

46:29 I really enjoy it.

46:30 It's lots of fun.

46:31 I get to go learn new things each time I come.

46:33 Yeah, we do every time we show up as well.

46:36 It's a lot of fun.

46:36 Yep.

46:36 All right.

46:36 And Brian, thanks as always.

46:38 Thank you.