Transcript #322: Python Packages, Let Me Count The Ways

00:00 Hello and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds.

00:05 This is episode 322 recorded February 7th, 2023. I'm Michael Kennedy.

00:11 And I'm Brian Okken.

00:12 I'm Calvin Hendricks-Parker.

00:14 Welcome Calvin. Awesome to have you here.

00:16 I'm excited to be back. I love this kind of my annual retreat.

00:18 That's right. That's right. Your annual drop in to share all your ideas with us.

00:23 You got some good ones to talk about, I know.

00:25 Also want to say thank you to Microsoft for Startups Founders Hub for sponsoring this episode.

00:31 If you want to connect with any of us, me, Brian, Calvin, or even just the show, you can find us over on Fostedon.

00:38 We're all over there actually on Mastodon.

00:40 So you can find links and show notes.

00:41 And if you're listening, but not watching, just go to pythonbytes.fm/stream/live where we will be streaming live every Tuesday, usually, but you can definitely subscribe to get notified there.

00:53 Calvin, I know you've been here before, multiple times.

00:56 You always come to put us to shame with your fancy camera that you got going, we love it.

01:03 Tell people who maybe haven't heard about you or met you before who you are.

01:06 - Sure, so I'm Calvin X. Parker.

01:08 I'm co-founder and CTO of Six Feet Up.

01:10 We are a software development company that helps digital leaders transform their impact.

01:16 We do a lot of interesting work now around impactful projects.

01:19 So it's like climate change, making the world a better place.

01:22 In addition, I've started a couple of communities.

01:25 So the IndiePy community here local at Indianapolis, and then we branched out from that into some little local conferences into now Python web conference.

01:33 So that's why the big thing you'll hear from me, at least this month and next month.

01:38 >> Yeah, we'll talk about that at the end of the show.

01:40 But that's an amazing conference you have.

01:41 I would say potentially more talks or on par as many talks as US PyCon.

01:47 It's really a lot going on there.

01:49 >> Last year, there were almost too many.

01:51 It was so good, I just kept saying yes to everybody.

01:54 This year, we pulled back a little bit.

01:56 We had more submissions.

01:58 We'll talk about this later, but it was really hard to choose.

02:01 >> Yeah, it's tough to say no to deserving people.

02:03 >> It's so much talent.

02:05 So many great people.

02:06 >> Indeed. All right, Brian, kick us off.

02:09 >> I just wanted to talk about packaging today a little bit.

02:12 I'm splitting it up into two segments, but I really want to focus right now on the PyPA has a packaging Python projects tutorial page.

02:22 It's linked in the show notes, of course.

02:25 But I think it's really pretty good, well-written.

02:29 I want to point people in this direction.

02:31 If you want to share your Python code with somebody else, packaging it is the right way to go, I think.

02:38 This tutorial walks through, it's not necessarily trivial to do it.

02:44 I wish it was easier, but it's not bad.

02:47 It walks through setting up your directory structure.

02:53 Then you have to have some package files with it.

02:56 You've got to have a license file and a pyproject.toml now, and a readme, and then it talks about your source directory and your test directory.

03:05 You've got the directory set up.

03:07 >> I know you're a fan of the source directory for packages.

03:10 It looks like-

03:11 >> Yeah, IPA is real.

03:13 >> Exactly. Your philosophy is winning out here is the official recommendation.

03:17 >> Yeah. I used to have good reasons behind it.

03:22 Now I just like having the top-level stuff just be packaging and having the actual source code be somewhere else in the source directory.

03:30 While you're developing, it's nicer.

03:33 If you have a single, there is caveats though.

03:36 If you have a single file that you're just sharing, that might seem like overkill.

03:41 I'm fine with, of course, I have packages as well that are single file that I don't do this with.

03:47 So you don't have to, but most projects are gonna have more than one file if you're sharing it.

03:53 So there's this, and then what do you put in it?

03:56 So it talks about what you put in the different things in the pyproject.toml and a little bit into the readme and some of the metadata.

04:04 One of the things we're gonna jump into a little bit later in the show is that there are lots of options for what build backend you use.

04:14 This project itself or this document talks about Hatchling, SetupTools, Flit and PDM.

04:22 And neither of them look that different.

04:24 And they kind of do the same thing, but you can have different metadata that goes with it.

04:28 So that's, I think both a good thing that we have options and a bad thing that some people don't know how to choose.

04:35 So--

04:36 - Is it really tough to like know why should you do it?

04:39 Maybe setup tools probably shouldn't be the choice.

04:42 I feel like that--

04:43 - Yeah, I don't know.

04:44 - Moving a little bit away from that, but--

04:45 - The history here is so deep.

04:47 Like, I feel like there's like a whole day, a whole college course you could give on why these things are the way they are now.

04:54 - Yeah, yeah.

04:55 So the project, the tutorial goes down into even distributing.

05:01 So all the way down to recommending that you use Twine to upload to the test repository, test pipe here.

05:10 And you have to get a token and it's not trivial and all this sort of stuff.

05:15 So it's still a little daunting, but this will walk you through all the steps.

05:20 I have a personal opinion, of course, and my opinion is read this and then check out Flit.

05:26 Because for simple projects, Flit is, I think, still the easiest.

05:31 It replaces the build step with Flit build, and it replaces the creation of some of these files with flit init, and it replaces the twine upload with just flit publish.

05:44 And I use it on a few projects and I haven't had reason to switch yet.

05:49 It's not perfect, but it's pretty easy for small projects.

05:54 So I would say read both of these if you wanna share some Python code.

05:58 - Excellent, yeah, very good recommendation.

06:00 And I think people need guidance here because there are so many choices.

06:04 There's going to be a lot of conversation in the Python space around this, actually.

06:07 - For sure.

06:08 - Yep.

06:09 - All right, let's move on though for right now.

06:12 - Real time follow up here.

06:14 As of about 30 minutes ago, Ofek, the creator of Hatch, and I are going to have a conversation March 1st on Talk Python.

06:22 So if people want to hear more about Hatch, that's coming up in about three weeks or so.

06:27 But what do I want to talk about?

06:29 Something fun.

06:30 Last time we had the Unrapple.

06:32 This time, we have the Untangle.

06:35 So Untangle by stchris on GitHub converts XML to dynamic Python objects.

06:43 So thankfully, we've largely moved past XML.

06:47 And really thankfully, we've largely moved past XML namespaces, because that just is a complication that never needed to exist.

06:55 But there's still plenty of XML stuff out there that people have to work with.

06:59 So here's a cool little library that people can use.

07:02 So we already have XML support in Python through the element tree API, which is pretty nice.

07:09 It works well.

07:09 You give it a XML document.

07:12 But then you do--

07:14 you sort of navigate it through like child, parent, or dot attribute.

07:19 There's like some-- it's not the totally cleanest, most straightforward API.

07:24 So with this untangle thing, you can treat it like a Python object that just dynamically adopts the nodes as fields of those objects as a hierarchy.

07:34 So super simple, you just get it to parse either a URL, a file or from a string, just an XML string.

07:44 There's an example that is just real simple.

07:45 It has a root and it has a child node.

07:48 And on the child node, it has an attribute where it says name equals child one.

07:53 So if you were to load that up with this untangled thing, you just, whatever you get back, you say dot root because that's the first element.

08:01 You wanted to say, for example, that name attribute, say dot root, because that's the name of the root, is whatever the first element.

08:07 Then the next element is dot child.

08:09 Then to get the attribute, you just treat it like a dictionary and say bracket name, and outcomes that value.

08:15 So that seems really clean and nice.

08:17 >> It does look clean, other than the fact you're still dealing with XML.

08:19 >> Yeah, well.

08:21 >> I was just dealing with XML this morning. Yuck.

08:24 >> Yeah, absolutely.

08:26 So I suppose there's an API for enumerating things.

08:31 We have multiple ones, you could have child, child, child, child with name one, child with name two, as a list of things in this XML.

08:39 So there's probably maybe if there's multiple ones, I don't know, maybe it's a list or something I have to look, actually realize I didn't totally look into that.

08:46 But it's a pretty cool library if people are doing things like that.

08:52 They can definitely check it out. Here we go.

08:54 >> I think I will try this out because I have used the elementary and it does the job, but I always thought maybe it could be easier.

09:04 >> What's the speed like? Do you know anything on if it's fast, slow?

09:08 >> Honestly, I would imagine what it's doing is it's probably using the elementary and just like a thin wrapper over top of it.

09:16 But I don't know. Let's see.

09:20 Oh, it's using defused XML.

09:24 >> Yes.

09:24 >> SACS. Okay. I don't know anything about SACS XML, but yeah, okay. Pretty interesting.

09:30 No, I don't know anything. Unfortunately, the readme doesn't have a lot of information about its performance and other things.

09:38 But yeah, they also have a blog post you can check out about why they built it, which is neat.

09:42 So also short and sweet.

09:45 But check it out if that sounds like something that would benefit you.

09:48 >> That's very cool.

09:49 Indeed. All right, over to you, Calvin.

09:51 Something else that could benefit you is going to be mypy is official 1.0 released.

09:57 I think this was as of about a week and a half ago.

10:00 So it's pretty new news. A ton of stuff here.

10:03 So for those who don't know, mypy is the static type checker for Python, basically a Python linter on steroids.

10:10 So if you are using type hints, this is going to be a great way for you to make sure your code is basically ready to go before it even runs. Something a lot of developers coming to Python from other statically or strongly typed languages really appreciate. And I think it's something that a lot of Python developers are moving to, especially with the surge in populative languages like TypeScript, because the IDEs are really supporting this so well. So you get a significant improvement in your quality of life if you move to a tool like mypy and start using my Python and Python hints. So type hints in Python. There's some things to watch out for.

10:49 So as this new release rolls out, they are changing their numbering scheme. It used to be like 0.999. They kept kind of extending this like 0.9 beta release window, but it's actually now 1.0. It is not Simver. So do not count on minor releases not having backwarding compatibilities, although they should be very minimal is what they claim. But some of you may you may want to be aware of if you are upgrading this.

11:16 And when you go in and pin your mypy version, you want to be really explicit about it because they're going to make sure they announce those significant backward incompatibilities in any kind of blog posts in the release before that feature releases.

11:28 So the idea--

11:29 - You say it has a gut feel-ver?

11:32 - Yeah, yeah, right.

11:33 - Instead of the official term, like, "Yeah, it feels like a big release.

11:36 Let's go ahead and change this part of the version." - But on that same note, they do have feature flags for this.

11:42 So if there is a new big feature coming that is going to be a backward incompatible feature, the previous release notes will note it, the release will have it, there'll be a flag to turn it on so you can then try it out easily in your workflow, see what breaks without having to downgrade, re-upgrade, downgrade, re-upgrade, so you can actually turn it on and off and make sure it works inside your workflow and your process.

12:02 Lots of performance improvements.

12:03 So 1.0 is 40% faster than 0.991.

12:07 There's about 20 optimizations in this list.

12:10 And this is, you can see by my scroll bar, there's a ton of stuff that happened when they went to release 1.0.

12:16 But some more things in here that are nice features, for example, warning about variables before definition and kind of goes along with this possibly undefined variables.

12:24 So for example, B is never true, X never happens, you'll never be able to print X.

12:29 mypy will tell you about that before you ever run your code.

12:31 So it's kind of like just a lot of those belts and suspenders, nice safety checks, keep your code nice and clean, combine this with tools like Black and things like that to just have super tight code that is easy to read for everybody.

12:44 Some new features that are supported as of Python 3.11, the new self type is supported.

12:50 If you are having a method that returns an instance of the class, you can use the self type so that if you have a subclass, you get the correct subclass return instead of having to explicitly redefine that for every subclass you are doing.

13:05 More convenience methods for those who are diving deep into the typing system here.

13:10 >> The self-type is so good.

13:12 >> Yeah. It really cleans up a lot of problems.

13:14 >> It was really janky.

13:16 If you're inside the class, you put in quotes, the name of the class.

13:20 Well, how's it supposed to find that one?

13:23 Or if it's going to create a circular dependency, you've got to do weird things.

13:26 So yeah, that's really, really nice.

13:28 I do wish that Python had a two-stage parser aspect, where it would say, go through, here's all the stuff that's defined in the class.

13:37 "Oh, here's some type things and other reference.

13:40 We might not know what that is.

13:41 Let's go all through and then go back again and see if we can figure out what that is." But the way it works now is when you're in the middle of a class, that class doesn't exist yet because it's not done defining it till you're at the end.

13:52 But if you gotta say, "I'm gonna compare this against another one of me." Well, how do you say that in types?

13:57 Self.

13:58 - Right.

13:59 - That's what we got.

14:00 - And I can see this being rolled in to proper Python.

14:04 It seems like at some point.

14:06 I just feel like types went on from being this pie in the sky idea to bolted on type feature to it's really getting ironed out and really well supported by the community and really well supported by all the editors.

14:18 There's some new parameters back stuff in here that I don't know too much about params to backs.

14:24 We're not heavy into types yet.

14:26 We're doing a lot with Django and doing some type ends there to start making some APIs.

14:30 But that's where we're still diving in.

14:32 A bunch of new miscellaneous features.

14:34 So if you are looking for features galore that has happened in mypy 1.0, there's a bunch of fixes to crashes.

14:41 So this seems like it's a really robust release.

14:43 Another important one here is going to be the Python 3.10 match statement support.

14:48 So it can compile those as well.

14:50 So they're staying on top of all the recent language features, which is obviously what's needed to get momentum behind a project like this.

14:58 I think anybody who's using Type-ins better using the mypy stuff.

15:02 For those who don't know, it's also supported by Dropbox.

15:06 When I said 40 percent faster, that was actually running against the Dropbox main code base.

15:14 So that's the benchmark against Dropbox itself.

15:17 >> Interesting.

15:18 >> Yeah.

15:19 >> Yeah. mypyc is pretty an interesting thing that opens up once you start to adopt types.

15:23 For me, the main reason I use types in my world and I I realized that I live in a different world than a lot of folks, but for me, it's really about making the editor tell me what it should know already and not, do I need to go look in that method?

15:38 Does this actually take, when it says user ID, is that a string or is that a BSON from MongoDB?

15:44 I can't remember what kind of thing this stores for this, it says, you know?

15:48 And then, oh, okay, that's obvious, right?

15:50 Just so you don't have to keep bouncing around, you just stay in the way you're trying to focus.

15:54 And I love it.

15:55 - Yeah, it's extra smart.

15:56 If you're using a PyCharm or VS Code and you thought it was smart before, once you introduce these types in there, it gets even smarter.

16:04 >> Absolutely. Awesome one.

16:07 Now, two pieces of real-time feedback.

16:09 One, I found out with the untangle that if you have multiple nodes of the same thing, like you would in an RSS feed, you have multiple posts or a blog post, same thing for the post.

16:20 You just still say root.mainthing.child, or whatever the name of that node is, that becomes a list instead of a single thing, if there's more than one.

16:30 That's pretty interesting.

16:33 Then the other one here is over to you, Brian.

16:38 Jeff asked, can an old project be converted to work with Flit?

16:41 >> Yes, probably, maybe.

16:44 It really depends on the project.

16:46 >> Some of it can be, there's not that much to it.

16:50 There's pyproject.toml and a few other things, and it's probably not that much work to migrate it, I guess.

16:57 - Yeah, but I mean, I'm assuming by old project, it's an old setup.py setup tools project, and you could do anything in that.

17:07 So if you were going crazy, you can't go crazy anymore.

17:11 But if it was a fairly simple, straightforward, then yeah, probably.

17:16 - If you were going crazy, you get what you deserve.

17:18 How about that?

17:19 (laughing)

17:22 So, all right, before we move on to our next topic, well, let me tell you about our sponsor for this week, Microsoft for Startups Founders Hub.

17:31 Really great program.

17:32 If you have a startup and you wanna get tons of support, not just technical support, but also networking with people who have founded their own startups, then you should check out Microsoft for Startups Founders Hub.

17:45 As Brian mentioned last time, we heard back from one of our listeners who signed up and had done several startups and said, "I was a little skeptical, but this is a really great program.

17:53 It's fantastic for us.

17:54 And I just wanted to let you know how well it's working out.

17:57 So definitely a good program.

17:58 People should check it out.

17:59 So what it is, is Microsoft created this program to help people become successful starting a business.

18:06 By some estimates, 90% of startups go out of business within the first year.

18:10 That's not what you want for your startup.

18:12 You want it to succeed.

18:14 So with Microsoft for Startup Founders Hub, you get access to free resources.

18:18 So this includes platform technologies like Azure, GitHub, a partner with OpenAI.

18:25 I've heard chat GTP is catching on, so that's cool.

18:28 So you get access to some of the resources from them, which is fantastic, but you also get access to a mentorship network which I think maybe is even the most important part.

18:38 You get access to a pool of hundreds of mentors across ranges of disciplines, across areas like idea validation, fundraising, management and coaching, sales and marketing, and others.

18:48 So you can go and book one-on-one meetings with these folks, and many of them are former founders themselves.

18:54 So you can make your idea reality today, the critical support you get from Microsoft for Startups Founders Hub.

19:00 Just visit pythonbytes.fm/foundershub2022 to go apply.

19:07 The link's in your show notes.

19:08 You don't have to be VC-backed, don't have to be third-party validated.

19:13 You just apply. If they like what you want, it's free, and they'll take you into the program and you can get started.

19:19 Thank you to Microsoft for supporting our show.

19:22 With that, Brian, over to you.

19:24 >> Yeah. Go back to the same article for a second.

19:29 >> Just scroll a little further down the same page. I got it.

19:32 >> Within the same packaging Python projects thing, I have a reason to bring this up.

19:39 But anyway, it lists a whole bunch of tools you can use for your build a backend.

19:45 It's got PDM, FLIT, Subtools, and Hatchling.

19:47 There's another page on the same in PyPI or PyPA that is the packaging flow.

19:54 This one talks about it as more of a high-level.

19:57 It's not a tutorial, talks about the high-level thing.

20:00 There, it lists all of those, FLIT, Hatch, PDM, Poetry, Subtools, and then also Trampoline and Way, which I've not heard of.

20:09 This can cause confusion, and that's really what we're talking about.

20:13 I have, and I'm bringing this up because just independent of my involvement with Python bytes, I've been involved with people having trouble figuring out how to package their Python code independently. I've got a web developer doing it, a tools backend person, and then somebody that's just trying, he's a C# person that just was trying to package a frontend to his tool, a Python frontend.

20:38 And he was like, I don't know how to do all this.

20:41 And it is confusing.

20:43 And it even is confusing for seasoned Python developers.

20:47 And here we've got, oh, I didn't link it right now.

20:50 But there was a Doug Hellman from Module of the Week posted that he's published lots of packages, and he went to do it again, and was a little lost and wanted some advice.

21:03 So new people or people that are outside of the scope are confused.

21:09 And I think it's fair to recognize that.

21:12 And so there's a lot of discussion around it.

21:15 And Pradyun put together a blog post, was talking about it on Disqus.

21:22 And he put together an article called "Thoughts on the Python Packaging Ecosystem." And I think it's a kind of a fair discussion around it.

21:31 And one of the points is that a lot of Python users are not software engineers.

21:37 And, but we also have to recognize that a lot are.

21:41 So it's a diverse ecosystem, and I think it's going to be a mess.

21:47 It's going to be a mess for a while.

21:49 And I think it's good that it's a mess in some cases.

21:53 It's good that we don't just have setup tools now.

21:56 We have a lot of ways that are working on different pieces of the problem, of how to make it easier around PyProject.toml, and you can have a different build back end.

22:09 But he breaks down the points.

22:13 It's a big article, but one of the things is, it kind of unintentionally, the packaging ecosystem unintentionally became a competitive space that it is today.

22:22 And maybe we need to make, the community needs to make an explicit decision if we should continue operating under that model that led to what we have now.

22:32 We could pick from different tools to do in different things.

22:37 that's a good model, according to this.

22:40 Or in approximately the same choices to get a really bad user experience, bad for some people, good for others.

22:49 And then, also maybe we should pick a default, because picking a default doesn't make the other approaches illegal, they just aren't the default.

22:57 And I'm kind of leaning towards that, I think it would be good to have something, but I'm not sure what.

23:02 Down, longer down the line, pardon me, of the scroll, is PIP.

23:09 One of the ideas is maybe we should have PIP.

23:12 Well, okay, I'll back up a little bit.

23:14 Some of these tools like Poetry and PDM are workflow tools, and other tools like Flit are more build.

23:23 Flit's more like I want to just build a wheel.

23:26 It also does publish, but mostly the workload is around building the wheel.

23:31 Whereas PDM and Poetry are like this whole workflow thing, control your virtual environment, do a whole bunch of stuff.

23:39 I know a lot of people from other languages like JavaScript, maybe Go, I don't know, are used to these high-end workflow things that do everything for you.

23:51 But we're already used to a multi-tool model now, so I'm not sure how to get from here to there.

23:56 It would be interesting to have pip do it.

23:59 we already all use pip, so could pip expand to do both the focus thing it's doing now, installing a package, and then also do some workflow stuff?

24:11 I guess I'd be open to explore that idea, but I was curious what you guys thought.

24:16 >> I'd be open to explore it.

24:17 Not to disparage anything, but I personally dislike the things that try to put their workflow onto you.

24:22 I'd much rather have something simpler.

24:24 As someone who teaches Python, this is a huge problem for people getting into language.

24:29 They feel like it's a big challenge.

24:31 Why is it so hard?

24:33 Why do I have to learn all these things?

24:34 I picked the wrong thing and now am I screwed?

24:37 No, you're not out of luck.

24:38 You just delete your virtual environment and start over.

24:41 But yes, I think having many of them, one, is a problem.

24:47 >> Yeah.

24:47 >> I think something that forces workflows on you, I'm not a huge fan of.

24:53 I feel like we've already got lots of tools that solve it pretty well.

24:57 I'm not against pip doing a little bit more.

25:00 I use pip-tools for managing my dependencies and it is so good.

25:04 It is so good.

25:06 You state just your top-level dependencies and when it's time to, let's see what new, let's make sure we upgrade to the new things.

25:11 You just run pip-tools upgrade and it'll regenerate.

25:14 You can evaluate what the changes are.

25:15 I really like pip-tools.

25:17 I think it's a much simpler scenario than a lot of us.

25:21 I know other people really like it.

25:23 But anyway, this is what I like.

25:25 We could possibly leave pip alone and expand pip-tools to do the rest of the workflow stuff.

25:29 Yes, potentially.

25:30 Yeah, we could, but I kind of agree with that, like Unix small tools do one thing really well, which is why I've kind of gone with the pip-tools approach as well, because I really feel like pip-tools does such an incredible job at that dependency resolution and getting exactly what you need way better and way faster than pip itself does.

25:49 I think that approach that Flit takes, it seems like, I've never used Flit, but the fact that it can just wrap around pre-existing tools to provide a common interface for users to be able to build, package, push the whole deal seems really nice because it guides them through step-by-step that process until maybe at some point they don't need it or they just enjoy it and it's using some of the standard tools behind the scenes.

26:13 >> Yeah.

26:13 >> I'm also with Michael, I don't like the predefined workflows that I'm not a huge fan of poetry necessarily because it just doesn't fit the way my brain wants to work down a project.

26:23 And I'm not surprised.

26:24 So I was I was thinking an analogy of like, if you're going to have a pocket knife, which tools should be in the pocket knife?

26:32 And if you go into like the medium pocket knives on the Victoria Knox site, oh, no, it won't load.

26:37 You get like even the small ones.

26:39 So the medium ones, you get like tons of different tools, which tools should be there.

26:43 Nobody's going to agree.

26:45 And I thought, well, at least in the small ones, maybe people can agree because, oh no, not even in the small ones, people can't agree, which should go together.

26:55 I want my Swiss Army knife to also have drywall tools and something for my car.

27:01 It seems like a combination of really small tools that do one thing really well with some tools that layer on top of it, whether it's DryIDE, whether it's Flit, and you can have the opinionated workflows layered on top of this. They rest on top of a solid set of identical tools that produce It's the identical thing.

27:21 Yeah.

27:22 I'm in that camp as well.

27:24 That's where it, but I totally get that.

27:26 That's confusing for people.

27:27 Oh, totally confusing.

27:28 I mean, the first time I go to explain to someone how they're going to manage like their dependencies on a Django project, I'm like, oh my gosh, I had to step back and think about the 20 steps that kind of go into how I learned to do this, that I'm now explaining to some new person.

27:42 I'm like, I'm sorry, this doesn't sound easy at all.

27:45 There's gotta be a better way, but I don't know if there is.

27:48 Yeah.

27:49 Anyway, okay, rabbit hole, but.

27:51 - Oh, totally.

27:52 I mean, Python packaging is the ultimate rabbit hole in the Python community.

27:56 - Yeah.

27:57 Speaking of packages, Michael, do you wanna jump into yours?

28:01 - I do wanna talk about more packages.

28:03 This is the simplified version, but also a bit of real-time follow-up as well.

28:08 Again, I'm having many of the folks that were on that discuss thread that you mentioned, Brian, about this topic on Talk Python, probably in a couple of weeks, has a big panel discussion about it.

28:19 So if you all want to hear more of this, but for a long time with the folks who are participating and dive on in.

28:26 Okay.

28:27 Oh, also, apparently I said chat GTP, rather than I meant ChatGPT, as Misharraf pointed out.

28:36 So thanks for that.

28:37 That's the one I meant, you know the one.

28:38 All right, so there's this cool GitHub page, I guess, called toppypi packages.

28:45 And so this is a pretty neat place.

28:47 It shows you the top 5,000 most downloaded PyPI packages updated monthly.

28:54 So you can scroll down here and you can say, show me the top 100, show me the top 1,000, whatever.

29:00 I'll pull up the 1,000 for you, Brian.

29:02 There you go.

29:03 So these are updated over the last 30 days.

29:05 And it actually says when it was last updated and that was February 1st, which is a week ago as of this recording.

29:11 So that's pretty fresh, really.

29:13 And so you can find things like pytest-check.

29:17 Is it in here somewhere, Brian?

29:18 - It's in the top 5,000, but not the top 1,000.

29:22 - Oh, okay, did I hit the wrong one?

29:23 Sorry, I tried to pull you up there.

29:24 But you see things like Boto3 and URLlib are the most popular.

29:30 So Boto3 is downloaded 522 million times every month, every 30 days or so, which is just insane.

29:38 You gotta be careful because some of these are really popular dependencies rather than directly really important libraries.

29:45 Like I use Boto3 all the time as a dependency, the less often directly, definitely for your OLIB3, I don't think I've ever used it directly, but obviously it gets installed, but I'm gonna use many things.

29:55 So this is pretty cool if people wanna check it out.

29:58 Calvin, what do you think?

29:59 - Oh, I'm excited about this too, 'cause just thinking about the number of times the Amazon cloud must go download Boto3 from like Wi-Fi, mind blowing. Like, there's all the serverless that's on server someplace downloading packages in the background. Now you get a feel for it with a big number.

30:18 Yeah, it's really nuts, which is also a little bit ironic because it comes from AWS and they sent it over to PyPI and then it comes back to AWS.

30:25 It's a weird, root goal world.

30:28 Don't they use Fastly for their CDN? I think they use Fastly.

30:33 PyPI?

30:34 Yeah.

30:35 I believe so.

30:36 - Yeah, it's like, you have to go to a competitor to get like, yes, exactly.

30:40 - Yeah, how interesting.

30:42 So this is interesting in and of itself, and it tells you which ones are there.

30:45 And there's an API which you can hit, let's not hit the minified one, let's hit the non-minified one.

30:50 And it'll tell you actually returns when it was updated, what the query was, et cetera, et cetera.

30:55 And so you actually can get this as a JSON endpoint, which is pretty cool.

30:59 It's just a static file that gets updated, you know, every couple of days on this GitHub page setup.

31:04 That's also an interesting way to think of an API.

31:07 My API is a static file that periodically gets refreshed, but there it is. I don't think you'll hear much there.

31:13 >> I love it though.

31:14 >> I do too.

31:15 >> Just a static cache.

31:16 >> Yeah.

31:17 >> Yeah. Think how scalable that bad boy is.

31:20 Especially if you put a CDN behind it.

31:23 Then related to that by the same person, there is HugoVic top PyPI packages on GitHub, which is a regular dump of the most downloaded packages from PyPI.

31:35 So it sounds like, "Yeah, Michael, you told us about that. Thanks a bunch." No, it's not the names, it's all of it.

31:42 It's all of the data that is here.

31:46 Where do they have it? This one?

31:48 I don't know if I want to open this up, but it's really big.

31:50 So it's all the XML.

31:53 I'm not sure where it is in here, but it's all the XML files.

31:57 The JSON files that Untangle got me thinking XML.

32:01 So it's all the JSON files of what comes out of the PyPI API, but all of them, it's when I checked it out and unzipped it, and it was like 22 gigs on disk.

32:12 So this is a big beast right here if you download it.

32:16 Actually, no, this is not the right link.

32:18 I have to find it. I'll put it in the show notes.

32:21 But there's this guy, darn it, this is the wrong one.

32:25 But there's also linked off of this.

32:27 There's one where you get all of the packages, which is all the JSON for every package.

32:31 I'll have to figure out where I got that from, unfortunately.

32:34 It's the wrong link I pulled up.

32:35 But anyway, these are both really cool resources, and you'll have to check the links for that other one.

32:40 >> Cool.

32:41 >> Definitely.

32:41 >> Yeah. All right.

32:43 >> Cowell.

32:43 >> Coming in at 57 on that list for the last 30 days is SQL.

32:49 For those of you who aren't familiar and do like Swiss Army Knives, this is the Swiss Army Knife of database tools.

32:55 It's not only an ORM, but it has a whole SQL toolkit built into it.

32:59 If you've never touched SQLAlchemy, but deal with databases, you may want to check this out.

33:03 More importantly, and this ties in with my previous announcement on the mypy 1.0 release, there's a lot more support in this release for types and typing.

33:13 There's native out of the box mypy experience has been improved.

33:18 Now, the author of SQLAlchemy, Mike Bayer is actually going to be joining us in a few weeks here at Python Web Conference and giving a tutorial-length session on SQLAlchemy.

33:29 He did this last time two years ago when he released 1.4.

33:32 For those of you who have been following along in the background, the 1.4 release was the first half of this major revamp of SQLAlchemy into this new world.

33:42 2.0 is the second half of that revamp.

33:45 It's really been four years coming for this major release of SQLAlchemy.

33:49 If you want to know what's new and hear it from the horse's mouth, Mike Bear is going to be joining us and he'll be happy to answer all of your questions.

33:57 Let's see, there's a ton of new things released in here.

34:01 I think if I just scroll down--

34:03 yeah, so plugin-free, PEP484 compatible ORM syntaxes.

34:08 So again, kind of ripping out the old stuff.

34:10 All the typing support in here is all fresh.

34:14 If you are using the mypy plugins from before, you're going to want to take all those out.

34:20 They're considered deprecated at this point.

34:22 So that's something that's very important for you to consider.

34:25 But now, SQLAlchemy 2.0 is using all the new Python 3 wonderfulness of data classes, enums, inline annotations.

34:33 It's just really a nice modernization of a really powerful package.

34:38 See what else is in here that was of super importance.

34:41 Oh, yeah, some performance stuff.

34:42 So this all new fully integrated ORM approach to bulk inserts is quite interesting.

34:48 You will get a major speed increase except if you're using MySQL.

34:52 The MySQL folks have not implemented the insert returning syntax, but if you're on MariaDB, you're good to go.

34:59 So that's something to pay attention to.

35:01 In addition to that performance, there's also the bulk optimized schema reflection.

35:06 So if you were using SQLAlchemy, it was basically doing a table by table reflection if you were asking it for metadata about the tables.

35:15 Now in all of the backends, there is basically a--

35:19 Oh, sorry, I think it's only for Postgres and Oracle right now.

35:22 So Postgres and Oracle support the bulk schema upgrades.

35:26 You can see right here for Postgres, a 250% improvement, and for Oracle, a 900% improvement.

35:33 I'm not sure what it was doing before, but it's a heck of a lot faster now at 900% improvement.

35:38 Always a benefit there.

35:39 And in addition, I'll point out some other things that were in the migration guide and the what's new in 2.0 guide.

35:46 You're going to want to make sure you check those out here.

35:48 there's like the migration process link from this article or blog post.

35:52 You're going to want to make sure you read through those because there's major incompatibilities from 1.4 into 2.0.

35:57 But the migration guide is really similar to like the 2.3 guide Python had.

36:03 It tells you how to mechanically go through and make sure you're going to be compatible for the latest version of SQLAlchemy.

36:09 Another speed improvement here is that all the native C extensions have been ported to Cython.

36:14 Some of them are, they're all as fast.

36:17 Some of them are slightly faster than their previous C extensions, but this also removes risks of memory corruption and stomping on each other and things that C does really well that you don't want. So that's also a nice benefit.

36:30 And then the kind of tying into that same benefit is the SQLAlchemy is now PEP 517 enabled.

36:36 So that means the PyProject.toml file that we just saw in the last couple announcements is supported, which means when you're on your local machine and doing a source build of SQLAlchemy, it can automatically go grab some of the dependencies such as Cython which can be tricky to get installed for some people.

36:52 That is simplifying a lot of people's lives to go to SQLAlchemy 2.0.

36:57 >> Yeah, SQLAlchemy 2.0 is a huge release.

37:00 >> Yeah, it's a big deal.

37:01 >> It's been years in the coming and it was in this intermediate 1.4 mode where you could choose either API and they finally removed some things.

37:10 But to me, the biggest deal is the async.

37:12 >> Yeah.

37:12 >> Right. In the prior versions, there was just no way to do any async things, which meant on all the scalable areas, you were stuck trying to find somewhere else to go.

37:22 Now, if you really like SQLAlchemy, you can just use the Async API.

37:25 >> Yeah. I think a lot of people are going to benefit from those types, all the type support in mypy native support.

37:30 >> Yeah, absolutely.

37:31 >> And get back to your editor.

37:33 >> The Vulcan search have always been a hassle in ORMs.

37:36 Sounds like that's really gotten a lot of love too.

37:39 >> Yeah, definitely. I look forward to it.

37:41 >> I do as well. All right.

37:44 Well, that's all of our topics.

37:47 Brian, what do you got for us as an extra?

37:49 I guess just one quick extra is like, one of the things I've been working on on the side is I'm going to build a, I'm going to create a new build back end for Python and a new workflow tool.

38:00 [LAUGHTER]

38:01 No.

38:02 [LAUGHTER]

38:03 I'm not.

38:04 [LAUGHTER]

38:05 But what will you name it?

38:06 [LAUGHTER]

38:07 Well, I think Flick, because I always thought Flit should have been Flick.

38:13 So like a little football. So yeah, yeah. So how about let's go to Calvin's extras.

38:19 Yeah, sure. So I kind of mentioned this in the intro, but we are putting on the Python web conference. This will be our fifth annual so big five years. I'm wearing one of the jerseys from last year. You can't see it, but it was really awesome. Last year we had 90 speakers.

38:36 we had over 400 attendees, something like 23 or 22 time zones. It was really a global event.

38:44 That's one of the things I'm really proud of. This event was meant to fill a hole, I felt like, in the Python community for folks who couldn't travel, couldn't afford to go to an in-person conference, or wasn't a regional conference near them. I know the PSF is doing a lot of work right now and getting some regional conferences into other areas, but this is still and will always be virtual and always be accessible to folks.

39:05 So if you go and register and you can't afford to pay for this, we do have a grant program.

39:10 We want to make sure that if you have a need to be in this room, you should be in this room and everyone is included. This year is going to be no exception. There's a lot of amazing speakers who have joined us. It'll be five half days from the 13th to the 17th. There'll be a keynote speaker kickoff every day, a keynote speaker at the end of every day, and there's some awesome speakers coming up. We've got some people from GitLab. We've been focusing on finding some folks from climate initiatives to actually come and give some of our talks where they're using technology for good in these areas.

39:38 There'll be a set of tutorials as well.

39:40 So I mentioned Mike Baer's tutorial already, but there'll be three or four other tutorials as well.

39:46 I know Matt Harrison's going to be giving one of the tutorials--

39:48 not tutorial, he'll be giving a keynote this year.

39:51 So he's back again from last year.

39:54 He gave a keynote and a tutorial.

39:55 Man's the hardest working man in Python.

39:58 But we really appreciate him joining us as well.

40:01 If you've got questions, make sure you just go to pythonwebconf.com.

40:05 You can also find me on @CalvinHP on Twitter.

40:08 There's also a Twitter account for Python Web Conf, so you can follow Python Web Conf on all the socials.

40:13 And join us. There'll be socials actually during the event as well.

40:16 So it should be tons of fun.

40:18 We'll have just everything for the community.

40:21 So I have a question right off the bat.

40:23 It says Web Conf. If I'm not a web developer, am I still going to be interested in this?

40:27 Or is it primarily web development?

40:29 That's an excellent question, Brian.

40:30 And we originally, the webconf name meant a conference for web people, but we switched it around, you know, because we felt like this was an opportunity for the conference for Python to be on the web.

40:42 And now we have app dev track, which is like not more than just web, we'll have a cloud track, we will have a PyData track, it's actually a officially sanctioned PyData event.

40:54 And we have a culture track, in addition to the tutorial.

40:56 So there is actually something for everybody in the Python world here.

41:00 We're really trying to, again, I've mentioned filling the gap for people who can't make it to specific conferences.

41:05 We're also trying to fill the gap for talks that I feel are a little more on the intermediate to advanced level.

41:12 If you're looking for very entry-level tutorials, this may not be the conference for you.

41:16 I mean, there'll be great community, lots of questions you can ask, there'll be tutorials, but I really wanted to make sure that the talks kind of scaled up that next notch.

41:24 When I go to a lot of regional conferences or even PyCon, there's some entry-level stuff there that's great for people who are new to Python.

41:32 This is that next step.

41:33 >> Cool. I don't need to cut you off, but can we see if Michael's got any extras?

41:39 >> Yeah.

41:40 >> I do. One while Calvin's thing is up, I'm going to be speaking at their conference about making your Python web apps fly with CDNs.

41:49 Speaking of CDNs earlier, so I'm really excited about that one.

41:51 That's a fun topic.

41:53 But I want to tell you a couple of tips and tricks.

41:56 I'll keep these short here, but they're kind of fun.

41:58 So I recently discovered that with many of the browsers, like Vivaldi or Chrome, sadly not Firefox.

42:07 Honestly, that's one of the reasons I quit using Firefox.

42:09 They support progressive web apps, which many people may know.

42:12 But even if you have a thing that is not a progressive web app, you can install it as a standalone from your perspective application that lives in the dock.

42:21 So the thing I got on my screen and I link to in the show notes is here's my Vivaldi, but then I have my different mail clients.

42:28 I got Proton Mail and Gmail and I got the Google calendar.

42:31 All of those are just regular web apps.

42:33 I don't think any of them are progressive web apps, but if you right click on the tab, you say, create shortcut and say, open in a window, they become their own little app that you have, you put in your dock or your taskbar or whatever.

42:43 So for example, like here's Proton Mail running.

42:47 You can see again, link in the show notes to the picture.

42:50 And it just looks like an app, like you would know it was an app.

42:52 It does.

42:52 You'd think it was an electron JS app basically.

42:54 But so if you want to have those as separate things, you're going to run in and alt tab or control tab over to.

43:00 There you go.

43:00 Oh no, I closed my browser and my mail is gone again.

43:03 I got to open it back up, right?

43:04 If that ever frustrates you, here's a cool fix for it.

43:06 All right.

43:08 Also speaking of Proton, I started using simple login.

43:10 Have either of you all used this?

43:12 No.

43:13 So I have a pro account at Proton, which is like, I don't know, a couple of dollars a month for a bunch of years.

43:18 if you pay like way in advance, I think it's not very much.

43:21 But along with that comes a thing called simple logon, which every time you go to a website, I don't think I have the extension in this profile in my browser, but if you go to a website, it suggests, hey, we'll create an automatic fake one-off email address that gets sent to wherever you decide it gets sent to.

43:38 And you can reply to it and it goes back through there and things like that, it's really cool.

43:42 So like if I went to rstechnica.com and I created, It would be like some random thing at RS Technica dash simple log on ish.io or whatever.

43:53 So you kind of know where it comes from, but it's, it's, if you want to just delete that and make it mailstop or you can figure out if it's sold or whatever.

44:00 So anyway, recommendation on checking that as well.

44:03 It's also open source.

44:04 You can host it yourself if, if you like hosting email.

44:08 I don't really feel like, I don't feel like doing that.

44:12 If you've heard about a new topic if you've heard about all the crazy stuff with chips the chips act in the US similar stuff in Europe if You've heard about biting banning chips or the US government banning chips to be used in certain things primarily in China But you know, there's a bunch of angles in which this kind of stuff is heating up not just with the US but primarily Around the US. There's an incredibly good documentary on on YouTube called US versus China, the war you can't see, that talks about the history of this.

44:46 It just reminds me of how much good stuff there is on YouTube.

44:50 I mean, I know there's a lot of stupid cats falling off of chairs and like idiots but hidden in there, there were some, this is a better documentary than you would find on professional top-tier television networks.

45:02 It's really good.

45:03 Anyway, if that's interesting to you.

45:06 >> I will second that documentary.

45:09 I have a 15 year old here who's obsessed with long form YouTube.

45:14 And some of the things he has shown me, it just blow my mind, like why the superconducting super collider didn't succeed.

45:20 Eight hours of like documentary on that.

45:22 I was like, what?

45:23 Yeah, there's there is no more Discovery Channel.

45:26 There's no more History Channel.

45:27 Yeah, I know they exist in name, but not in what they were, where they actually have real science and real information.

45:33 They're just reality TV and other silly stuff.

45:35 So all right.

45:36 I know we're getting short on time, Brian, so I'll go quick.

45:39 Talk Python's hiring, so I'm looking to hire somebody that does Flutter and Dart, not Python, because if it's Python, I'd be doing it myself.

45:48 But if you're out there and you have experience with publishing apps and Flutter apps in particular to the various app stores, if you're used to working on Android and iOS and others with Flutter, and you can show me an app in that store, I would love to work with you.

46:02 I'm going to post this somewhere pretty soon, but I want to put it out to the community first 'cause I'd rather hire somebody from our community than just some random person off the internet.

46:11 And it's primarily to do a complete rework of our mobile apps for Talk Python Training.

46:16 So that'd be a lot of fun if people are interested, just email me, michael@talkpython.fm.

46:22 That's it, those are all my extras, yeah.

46:25 - Nice.

46:27 - Calvin, thanks for being here.

46:28 - Thanks for having me, I really enjoy it.

46:30 It's lots of fun.

46:31 I get to go learn new things each time I come.

46:33 - Yeah, we do every time we show up as well.

46:36 It's a lot of fun.

46:36 All right, and Brian, thanks as always.

46:38 - Thank you.