Transcript #297: I AM the documentation
Return to episode page view on github00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.
00:04 This is episode 297, recorded August 16, 2022, and I am Brian Okken.
00:11 I'm Michael Kennedy.
00:12 It's good to be back and be with you, Michael.
00:15 Yeah, it's great to be back. Not in the usual location today.
00:19 You may hear some nature sound shmaiya and apologize if animals go crazy,
00:23 but there's construction, which is guaranteed to be a problem in my office.
00:27 So I'm sitting in the backyard. We'll see how it goes.
00:29 Yeah, for people that are listening to the podcast, the live stream people get to see his lovely view from his backyard.
00:36 Some nice trees.
00:37 Yeah, big trees. It's all Oregon backyard here.
00:40 Nice.
00:41 But also, I also want to say this episode is brought to you by IRL podcast from Mozilla.
00:48 So we'll tell you more about that later, but thanks to Mozilla and the IRL podcast for supporting the show.
00:53 Thank you.
00:53 Let's talk about writing code by not writing code.
00:58 Does that sound like a good idea?
00:59 Well, okay.
01:01 Not in the low code, no code sense.
01:04 There's actually some pretty cool tools in that space, but that's not what I'm talking about.
01:07 Imagine, Brian, somebody says, we used to be a .NET shop and we have this huge database and all of our code to talk to it is in some other language or it's Ruby or it's Java or whatever.
01:20 It's not Python.
01:21 And you decide the best way to talk to this database is with some flavor of SQLAlchemy.
01:26 SQLAlchemy straight.
01:28 Or if you prefer, you could have it with data classes or you could even mix in a little sprinkling of SQL model if you're doing async and FastAPI and pydantic.
01:37 Whatever choice you want to make from that perspective.
01:41 If you're looking at a database with 150 tables and all sorts of gnarly relationships, you might say, well, I'm going to have to spend the next week planning out how to model those out so I get them to exactly match the database.
01:54 Is it a Varchar?
01:55 Is it a Varchar with a limit like Varchar 10 or, you know, what?
01:59 That doesn't sound fun, does it?
02:00 No, no.
02:02 Not, not.
02:02 I mean, at least for me, maybe some people that's a special kind of fun for them.
02:05 But Josh Thurston sent over this project called SQL A for SQLAlchemy code gen.
02:13 So I'm taking that he doesn't think it sounds like fun.
02:15 I certainly don't think that it does either.
02:17 So people should check this out.
02:19 It's it looks really cool.
02:22 And what you do is it's it's an automatic model generator for SQLAlchemy.
02:26 What does it generate it from?
02:29 So what you do is you go through and you point it at some database.
02:33 OK.
02:34 Just say SQL A code gen and you give it your connection string.
02:38 For example, Postgres SQL colon triple slash some database connection string.
02:43 And then magic happens and you have a whole bunch of Python classes that are attempted to look handwritten.
02:50 So instead of taking all your time, like a week to model out the database and the relationships and all that stuff in Python, you run this one command line thing and then you have all the classes and then you can tweak them a tiny bit if you see fit.
03:04 OK, so this is OK.
03:06 You probably said this and I missed it.
03:08 But so you already have data in a database and you're trying to hook up an application to it.
03:12 Yeah.
03:13 Yeah.
03:13 Which is why I went on my theoretical theoretical example.
03:16 I have a database.
03:17 I have code that talks to it, but it's not Python.
03:18 And so there's not really something to work from.
03:21 But I've got a really complicated database that's been around for a while.
03:24 It doesn't have to be complicated.
03:25 But the more gnarly the database, the more you will appreciate this tool making, you know, doing it for you.
03:30 I think I have this situation.
03:32 I totally want to use this.
03:34 Yeah.
03:35 Yeah.
03:35 Yeah.
03:36 Yeah.
03:36 So it does a bunch of neat things.
03:40 It's written to read the structure of an existing database and generate the appropriate SQLAlchemy model code using the declarative style if possible.
03:47 So deriving from SQLAlchemy declarative base.
03:50 It's also there was some other tool called SQL Auto Code, which apparently has some limitations, such as, for example, it doesn't support Python 3 or recent versions of SQLAlchemy.
04:00 That seems like a pretty large limitation, but whatever.
04:03 So this supports the newest version.
04:07 It produces PEP 8 compliant code, tries to make it look like you're writing code by hand so it doesn't look auto-generated.
04:14 It automatically detects join table inheritance and all kinds of things.
04:20 So if I scroll down here, it's got these different generators.
04:23 So you can generate table objects for people who don't want to use the ORM because SQLAlchemy has these two flavors, like a low level, slightly above just raw SQL, and then the ORM, which is the one that I use all the time.
04:34 By default, it uses the ORM one, but you can also use, like I said, data classes, which is pretty excellent.
04:39 Yeah.
04:40 Instead of, you know, in case that's what you want your code to look like, or even better than that, you can use SQL model models or using SQL model, which is the project.
04:50 I'm sure we've discussed it before by Sebastian.
04:54 It's based on Pydantic and Async, but then it's built really on top of SQLAlchemy.
05:01 So if you're looking to do the newer version of that, you'll get this.
05:05 By the way, just thinking about it while I'm looking at this, maybe I have actually a SQLAlchemy generated database, but it's written in the older style of SQLAlchemy.
05:15 And it's not using SQL model and I want to just upgrade to SQL model.
05:18 You might be able to use the database to just go rewrite it for me again.
05:21 But in this, in this flavor, you know, or use it to generate the data class version.
05:25 That actually looks pretty cool.
05:26 Or do all of them and look at it and see which one you looks, looks like it's more fun to maintain.
05:31 Yeah, exactly.
05:33 So there's a whole bunch of options and stuff that you can use, but you can basically pass a bunch of command line arguments and stuff to change how it works.
05:40 Cool.
05:40 Like change how it names objects or change how it names fields, et cetera, et cetera.
05:46 People, if they really want to look into it and use it, I think they all got the idea from this.
05:50 Okay, cool.
05:52 What do you think?
05:52 Sound cool?
05:52 Yeah.
05:53 Looks really cool.
05:54 Yeah.
05:54 Anytime you've got a database, they're so hard to model because you've got to get the SQLAlchemy code to match it just right or it won't work at all.
06:01 And yet, you know, do you really want to do that by hand?
06:04 No, I don't.
06:05 And so SQL A code gen.
06:08 Thanks, Josh.
06:09 Well, I am going to talk about package.
06:11 I've been, my headspace is in packaging lately because I'm preparing a talk for next month and it's going to have some packaging stuff in it.
06:20 So this is really exciting.
06:22 I heard from Juan Luis Cano Rodriguez.
06:25 Rodriguez?
06:26 Sorry.
06:28 SetupTool664.00 is out and it ships with PEP660 editable installs.
06:36 So this, the big headline is not that, although that's really cool.
06:40 It's that most projects don't need a setup.py or a setup.cfg anymore.
06:46 Those can be gone.
06:48 And not that setup.py was evil, but it kind of was evil because what it does is it runs Python, runs Python while you install something.
06:57 Like when used normally, it's fine.
06:59 But it has this tremendous gaping hole for abusing things at different levels.
07:05 Yeah.
07:06 And the, the, okay.
07:08 So the, the caveat on this is the reason why it has that is sometimes it goes out and compiles stuff if it's not just pure Python stuff.
07:16 But you don't need that for that.
07:19 You, you, a lot of projects, most projects don't need that anymore because they're not really compiling stuff.
07:25 If they're on during pip install, they're, they're compiling stuff ahead of time and they have their separate wheels for different architectures.
07:31 So I like that model better.
07:33 So anyway, the, I'm like pretty excited about this.
07:37 So there's a, so yeah, congrats to the pipe PA for getting that done.
07:42 We've got, there's an article called development mode, editable installs.
07:46 So here you've got pip install editable that works with setup tools without a setup.py.
07:54 Used to have, have to have a shim.
07:56 So everything can be in piproject.toml now.
07:59 And, and so one of the cool things, and I actually discovered this also at the same time I was researching packaging stuff.
08:08 The pipe PA has this really cool guide packaging Python projects.
08:12 And they, what's neat is they keep it up to date fairly well.
08:16 And this whole tutorial, it's a simple, so you got a simple Python project and you want to try to learn how to package.
08:23 They have this, this page here.
08:25 It's nice.
08:26 There's no mention of setup.py or setup.cfg.
08:29 It's all pyproject.toml.
08:31 Oh, wow.
08:31 That's awesome.
08:32 Here's how you do it.
08:34 And you don't even talk about the older, more troublesome way.
08:38 Yeah.
08:38 And since setup tools and pip are not part of Python proper, they're separate things.
08:44 Well, I mean, you get pip when you, when you download Python, but you get a version and you usually upgrade anyway.
08:51 But setup tools is separate.
08:53 So they can move at a faster pace than Python itself.
08:56 Oh, right.
08:57 Okay.
08:58 Excellent.
08:58 Well, this is great news.
09:00 Anything that makes the supply chain side of Python stronger is good.
09:06 Yep.
09:06 Indeed.
09:08 Well, before we move on to the next thing, Brian, let me tell you about our sponsor.
09:12 All right.
09:13 So this episode of Python Bytes is brought to you by the IRL podcast, an original podcast from Mozilla.
09:19 If you're like me and Brian, we care about ideas behind technology, not just the tech itself.
09:24 So we know that tech has an enormous influence on society.
09:28 Many of these effects are hugely beneficial.
09:30 Just think about carrying all of the world's information in our pockets sort of thing.
09:34 But other tech influences can have negative effects.
09:37 And I really appreciate that Mozilla is always on the lookout for and working to mitigate negative influences of tech on all of us, all the tracking stuff they're doing, but a bunch of awareness things as well.
09:48 And so if these ideas resonate with you, you should definitely check out the IRL podcast.
09:52 It's hosted by Bridget Todd.
09:54 And this season is very much in the focus of Python.
09:57 It's AI in real life.
10:00 So who can AI help?
10:02 Who can it harm?
10:04 The show features fascinating conversations with people who are working and building more trustworthy AI.
10:08 For example, there's an episode about how our world is mapped with AI.
10:12 And it's the data that's missing from those maps that tells as much of the story as the maps themselves.
10:17 Another one's about gig workers and how they're pushing back on algorithms to create better working style.
10:22 And for political junkies, there's an episode about how the role of AI plays when it comes to spreading disinformation about elections.
10:30 Obviously, huge concern as just across the world for all the democracies.
10:35 I also just listened to the tech we won't build, which explores when developers and data scientists should consider pushing back on projects that can be harmful to society, even though the machine learning can easily be turned on them.
10:49 If this sounds interesting, try an episode for yourself.
10:51 Just search for IRL on your podcast player or visit pythonbytes.fm/IRL.
10:56 The link is in your podcast player show notes.
10:58 Thank you so much to IRL and Mozilla for supporting the show.
11:00 I've been listening to it.
11:01 It's a really great show, too.
11:03 Yeah.
11:03 Yeah.
11:04 I enjoy it as well.
11:05 It's not super, super technical where it's all about APIs and stuff.
11:09 You can kind of just kick back and enjoy it.
11:11 Not like this podcast where we're super technical.
11:15 Exactly.
11:15 Yeah.
11:17 We talk about a bunch of technical things, sometimes not too deeply, huh?
11:19 Oh, I like it.
11:20 I like our level.
11:21 Yeah, I do, too.
11:23 I do, too.
11:23 Before we get on to the next topic, I just want to do a quick audience comment from Anna here.
11:29 It says, hello from London, UK.
11:30 SQL code gen sounds like it could save a lot of headaches.
11:33 Yeah, I think it's going to be great.
11:35 Yeah.
11:35 And feedback for the tutorial that you highlighted, Brian, on python.org.
11:41 Yep.
11:41 Henry Schreiner says, it took around six months for my rewrite of that page to get accepted.
11:45 Well, thank you for all the hard work, Henry.
11:47 That's awesome.
11:47 Great job.
11:48 It looks great.
11:49 Yeah.
11:50 Very, very cool.
11:51 Previously, I had talked about async cache.
11:55 Remember that?
11:56 Where it's like the functools, LRU cache, but a little bit more.
12:00 However, you can apply it to async methods.
12:02 Owen Lamont said, you may also be interested in AIO cache.
12:07 What this one does is this lets you use proper distributed backends for caching.
12:12 So, for example, if you're on a web app, you might have five, six, ten, maybe many more worker processes,
12:19 either on one machine using the supervisor mode of like G-Unicorn, or it could be even across different computers.
12:26 If you're using that in-memory version of cache, every time the request goes to a different part of your site,
12:32 or different runtime, different process running your site, you've got to recompute it, right?
12:37 Well, this one also supports Redis and Memcached.
12:40 Oh, sweet.
12:41 Or Memcached.
12:41 And it has a common API across all of them, which is pretty fantastic.
12:45 And they're all async and awaitable, which is cool.
12:48 So, it aims for simplicity rather than trying to highlight all the nuances of that particular, say, to Redis versus the others.
12:55 So, it has an add, a get, a set, a multi-get if you need to say, give me the values corresponding to these four IDs,
13:02 or does this thing exist or not, delete, clear, even increment a value, like how many people view this page, increment that in the cache.
13:10 And it's shared across, like I said, ten worker processes across machines instantly.
13:15 How cool is that?
13:16 That's pretty cool.
13:16 Yeah, so, super easy to work with.
13:18 You can install it, but then you should also reference probably the specialization that you're using or the backend that you're using.
13:26 So, for example, you can say pip install AIO cache, but if you want to use Redis, it's bracket Redis.
13:31 If you want to use Redis and Memcached, you might say, you know, bracket Redis, Memcached.
13:35 They have message pack and different formatting.
13:37 So, depending on how you're using it, you might have to install some dependencies, okay?
13:42 The optional install mechanism of pip is pretty cool, though.
13:46 It is.
13:46 It is pretty cool.
13:47 So, you just import asyncIO, easy.
13:50 And then you import AOCache, and you've got to, you know, just basically run your loop somewhere.
13:56 Or if you're using something like FastAPI, just that thing's generating or managing the loop for you, so you don't have to worry about it.
14:02 So, you can just say await cache.set, key, comma, value, await, cache.get key.
14:08 Sounds pretty straightforward, right?
14:09 You can even use it as a decorator.
14:11 So, if you put at cache on a function, you can give it time to live, the target, which is Redis, the key to use for that particular thing, and so on.
14:21 And then off it goes.
14:22 It's the serializer.
14:23 You have pickles, or you have message pack, or JSON.
14:25 And then, there you go.
14:27 Pretty cool, huh?
14:28 So, does it...
14:29 Okay.
14:29 So, for a function, does it cache the input and output of that function, then?
14:33 I think it caches the output.
14:35 Okay.
14:37 But it doesn't look like it varies by argument.
14:43 At least in this example, it's not very...
14:44 At least, yeah.
14:45 And I don't see how you would...
14:47 Like, this key is the lookup value, right?
14:49 So, you might call that cache call key result or something.
14:52 I don't see how you dynamically do that.
14:56 It's got to be, like, a void.
14:57 But, you know, a lot of times, that's...
14:58 You just, like, show me all the products in this database or whatever, right?
15:02 Yeah.
15:02 Yeah.
15:02 Cool.
15:03 Cool.
15:04 Yeah, pretty neat, huh?
15:05 Very neat.
15:06 Yeah.
15:07 And then you have different...
15:08 Three basic ideas to think about.
15:10 You have these backends.
15:10 So, you have Redis backed by AIO Redis.
15:13 You have Memcache backed by AIO Memcache.
15:16 And then serializers.
15:17 Like, you can serialize just string, pickle, JSON, message pack.
15:21 But you can also build your own.
15:23 And you can also plug in.
15:24 There's a bunch of examples and documentation people can check out.
15:27 So, this looks really neat to me.
15:29 Yeah.
15:29 Nice.
15:30 It's not quite something that I need.
15:32 But if I did need it, I would definitely go...
15:35 I know.
15:36 And I'd be all over it.
15:37 Yeah.
15:37 Regarding the level of detail we have in our podcast, SE Steve says, I only listen to podcasts
15:45 about APIs.
15:46 Nice.
15:47 Nice, nice.
15:50 What's your last one, Brian?
15:51 My last one is, well, the Python Packaging Project.
15:56 Packaging Python Project.
15:57 Same thing.
15:57 No, I have a new one.
15:59 But I got it from this.
16:00 So, when I was reading this article or this tutorial again, I came across something I wasn't familiar with.
16:06 So, I had to go check it out.
16:07 So, down with creating a PyProject.toml, one of the options is Hatchling.
16:14 Hatchling?
16:15 Yeah.
16:16 I've heard of Hatch.
16:17 Is this somehow related to Hatch?
16:19 Yeah, it is Hatch.
16:20 So, have we already covered Hatch?
16:22 Actually, no, I don't think we have here, but I love the idea of it.
16:25 Okay.
16:25 So, Hatch is a modern, extensible Python project manager with a whole bunch of cool features,
16:32 like a standardized build system and reproducible builds by default, and environment management,
16:39 which, you know, okay.
16:40 So, I'm not sure if this is similar to Poetry's environment manager or not.
16:45 I haven't played with it much.
16:46 But you don't actually have to care, which is nice.
16:50 Because poetry, you have to care about it, because that's part of the whole thing.
16:53 Anyway, publishing is easy to PyPI and other sources.
16:58 Version management, project generation with same defaults, which I haven't tried that, so
17:03 I want to try that.
17:04 And supposedly a responsive CLI that's two to three times faster than equivalent tools.
17:10 So, this I definitely need to try.
17:11 So, this is, I would think it's similar on the line of Flit, I think, but with some extra
17:19 things thrown in.
17:20 And one of the reasons why I love Flit now is because even though Setup Tools does now support
17:26 PyProject Tommel properly, completely, Flit's like twice as fast for building stuff.
17:31 But, so I definitely want to try out Hatch and try this.
17:35 I did try one little small project, just converting a Flip project to Hatch, and it took me like
17:39 five minutes just using, the documentation here is great.
17:43 So, the excellent documentation here about how to use the different pieces of it.
17:48 So, it's pretty neat.
17:48 Have you tried it?
17:49 I have not tried it.
17:50 I've looked at it, and it looks neat to me.
17:52 I don't make many packages.
17:55 I more build applications and web apps and stuff.
17:58 So, I'm less in the, what's the right tool to build packages properly?
18:03 You know, and I know you're doing that a little bit more.
18:05 So, I just want to learn from you.
18:07 I guess, good mix.
18:09 I do more packages and less applications, and you do more applications.
18:12 So, it's good.
18:13 Yeah.
18:13 Some live feedback.
18:15 Henry Schreiner says, you can use any PEP 6.21 backend, Hatchlane, PDM, FlitCore, and so
18:23 on with Hatch or with PDM2, one of the fantastic results of standardization.
18:27 And that Hatchlane does a much better job of getting source files right than FlitCore.
18:31 Okay.
18:32 Interesting.
18:33 Yeah.
18:33 Yeah, there was a lot of cool options with the Hatch that you could specify exactly which
18:39 modules and packages to pick up if you need to.
18:42 That's one of the things that's a little bit mysterious with Flit, how to figure it out,
18:45 because it just sort of knows somehow.
18:49 I think it's the stuff that's in Git, but it's interesting.
18:54 So, the main thing is, is I really like, with the standardization, that Hatch is possible,
18:59 that Flit's possible, that PDM is possible, that we can do new things, and they're not that
19:06 different.
19:07 Like, it's kind of the backend of packaging, and the frontend is the PyProject.tom.
19:12 That's a better world to be in.
19:15 Yeah, that separation lets there be a lot more exploration.
19:19 A lot more variation.
19:20 Yep.
19:20 All right.
19:21 Well, that's it for our items, isn't it?
19:23 I think so, yeah.
19:24 You got any extras?
19:26 No, but hopefully I will soon.
19:30 So, I've got something I'm working on.
19:31 Oh, right on.
19:32 Yeah.
19:32 Yes, I know.
19:33 I think I know what you're alluding to.
19:35 Very exciting stuff coming quite soon.
19:37 Yeah.
19:37 How about you?
19:38 I have one extra.
19:39 This one's quick, but quite cool.
19:41 So, I'm sitting here on my MacBook Pro with the M1, you know, MacBook Pro, the M1 Max.
19:47 And until recently, I wasn't able to use PyPy.
19:52 Now, PyPy is the JIT compiled, often faster version of Python.
19:57 Sometimes you'll hear people say PyPy when they are referring to PyPI, but all the people
20:03 who work on PyPI pronounce it that way.
20:05 And it leaves space for PyPy to be pronounced it like it should.
20:08 So, PyPy is the fast JIT compiled version of Python.
20:13 And the big news is, a couple weeks ago, they announced support for M1, which is pretty cool.
20:19 So, if you're on Apple Silicon, you can now use PyPy.
20:22 That's very cool.
20:23 Natively.
20:23 Yeah.
20:24 It was done by Fajal and supported by contributions from the Open Collective, which is pretty cool.
20:32 And it's based on support for ARCH64, which is ARM64 and Linux, with some variations on
20:39 how this works.
20:40 So, they've got 3.8 and 3.9 working on macOS ARM64 platform, which is pretty cool.
20:46 So, if you're using that and you've been waiting for this, it should make your code run faster,
20:49 maybe use less memory, that kind of thing.
20:51 Very cool.
20:51 If people are interested in PyPy, testing code on episode 190, I interviewed Carl Frederick Bowles
21:00 about testing PyPy.
21:02 Oh, cool.
21:03 Yeah.
21:04 There's a lot of testing.
21:05 I mean, it's the entire Python runtime, basically.
21:08 It's like in much of the standard library.
21:10 That's a lot of work.
21:11 Yeah.
21:11 It's an interesting story.
21:12 So, yeah.
21:13 Would you say that testing and documentation are often really good things to add to your
21:19 project that go along together in some ways?
21:21 Well, hopefully you're doing it at the same time, but yes.
21:24 Yes.
21:25 You know, we've all worked with different types of team dynamics, the sort of flat hierarchy.
21:30 People would just take over the projects, the parts of the projects that they seem best suited
21:34 for.
21:35 And there might be more hierarchical versions.
21:39 So, our joke this week is about a somewhat dominating senior developer here.
21:46 And there's a junior developer just hired onto the team.
21:49 This is a picture.
21:51 People can check it out.
21:51 Just follow the link in the show notes.
21:53 The junior asks, where's the documentation?
21:55 In a very stern face, the team says, I am the documentation.
22:01 Hopefully, you're not currently working in this situation, but it's pretty funny.
22:06 You know, there's always bits.
22:10 There's always pieces of the system that are like, well, how does this work?
22:14 Oh, you've got to ask that guy.
22:15 He's not even on our team anymore.
22:17 Yeah, but he's the one that wrote it.
22:19 And luckily, he's still with the company.
22:21 So, go talk to him.
22:22 Exactly.
22:22 No one understands that we don't touch it anymore.
22:24 It seems to still work.
22:25 Yeah.
22:26 Exactly.
22:26 All right.
22:27 Well, what seems to still be working is our podcast, Brian.
22:29 It does.
22:30 Thanks for being here.
22:31 Yeah.
22:31 Thanks for being here.
22:32 Thank you.
22:32 Yep.
22:33 Thanks for everyone to listen.
22:34 See y'all later.