WEBVTT 00:00:00.020 --> 00:00:05.140 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. 00:00:05.340 --> 00:00:07.040 This is episode 496, 00:00:07.520 --> 00:00:08.280 recorded Tuesday, 00:00:09.000 --> 00:00:09.840 September 15th. 00:00:09.860 --> 00:00:10.400 I'm Michael Kennedy. 00:00:10.800 --> 00:00:11.840 And I'm Calvin Hendryx-Parker. 00:00:12.380 --> 00:00:15.380 This episode is brought to you by Logfire from Pydantic. 00:00:15.920 --> 00:00:19.200 If you want observability for your apps and your AI agents, 00:00:19.460 --> 00:00:21.280 Logfire is the business. 00:00:21.680 --> 00:00:23.140 I will be telling you more about them later. 00:00:23.260 --> 00:00:25.020 Find the link at the top of the show notes. 00:00:25.280 --> 00:00:26.700 Follow us on the socials. 00:00:27.000 --> 00:00:32.099 All the various things you can think of are there on the episode page as well. 00:00:32.480 --> 00:00:33.320 And sign up for the newsletter. 00:00:33.660 --> 00:00:37.180 I just sent out the most recent one a couple days ago. 00:00:37.300 --> 00:00:37.920 It was a little bit late. 00:00:38.060 --> 00:00:38.140 Sorry, 00:00:38.300 --> 00:00:38.380 folks. 00:00:38.580 --> 00:00:45.380 But really cool stuff that we add, like extra information that doesn't even appear in the show that helps you get a little more out of the show. 00:00:45.680 --> 00:00:45.820 Yep. 00:00:46.040 --> 00:00:47.320 I love all the context it adds. 00:00:47.620 --> 00:00:48.040 I do, too. 00:00:48.320 --> 00:00:48.660 I do, too. 00:00:48.740 --> 00:00:50.120 I'm like, well, that's pretty good. 00:00:50.200 --> 00:00:51.000 We found some good stuff here. 00:00:51.800 --> 00:00:56.360 I would say that this newsletter that we're writing here, it shouldn't go extinct. 00:00:57.160 --> 00:01:00.560 But some things might need to go extinct. 00:01:01.240 --> 00:01:02.120 What's going on here? 00:01:02.560 --> 00:01:07.040 So I found, so this is a blog post from, 00:01:08.020 --> 00:01:09.440 what's Eddie's last name? 00:01:10.060 --> 00:01:11.280 Hold on, it's down here at the bottom of his copyright. 00:01:11.550 --> 00:01:12.300 Eddie Atkinson. 00:01:13.020 --> 00:01:17.420 He gave a talk at the most recent latency conference. 00:01:18.180 --> 00:01:21.460 So it's actually a talk from last year, but I think he kind of brought it back a little, 00:01:21.600 --> 00:01:26.480 evergreened it into a blog post last week about pandas that should go extinct. 00:01:26.580 --> 00:01:31.580 and we're not talking about the cute little fluffy things that are used for international diplomacy, 00:01:32.160 --> 00:01:34.860 but the Python data frame library. 00:01:36.040 --> 00:01:38.540 Michael, how many times have you thought you had big data, 00:01:39.040 --> 00:01:43.220 only to find out you were ready to defunstrate your laptop because pandas was the problem? 00:01:43.500 --> 00:01:44.760 You know what? It's happened. 00:01:45.080 --> 00:01:45.400 It's happened. 00:01:45.760 --> 00:01:48.760 I'm going to get my dictionary real quick, and then I'm going to know that that happened. 00:01:50.100 --> 00:01:54.520 I think the issue is a lot of folks really don't have truly big data problems. 00:01:54.720 --> 00:01:58.760 I mean, we've done some big data projects in the past, which were 10,000 tables, 00:01:59.360 --> 00:01:59.940 petabytes of data. 00:02:00.160 --> 00:02:01.020 That's truly big data. 00:02:01.640 --> 00:02:04.760 Most folks probably lie in the medium-sized data, 00:02:05.280 --> 00:02:06.620 but Pandas definitely tops out. 00:02:06.730 --> 00:02:10.960 I mean, he does some interesting benchmarks in here, gives a couple good code examples, 00:02:11.640 --> 00:02:15.840 actually shows a really interesting post from Amazon Redshift team, 00:02:16.340 --> 00:02:21.540 where they were looking at the composition of many of the tables that are out there in the Redshift environment. 00:02:21.600 --> 00:02:25.920 If anybody's going to have a good view on what the size of data is and what big data could be, 00:02:26.460 --> 00:02:28.320 they're probably the ones to look at that. 00:02:28.380 --> 00:02:29.400 But if you look at this chart, 00:02:30.220 --> 00:02:33.480 they basically say on a continuum of data size, 00:02:33.920 --> 00:02:35.700 most folks start over here in Excel. 00:02:36.220 --> 00:02:39.020 You've got like under a gigabyte, around a gigabyte of data. 00:02:39.320 --> 00:02:41.560 About that point in time, Excel falls over. 00:02:41.980 --> 00:02:44.520 It's probably time to pick up another tool to handle that. 00:02:44.680 --> 00:02:47.359 And a lot of people reach for pandas because I think there's just a lot of 00:02:47.400 --> 00:02:52.760 built up inertia or momentum in the community around the pandas and data frames and it's an easy 00:02:53.040 --> 00:02:58.520 ui it's been taught in a lot of universities so there's just not a lot of like need to kind of 00:02:58.540 --> 00:03:01.600 move out of that space because there's a lot of good code examples a lot of blog posts have been 00:03:01.680 --> 00:03:06.560 produced a lot of data science is based on pandas but they're really based on data frames and there's 00:03:06.700 --> 00:03:11.040 more than one library out there to handle data frames and probably do it more efficient so if 00:03:11.040 --> 00:03:15.839 you actually looked at the chart here they're basically saying when you get up into like the 10 00:03:15.860 --> 00:03:19.040 10 gigabyte range for data sizes. 00:03:19.380 --> 00:03:20.360 Pandas is probably still pretty good, 00:03:20.940 --> 00:03:21.580 but then there's a gap. 00:03:21.740 --> 00:03:25.480 It falls off somewhere between 10 and 100 gigabytes of data. 00:03:25.820 --> 00:03:28.620 And 100 gigabytes of data these days is not unfathomable. 00:03:28.820 --> 00:03:30.760 You can easily go find sample data sets 00:03:30.760 --> 00:03:32.580 that are in that realm, in that range, in that size. 00:03:33.200 --> 00:03:35.000 And so the next thing they reach for 00:03:35.320 --> 00:03:37.640 is typically a commercial tool like Databricks, 00:03:37.920 --> 00:03:38.400 Snowflake, 00:03:39.020 --> 00:03:40.100 Dask, or some of these other things 00:03:40.220 --> 00:03:41.040 that are like Spark. 00:03:41.280 --> 00:03:43.819 So you're distributing the memory 00:03:43.840 --> 00:03:45.560 of that data set across many machines 00:03:45.960 --> 00:03:48.340 or maybe even across one very large machine, 00:03:48.540 --> 00:03:49.580 but doing it in a distributed manner. 00:03:50.160 --> 00:03:52.160 Most people probably don't need to go that far. 00:03:52.400 --> 00:03:54.520 Like most people probably are still sitting in the range 00:03:54.560 --> 00:03:56.500 where you can see on this chart that Polars handles. 00:03:56.840 --> 00:04:01.040 Polars can handle straight up into 100 gigabytes of data 00:04:01.340 --> 00:04:02.260 easily on a single machine. 00:04:03.340 --> 00:04:05.460 And then DuckDB takes it to the next step, 00:04:05.660 --> 00:04:08.360 which actually kind of fitting for this episode. 00:04:08.420 --> 00:04:09.980 I think this will be an interesting episode 00:04:10.060 --> 00:04:11.259 because there's a lot of information here 00:04:11.280 --> 00:04:13.420 about DuckDB later on in the show. 00:04:13.940 --> 00:04:17.140 But he kind of goes on again and shows that basically 00:04:17.390 --> 00:04:20.220 the average size of a row in Redshift is about a kilobyte. 00:04:20.500 --> 00:04:22.680 Every Redshift cluster has like 10 machines in it. 00:04:22.940 --> 00:04:25.820 They're capable of guzzling 8 gigabytes per second 00:04:25.960 --> 00:04:26.360 from S3, 00:04:26.770 --> 00:04:27.940 but really in actuality, 00:04:28.520 --> 00:04:31.280 almost 95% of the tables in Redshift 00:04:31.560 --> 00:04:33.500 contain fewer than 100 gigabytes of data. 00:04:34.120 --> 00:04:36.040 Most people are still in the range 00:04:36.040 --> 00:04:37.900 of just using a single machine with DuckDB 00:04:38.220 --> 00:04:39.260 or even just Polars, 00:04:39.490 --> 00:04:40.260 which is probably similar, 00:04:40.820 --> 00:04:41.940 simpler to maintain and manage, 00:04:42.320 --> 00:04:43.860 but it's above the reach of pandas. 00:04:44.040 --> 00:04:45.300 That's why the post is kind of 00:04:45.360 --> 00:04:47.120 going on about pandas needing to go extinct. 00:04:47.780 --> 00:04:48.580 Another interesting bit, 00:04:48.920 --> 00:04:49.040 again, 00:04:49.360 --> 00:04:50.000 kind of good code 00:04:50.160 --> 00:04:50.720 examples in here. 00:04:51.220 --> 00:04:55.720 When we get down into some of the tables for the performance, 00:04:56.100 --> 00:04:57.260 what strikes you 00:04:57.380 --> 00:04:57.460 here, 00:04:57.620 --> 00:04:57.740 Michael, 00:04:57.940 --> 00:04:58.820 on their memory usage? 00:04:59.300 --> 00:05:00.500 The pandas library, 00:05:00.800 --> 00:05:02.480 we're talking about 30, this is one of 00:05:02.520 --> 00:05:03.800 those examples, I can't remember which one, 00:05:04.300 --> 00:05:06.520 but four minutes basically for the duration of the 00:05:06.480 --> 00:05:06.760 processing, 00:05:07.380 --> 00:05:08.320 38 gigabytes of RAM. 00:05:08.680 --> 00:05:11.580 If you get into pullers, that gets halved, 18 gig of RAM. 00:05:11.890 --> 00:05:14.360 And if you go into DuckDB to do the same operation, 00:05:14.920 --> 00:05:18.160 five seconds at 1.93 gigabytes of RAM. 00:05:18.920 --> 00:05:20.700 So even the- 00:05:20.700 --> 00:05:23.780 It's like nine to 20 times as much, 00:05:23.980 --> 00:05:24.120 yeah. 00:05:24.400 --> 00:05:24.940 Yeah, I mean, 00:05:25.150 --> 00:05:27.040 an iPhone could do this operation 00:05:27.500 --> 00:05:28.880 against 100 gigabytes of data. 00:05:29.680 --> 00:05:30.420 It's fine though, 00:05:30.450 --> 00:05:31.340 'cause you can just get more memory, 00:05:31.460 --> 00:05:32.260 memory's cheap these days. 00:05:32.820 --> 00:05:33.120 Memory's cheap. 00:05:33.280 --> 00:05:34.680 Totally cheap, totally, totally cheap. 00:05:35.880 --> 00:05:38.280 So I just think folks need to put this one to bed. 00:05:39.220 --> 00:05:40.740 Pandas was probably a good way to start, 00:05:41.180 --> 00:05:42.520 but you'll notice in here, 00:05:42.600 --> 00:05:45.180 like the Polars notation or syntax, 00:05:45.600 --> 00:05:46.200 really, really similar. 00:05:46.620 --> 00:05:48.420 I like the DuckDB syntax. 00:05:49.280 --> 00:05:50.560 I think he's got some examples in here 00:05:50.700 --> 00:05:53.120 where he reads in and does some more operations. 00:05:53.540 --> 00:05:55.420 He does a couple against some larger machines, 00:05:55.800 --> 00:05:57.480 then falls back into an older laptop, 00:05:57.740 --> 00:05:58.640 like a framework 13, 00:05:59.240 --> 00:06:02.260 to show that this is still useful as a developer tool. 00:06:03.340 --> 00:06:06.140 So I think here, Pollard versus DuckDB kind of comes down 00:06:06.280 --> 00:06:07.140 to your workload, 00:06:07.300 --> 00:06:08.380 your experience, and your preference. 00:06:09.040 --> 00:06:09.540 It's a good post. 00:06:09.620 --> 00:06:12.200 I really liked all the code snippets he links over 00:06:13.280 --> 00:06:15.380 into the GitHub where you can actually try it yourself. 00:06:16.160 --> 00:06:19.320 It goes against the New York City taxi data set. 00:06:19.640 --> 00:06:20.700 It's got a ton of data in there, 00:06:21.020 --> 00:06:21.980 so it's fun to play with. 00:06:22.300 --> 00:06:24.120 And you can see that we want to wait minutes 00:06:24.300 --> 00:06:25.140 or do you want to wait seconds? 00:06:25.640 --> 00:06:27.360 And would you want to use all your memory for this? 00:06:28.200 --> 00:06:28.560 And actually, 00:06:28.760 --> 00:06:30.200 another thing that Pollard 00:06:30.220 --> 00:06:32.920 and DuckDB did much better was utilizing the CPU. 00:06:33.300 --> 00:06:36.480 PurePandas in this case was using like a multi-core machine, 00:06:37.020 --> 00:06:38.200 146% CPU, 00:06:38.720 --> 00:06:39.960 where if you go to PureDuckDB, 00:06:40.400 --> 00:06:42.380 there were over 800% CPU. 00:06:42.580 --> 00:06:47.020 So obviously eight to 10 cores are being fully utilized as opposed to 00:06:47.280 --> 00:06:48.900 basically one and one and a half cores. 00:06:49.180 --> 00:06:49.460 Yeah, 00:06:49.660 --> 00:06:49.940 that's awesome. 00:06:50.520 --> 00:06:51.720 I feel like this is a pretty 00:06:52.440 --> 00:06:57.400 data heavy episode for the data science crew out there. It wasn't on purpose, but we ended up that 00:06:57.560 --> 00:06:57.620 way. 00:06:57.840 --> 00:07:00.500 I have a little bit of a real-time follow-up for you, Calvin, 00:07:00.920 --> 00:07:02.059 for people who... 00:07:02.140 --> 00:07:02.160 Yeah, 00:07:02.860 --> 00:07:04.160 I was gonna ask the one last bit in here. 00:07:04.240 --> 00:07:05.960 He does mention Apache Arrow, 00:07:06.070 --> 00:07:06.980 and if you've not played with it, 00:07:07.270 --> 00:07:08.840 it allowed him to switch back and forth 00:07:08.990 --> 00:07:09.800 between pandas, 00:07:09.980 --> 00:07:10.360 Polars, 00:07:10.580 --> 00:07:11.420 and DuckDB 00:07:12.350 --> 00:07:14.540 without having to reload or copy the data into RAM. 00:07:14.860 --> 00:07:16.660 So you could actually do the same operation 00:07:16.940 --> 00:07:17.660 with each of the libraries 00:07:17.870 --> 00:07:19.620 without actually having to take the data back out of RAM. 00:07:19.800 --> 00:07:22.060 So check out Arrow, 00:07:22.580 --> 00:07:23.120 which is really cool. 00:07:23.700 --> 00:07:25.540 It's kind of a little bonus side bit 00:07:25.550 --> 00:07:27.000 that was in the blog post. 00:07:27.800 --> 00:07:30.320 So data folks who got medium-sized data, 00:07:31.020 --> 00:07:32.800 This is going to be a godsend for you. 00:07:33.200 --> 00:07:33.280 Yeah. 00:07:33.490 --> 00:07:35.480 I think Arrow is the foundation of pandas too, 00:07:35.660 --> 00:07:36.340 if I remember correctly, 00:07:36.580 --> 00:07:37.720 and also a polar. 00:07:37.960 --> 00:07:38.700 So that's pretty sweet. 00:07:39.140 --> 00:07:39.300 Yeah. 00:07:39.580 --> 00:07:40.660 My real-time follow-up here. 00:07:40.680 --> 00:07:41.680 It was a real-time follow-up here. 00:07:42.070 --> 00:07:42.180 Oh. 00:07:43.240 --> 00:07:47.720 If you were working with one of these and you want to switch to the other, 00:07:47.980 --> 00:07:48.680 I had Marco 00:07:48.820 --> 00:07:49.320 Garale on, 00:07:49.680 --> 00:07:53.020 really on Talk Python a while ago to talk about narwhals. 00:07:53.440 --> 00:07:58.540 And narwhals is a facade adaptive layer that speaks native polars, 00:07:58.810 --> 00:08:00.920 but also talks pandas. 00:08:01.340 --> 00:08:03.060 So if you want to try like, 00:08:03.280 --> 00:08:03.380 oh, 00:08:03.540 --> 00:08:04.260 let's see what we're doing. 00:08:04.290 --> 00:08:07.420 You know, you could use this as a intermediate layer to kind of swap that out a little more 00:08:07.700 --> 00:08:08.760 easily than rewrite and everything. 00:08:09.060 --> 00:08:09.260 Yeah. 00:08:09.530 --> 00:08:11.760 I think people just need to drop pandas. 00:08:11.810 --> 00:08:12.100 I mean, 00:08:12.470 --> 00:08:13.300 it was great. 00:08:13.370 --> 00:08:14.620 It was great 10 years ago. 00:08:15.760 --> 00:08:16.120 Yeah. 00:08:17.460 --> 00:08:17.560 Yeah. 00:08:18.100 --> 00:08:19.380 I have some funny jokes, 00:08:19.540 --> 00:08:20.860 but let's carry it. 00:08:20.960 --> 00:08:21.720 Let's move on. 00:08:22.040 --> 00:08:24.240 Let's move on to Pydantic Pint. 00:08:24.680 --> 00:08:27.180 So Pydantic Pint is really interesting. 00:08:27.640 --> 00:08:28.340 Do you know Pint? 00:08:28.720 --> 00:08:29.500 Are you familiar with Pint? 00:08:29.520 --> 00:08:34.780 actually i've never i've never used pint so pint we've covered that on the show back in the day 00:08:34.780 --> 00:08:40.599 and pint is interesting because if you're i mean all you got to do is say mars lander sample return 00:08:40.880 --> 00:08:46.200 whatever and it's like the 100 million dollar plus fail because somebody used feet and somebody used 00:08:46.580 --> 00:08:52.040 neighbors or something like that right and so pint lets you do math in python with units attached 00:08:52.480 --> 00:08:58.540 which is pretty cool right so i can say instead of just having a distance i have 42 i have 42 00:08:58.560 --> 00:09:04.280 kilometers and you can say like two miles to whatever and so on so it it basically mean forces 00:09:04.390 --> 00:09:09.880 you to work in units right a lot of times we don't do this as regular programmers but if you do 00:09:10.040 --> 00:09:15.280 anything scientific well there you go right so that's the background on pint but what i want to 00:09:15.280 --> 00:09:21.520 talk about is actually not pint it's called pidantic pint because pidantic is an awesome 00:09:21.720 --> 00:09:27.640 library that lets you validate the inputs and parse them and everything whenever you read some sort of 00:09:27.520 --> 00:09:31.340 JSON, right? Like if it's FastAPI or just a JSON file or whatever, 00:09:31.740 --> 00:09:32.340 you need database, 00:09:33.260 --> 00:09:34.440 SQL model, 00:09:34.680 --> 00:09:35.320 all those things. Yeah. 00:09:35.550 --> 00:09:41.800 So Pydantic Pint takes this idea and adds units to your data validation 00:09:41.980 --> 00:09:42.160 libraries. 00:09:42.760 --> 00:09:45.800 So instead of saying I have a box that has a length and a width, 00:09:46.050 --> 00:09:46.960 I could say I have a box 00:09:46.960 --> 00:09:53.360 that has a length and a width that is a Pint quantity and the validation is to convert it to 00:09:53.300 --> 00:09:53.420 meters. 00:09:53.870 --> 00:09:57.000 So even if you parse something that says feet or centimeters or whatever, 00:09:57.260 --> 00:09:58.200 it will show up 00:09:58.500 --> 00:09:58.660 correctly. 00:09:59.020 --> 00:09:59.620 What do you think? 00:10:00.380 --> 00:10:01.300 That's definitely handy. 00:10:01.460 --> 00:10:01.720 Yeah. 00:10:02.040 --> 00:10:03.200 It's the kind of thing that's like, 00:10:03.370 --> 00:10:03.860 you don't have to, 00:10:04.240 --> 00:10:05.860 you're not going to use it a lot unless you're really in, 00:10:06.120 --> 00:10:06.260 you know, 00:10:06.310 --> 00:10:07.180 some kind of engineering or something. 00:10:07.980 --> 00:10:09.560 Oh my God, this is so good. 00:10:09.720 --> 00:10:10.540 It's like so perfect. 00:10:10.730 --> 00:10:12.940 This has got to solve so many, like you said, 00:10:13.140 --> 00:10:15.560 small mistakes that end up in huge damage. 00:10:16.120 --> 00:10:17.300 A hundred percent. So yeah. 00:10:17.540 --> 00:10:18.780 And to be able to validate with it too. 00:10:19.130 --> 00:10:19.940 Yeah. Just automatically, 00:10:20.140 --> 00:10:23.260 right. Just all the Pydantic validations have a Pydantic base model 00:10:23.280 --> 00:10:25.260 And it parses over to whatever it is. 00:10:25.420 --> 00:10:26.660 And if you put, I don't know, 00:10:27.060 --> 00:10:28.880 liters into the length, 00:10:29.450 --> 00:10:30.180 well, liters, 00:10:30.390 --> 00:10:31.780 you can't convert liters to meters. 00:10:32.420 --> 00:10:33.780 So I don't know. 00:10:34.260 --> 00:10:37.360 And it kind of fits perfectly under the Pydantic scope 00:10:37.720 --> 00:10:40.120 of the data validation and serialization. 00:10:40.560 --> 00:10:41.400 Like it just, 00:10:41.660 --> 00:10:42.180 there's natural, 00:10:42.470 --> 00:10:44.820 like this should exist and they made it exist. 00:10:45.200 --> 00:10:45.700 Yeah, it's really cool. 00:10:45.700 --> 00:10:47.140 So you can have say a FastAPI endpoint 00:10:47.290 --> 00:10:49.040 that just automatically just takes units 00:10:49.240 --> 00:10:50.440 and automatically converts units. 00:10:50.550 --> 00:10:52.380 And yeah, it's a really nice one there. 00:10:52.720 --> 00:10:53.720 Speaking of really nice, 00:10:53.920 --> 00:11:00.240 now this transition here has nothing to do with the sponsorship, 00:11:00.400 --> 00:11:01.000 the previous one. 00:11:01.140 --> 00:11:03.820 They just happen to do great stuff in open source code too. 00:11:04.480 --> 00:11:09.320 But Pydantic also happens to create Logfire, which I told you about at the beginning. 00:11:09.720 --> 00:11:13.020 So let me go ahead and tell you about our sponsorship offer, 00:11:13.280 --> 00:11:13.620 Logfire, 00:11:13.820 --> 00:11:14.560 not Pydantic 00:11:14.700 --> 00:11:16.780 Pint, which is not even from them, but it's based on Pydantic. 00:11:17.140 --> 00:11:18.000 So here's the deal. 00:11:18.600 --> 00:11:19.620 It's 2 a.m. 00:11:19.780 --> 00:11:20.940 Your AI agent failed. 00:11:21.280 --> 00:11:21.900 Was it the model? 00:11:22.040 --> 00:11:22.620 A tool call? 00:11:22.660 --> 00:11:23.120 the database, 00:11:23.570 --> 00:11:25.960 just the general unreliability of, 00:11:26.320 --> 00:11:28.180 hey, I think a new model is coming. So the current 00:11:28.190 --> 00:11:29.560 one starts breaking periodically. 00:11:29.890 --> 00:11:31.420 So most observability tools, 00:11:31.650 --> 00:11:32.700 they can't tell you because 00:11:33.070 --> 00:11:34.200 they only see part of your stack. 00:11:34.560 --> 00:11:35.760 PyDentaglogfire sees all of it. 00:11:36.080 --> 00:11:37.160 One trace across your agents, 00:11:37.380 --> 00:11:39.700 LMs, APIs, and databases down to the infrastructure, 00:11:40.160 --> 00:11:40.420 services, 00:11:40.760 --> 00:11:41.100 Kubernetes, 00:11:41.300 --> 00:11:41.520 hosts. 00:11:41.980 --> 00:11:42.540 It's built on open 00:11:42.760 --> 00:11:44.820 telemetry with SDKs for Python, 00:11:45.380 --> 00:11:45.760 TypeScript, 00:11:45.960 --> 00:11:46.440 and Rust. 00:11:47.160 --> 00:11:49.359 And it works with any OTEL compatible 00:11:49.380 --> 00:11:49.640 language. 00:11:50.140 --> 00:11:50.540 Every prompt, 00:11:50.880 --> 00:11:51.300 token count, 00:11:51.500 --> 00:11:54.620 and cost right next to your vector searches and API calls. 00:11:54.940 --> 00:11:59.420 You query everything with Postgres compatible SQL to understand what your app is doing, 00:12:00.400 --> 00:12:02.420 and so can your coding agent. 00:12:02.610 --> 00:12:05.100 It can use the same way because it talks SQL really well. 00:12:05.260 --> 00:12:05.500 If you 00:12:05.720 --> 00:12:06.720 connect it to the MCP server, 00:12:07.050 --> 00:12:09.940 your agent can also figure out what is going on. So stop guessing, 00:12:10.250 --> 00:12:10.720 read the trace, 00:12:10.940 --> 00:12:11.660 Pydenic, Logfire. 00:12:11.960 --> 00:12:12.400 AI, 00:12:12.670 --> 00:12:14.140 it is still just engineering, 00:12:14.750 --> 00:12:16.039 even if it's weird engineering 00:12:16.060 --> 00:12:16.400 these days. 00:12:16.690 --> 00:12:20.260 So visit pythonbytes.fm/logfire today and sign up. 00:12:20.540 --> 00:12:21.540 Get 10 million records free 00:12:21.810 --> 00:12:22.220 every month, 00:12:22.440 --> 00:12:23.340 no credit card required. 00:12:23.710 --> 00:12:24.700 You can even click, 00:12:24.870 --> 00:12:26.260 and I really like this, there's a little 00:12:26.480 --> 00:12:27.340 copy of this text, 00:12:27.560 --> 00:12:29.860 onboard you with your agent. 00:12:30.340 --> 00:12:32.180 Click that and it gives you a prompt you can drop 00:12:32.320 --> 00:12:36.460 in to Claude Code or Codex or whatever and it automatically knows what to do to set up Logfire 00:12:36.520 --> 00:12:38.660 in your app. So thank you to Pydantic for supporting the show. 00:12:39.000 --> 00:12:41.359 Calvin, I know you're a big fan of the 00:12:41.420 --> 00:12:42.580 visibility into the token. 00:12:43.080 --> 00:12:43.180 Yeah. 00:12:43.820 --> 00:12:47.280 I was just curious now if copy the setup prompt is the new 00:12:47.740 --> 00:12:48.720 pipe to bash, 00:12:49.130 --> 00:12:50.920 like pipe some curl to bash. 00:12:51.380 --> 00:12:51.540 Yes. 00:12:51.790 --> 00:12:53.860 This is the replacing that. I think it is. 00:12:54.260 --> 00:12:54.960 That's why I was getting it. Yeah. No, 00:12:55.120 --> 00:12:56.780 no. I think it is. And it's amazing. 00:12:57.780 --> 00:12:58.860 I have some stuff that 00:12:58.870 --> 00:13:00.120 I'm working on. I'm like, oh, this is like, 00:13:01.020 --> 00:13:02.000 this idea is perfect. 00:13:02.150 --> 00:13:02.920 I love it so much. 00:13:03.210 --> 00:13:03.740 So yeah, 00:13:04.210 --> 00:13:05.980 pretty cool. Thanks to Pydantic for sponsoring the show. 00:13:06.370 --> 00:13:06.680 Thank you. 00:13:07.000 --> 00:13:09.399 And let's jump over to 00:13:09.820 --> 00:13:13.460 your topic next. You've got in the wrong order in the other. So what's next? 00:13:13.620 --> 00:13:13.960 Well, 00:13:14.980 --> 00:13:15.440 speaking of 00:13:15.800 --> 00:13:16.120 Pydantic, 00:13:16.920 --> 00:13:22.000 this comes from Bob Builderboss, a friend of the show. I know you've had him on numerous 00:13:22.240 --> 00:13:24.780 times for other events and things, 00:13:25.240 --> 00:13:27.520 but this one is about how to run, 00:13:27.980 --> 00:13:29.240 how Rust code becomes something 00:13:29.280 --> 00:13:29.920 you can import. 00:13:30.340 --> 00:13:32.840 I think it's interesting that we can, 00:13:33.080 --> 00:13:34.980 if people are complaining about performance, 00:13:35.260 --> 00:13:41.440 the first news article I had about getting rid of pandas and bringing back in with polars and 00:13:41.660 --> 00:13:43.300 DuckTV was about performance. 00:13:43.560 --> 00:13:45.080 This is similarly veined. 00:13:45.360 --> 00:13:48.400 Like if I've got a very computationally 00:13:48.900 --> 00:13:53.660 intense data structure or function that's happening in my program, 00:13:53.900 --> 00:13:55.360 it'd be sure be nice if I could 00:13:55.440 --> 00:14:00.240 maybe replace it out with a Rust version of that, but have it act natively inside of my Python code. 00:14:00.580 --> 00:14:06.780 So this blog post from Bob goes over basically what Pydantic V2 does, which is a data validation 00:14:06.940 --> 00:14:09.480 library that most Python apps are using these days. 00:14:09.660 --> 00:14:11.480 It's actually a Rust extension under the 00:14:11.560 --> 00:14:12.360 covers that does the work. 00:14:12.620 --> 00:14:13.180 Its core, 00:14:13.360 --> 00:14:15.860 Pydantic core, is all built with Py03, 00:14:16.080 --> 00:14:17.340 the same tool chain we're 00:14:17.340 --> 00:14:18.340 going to use here in this example. 00:14:18.800 --> 00:14:23.120 So it goes over some examples of basically you write a normal Rust 00:14:23.240 --> 00:14:23.520 module, 00:14:23.820 --> 00:14:26.200 you annotate it with some specific macros, 00:14:26.560 --> 00:14:28.719 and then you'll actually be able to import that 00:14:28.780 --> 00:14:32.360 into your Python code fairly naturally. 00:14:32.770 --> 00:14:37.040 I think there's basically the Rust parser is incredibly fast. 00:14:37.640 --> 00:14:41.060 Python never touches anything until the boundary crossing. 00:14:41.130 --> 00:14:42.680 So if you call for some data, 00:14:43.210 --> 00:14:44.440 that all happens over in Rust. 00:14:44.860 --> 00:14:48.220 The thing that I think the article covers that's really important is that 00:14:48.520 --> 00:14:51.100 if you pull back that data across the boundary, 00:14:51.760 --> 00:14:55.380 those Rust results get turned into Python objects. 00:14:55.780 --> 00:14:59.640 And so you're going to want to think carefully about how you bring back parts of that, because 00:14:59.800 --> 00:15:03.480