Transcript #216: Container: Sort thyself!

00:00 Hello and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds. This is episode 216 recorded January 13th 2021 I'm Michael Kennedy. I'm Brian Okken and Brian. We have a special guest Yusef. Welcome Hey, great to have you here. You want to just take a quick moment and tell folks about yourself maybe about your podcast real quick Yeah, sure Thanks for even being able to participate in this podcast So my name is Joseph and I might not be well known as you guys are for sure I'm a mechanical engineer from Germany and based in Germany as well And I'm working for IT company called SimScale who are providing cloud-based simulation technology on the site I'm hosting a podcast called engineered mind and I'm working on a bunch of other stuff For example my thesis which will extensively cover. Let's see him this podcast. Yeah. Yeah, I used a couple of cool libraries and stuff over there, which we'll feature here.

00:55 All right, well, the very first item, Brian, Ron, let's talk about maybe doing, what, should I do a pip search a lot?

01:04 - Yeah, well, I kind of forgot pip search was a thing, 'cause when I'm looking for PyPI stuff, I go to pypi.org and just--

01:13 - Yeah, so do I, that's, yeah, exactly.

01:15 It's really fast there.

01:16 - Yeah, but there's a feature called pip.

01:20 you can do a pip search, which the documentation says it's supposed to search for PiPI packages whose name or summary contains whatever query.

01:30 So I can say pip search, Pytest, for instance, and it should show me what if Pytest is a package on PiPI.

01:38 But right now, if you do that, it comes back with a big trace back and it says, fault, what, minus 32500, runtime error, the pipe is XML. Anyway, the API is broken. So this is on purpose. The what happened is actually I don't know really what's happened, but there's the it's the services getting swamped the the search endpoint is getting it extremely hard. So I saw some message or some tweet that was to the effect of is somebody out there running an insane number of searches against this endpoint? Please don't. Yeah, I don't know what's going on. But so there's some guesses, maybe it's a rogue continuous integration server or something weird's going on. But in the meantime, right now, we're going to link to a Python infrastructure status page which has an update on this. So if anybody wants to follow, you can check that out. It says that the search endpoint remains disabled due to ongoing request volume. And I think This really started becoming a problem mid-December, and so I'm not sure what happened then.

02:54 And then there's a related issue on GitHub for pip.

03:00 So there's an issue open saying, "Remove the pip search command." So I think the end result is, and even the error message says, "The search endpoint will be deprecated in the near future." So I think that this way to do pip search is just going to go away.

03:18 So, and--

03:20 - That's actually a little surprising because usually a lot of these things are so backwards compatible.

03:24 - Yeah, and there's quite a discussion on the issue thread, but there's the gist of it is, is the current architecture is not, it was never designed to handle the volume it's getting right now.

03:37 So there's a comment at the end of the thread that says, if you've got an idea for how to do this algorithm better, or a way to do it scaled, go ahead and discuss it.

03:49 But there's a link to, we're not gonna put that link in the show notes, but in the IPI thread or the GitHub thread, there's a link to, if you wanna comment on that.

04:00 But basically, we're bringing this up.

04:02 You may have figured out it might be a fluke or whatever, but it's really going on.

04:07 And a plea to look at your continuous integration scripts.

04:11 And if you're doing a pip search in there, take those out.

04:14 - It ain't gonna work anyway.

04:15 - It's gotta be some kind of bot, some automatic thing like this, because it's already given the error message.

04:21 Like people would stop, you know, if it wasn't.

04:25 Maybe somebody's trying, like constantly trying to scrape all of the PyPI data out, I don't know.

04:30 - Yeah, why do it with search?

04:32 That's just weird.

04:33 - Yeah, exactly.

04:34 I don't know what's going on here, but I guess don't do it.

04:37 - Don't do it.

04:38 (laughing)

04:39 - Doctor, it hurts when I do this, stop doing that.

04:41 So the next one that I wanna talk about is QPython, not QTPython or Qt or anything like that, but QPython, which is a way to do Python on Android.

04:52 So we've talked about a couple of interesting applications.

04:55 We've talked about Carnets or Carnet.

04:58 I think it's French pronunciation, I've been told.

05:00 And that's a really cool way to do like Jupyter on iPad.

05:03 So local, all of these are local, obviously not just running in the browser.

05:08 There's Pythonista, which is really interesting.

05:11 And QPython is also an interesting one for a couple reasons, 'cause you get an SDK and a REPL for your Android device, which is pretty interesting.

05:20 But the reason I'm covering it, I think it's interesting.

05:23 Somebody, I think somebody sent this over.

05:25 No, I ran across this myself.

05:27 Anyway, it allows you just to integrate with the underlying Android APIs and features for automation.

05:35 Yeah, cool, right?

05:36 So you can do things like check the system, you can send out toast notifications, you can interact with applications, you can mess with the clipboard, you can do barcode scanning, speech recognition, send emails, like all those kinds of things around even screen brightness, checking your battery or whatever.

05:58 So if you wanna get access and automate your Android things, Python, well, here's a cool little app to do it.

06:06 - Okay, wait a second.

06:08 So I'm not an Android user that much.

06:10 I've got like one Android tablet, but I didn't know it can make toast.

06:13 - Yeah, well, it really prefers sourdough, but it will go even as far as rye if you have to.

06:22 - No, what's toast?

06:23 Do you know what toast is?

06:24 - It's like a pop-up notification, I think.

06:26 - Oh, okay.

06:27 (laughing)

06:29 - Josef, are you an Android person or an iPhone person?

06:33 - I have to confess I'm an iPhone person.

06:35 I used to be completely against iPhone, but once you're in the ecosystem, you never get out.

06:41 - It's like the godfather, they just keep pulling you back in, man.

06:44 Yeah, I just recently got a new iPhone as well, and I'm general about it, but because we have our mobile apps, the training for the courses, I've got an Android tablet, and I've got an Android phone and so on.

06:57 Oh, also got a comment here on YouTube.

07:00 So is it an own framework, can you use it in Android, Kotlin and Java?

07:05 I believe it's more like an app that you run, and then within that you can do little jobs and stuff.

07:11 So way to aesthetic, it's not something you can bring in that I'm aware of, 'cause you install it from Google Play, for example, to get started and so on.

07:19 But maybe you can plug it in.

07:21 They do talk about having SDK, so possibly, but I got the sense that's more for writing code outside than getting it on your device.

07:27 But yeah, pretty cool.

07:29 So if you're into Android, You want to do Python automation on it.

07:33 This is pretty cool.

07:34 It's free, get it on the Android store.

07:35 It apparently has ads, but it's also open source.

07:38 So go with that.

07:39 - Do you know if there's a counterpart for iOS?

07:42 - I don't know about the automation side.

07:44 There's a thing called Carnets, which is really cool.

07:48 Let's see if I can find that.

07:50 Carnets app, I believe that's how you spell it.

07:53 Yes, that's Jupyter on the app store.

07:55 And that thing, I don't really want to open the app store, but apparently I have to.

07:59 Well, so much for that.

08:00 But Carnets, it's here.

08:02 Oh, and it's also on Google Play.

08:06 Is that the same thing?

08:07 No, that's a totally different thing.

08:08 But Carnets or Carnet is a very cool app that lets you do something similar.

08:13 And there's also Pythonista.

08:15 Those are the two I know for iOS.

08:17 All right, so moving along.

08:19 Yosef, maybe tell us a little bit about your research and then some of the, one of the libraries you've been working with here.

08:26 - Yeah, sure.

08:27 So Open3D is one of the possibilities to visualize.

08:30 - There you go.

08:31 - 3D point cloud.

08:31 - I had it out of order.

08:32 Yeah, yeah, let's talk about PyTorch first.

08:34 - That's fine.

08:35 That's just fine.

08:36 So PyTorch 3D is basically an option.

08:37 Let's say if you work with meshes, let's say a mesh consists of edges and points, for example, and these edges connect all the points and what you get at the end is a mesh.

08:47 So PyTorch, which is for Facebook, Facebook AI research, and they created this framework, so to speak, to be able to work efficiently with 3D data.

08:57 So unfortunately I'm using point cloud data, But the beautiful thing is that if you use PyTorch native application, which you produce for your 3D geometry, it runs, I wouldn't say significantly, but roughly 10 times slower than this PyTorch 3D, which is implemented especially for 3D problems.

09:17 - Wow.

09:18 Okay, so what kind of problem do people solve?

09:20 What problem are you solving when you're working with this?

09:23 - Yeah, so in the beginning, it was like I was doing some kind of research.

09:26 Unfortunately, they are coming out paper like every day And not too many, actually, in the field of deep learning, especially when it comes to point cloud or geometric data.

09:35 And the goal, just to inform the audience a bit, is my goal is basically to use deep learning and use some kind of or create an assistant system for engineers and designers.

09:45 That means, let's say you're an engineer, and we have this CAD model, so C-A-D, which stands for Computer Aided Design.

09:51 So we'd create a model, for example, of a gear, and then you would have that gear.

09:54 But sometimes we have this differentiation between implicit knowledge and explicit knowledge.

09:59 explicit knowledge means this is existing knowledge, which we already know about, let's say this knowledge, this knowledge can sit in a database.

10:06 And sometimes we are not making use out of it.

10:08 And then we have this implicit knowledge, let's say an engineer comes into a company is completely new, and he brings knowledge with him to the company.

10:14 Now, the problem I want to tackle is because we're having so many data, and we're accumulating geometric data in a company, we have to make use of that.

10:21 And my approach is hopefully when I'm at the end of the thesis, which is like in roughly two months, is that I have a system or web application as a front end, where the engineer or designer picks or starts a design or picks a point cloud or a design.

10:36 And then it would suggest the engineer or designer with a probability of what they want to model.

10:42 Let's say he picks a gear, or maybe you want to have like an arrangement of gears, or any specific big component, or let's say you take a wheel.

10:50 Okay, for example, what would you...

10:51 Like for a transmission or something?

10:52 Exactly.

10:53 For a transmission.

10:54 pick a wheel and it could be a Tesla or it could be any other car. And then it would give you a probability, okay, this wheel is maybe from a Tesla. And then it would give suggest you Tesla with a, for example, 89% probability. And then you would click on the web application. This is the idea. And then it would pop the geometry into the web browser in the front end.

11:13 Oh, that's pretty cool. So it basically it's like image recognition, but instead of for pictures, it's image recognition for 3d CAD outlines.

11:21 Exactly. It's so cool that you mentioned it, because there's a big difference between doing convolutional neural networks or deep learning for images, because images are 2D. It's like a 2D matrix. But if you have a point cloud, then you have a tensor of higher dimensionality.

11:35 And then you are kind of forced to use, for example, NumPy and all these kinds of things.

11:40 And if you're lucky, you could use something like PyTorch, PyTorch 3D, which you can also use CUDA on to be way more efficient.

11:46 Yeah, yeah. Wow, that's really cool. So it looks like a neat thing. This is, you know, I haven't done any 3D work for a while, but yeah, it looks pretty cool.

11:55 I would love to see, I don't know, some pictures and stuff.

11:58 It would not be neat, but yeah.

11:59 - They have a very good, like if someone is interested in seeing what PyTorch 3D can do, Facebook AI Research has an own YouTube channel and they pitched PyTorch 3D on that channel and they really do a nice, they show you what you can do with it.

12:12 So it's really interesting.

12:13 - Yeah. Oh, awesome.

12:14 Well, I guess I'd never really thought about applying AI, ML stuff to 3D meshes, but it makes perfect sense.

12:21 And I can see it's totally different than images.

12:23 - Yeah. - Yeah, very cool.

12:24 Brian, you don't do any CAD stuff with your devices, do you?

12:27 - Well, I mean, yeah, some people do.

12:29 (laughing)

12:30 Not me, though.

12:31 There's a lot of CAD that goes on in the ASIC design and stuff.

12:34 - Yeah, I can imagine.

12:35 Yeah, cool.

12:36 All right, now before we get to the next one, I wanna get something sorted out, Brian.

12:40 - Okay.

12:41 - I wanna talk about Datadog.

12:42 So they're back to support the show.

12:43 Thank you, Datadog. - Yay!

12:44 - Yay!

12:45 And so they're really about helping you troubleshoot latency, CPU, memory bottlenecks in your apps.

12:53 And if you don't know where it's coming from, Datadog will seamlessly correlate the logs and the traces at the level of individual requests cross systems, allowing you to quickly troubleshoot your Python app.

13:03 And they have a continuous profiler that allows you to find the most resource consuming parts of your app in production, just running all the time at any scale.

13:12 And it has very little overhead.

13:14 So that's pretty cool.

13:15 Instead of trying to debug it and then deploy it and hope that kind of translates to production, just turn it on and watch.

13:20 So yeah, that's cool.

13:22 So be the hero that got that app back on track at your company, get started with a free trial and support the podcast at pythonbytes.fm/datadog or just click the link in your podcast player show notes.

13:33 Now that that's sorted out, Brian.

13:35 - Yeah, yeah, so sorting, sorting's a thing and the default Python containers are not sorted and there's reasons behind that, but sometimes you need to sort stuff.

13:49 So there's a Python library called, or a package called sorted containers.

13:54 It's a, I like it.

13:55 It's a very, I mean, I like the name at least.

13:57 It's a very easy to remember sort of thing, but it's, this is amazing.

14:03 I looked into this.

14:03 So this was recommended by Fanchin Bao recently for us to take a look at.

14:09 And it's a pure Python based sorted collections library.

14:14 And it's as fast as other packages that are built using C extensions.

14:20 - Wow.

14:21 - That's the impressive part.

14:23 It's also fairly memory safe, but the documentation is pretty cool.

14:29 There's a whole bunch of different benchmarks.

14:32 So you can take a look at how it deals with large things, but it's really pretty zippy.

14:38 It was pretty cool.

14:40 Right on the front page, there's an example, and we're gonna throw this in the show notes too, of just, you've got, it handles a handful of different data types.

14:50 It shows sorted list, sorted dictionaries, and sorted set.

14:54 There's also a sorted key list, and I had to look that up to figure out what that was.

15:00 So, sorted, the sorted function within Python allows you to pass in a key, which the key really is a function to use to create a key for sorting.

15:13 And--

15:14 - Right, 'cause the things in there might not have a natural sort, right?

15:17 Like you put a bunch of order objects in there, well, how do you sort those?

15:21 Do you sort them by price?

15:22 Do you sort them by date?

15:24 - Right. - Right, so you select out that element, yeah?

15:26 - Yeah, you should have it selected out.

15:27 Or you can do something like, they might be sortable by default, but you want it to be like a reverse sort or something like that.

15:35 - Right. - So, and there's some caveats listed so that you have to make sure that your, the key that you pass in, it follows some conventions like two identical items should be, should have the same key, stuff like that.

15:48 That's all reasonable things.

15:50 But it's a fairly easy and complete package to just use.

15:55 It looks, it acts just sort of like a normal, the normal thing, containers like lists and dictionaries and sets.

16:03 It just remains sorted all the time.

16:06 And this is pretty incredible, so.

16:07 - Yeah, I can totally see bugs get into your code 'cause you're like, well, we put stuff into this list and oh, I want the latest one, so it's the last one, but maybe you forgot to sort it before you did that or the first one's the last 'cause you reversed it or whatever.

16:21 - So one of the things that confused me when I first looked at this, I was scratching my head for a second 'cause it looks like a fairly simple set of examples with just like small set of elements in it.

16:32 So like the first one is a list of like A, B, E-A-C-D-B, just a few characters.

16:40 And it's a whole bunch of these examples with just a little small amounts.

16:43 And it says, underneath it says, all of the demo listed above takes a gigabyte of memory.

16:50 And I'm like, what the heck?

16:52 Why is it taking so much memory?

16:54 (laughing)

16:55 - It's only five things, come on.

16:57 - Yeah, I mean, like why?

16:58 - It's cheap, don't worry about it.

16:59 - But there's, hidden in there, there's an example of that five character list sorted list that gets multiplied by 10 million.

17:11 So it's a 10, like a 50 million element.

17:14 >> 50 million characters in a list that got sorted, right?

17:17 >> Yeah. So and then all the operations like count.

17:22 So you can say, count all the C's in there, it'll tell you how many there are.

17:26 Although a lot of these operations like counting stuff with a sorted set take less than linear time.

17:33 So yeah, so there's times you need sort and this is a cool one to check out.

17:39 - Yeah, it's cool.

17:39 It's nice that it's pure Python, super easy to install, right?

17:43 And it's not gonna have any like weirdness around that.

17:45 Like if you say got an M1 computer and thing won't compile or whatever.

17:49 No, this looks really cool.

17:51 Yousef, what do you think?

17:52 - Yeah, this looks amazing.

17:54 I'm also, I'm in touch with my brother on the side and he's also watching our podcast at the moment.

18:00 And he's also saying because of the one gigabyte memory for sorting is incredible, it's crazy.

18:04 - Yeah, that's pretty awesome.

18:07 I guess it's just showing like you can have a ton and it's all nice.

18:10 So it's, I mean, it seems really straightforward, but having these things sorted, we just got dictionaries that would stay put.

18:17 So having sorted dictionaries is also cool.

18:20 (laughing)

18:21 - Yeah.

18:22 - Right, it used to be that they sort of, if they had the same keys and stuff, or they wouldn't necessarily retain their order of the things you added, but now they do.

18:29 - Right, so if people are confused and think, - Well, aren't dictionaries already sorted?

18:33 No, they just stay in the order that they were created.

18:37 - Exactly.

18:38 - So.

18:39 - Yeah, similar, but not exactly the same thing.

18:42 - Yeah.

18:43 - All right, so I wanna, this next one, I wanna riff a little bit on typing and I wanna do that around a tweet, which I think I've got to put into a different type of, hold on, for some reason, Twitter has stopped showing me like the entire conversation of things.

19:00 I don't know why, but I guess it doesn't really matter.

19:02 So Lucas Lenga responded to a tweet that went out there.

19:07 You know, Lucas is obviously his core developer.

19:09 He's been doing really important stuff, but one of the main focuses that he's been working on is around type hints and typing with things like mypy.

19:17 He was instrumental in bringing typing to Facebook and the Instagram code bases and things like that.

19:23 So there's a tweet that says, "Controversial take, types in a Python code base are a net negative.

19:30 That's not Lucas.

19:31 This is his, he's about to have a whole long conversation about this that I'm gonna talk about.

19:35 But Brian, what do you think?

19:38 You retweeted this.

19:39 - Them's fighting words.

19:41 - Them's is fighting words.

19:42 Yeah, so what do you think?

19:46 - I think that they're good.

19:48 - Yeah, I do too.

19:49 I think when I first saw them, I was a little concerned.

19:52 Like, oh my goodness, this is going to potentially, you know, turn Python into something like TypeScript.

19:59 And while I appreciate what TypeScript does to make JavaScript much better, I almost always walk away from working with TypeScript with a feeling of like, ah, that kind of hurt and was painful, I wonder why it had to go that way, you know, because the TypeScript requires, it's like, it's like C# or C++, the types have to match and they have to be there.

20:18 And if they don't match at all, then it just won't work, right?

20:21 It's super frustrating.

20:22 I know this thing is not defined and you know, 'cause there's libraries that might not have types and then how do you work with them?

20:28 It's just, if I find it, there's always some little edge case that's like, oh, this is frustrating.

20:32 But I never feel that way with Python and I really have come to love Python's type hints.

20:37 And obviously, Lucas starts out his conversation saying this is easily disproven.

20:42 If you ever use PyCharm or VS Code, the code completion in there is based on type annotations.

20:48 If you've ever seen your editor highlight a function and squiggly say this expects something else then what you're giving it, besides the number of variables, but like you're giving it a string and it wants a number or something like that, you're using type annotations and you can enhance your code by doing that, right?

21:05 So I was actually talking to Yusuf about this yesterday.

21:08 My philosophy, or maybe my rule of thumb is you don't have to always do it this way, but if you're working in your editor and you have to type more than say three characters to get some kind of symbol to come up, you're probably doing it wrong.

21:21 So like if you have email service and you wanna have email service send account email, you should be able to say .sae, S-A-E, right, send account email.

21:31 And it should know the type that's been returned, what an email service is, that it has this property, and just write it for you, right?

21:39 So to me, a lot of the typing stuff, I know, I mean, this comment is somewhat about bugs, like I never found a good bug because of this.

21:45 To me, that's almost like a side benefit.

21:48 It's about quickly generating code without stopping to go look at the code definition, without going over to the documentation to see what I could have typed over here.

21:57 You know, it's, for example, AWS people, this is insanely frustrating to work with AWS because you get these like weird, create this service and you give it a name and then you get an S3 service back, but it has no idea that it's an S3 service.

22:11 So you get zero help on what anything, even I think go to definition doesn't quite work because it's, you know, use some factory method to reach down some weird place and get the thing.

22:21 So I think really driving the code generation experience without being in documentation, without jumping around and reading all the source, just go for it.

22:30 I think it's super nice.

22:31 So to me, that is the biggest win of all of this stuff.

22:34 So let me give you--

22:36 - The entire thread is very interesting.

22:37 So-- - Yeah, yeah.

22:39 So let me touch on a couple of the points of the thread 'cause I can't get it to come up in the screen share, but that's fine.

22:44 I took notes, luckily.

22:45 So some of the things he pointed out, he's like, here's tweet one of 10.

22:50 So number one, put enough annotations, and then the tooling will connect the dots and make plenty of errors evident, as well as like heighten this code generation, auto magic, right?

23:01 That's one.

23:02 The most common types of errors though, that'll creep in is if none is being used where you expect a concrete type.

23:09 And things like mypy will say, you're using a type that is an optional of something, but you're not checking to see if it's none before you dereference it.

23:17 You're probably gonna end up at some point with an attribute, none type does not contain attribute, whatever you tried to do, you know, upper or whatever, right?

23:26 - None is not scriptable or something.

23:29 - Yes, yeah, something like that, right?

23:32 Or callable or any of the things.

23:34 Also, another common bug is the return case.

23:38 So if you've got a function, you can, you know, maybe check something and return one value, check something else, return another value.

23:46 But if you forget at the end and you fall through and you don't put up some kind of concrete return type, Python functions just return none.

23:53 Like this actually blew me away when I learned Python and I learned about functions that they always, always, always return something.

23:59 There's no such thing as a void function in Python.

24:02 - Yeah.

24:03 - As a C++ person, that probably surprised you too, right?

24:05 Like with your C++ background.

24:07 - I did a bunch of other Perl and stuff like that.

24:09 So it wasn't too bad. - Oh, okay.

24:11 - But the return type is actually one of the greatest documentation features as well, because sometimes you can kind of figure out what the parameters are going to look like.

24:21 Think, you know, you can guess.

24:23 But what's the return type?

24:25 Is it going to be a list?

24:26 Is it going to be a tuple?

24:27 Is it going to be a single element?

24:28 What if there's more than one element?

24:30 Having type hints around the return types is a great feature.

24:35 Yeah, absolutely.

24:36 All right, let me touch on a couple more.

24:38 And I see some listener comments in the stream as well.

24:41 Squiggly lines in your editor, anyone?

24:43 Like I just got this the other day.

24:45 I thought I was supposed to pass an object ID, the primary key in Mongo, but we'd overridden it, and it's actually a string, and it said you're passing object ID when you expect a string.

24:55 I'm like, oh, yeah, I guess I am.

24:56 All right, well, change that.

24:58 Right, that's really nice, instead of that being a runtime error.

25:01 And he talks about the work with TypeScript and Anders Halsberg and what he did to help build that.

25:07 And TypeScript, like I said, is pretty neat.

25:09 But he also points out that, you know, the same company, Microsoft, is developing powerful type checking and code completion for Python with VS Code.

25:17 And they have one of the Python steering council folks working on there, hint, maybe that's Brett.

25:23 And also possibly the Python creator himself, Guido.

25:27 So do you think those two people would be working on something that just provides the illusion of productivity?

25:32 Probably not.

25:33 So let's see, a couple comments.

25:36 Chris May, hey Chris, happy to see you out there.

25:38 He says, "Code completion is such a confidence builder too." I think it's so awesome because for me, it's both amazing for beginners 'cause they can type dot and go, "Now what?" And for experts, they can just blast out code so quickly because you just type dot a few things and you know, like you said, with confidence, you just keep going.

25:54 - Oh, sorry, a lot of these features, you get them if everybody around you writing the code that you're using is using type ints.

26:05 You don't necessarily need to use type ints yourself, but then you're being a bad citizen and not helping the people out that you're sharing code with.

26:13 So if you don't share code at all, and you're only working on projects with yourself, then, you know, go ahead, don't use type-ins.

26:19 - It's up to you, right?

26:20 Yeah, absolutely.

26:21 Yousef, what do you think?

26:23 Do you guys use type-ins on your project?

26:25 - No, not really.

26:26 Like, it's not something that's in our conscious mind, I would say.

26:30 I'm not sure if it's also something related because you're an engineer.

26:33 I wouldn't want to generalize, but engineers are usually bothered with the problem itself rather than digging down on types, for example.

26:40 It depends also on what language we use.

26:42 - Yeah, it's a bit of a computer science-y topic, I can see, but yeah, like I said, I love how it generates the content so much easier.

26:52 Magnuson also commented, "I love, for example, Pydantic, "but I agree with Romalo," Luciano Romalo, who was in this thread, "Hopefully it won't be required in Python "to help people get started." Yeah, so I think the typing stuff is really interesting.

27:06 Like Pydantic, we've talked about a bunch.

27:07 It's a super interesting example of really using typing to generate cool data ingestion and processing.

27:14 Like if you say I've got a Pydantic model and one of its fields is a list of integers but you give it a list and the things in the list happen to be strings that could be integers, it'll automatically convert it.

27:24 Stuff like that is really fantastic.

27:26 I think that's always going to be an add-on type of thing.

27:29 Yeah, even though I'm a fan of type ints, I don't use them all the time and I would be very opposed to having them be required.

27:37 Yes, I would too. I would too.

27:39 I don't think they need to be on the whole code base.

27:41 I mean, it depends if your goal is to say I want to use them for mypy or mypyc and like completely generate stuff.

27:48 But if your goal really is to get a little bit of help with editors, just having it on the boundaries.

27:52 Like here's the data access layer.

27:54 The things that come out of there return whatever and they take the, like you don't have to do anything else and the editors will pick it up and run.

28:00 Yeah.

28:01 Alright, one quick question.

28:02 What is a function return if there's no return?

28:05 It returns none.

28:07 So that's why you don't have to just say whether there's a return type.

28:09 It always returns none.

28:10 All right.

28:11 Next up, I guess we got the one I tried to open with there.

28:15 Yusef is, open 3d.

28:17 That looks fun.

28:18 Yeah, this is, basically a library which you could use in Jupyter, which I tried to use, but somehow they at the moment have problems using open 3d.

28:26 So what you can do is you call open 3d in your Jupyter notebook and then have it have the point cloud visualized.

28:31 however, there are some ways around it, but open 3d, I think.

28:35 If I would start all over again, I would probably use Open3D to visualize my point cloud, which I'm actually working with in my Jupyter notebook.

28:41 I'm not sure if using a Jupyter notebook is also something you would recommend personally, maybe Brian and Michael, if you're a fan of Jupyter notebooks.

28:48 I think it depends on the application, right?

28:50 Yeah, I think it depends as well. And to me, it really depends on what I'm trying to do and like the kind of code. Am I trying to explore data?

28:59 And does it have a really strong visualization component?

29:02 Or is it like a utility type thing?

29:04 So for example, one of the things that I wrote recently that I would never put into a Jupyter notebook, but I find really helpful is we've got literally thousands of video files, MP4s and whatnot for the online courses.

29:19 And in order to import them, one of the things I have to tell the database is how long in seconds is each file and where does it live and stuff.

29:27 So I've got a little script and I just say, go to this directory and generate a little JSON output for all of the files and parse them and tell me how long they are.

29:37 Like that kind of app doesn't belong there, right?

29:39 It's just, that's a command line type of utility type of thing.

29:42 But if I want to visualize something like this, I think it may well be really good for it actually.

29:46 So I think it varies.

29:48 - Yeah, there's a lot of application parts of my work that I think using a Jupyter Notebook actually might be more beneficial.

29:56 So I'm often taking big, huge trace datas and stuff for like spectrum traces.

30:02 And those could easily be driven from a Jupyter notebook and with the visualization stuff would be good.

30:09 - Yeah, cool.

30:10 So this thing is a set of both C++ and Python libraries for basically working with 3D meshes, right?

30:20 - Mostly 3D data.

30:21 For example, if you use a LIDAR, so when you work with a laser and this looks, for example, great, I never watched the video, but if you scan objects in your surroundings, Usually what you get is a point cloud and which you can then visualize using Open3D.

30:35 And the big disadvantage with point clouds is that they're kind of unstructured.

30:39 So you could have one matrix representing one point cloud and you could have the same matrix switching two points but the matrix would be different.

30:47 This is also a problem that a lot of papers try to tackle and make sure that, yeah, get around the spot.

30:53 - Nice, you have the example video here using Open3D for 3D object detection, which is, that's pretty wild.

31:03 Yeah, nice.

31:03 - The things people do these days.

31:05 - I know, I think it's really interesting, all this image processing and analysis stuff, yeah.

31:11 - Good question, Brian, by the way.

31:12 This is what I ask myself when I listen to Python Bytes.

31:14 As an engineer, what are you guys doing?

31:16 It's great.

31:17 (laughing)

31:19 - Absolutely.

31:20 Cool, all right, well, that's it for all of our main items.

31:24 Brian, you got anything extra that you want to throw out there?

31:26 - Yeah, I just wanted a couple things.

31:28 One, 2021 has been exhausting so far.

31:32 - Yeah.

31:33 - I don't know if anybody else has got the same experience, but wow.

31:36 And also, I've got a lot of extra projects, side projects that I'm working on right now.

31:40 Python Bytes is one of them, but there's other stuff going on as well, trying to do more writing.

31:45 And because of that, testing code has shifted to an every other week cadence.

31:51 So it's not going away.

31:53 I know a lot of, oddly enough, I've had a lot of feedback in the last couple of months of people saying, "Thank you for the podcast, I've learned so much." So I do not, I don't wanna shut it down.

32:03 I wanna keep it going, and there's no plans on shutting it down.

32:06 It's just slowing down so that I have room in my life for other projects as well.

32:10 So just wanted to let people know that.

32:12 - Yeah, well, yeah, I try to, for a Talk Python, batch it up and do a whole bunch.

32:16 Just to say, "This week, I'm just gonna get nothing done, "but I'll do a ton of recording," and then just roll them out.

32:21 I had three months of stuff done in like a week and a half.

32:24 I was, I really needed a break after that, but then I was good.

32:27 - Cool.

32:28 - Yeah, cool.

32:29 Well, thanks for the update.

32:30 Yousef, anything you want to share on the way out into the show?

32:34 - I just want to say thank you for letting me or being able to participate in this quick and brief podcast and keep doing what you do guys.

32:41 I follow you both on Twitter and what you share and what you do is really amazing.

32:45 So it's really inspiring for an engineer who wants to delve into the field of Python and all fancy kind of things to listen to your podcast, taking your courses or following you on social media.

32:54 It's really great.

32:55 You learn a lot.

32:56 And I actually have to learn more to be honest to myself.

32:59 - Yeah, I have to learn more.

33:01 - Yeah.

33:01 - It never stops.

33:03 It never stops.

33:03 - Yeah, that's right.

33:04 - Yeah, but that's awesome.

33:05 Thank you so much.

33:06 Really appreciate that.

33:07 - How about you, Michael?

33:08 - I have one quick thing driven by yours, a comment you had last time.

33:13 So Francisco Silva pointed out, we had talked about some of the numpyc, Pythonic, the idiomatic numpy stuff that you might do and how instead of looping over stuff, you can just add, say, two numpy arrays and it'll add them or you can dot product them and whatnot, right?

33:30 So one of the things you can do, I guess we also talked about ones and zeros to generate a pre-built list of those.

33:37 So one of the things he talked about is the all-close method.

33:41 So if you've got floating point numbers, one of the things that's really frustrating like, are these equal? Well, does it mean floating point numbers equal, right?

33:48 Like, they could be so nearly the same, but not the same, right?

33:52 They could be within an insane amount of closeness, right?

33:56 Like, 10 decimal places and then a one, right?

33:58 So, all close is like, well, if they're within, you know, one one thousandth of each other, consider them the same.

34:04 Well, all close takes a bunch of parameters that you can, you can specify the tolerance though.

34:09 So, yeah. Anyway, I thought that was cool.

34:11 Yeah. Hey, while we're on the topic, I may as well throw out, I've got it. So I tried to use this method of using NumPy, and I ran into a problem. So I'm hoping some data science people can help me figure out how to solve it. So my problem is just the simple thing if I've got two arrays, I want to see if all of the elements are element-wise less than or equal to the other element in the other array. Okay, I can do that with NumPy.

34:35 But what I can't, that assumes that all of the elements are the same data type, like comparable.

34:42 If there are strings thrown in there, It doesn't work.

34:45 So obviously, I don't know if it's obvious, but so I had to do some cleanup at a time, but I don't know what the most, the best way is.

34:54 Reach out to me if you've got an answer.

34:56 - Awesome.

34:57 Yeah, I don't have an answer, but I'm sure people do.

34:58 And a quick, quick comment here.

35:00 This is the one I wanna show.

35:02 Magnus Carlsen says, "Tip, I found out about Copier, "an alternative to cookie cutter that can be run later "as well to update the project to a newer template." That's pretty cool.

35:14 I hadn't heard of that.

35:15 And also Toml spec has reached 1.0.

35:18 Parser might be added to the standard lib.

35:21 Also haven't covered that, but that's cool news.

35:22 Thanks for sharing you guys.

35:24 - Yeah.

35:24 - And I guess, thanks for being here.

35:27 Yousef, thanks for joining us.

35:28 And Brian, thank you as always, man.

35:30 - Thank you.

35:31 Thank you so much, guys.

35:31 - Bye everyone.