Transcript #213: Uh oh, Vulcans have infiltrated Flask

00:00 Hello and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds.

00:05 This is episode 213 recorded December 23rd or Anthony, Anthony Shaw here. Is this December 23rd or 24th? You tell me. It's the 24th. It's Christmas Eve.

00:16 Yeah, of course. Yeah.

00:17 In Australia. Awesome.

00:18 In the future.

00:19 Yeah. 23rd for us, Brian and me here in the US, 2020. And yeah, this episode is brought to you by us so we'll talk more about that later. And I'm Michael Kennedy.

00:31 And I am Brian Arkin.

00:33 And Brian we got a special guest here, friend of the show, Anthony Shaw. Welcome Anthony.

00:37 Hi there, great to be on.

00:38 Yeah, it's good to have you here. Thanks for taking time out of your holiday.

00:42 Yeah, no, it's a pleasure. It started last night so the Christmas holiday has now started. I'm off for two weeks. Should be lovely.

00:48 Yeah, yeah, it should be very lovely. So you've got, I think everyone on Twitter is jealous of of all the pictures that you post when it's like cold and gray and you're like, "Oh yeah, there's this beautiful sunny beach here in the summer in Australia that I happen to be suffering through." >> Yeah, it's the middle of summer here and it's beautiful weather and five minute walk to the beach.

01:11 >> Yeah, not too bad.

01:14 Cool. Well, welcome to the show.

01:15 Happy to have you here. I want to kick us off with this project called Django Ledger.

01:21 Django Ledger is, have you guys heard of this?

01:24 - Yeah, so Django Ledger, you've probably heard of QuickBooks or FreshBooks or some of this accounting software, right?

01:31 That you've got to work with.

01:32 You know, it's allows you to do either, it's a desktop app or some kind of online thing where you create purchase orders, you keep track of who your customers are, when they owe you money, you accept payments, all that kind of stuff, right?

01:46 So Django Ledger is something like that built in Django, which I think is really cool because it could serve multiple purposes.

01:54 You could take it and just run it for yourself or for your company and then customize it.

01:58 Or you could actually use it to extend something, build something on top of it, or maybe even offer services in it.

02:05 If you're like Stripe, maybe it makes sense to integrate some sort of plugin here 'cause then you get 3% of everything that company makes, basically, the way credit cards go.

02:15 So it's a bookkeeping and financial analysis engine for the Django framework, which pretty sweet.

02:20 It's open source.

02:21 And if you look through its features, It has a chart of accounts, basically customers, financial statements, it has multi-tenancy support, it has stuff for operations, for investing, it has bills and invoices and bank accounts, all that kind of stuff.

02:37 And yeah, pretty awesome, right?

02:39 Wait, multi-tenancy?

02:41 So is there like North Tennessee and South Tennessee or?

02:43 That, that would be Dakota.

02:45 I think Tennessee, there's only just one Tennessee.

02:46 No.

02:46 Yeah.

02:48 I think if you wanted to run this as like a platform as a service type of thing, and you wanted to offer up, if you wanted to basically create your version of FreshBooks, I feel like that's what it would be.

02:58 - If you're an accountant that serviced multiple clients, you'd probably do it that way, I guess.

03:03 It looks pretty cool.

03:04 I thought I'd add, there's a project called Ledger, which is open source, but it's not a, I don't think it's a Python project.

03:10 I think it's written in something completely different.

03:13 So yeah, this is really cool and great, I think for businesses who maybe have someone who can set this up, I'm guessing it's not available as a service.

03:22 Do you have to kind of?

03:23 - I don't think so.

03:24 Especially, yeah, yeah, if you look at it, it says this project is under active development.

03:28 It's not quite ready for production.

03:30 So this comes to us from Miguel Sanda.

03:34 I believe that's his project.

03:36 And I would love to see what the roadmap for stability is and when this is ready, when it's coming out.

03:42 But it's basically all Python and HTML and just a tiny bit of TypeScript, like a salt level.

03:48 Stuff that's coming is like inventory management, cash flow, taxes, all the fun stuff.

03:54 And Brian, even BDD's coming.

03:56 - I don't understand.

03:58 Behavior driven, oh, behavior driven development tests.

04:00 - Yeah, yeah, yeah.

04:02 And so they're also actively looking for contributors, especially anyone with financial or accounting experience.

04:08 So if you're looking for some project to contribute to, you know, that'd be great.

04:12 You could contribute to this one.

04:14 It's not super well known yet, but I thought I would shine a little bit of a light on it because it seems like it's a cool idea.

04:19 - Well, it's even got invoicing and stuff too.

04:21 So I think this is pretty neat.

04:24 - Yeah, absolutely.

04:25 So what's up next for you, Brian?

04:27 What do you got for us?

04:28 - Next up, oh, so another web sort of thing.

04:31 But this, I ran across Flask Meld.

04:35 - Flask Meld, I've never heard of this.

04:37 So is this like a Vulcan thing from Star Trek?

04:41 - No, the idea is like melding the front end and the back end.

04:45 And it's a pretty cool, You should click on the example article.

04:52 But yeah, that first link.

04:54 Right there.

04:55 Oh, right here?

04:56 Okay.

04:57 Yeah.

04:58 There's a...

04:59 It's basically fairly simple interactive stuff where that JavaScript gives you.

05:05 But it's super fast and it just...

05:10 You don't have to write the JavaScript code.

05:12 So the article in this article talks about how--

05:16 this is from Michael Abrahamson.

05:20 But he wanted to avoid writing JavaScript, so he wrote a whole bunch of JavaScript to make this plugin.

05:28 So it's a Flask--

05:29 his first Flask extension.

05:32 But you modify the Flask templates to insert these elements.

05:37 And then the elements just show up as objects in Python.

05:42 and you can program them like that.

05:43 So all these elements are all implemented in Python.

05:46 Oh, cool.

05:47 And there's a little video on there, but there's a demo as well.

05:51 So there's a working example.

05:53 And then the code for the example is up on GitHub as well, so you can play with it.

05:58 One of the fun parts in one of his working examples is the dropdown for search.

06:06 So if you start predictive search sort of thing, so if you start, the example shows states.

06:11 So if you start typing states, it'll like start filling in this list of states that you might mean, and it's like super fast.

06:20 - Nice, yeah, apparently it says right here that it utilizes WebSockets and Morphdom to create server-side rendered HTML and swap out DOM elements without refreshing the page.

06:30 That's pretty killer actually.

06:31 - Yeah, and he's nice enough, I think this is cool, he's saying that he basically stole all the ideas from other people and put them together for this, or standing on the shoulders of giants, so to speak.

06:44 And there's a similar project called Django Unicorn for Django people.

06:48 I played with that a little bit.

06:50 The Django Unicorn is, the documentation site's amazing.

06:56 It's got a bunch of examples.

06:58 But it seemed, I don't know if it's the server that it's hosted on or what, but the Django example seemed a little bit slower.

07:06 It seemed like there was this round trip thing going on, whereas the Flask example, Flask meld was zippy enough that that would be completely sufficient for a lot of the applications I'm thinking of throwing in some interactive stuff.

07:19 - Yeah, this is really neat.

07:21 This idea of components is pretty cool here.

07:24 Anthony, what do you think?

07:25 - Yeah, it's interesting.

07:26 I don't really use Flask, but in the Django world, quite a lot of components and stuff that I've pulled in and plugins to do interactive elements.

07:36 And each one is implemented completely differently, which is really frustrating, whether it's like a search box or a type ahead or like a multi-select field, just something that I need, which is not available in the standard forms model.

07:50 And each one tends to have its own JavaScript and its own collection of bugs.

07:55 And yeah, it is quite frustrating to be honest, because you kind of think, oh, I don't need to write this from scratch in JavaScript.

08:02 I'll use an extension that already exists out there.

08:04 And then you kind of get stuck in, does it work in Django three?

08:08 Right.

08:09 You know, what state is the JavaScript in?

08:11 And then you read through the source code and you stumble across security issues.

08:15 And it's like, okay, I wasn't, you know, there's a, there's a line between it's easier to write it myself and you end up, I ended up just maintaining folks of all these extensions and then trying to get pull requests back into them to kind of fix things up.

08:27 But, yeah, it's interesting to see, I guess, a different model for doing it.

08:31 which could be applied to Django.

08:35 >> This thing brings me back to the year 2001 web with ASP.NET web forms, which are such a weird way to build web apps.

08:45 They tried to mirror desktop apps, but you could put a little Ajax tag onto a section and just that part of the page would automatically refresh and interact differently.

08:56 This gives me that feel, but not old school web, but more modern web, which is nice.

09:01 Yeah, I mean, like I was thinking of it, and now a lot of people that are comfortable with like jumping into some JavaScript, this seems silly, I'm sure. But for instance, like me, I never touched JavaScript. So if I've got a little, I've got like a little flask app that pulls up, like test result data, and it'd be great to just have a be able to get a little form there that says, hey, here's here's the version I want to see the results for and be able to pull that up. And it, it doesn't have to be pretty. But this way, I could implement it without having to go in and learn JavaScript.

09:34 - Yeah, yeah, and I think this is super neat.

09:37 I'm with you, Anthony, on the trade-off of grabbing some of these cool plugins, Flask, Django, whatever, and go, okay, this is just now adding functionality, but then you've kind of gotta understand its assumptions, when does it work, like why does it not quite work for what you're doing, it's always a trade-off.

09:53 I usually go for the vanilla version of the web and just build it myself until it's like really clear that there's a big benefit.

10:00 - Yeah, and you bring up an interesting point with the security concerns, especially anytime you've got dealing with input fields and stuff, you've got to be careful with that.

10:09 - Yeah, because they normally run queries in a database.

10:12 So you've got to kind of audit these things to make sure that they're not using raw queries and that they're not using weird templates and stuff like that.

10:21 - Right, they better be parameterized queries and not little Bobby Table type queries.

10:26 - Exactly.

10:26 - Yeah, yeah.

10:27 All right, wrong one, this one.

10:30 So what about what's next?

10:31 maybe they're using even bitwise operators in there, Anthony.

10:35 Possibly.

10:35 Yeah.

10:36 so my next, link is bitwise operators in Python, by Bartosz Zatrznicki, and this one's really cool.

10:46 What she, I think if you've ever used, bitwise operators or seen them in the Python language.

10:52 So this is where you'd use, the pipe symbol to, less than less than, which kind of looks like two arrows or right, right, right, which is greater than greater than there's a X or is, is there another way to say X or I just use X or.

11:09 Oh yeah.

11:10 That's what I say as well.

11:11 You didn't say Xor or something.

11:12 No.

11:13 Which is the hat sign.

11:14 I can't, well that symbol is called the tilde is for not.

11:19 So yeah, these are basically used for specific types in Python, which support bitwise operators.

11:26 I like this article because they're rarely used in Python because you typically use types where you'd use a method to do a lot of these things.

11:36 Or most of the time you wouldn't necessarily need to work with data which is binary.

11:42 So you wouldn't necessarily need to do these things.

11:46 But if you are working with binary data, they're super useful.

11:49 And it takes a bit of time to get your head around.

11:52 So I've got a couple of examples, but yeah, I love this tutorial because it actually, it doesn't assume you know anything about this topic explains what the binary system is.

12:03 And it uses these icons of hands like it uses emojis, kind of like a mini sign language to explain the the the kind of the bytes and stuff like that, which is really cool.

12:16 See I really like it for that reason.

12:18 It's really illustrative and kind of takes you through a few concepts and then how you You can use these special operators for some of the built-in types, Python integers, bit strings.

12:31 You can also use them for byte arrays, which is really helpful.

12:35 And then it goes into things like bit masks and stuff like that.

12:38 So if you're ever working with any lower level data, this is super helpful to understand.

12:45 And also a little trick I discovered a while ago was if you've ever used the enum built-ins.

12:55 So enum is in the standard library. And if you want to describe an enum, you basically create a class and inherit from an enum type, which is in the enum package. And you can do a whole bunch of things and just represent things as enums in Python instead of other weird ways of doing it. I don't know. Introduced in Python 3.7, I think it was. But there's a little known feature in the enums, which is there's an int flag and a flag type. An int flag is an enum, which you can represent as a flag, which means you can combine multiple of them. So for example, if you had a color, enum, and you had red, blue and green, you could represent white as being red, blue and green, and by combining the colors together.

13:51 So nice. Wow, that's cool.

13:53 Yeah, you probably wouldn't make a paint mixer in Python. But there's lots of things, flags, for example, like if you're representing system flags, or compiler flags, or any other kind flags. This is really useful because it actually implements all the binary operators. So yeah, flag and int flag are super helpful types which are built into the standard library. So if you ever find yourself doing something like this, or using a package which has implemented its own weird version of this feature which is built in, then you can upgrade it to this new syntax and and then you're done.

14:29 - So, tell me what does this auto do?

14:32 - Oh, so in enums, if you're gonna say, you know, you need to make a representation of the enum name.

14:43 And if it's an int enum or int flag, you'd say, you know, one, two, three, four, five.

14:49 Like, so if I save it to a file or pickle it or something, then it's, this is the number that it represents.

14:54 So I can convert.

14:55 So if you converted the number one to your enum, it would know that red was one.

15:00 So, you know, you could basically use it for storage or something.

15:03 It's also really helpful, like if you're reading from XML files or JSON files or something, and there's like, there's a field which is stored as an integer, but you know that it actually represents something a bit more logical, like enumeration, then you can do it that way.

15:20 However, for flags, then, because they support bitwise operators, they should follow the binary sequence.

15:26 So 1, 2, 4, 8, 16, 32, 64, et cetera.

15:30 And once you get past 1,024, most people start to forget.

15:36 So instead of working that all out in your head, you can just use auto, which is a function built into the enum.

15:45 And it will basically just work out what that value should be for you.

15:48 So you don't have to work it out in your head.

15:50 - It'll let you like reorder stuff and not have to go, oh, now I just wanna go one, two, three, or I want to add another one, but in the middle and accidentally messed it up or whatever.

15:59 I think I might use those anyway just to indicate that the actual number isn't important.

16:04 It's just that they're unique.

16:07 Exactly.

16:08 I think if you don't assign it, then it works, but with flags you need to assign it to something.

16:15 Yeah, just for people listening who don't necessarily see the code, the show notes, The idea is you create a new class and you say like, category equals just lowercase auto open close.

16:25 That's the way that you sort of invoke this behavior.

16:28 Also a couple of questions from the listeners who are in the live stream.

16:31 Anthony Lister says, useful for MicroPython or hardware maybe?

16:34 What do you think?

16:35 Yeah, absolutely.

16:36 If you're working with embedded systems or anything, I mean, Brian, you're the expert on this.

16:42 Yeah, you have to work with binary a lot.

16:44 So yeah, this is super helpful because you can represent stuff which is different states in the system or if you want to read inputs and stuff from multiple channels, you're going to need to use Bitwise operators.

16:57 So both the Bitwise operators are really helpful as well as the flags enums.

17:02 Yeah, and Brandon Rayner says Bitwise always confuses me.

17:06 When I see it, I tend to roll my eyes back and ask why. I'm sure there's a reason.

17:10 Well, I mean, one of the things that comes to mind for me is, you know, you're thinking about storing stuff in memory.

17:15 you know, if I wanted to store, say, a number or something like that, or, you know, you put that into a Python number, that's like 28 bytes.

17:22 But if you create an array of bytes and you know, the size of them are going to be, you know, packed into little bits there, you know, like one to 10 or something, a whole bunch, you could be way more efficient by, you know, creating little smaller containers and then bitwise oring them together and whatnot.

17:36 Yeah.

17:36 Like bitfields for instance, are a really cool thing, but even just, just straight numbers with bitwise operators are important for hardware because you're like a lot of times you just have register access to something or you have memory memory mapped registers and you you know just uh kind of read those out and there may be a whole bunch of data so each bit might represent completely wildly different things so you can't really just check for equality you have to check is this bit on or off um or i need to set this bit and leave all the rest alone things like that - Yeah, yeah, absolutely, absolutely.

18:09 All right, before we get to our next item, let me just let you know this episode is brought to you by us, things that Brian and I are doing.

18:15 And so one of the things we just launched over at Talk by Then Training is our brand new FastAPI course, which is a super cool new API framework that I think brings together a lot of the things that we've been proponents of, you know, things like Pydantic, TypeHints, Async and Await, all those, they all come together really nice over there.

18:33 So if you wanna learn that, check that out.

18:35 Brian has some book on pytest, so yeah, you can check that as well.

18:38 Links in the show notes.

18:40 The next thing I want to talk about, though, is why you should use an ORM.

18:45 Here, there we go.

18:46 So, Anthony, Brian, what do you guys think?

18:49 Raw SQL ORMs, what are your thoughts here?

18:52 Well, I've been using, like, document databases lately, and I don't really need an ORM, so...

18:58 I'm with you. So, over there, maybe the R is a D, at least if it's a document database, it's an object document mapper, maybe, But yeah, I find being able to work with classes, like the way I think of it in Python to be, that's how I want my data to be and just something else can figure out how the database has to break that apart into relationships and stuff.

19:15 Super neat.

19:16 You know, like SQLAlchemy, go and say create, like if I had a user and the user had orders, like they might have an orders list on the user class.

19:26 To create a new order, you can just go to the user.orders.append, that new thing and hit save, commit changes, and then boom.

19:32 you know, it like figures out that has to be inserted and the relationship has to be said and all that.

19:36 Anthony, what do you think?

19:37 I use the Django ORM quite a lot and really like it.

19:42 Learning all the edge cases where it creates queries which are not super efficient.

19:48 But so I guess there's pros and cons like if you were to write raw SQL, you know, sometimes if you know SQL really well, then you can write more efficient queries.

19:56 However, there are typically ways around that.

20:00 And the n plus one problem in Django.

20:03 The n plus one is the biggest problem for sure.

20:06 You want to describe the n plus one problem or take a shot at that?

20:09 Oh, yeah. So if you sort of have a foreign key and you reference an entity, which might be a many to many relationship, and then you reference a property of it in the query, not actually not in the query, but actually in the view, you, you, you mentioned something or you look up a field which is part of a mapping to another table.

20:28 Kind of like I described, like if you gave a user to the view and the view wanted to know about its orders.

20:33 Yeah.

20:34 So more probably if you gave a list of users and it wanted to know about the orders for each of them, right?

20:39 That's the real bad case.

20:40 Yeah.

20:41 So you had it for like, you know, which group people are in or like which team they're in or something.

20:46 And it's just, it's the team ID.

20:48 And then you actually want to show it on the table was the team name.

20:51 So what ends up happening is that for every row in the table, it does another query to look up the team name, even though they're the same across the board.

21:00 Whereas if you're writing that in SQL, I'd hope you'd do that as an outer join.

21:06 But yeah, so, sorry, not an outer join.

21:11 So yeah, OMs are really useful in that sort of thing, but they do have edge cases.

21:15 For N+1s, pretty sure you covered this one before, but there is an awesome tool called N plus one, which you can run in your test suite.

21:22 So when you do all your integration tests with Django, it fails the test if it detects an N plus one query, which I use quite extensively.

21:32 I think Adam.

21:34 - Oh, I had not heard of this.

21:36 I mean, this is not such a problem for me because I work like Brian in document databases, but to the extent that I work with like SQLAlchemy and stuff like this is really, this is the one you're talking about the one.

21:47 - Yeah, yeah, that's it, yeah.

21:48 L-U-S-O-N-E. Yeah, that one. Yeah. So I add it. I only add it in the test suite. So you don't, you don't really need to put this into production. So you just load it into Django.

21:58 You're assuming that you have tests, Anthony. Yeah. Well, that's probably a starting point.

22:02 So, or if you're, if you're, you're developing an environment, you can just turn it on, but it would, yeah, it basically, it would print a warning and stuff like that. However, there is a config option that says that it can raise an exception if it detects one, which is awesome because when you run all your tests, it will fire off and fail the test if it detects this type of query. And the workaround in Django is actually really simple. You just add another function to the chain, the query command basically, which indicates that I'm going to use this field in this other table. So it kind of pre-looks up.

22:40 Right, right. Do the join or sub query load or whatever it needs to do. Yeah.

22:45 And the other thing about IRMs that I really like is the migration ability.

22:50 Like, it's fine when you initially design a system, you could, yeah, fine, you can write your own custom SQL queries and stuff.

22:57 But, you know, within a week or two, you've added like five extra fields.

23:01 And actually, that one field you added now needs to be a different type or, you know, databases are not static things like database structures change all the time in a real application.

23:12 So that becomes an absolute nightmare if you've handcrafted all your SQL, especially if you don't have tests.

23:21 So yeah, I mean, for different database engines, there's different ways to do this.

23:25 I mean, SQL Server like has DAX, so you can do DAQ packs and DAQ pack migrations, but most of the other database systems don't have that.

23:35 Whereas if you're using the ORM, then, you know, Django and Flask and stuff would create the migration for you.

23:42 you're kind of pretty sure that you can add fields as you wish, without necessarily breaking things, or you can add types and you can describe what to do with the old ones.

23:53 So I think it saves so much time in just working with a live system where you're actually constantly making modifications to the table structures and stuff.

24:03 - And maybe you have different developers at different stages, or you want to go back in the brain, you say, I'm going to go back here and then work on this version of the app at this stage, you want to be able to go to just run the migrations and get to the right structure.

24:17 I find it to be super, super helpful.

24:19 I mean, there are times where you want to get a 100,000 records back from the database and ORM doesn't make sense.

24:26 But to me, I feel like you should 80, 90 percent of the time use an ORM and then there's that little edge case where something slightly different maybe needs to happen.

24:33 But you shouldn't start there because there's that 10 percent or five percent chance.

24:37 Yeah, that most of our rooms give you the ability to write a raw query. Yeah. And and yeah, you should also check those raw queries for SQL injection. And I do have a tool for that.

24:48 If you if you want to, you can run it through the Python security thing that I've built, which will scan your code and spot where you've used the raw SQL functions in the flask or m and the Django or m. And it will see if there's anything in them, which is pretty likely to leave give you subject to a secret injection.

25:08 - Yeah, is that built into your PyCharm security plugin?

25:13 - Yes, yeah, it is.

25:14 - Yeah, I wanna just mention this conversation, although not exactly, we didn't pull that much out of it, but comes from an article I wanna highlight called "Why Should You Use an ORM?

25:24 Object Relational Mapper," written by Karim.

25:26 I'll leave that in the show notes.

25:27 There's a bunch of details that walks you through it.

25:29 So I wanted to cover this so people, if they're having this debate on their team, or they're not sure which way to go, they're new and they're like, what are all these funky acronyms about?

25:38 You know, they can check that out.

25:39 There's the obligatory little Bobby Tables joke in there.

25:44 I gotta go with that.

25:46 This is what your plugin is supposed to detect.

25:48 Is this cool XKCD.

25:50 Cool.

25:51 If you're not on the receiving end.

25:53 One of the things I'd like to bring up around ORMs is that like a lot of the ORM tutorials assume that you know SQL.

26:00 And I'd like to see more tutorials on how to use an ORM and how to use it correctly without any SQL examples.

26:08 >> Yeah. You know classes in Python.

26:11 >> Yeah.

26:11 >> Here's your way, you don't have to know SQL.

26:13 >> I thought that was the point, is that you can think in objects instead of in SQL.

26:19 Why do I got to learn SQL in order to understand the tool?

26:24 >> Another thing that you can turn on that's helpful for the N plus one problem, at least with SQLAlchemy, you can go to the engine when you create it and you can say, echo, echo equals true, and it'll echo every single underlying SQL command sent to the database.

26:39 You can do something in SQLAlchemy and it'll say, here's what the actual thing is.

26:42 And if you've written your code well, and you've done the right join stuff, you'll just see like a couple of entries for each page or interaction you're having.

26:51 You've done it wrong, you'll see your output just scream by full of these things.

26:55 Like, ah, there's one of these problems.

26:56 So that's an easy way as well.

26:58 - Is there like a test way to like, Like for instance, the check to see how many for a certain test sequence, how many database interactions you've done?

27:07 Yeah, there is. Yeah. There is in Django, you can kind of hack a bit of middleware, which kind of catches SQL queries and stuff like that. It's not built in, but you can write it. I think I've got code sample somewhere that does that. And just say how many queries this page executed.

27:26 Obviously, it depends on that's why it's important to see the database first with test data, because if you just run it on an empty database, then typically, or you've got like one row or something, then it's going to be like, "Yeah, it's fine." But then when you deploy it to production, it's actually running thousands of queries for every page.

27:44 Yeah. Did you end up with what you got in the United States when we tried to roll out the healthcare.gov and the entire system went down? And yeah.

27:54 Yeah, whatever.

27:55 They knew it was going to be the entire country using it and they didn't load test it.

27:59 Exactly.

28:01 One more thing, a comment from Dave.

28:03 It's also useful when using Django to add the Django debug toolbar or Django Silk to be aware of what queries are going on.

28:11 And also I would add for Pyramid, there's a Pyramid debug toolbar.

28:15 And you can actually see the underlying queries and how many of them there are and the timing.

28:19 It's really nice, those things.

28:21 All right.

28:22 Well, let's move on to the next one.

28:24 Sticking with the database stuff, Brian.

28:26 Definitely sticking with databases.

28:28 And SQL. So this one's from Simon Wilson.

28:31 And he's got a tool called, or a library called--

28:35 Dataset.

28:36 Well, yeah, sure, Dataset. But SQLite Utils.

28:40 This is, it's developed as part of the Dataset project, but it's usable by anybody that uses SQLite.

28:50 And it's a couple things.

28:52 It's a command line utility, which a lot of databases do have a command line way to query the database, but I don't know if SQLite does.

29:02 But this is a pretty cool command line. You can interrogate, analyze tables and dump things and do all sorts of stuff.

29:09 And even search, it's pretty useful in pretty simple examples.

29:15 One of the things I really like though is the API that it has.

29:20 So there's a Python API to, I mean, you can use SQLAlchemy, for instance, to interact with SQLite.

29:28 But another way to do it is to use the SQL Utils as an API for SQLite.

29:35 It's a pretty clean, simple interaction and I think that's something I want to try because it looks like, I do need to get back into some SQL database work, and I think this would help a lot.

29:50 >> Yeah. This comes from the dataset, it's one of the foundational tools from dataset, this project that he's working on, which is really interesting.

29:57 It's like empowering data-driven journalism and data exploration across all these different data sources by converting them all down to SQLite databases.

30:08 Then once they're there, you can explore them in interesting ways.

30:10 So there's like Twitter to SQLite, various other things to SQLite, Gmail to SQLite, and then you can like explore all these different things like GeoJSON to SQLite.

30:22 What else have we got here?

30:23 There's just tons and tons of these things that plug into other stuff and then gets into this common format.

30:28 And then he also has this cool tool called DogSheep, which builds on top of those databases that creates like a personal search engine for your life.

30:36 So you could like plug in Twitter, you could plug in your iPhoto's library, You could plug in your Gmail and then there's a search engine that just says, search that.

30:44 So it's like one of the foundational building blocks of like, how do I get everything from its original source into SQLite?

30:51 Pretty cool.

30:51 - Yeah.

30:52 - Yeah.

30:53 So, Anthony, what do you think?

30:54 - I'm actually really interested about the next link on his blog, which says I commissioned an oil painting of Barbra Streisand's cloned dogs.

31:02 (laughing)

31:04 - It's really nice, actually.

31:07 I kind of like it.

31:08 I thought it was a joke.

31:09 He actually did.

31:10 For gazing the tombstone of the dog that they are.

31:12 Yeah.

31:13 Okay.

31:14 Interesting.

31:15 Yeah.

31:16 I'm not sure what it has to do with SQLite, but it's pretty cool.

31:17 I think, yeah, more tools for journalists and stuff as they start to work with data and data becomes more readily available, or at least there's more of it.

31:26 So yeah.

31:27 And people aren't necessarily, yeah, don't necessarily have the technical skills to work with massive data sets and stuff like that.

31:34 So yeah, it'd be really interesting to use some of these tools.

31:39 For sure.

31:40 Simon Wilson was one of the co-founders of Django.

31:43 I had him on Talk Python recently, but it's not published yet.

31:46 And we talked about Datasette and DogSheep.

31:48 And there's just a lot of interesting stuff.

31:50 But one of the stories that he tells, so people can listen to the episode, but he has a dog, which may be this commissioned thing.

31:56 He's like really got this dog that has a Twitter.

31:59 The dog has its own Twitter account.

32:00 So for example, some of the interesting stuff that he did was his dog in the Twitter account will tweet when it goes to the veterinarian, like how much it weighs and stuff.

32:08 And then it will tweet pictures when it's on a walk.

32:10 And he was able to do things like create a graph over time of his dog's weight by just doing a SQL query against the dog's Twitter account that got pulled in.

32:21 Like all sorts of weird connections of like pulling data together that you just couldn't imagine.

32:26 So yeah, anyway, really, really neat stuff there.

32:29 Okay.

32:30 Yeah.

32:31 All right.

32:32 People would check that out.

32:33 And then you want to wrap it up with a talk that is very work from home-ish.

32:40 Conference is very work from home-ish.

32:41 A talk from there, maybe?

32:42 Yeah.

32:43 Yeah.

32:44 So my next one is from the Pyjamas conference, which happened a couple of weeks ago.

32:51 And this is, yeah, lots of online conferences happening at the moment, which I kind of sign up for and look at the, you know, the talk list and think, oh, great, that'll be really interesting and then never actually get around to watching any of them.

33:04 Exactly.

33:05 Yeah, I'm just kind of struggling with this online conferencing thing.

33:09 The ideas are really cool.

33:10 The concept is cool, but the practicalities of it.

33:13 Yeah, especially like, I don't know if it's different in our house, but the conference run over a weekend and I just wanted, you know, maybe an hour or so to watch some of of talks and things, but just sitting down on the couch and turning the TV on, putting on YouTube and then sitting and watching talks.

33:29 And it was like, the kids isn't making so much noise.

33:31 My wife's like, why are you watching TV?

33:33 There's loads of stuff to do.

33:34 And I'm like, no, but I'm, this isn't.

33:37 And then...

33:38 Is this work?

33:39 It actually is work.

33:40 I'm sure it is.

33:41 Yeah, yeah, sure.

33:42 It looks like work.

33:43 Yeah.

33:44 So, yeah.

33:45 I see pajamas.

33:46 There's pajamas up there.

33:47 This doesn't make it, this doesn't feel like work.

33:48 Yeah.

33:49 And I definitely wasn't in my pajamas.

33:50 10 in the morning.

33:51 So, yeah, I don't know.

33:54 I know I'm kind of talked about this and a couple of other people said they're having similar challenges with these online conferences.

34:00 Just like, where do you fit them in?

34:02 And if you actually go to a conference, it's like you've got an excuse that this is more, this is like my time is now set out.

34:10 Whereas if I was to say I'm going away for a few days to a conference, that's fine.

34:14 But if I try to attend that conference at home, it's like that's not.

34:18 Yeah. Yeah.

34:18 I'm going to lock myself in my room and watch TV for two days.

34:21 I'll be back.

34:22 Yeah, yeah, exactly.

34:23 It's not the same story.

34:24 Yeah, someone at work actually suggested that we have like a work hotel sort of thing where people can attend virtual conferences.

34:33 Where, you know, especially if you're on if they're late at night or they're in weird time zones and stuff.

34:40 So even if it's just in your local city, you can go and stay there.

34:44 So you can at least dedicate the time.

34:46 Anyway, the pyjamas, which I really wanted to attend, I managed to get through half a talk.

34:51 And then I watched the other half like a week later. And so that was not very successful.

34:58 However, the talk was brilliant. So I wanted to share a link. It was called what the struct by Zachary Anglin. And it's talking about the struct library, which is built into the standard library. I didn't really realize there was a struct library, right? Like there is an array library that says this will hold floats contiguously. Same thing for structs, right?

35:22 Yeah, exactly. And I think, you know, Brian was talking about data classes before we went live, actually. But, you know, data classes are really cool. Like if you wanted to represent interesting structures and stuff like that. ORMs are really important for this sort of thing.

35:37 But if you're working with binary data, reading the binary data and then converting it into native Python types. Often, you know, you do C for that sort of thing. And in C, you just declare a struct and just say, there's these fields. And you can just read and write that into binary structures, which is kind of built in. But sometimes you need to do that in Python, if you're working with... Actually, I've got some examples. So this explains what the struct library is. And it also has this macro language for describing what the underlying type is.

36:10 And then you can kind of like pack and unpack it basically. So you can say, here's my class or whatever. And here's the fields and the underlying binary structure is a float 32 bit integer or binary Boolean or, you know, it may be a ASCII string or something. So yeah, there's basically all these like little characters for packing and unpacking data. So yeah, it's really helpful.

36:37 If you're getting into this topic of working with binary data structures, or something that you've wanted to do, I recommend if you're on a Mac using a tool called Sinalyze it, or Spice Sinalysis, which is basically a GUI for working with binary data. And it has a database of grammars.

36:58 So you can basically like open a compiled executable or, you know, something like that.

37:05 And it would be like, oh, I know what this format is. And it would then just represent it in actually something more understandable. And you can edit fields and hack around with stuff.

37:14 It's great if you're doing capture the flags, or if you're doing some hacking for good.

37:21 It's also really cool if you want to edit files, which are not in a human readable format, if they're in a binary format. And in the database, they've got a whole bunch of examples, including a lot of save game formats. So this is actually how I got into this topic.

37:40 Originally, it was when I was a lot younger playing games and stuff. And I'm like, what is this file? And then, you know, trying to echo it on the shell, and it would be like, oh, that's all garbled. And then realizing it's all in binary. And then, you know, trying to figure out if you open it up in a hex editor or something, what it is. And then, oh, if I change this field, than I can give myself more gold or I can make my character invincible or...

38:03 So basically like hacking the game by modifying the save game.

38:07 This end would also be so much easier if I had a hundred thousand hit points or whatever. So let's just increment that number.

38:13 Yeah, exactly. I think modern games are a bit, they kind of put protections around the save game because there's a lot more online aspect, but definitely older games, you can hack the save games and most of them give yourself whatever it is that you feel like. So yeah, understanding binary structures is really helpful on the structs.

38:30 You can even automate it using the structs much more.

38:34 Yeah, nice. Yeah, there's also some good comparisons between like the values of structs and then tuples and named tuples and data classes and pydantic. So there's sort of a spectrum that gets covered there as well.

38:46 Yeah, interesting. Goes in line well with the bit manipulation topic you were talking about. It's like for instance, Very structured structs are very important for message systems, like communication systems.

39:03 We have very defined fields within each, within the, you know, we've got 128-bit sequence or 1,000 bits in there. Each bit is specifically assigned to different things. And those are, yeah, you definitely don't want to just hope that it's right. You have to put it exactly where you - Yeah, exactly.

39:25 Like little network packet headers and things like that.

39:28 - Yeah, I'm using this at the moment 'cause I'm writing a disassembler.

39:32 So it's quite helpful.

39:34 - Yeah, awesome.

39:37 Yeah, you're definitely doing some low level stuff these days.

39:40 So I guess that's our main items for today.

39:43 Brian, you got some extras you wanna share?

39:46 - Yeah, we had poked by the Python Software Foundation that we should probably plan some events because we've got a meetup, the Python PDX West meetup, but I can't access the building that we normally held it in and nobody would come anyway, right?

40:03 So we're doing virtual.

40:04 So we've got January 14th, we're gonna start seeing what it's like to do a virtual thing.

40:10 And I thought, normally when we did the meetups before it was in the evening, I don't really wanna hang out in the evening doing this, but at lunchtime would be fine.

40:19 So I thought like a lunch and learn thing.

40:22 So we're gonna try that, see what it's like.

40:23 - Yeah, it's perfect.

40:25 Being virtual, you could just do it during lunch, right?

40:27 You don't have to go, okay, well, I'm gonna drive out, you know, fight the traffic or whatever, right?

40:32 - Yeah, and I don't have to convince my boss to buy a bunch of pizzas.

40:35 So it's good. - Yeah, exactly, exactly.

40:38 All right, I got a couple of extras as well.

40:40 I ended up doing a FastAPI webcast with the folks over at JetBrains.

40:44 So people can check that out.

40:46 They wanna go learn more about that.

40:49 And then also, let me close my notes here, but I'll get them back.

40:54 Brian, you go next.

40:55 I gotta pull this up somewhere else.

40:57 - I just went, so let's--

40:58 - Sorry, yeah, you got it.

40:59 Anthony, what are you up to these days?

41:01 What extra you got to share, folks?

41:03 - Yeah, so I'm gonna be starting at Microsoft in February, which is big news for me.

41:09 - Yeah, congratulations, that's awesome.

41:11 - Yeah, yeah, I'm really excited about it.

41:13 So yeah, I'm gonna be the Python developer advocate working with Nina Zakharenko.

41:17 So Nina is the current Python developer advocate, and I'm gonna be helping out.

41:22 So yeah, really looking forward to it and working with a bunch of smart people over at Microsoft.

41:28 So yeah, all things Python.

41:30 - Do you know any particular projects you're working on?

41:33 - So I'm not gonna be, I'm not gonna be, I'm not on the engineering team.

41:37 And I'm definitely not working on Pigeon.

41:41 This has got nothing to do with Pigeon.

41:43 Contrary to the rumor, I guess.

41:46 I heard or a compendium. That's what I heard. Come on.

41:49 Yeah, I yeah, I'm definitely not. So yeah, this is kind of a side thing. But yeah, it's kind of definitely the Azure platform and like using Python and Python working with Azure and on Azure VS Code and the Python extension and just the tooling and in around that, as well as the Python ecosystem with Microsoft products in general, or Microsoft technologies, and Python in general. So all things Python and yeah, really looking forward to starting on the 1st of Feb. And yeah, I have a long list of things that I want to want to get stuck into.

42:26 Yeah, I'm guessing that's remote, not just now, but permanently, right?

42:30 Yes. Yeah.

42:31 Yeah, that's super, that's super exciting. But that does mean, Brian, that he'll be somewhat in our neighborhood more often. I suspect you'll come up to Seattle and Redmond sometimes when that happens again, when the world is let loose.

42:44 Yeah, I expect so when we're not even allowed to leave the country at the moment.

42:49 Yeah.

42:50 We'll see you in the Northwest once in a while.

42:53 Yeah, yeah.

42:54 When, when, maybe 2022.

42:58 Yeah.

42:58 What 2030 we'll see.

42:59 We'll definitely see you 2030.

43:00 No problem.

43:01 Also, from, from, from, the YouTube stream, Piling says, congrats, Anthony, on the new job.

43:06 Thanks Piling.

43:07 Yeah.

43:07 Yeah.

43:08 So we should have like a, like a webpage with like a, graphic to show like how many, Like well-known Python people there are, and the, like maybe a death star, Microsoft sucking all the Python people into Microsoft.

43:22 Yeah.

43:22 That's pretty evenly distributed.

43:26 Yeah.

43:28 No, it's all good.

43:30 All right.

43:30 Well, I feel like, might be time to wrap this up with the joke.

43:34 You think?

43:34 Sure.

43:35 Yeah.

43:35 So, this one came to us over Twitter from Lars and there was a, apparently a question, someone named Kate Maddox asks, what's the best programming language for coding your own therapist?

43:47 You know, like we're all stuck at home.

43:49 People are depressed, have social anxiety or just anxiety about the world in general.

43:54 And Wes comes along with an appropriate answer.

43:57 What do you guys think here?

43:58 Python of course is the language.

44:00 So you can call it there up.

44:01 I E H R a dot P Y.

44:04 Yeah.

44:05 I think it's pretty much on par for our standard jokes though.

44:09 [LAUGHTER]

44:12 Yeah.

44:13 Might be better than average, actually.

44:15 Yeah.

44:16 Yeah, it could be better than average.

44:17 You need a little drum kit in the background, Brian, that you can--

44:19 Yeah.

44:20 [DRUMMING]

44:21 Some cymbals.

44:22 We'll set you up with that little--

44:23 It's a business expense.

44:24 I'll totally get a drum kit.

44:25 Yeah.

44:27 Fantastic.

44:28 All right.

44:28 Well, Brian, thanks as always.

44:30 And Anthony, thanks for joining us.

44:32 Thank you.

44:32 Yeah, always a pleasure.

44:33 Yeah, you bet.

44:34 And everyone listening, see you next time.

44:36 Thank you for listening to Python Bytes.

44:38 Follow the show on Twitter via @pythonbytes, that's Python Bytes as in B-Y-T-E-S.

44:43 And get the full show notes at PythonBytes.fm.

44:46 If you have a news item you want featured, just visit PythonBytes.fm and send it our way.

44:51 We're always on the lookout for sharing something cool.

44:53 On behalf of myself and Brian Auchin, this is Michael Kennedy.

44:56 Thank you for listening and sharing this podcast with your friends and colleagues.