Transcript #86: Make your NoSQL async and await-able with uMongo

00:00 Hello and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds.

00:05 This is episode 86 recorded July 11th, 2018. I'm Michael Kennedy.

00:10 And I'm Brian Atkin.

00:11 And Brian, we have a new special guest, Bob Belderbose.

00:14 Yeah, pretty excited.

00:15 Yeah, welcome Bob.

00:16 Hey, nice to be here. Thanks.

00:18 Yeah, so people may recognize Bob from his PyBytes with an I code challenge stuff that he does, as well as the 100 days of code course that we did together, right?

00:27 Yeah, we did a major course on 100 days of code in Python.

00:33 It was very fun.

00:34 Yeah, it was only what 20 hours of video and 80 hours of exercise.

00:38 Yeah, that was awesome.

00:40 So welcome to the show.

00:41 We're super excited to have you here.

00:43 Before we get into everything, let's just say thank you to DigitalOcean, our customer and their sponsor, our show as well.

00:49 Check them out at pythonbytes.fm/digitalocean.

00:53 You'll get $100 free credit if you're a new user.

00:55 That's pretty awesome.

00:56 Brian, if I had to guess if I were just going to like pick out of like thin air, what kind of topic you might have, it might have something to do with testing.

01:02 I do like testing quite a bit.

01:05 So what do you got for us?

01:06 Well, we have responses. And this is it. There's a funny story around it. But this is a responses is a utility library for mocking out requests Python library. And it's from it's from get century. So it's under the get century, GitHub account. And it looks pretty cool. It's actually, we'll include a small snippet in our show notes, but there's, it's a decorator approach to, and other ways to be able to test your, pretty much mock out if you have an application that uses requests to access any, really any end point on the internet, you can use this to mock out those. And it has some dynamic, dynamicness to it. And it's, it looks actually pretty complete and pretty nice.

01:52 It's pretty sweet.

01:53 Yeah, you put an attribute on your your function.

01:56 And during that one, any call to requests like request I get or the response JSON or whatever is going to be basically overridden as if you had done that with patch, but it's a little more focused on just requests, right?

02:09 Yeah, it's focused on requests.

02:11 And it's also it's designed to go well with the pytest framework.

02:16 So it cleans up after itself really easily.

02:19 Nice.

02:20 with pytest? Like what's special about that?

02:22 Yeah, the decorator you put on your test is like responses activate. So it'll and then you're going to have it you register an end point to mark and what response it should give. And then all of that gets cleaned up at the end of the test. You don't have to undo it or anything like that. Yeah, that's awesome. Very nice. The funny thing is, is this is exactly what Anthony and I were considering building together. We just didn't know this one existed.

02:51 And so I wanted to point it out because your responses is a clever name to go with requests.

02:56 I get it.

02:58 But if I'm trying to find something that goes well with pytest and goes well with requests, I might not know to search for responses.

03:06 So that's why I'm violating it out.

03:09 So I think anybody that's trying to do something like this should check it out.

03:12 It looks pretty complete.

03:13 - Cool. It looks very useful.

03:15 So it's not making any call over the network, right?

03:18 That's the whole point of patching it out, right?

03:20 Well, it's patching, it'll patch out the endpoints you specify.

03:26 I don't really know, I don't know what the behavior is for other ones.

03:28 It probably lets everything else go through.

03:30 Yeah, that's one of the things I like about it.

03:32 As you say, this, if I request this HTTP endpoint, then do this, rather than I'm just replacing, you know, request.get with everything.

03:41 Right, right.

03:42 Oh, yeah, pretty nice.

03:43 I do see your point about SEO and discoverability for responses, because like response and responses is just such a common word on so many API libraries.

03:54 Like, how are you going to find the testing one?

03:56 Yeah, I was looking, it says in its description that it's Python 2.7 or newer.

04:01 Well, so it isn't obvious to me that it was 3x.

04:05 But it looks in like in the investigate the talks any file, it's tested up through three five at least.

04:12 So it's probably three, six, three, seven compatible too.

04:15 Haven't tried it, but yeah, probably.

04:17 Yeah, very nice.

04:18 All right, Bob, it's time for yours.

04:20 What one did you bring?

04:21 Yeah, I found an interesting decision tree article 29 common beginner Python errors on one page.

04:30 And I found this interesting because these kind of decision trees or graphics, you get a lot of information in one page, you can print out or study in detail.

04:41 It's like a flow.

04:42 It's like an infographic for like getting my code to work.

04:46 It's amazing.

04:47 Yeah.

04:48 It didn't land directly on the article actually.

04:49 It was a Microsoft devs that tweeted it out and that's how I found it on the Python has hashtag.

04:55 And it goes through this whole flow.

04:56 Like what kind of error do you get?

04:59 And then it goes into these common Python errors, like attribute error, syntax error, type errors.

05:05 And for each one, it gives like a couple of plausible causes.

05:09 Yeah, I mean, if you have more experience, some might be obvious, but some are very subtle.

05:14 And that was like a great reminder, like comparing three int with a three string that returns false.

05:22 And for example, when I was going through it also could relate to similar things like when you exhaust a file that also can happen with a generator, right?

05:32 A generator, you go through it once and then if you do that again, then hey, it's like an empty list because the generator only, you can only consume once.

05:42 Right.

05:42 So it was a nice, I guess, exercise to take a lot of info in from one infograph.

05:48 And I think for a beginner, it's very useful to go through that at least once.

05:52 And I think the author made the infograph after having teached Python for a long time in the biology space.

06:01 So that was also inspiring for me, like teaching others.

06:06 Yeah, how you could summarize that information at some point.

06:09 I'll try to give a quick summary just so people can get the visual.

06:12 So it's like an infographing. It says, "Start here. Do you get an error when you run your code?

06:16 Yes. What type of error do you get?" And there's a whole branch there. "No.

06:19 Does your code use loops or if statements? If you use an if statement, go check this." You know, it just like walks you through some pretty basic stuff. I kind of like it.

06:26 Brian, what do you think?

06:27 Yeah, actually, this is one that I've run across before.

06:31 And when our team started using Python for testing, one of the other engineers actually printed this out and posted it on the side of their cubicle so that people that came and asked them questions could just look at that first.

06:43 Wow, that's pretty cool.

06:44 And I like the fact that it's coming from Python for biologists.com.

06:49 So we're seeing Python being taught in a lot of different domains, which is awesome.

06:54 Yeah, that actually is a really good point.

06:55 It's pretty awesome.

06:56 So the next one that I have for us is this thing called micro Mongo, which somehow I have not heard of previously.

07:03 But Brian, you know, I go on and on about async programming stuff and parallelism and things like that, right?

07:10 Yes.

07:10 And I do know that you kind of like Mongo DB also.

07:13 This is like, this is like, I get to put them together.

07:17 It's amazing.

07:18 So one of the main challenges of this async await stuff actually becoming super useful and super powerful.

07:26 There's two fundamental problems limiting the serious adoption of this.

07:30 One is the web frameworks don't deeply support it.

07:33 So I can't have an async flask method.

07:36 I have to switch to something like Court or use Fibora or some other framework, like the main Django pyramid flask, they don't support async.

07:43 So that's thing one, but we're seeing some movement there.

07:47 But even if you could get that to work, a lot of the ORMs don't have great ways to take advantage of their parallelism or their potential parallelism, why they're waiting on the network talking to the database, because usually that's what your web app is doing.

08:00 And that's why you would want the async is like you could let it do other processing while you're waiting on a database response.

08:05 So what you need is if you're using an ORM or something equivalent called an ODM for Mongo is the ability to interact with those databases is in an async way.

08:15 And so this micro Mongo is a very lightweight and small as you might imagine from the name object document mapper for MongoDB and Python that brings this ability to do async operations against MongoDB.

08:27 So if you wanted to use say, Quart or Vibora, and you want to use MongoDB as the back end, you have to use something like this, which is pretty sweet.

08:35 Very interesting.

08:36 I need to go on the async train myself yet.

08:39 So maybe this is a good opportunity to learn it.

08:41 So definitely, it's not that, I mean, Mongo itself, you don't need, that's on the other end.

08:47 So that's already asynchronous or can be, but it's the document mapper that's the problem.

08:53 - You need the library that talks to the database to have an async option.

08:57 So for example, like let's take something that people know more about probably SQLAlchemy, right?

09:02 If you go to SQLAlchemy and you create a query and you say like order by or filter or whatever, you need SQLAlchemy to be rewritten so that filter is async, is an async method or it's useless in terms that, right?

09:16 So it's like the place you want to get to is interact with your database asynchronously.

09:20 But if the ODM or ORM that you're working with in the middle doesn't support that there's zero workaround, you're done, right?

09:27 You cannot do that.

09:29 Whereas most of the things you're doing is waiting on either web services, or you're waiting on databases.

09:34 And that's where you would get like the massive scalability.

09:36 Another thing that's really cool here is there, there is something called motor, which is another async way of programming against MongoDB.

09:43 But this one will let you choose the foundational bits.

09:46 So you can say I don't care about asynchronous programming.

09:49 So I just want the base to be pi Mongo.

09:51 Or you can have TX Mongo, or you can use motor async, which is something from Jesse Davis, I believe, at MongoDB.

10:00 Or you like this, Brian, you can even say the driver, the foundational database access part of this ODM is Mongo mock.

10:08 So if you want, you can just say actually replace the internals with this mocking database later for testing. Oh, actually, that's pretty cool. Yeah. Yeah. So this is this is not I haven't used it. I generally use Mongo engine, but this is really looking quite promising and powerful.

10:25 So definitely want to give a shout out to that. Speaking of shout outs, let's give a shout out to DigitalOcean. So DigitalOcean is sponsoring this episode and they have a great, very affordable, very reliable service over at DigitalOcean.com. So our stuff runs on DigitalOcean. We have a couple servers doing all the magic including one doing MongoDB back there somewhere in New York City, I think it is. Anyway, they're super great. Go over there and create a virtual machine, get an awesome machine for five bucks a month, get it set up in 30 seconds, and off you go.

10:57 If you want to create something more pre-configured, they have a bunch of one-click apps to say set up like a ghost blog for five bucks a month for one click, things like that.

11:06 So check them out at pythonbytes.fm/digitalocean. You get $100 credit for a new user and you'll So let them know you're supporting the show, and thanks for them supporting the show.

11:16 So Brian, what's up with this statistics stuff here?

11:19 I've not done statistics for a really long time.

11:21 - I'm one for saying that I've been known to oppose the stringent math requirements in a lot of computer science and other science fields, but regardless, there's a lot of people that end up being, that end up having to learn statistics.

11:37 And I think statistics is a good thing to learn, at least some of the basics.

11:40 I think we often come into like even going through computer science. You might not even take statistics You probably took calculus and differential equations and other stuff that you never ever use again, right? Yeah, exactly. Yeah Let's not go down that tangent too far Crying about remembering how much work we put into learning those stuff, right?

11:58 I know I have never needed to factor a polynomial ever in real life How about computing the inverse of a non-secular matrix? Sorry, let's keep going on statistics However, sometimes I do need to know like averages and medians and stuff like that.

12:14 So this is, I'm highlighting this article, it's called the basic statistics in Python, descriptive statistics.

12:22 And I like it, it's just using some simple Python to demonstrate how you would build up some simple statistics concepts to teach the concepts like min max, mean, median, mode, some of the others like standard deviation.

12:38 It's just a handful of them, but those are kind of hard things to get your head around when you're first looking at it.

12:43 So being able to play with the numbers with a computer, I think is a neat thing.

12:47 So they give some descriptions on how you would implement some of these functions in Python.

12:53 >> Yeah, that's nice.

12:54 >> It does a nice job.

12:55 Like for instance, if you wanted to figure out the average, you can take the sum of all of everything and then take how many scores and it's like the sum divided by how many.

13:06 Those are good concepts to look at.

13:08 The thing that was missing out of it is the little bit at the end, I think, that says, by the way, don't ever do this if you're actually writing Python code, because all this stuff's already built in.

13:20 Exactly.

13:21 Yeah, you've got the various numerical libraries for it, right?

13:23 Isn't this stuff already built in?

13:25 And Min and Max are already built in just normally, and some of the others.

13:29 But I actually went and looked and found a...

13:33 I didn't know about the statistics module.

13:36 built in, I think as of 3.4 or 3.5 or something.

13:40 That's right.

13:41 It's quite new.

13:42 I don't remember, but I think it might be 3.5 actually.

13:44 3.4.

13:45 3.4.

13:46 Thank you, Bob.

13:47 Okay.

13:48 But I threw in some examples, but if you're just using that, it doesn't help you learn what statistics is, so that's why I think the article is still good.

13:57 I stuck up, I was looking at the statistics and I noticed that it had both standard deviation variance and population standard deviation and population variance. I don't know enough statistics to know what the difference in those are.

14:11 I know P goes in front of the method name. No, that's awesome that that's in there. Yeah, I think it's interesting that they added that module into the standard library in 3.4. It's cool.

14:23 Yeah, I guess maybe they were tired of people implementing them in crazy hand-coded ways.

14:27 Yeah, and it's got to be faster written in C to compute the variance than to do it in in like a Python loop.

14:33 - That's a good point.

14:34 Yeah, and it again goes to say that always keep up with the standard library because there's so much stuff in there that can save you writing code and probably gain you performance.

14:45 - Yeah, absolutely.

14:46 All right, Bob, what's the next one you got here?

14:48 - This is a big piece.

14:49 Strings and character data in Python by RealPython.

14:54 I don't know if--

14:54 - If it's by RealPython, you know it's kind of like a novella.

14:58 - Yeah, exactly.

14:59 Because I want to compare it with that it or tools article you featured a couple of weeks ago.

15:05 That was like this long piece.

15:06 It took me an hour to read, but I got so much out of that.

15:09 And it's the same here.

15:10 I didn't even honestly didn't finish it yet.

15:13 I've been like 60, 70% in and always already got so much value out of it because it does this great detailed tour of strings in Python and all the methods you can use on methods you can use on them.

15:26 And I put some snippets in the, in the show notes because I mean, for a beginner, it's great because you will be working with strings from day one.

15:33 But even if you know Python, there's like a lot of these little tricks in there that can just make you faster in Python.

15:41 if a string is a digit, you can use is digit on the string, which returns a Boolean rather than try, try to convert it to an int, catch the exception and go to the exact, if not an int, yeah, that kind of thing.

15:53 Yeah.

15:53 So you can use that method.

15:55 or for example, you want to look for white space, a space, a tap or a new line, there's also an is space method that does that for you, or, commonly with, terminal apps, you want to have this banner with a tech centered.

16:09 Well, you can just do string center and then the width and then the, the, the filler character.

16:16 So there's, there's a lot of good stuff in there.

16:19 And, I think it's worth to go and spend that hour and learn all that stuff.

16:23 because you mean that we'll just shave off time when you're actually coding.

16:28 And another thing I also found kind of the linking you do when you read such an article, for example, polymorphism, right?

16:35 If you have the index method works on a string, but also works on a, on a list.

16:39 So I put a snippet in the show notes that I use index on a string and a list, and you see that they behave in a similar way.

16:48 Same goes for count.

16:49 So I found that also an interesting point to highlight.

16:53 Oh yeah.

16:53 - Yeah, this is really interesting.

16:54 I just threw that into Instapaper and Instapaper says reading time 33 minutes.

16:59 Yeah, this is definitely, it's like a book chapter almost on strings character.

17:03 - I might be a slow reader, but.

17:05 - Well, I'm sure I would too.

17:06 And you know, that's not counting that it's like code.

17:09 Code you gotta analyze, right?

17:11 - Yeah, you want to try some things.

17:13 Yeah, yeah, yeah.

17:14 - Exactly, exactly.

17:15 No, it's a great find.

17:16 It's really cool.

17:17 - And people coming from different languages, I know I was like this.

17:21 I did not expect that strings would have all of these cool operations on them.

17:26 Yeah, especially if you're coming from C where strings are really just, you know, character pointer array type things.

17:32 They're basically just memory, right?

17:34 Yeah.

17:35 Yeah.

17:36 But imagine you have to do by leading zeros to a number.

17:40 And that would take you a couple lines of code.

17:42 In Python, it's just number string Z fill, and then the number of zeros you want to be penned.

17:47 And it's it just makes for shorter code.

17:48 Yeah, it's wonderful.

17:49 fill this he. Yeah. So you want to have a 42, for example, and it needs to be five, a width of five, you can just set fill it with zeros.

18:00 Nice. So I'm going to bring I'm going to close this out with something that is surprisingly controversial. I don't know, I'll see what your thoughts are. So there's a new PEP in town, PEP 572 and whenever I think about these things always think you know someone has worked super hard on this and that's really awesome they're making a contribution and then you know to see people react and I don't know not totally excited ways I don't know it's sort of I don't know how to feel about let's say but there is a new piece of Python syntax that will allow you to in a single expression, create a variable and assign to it. So, so often you have to do something to the effect of like, I'm going to create a variable, set a value, and then if that value is something I'm going to do a thing or I'm going to do something else. So the new syntax lets you put a colon equals to define that and put it all in one line. So instead of saying like match equals pattern dot search if match is not none, etc, etc. You can just say if match colon equals pattern dot search is not none, all in one. I'm actually even confused about your example. I thought that the point to that new operator was explicitly to check against none. The point of the thing is to allow you to both create the variable and set its initial value and check it at the same time whatever you're checking it for. Oh, right. And you don't even have to check it. For example, one of the really common use cases for this is to use it in like a list comprehension or something like that. And anytime you need more than one line, that cannot be put into a list comprehension or lambda expression, right? But now with this one line variable creation assignment potential test behavior, you can put these into inline expressions like list comprehensions or lambdas or so on. I think that's it. I've only skimmed some of the use cases but that's That's what I'm thinking it's for.

20:03 Okay.

20:04 So this is interesting.

20:05 The syntax is not terrible.

20:07 I think it's kind of okay.

20:08 There's already lots of examples of this in other languages that had pretty decent implementations.

20:14 So one of my thoughts was like, well, why didn't that get adopted?

20:17 So both in like C# link expressions, as well as JavaScript expressions, you have let something equals a value in terms of like a, like in terms of for loop or something like that, that could have also been an interesting option.

20:30 But I don't know.

20:31 To me it's okay, but I don't know that it's necessarily needed for the language.

20:35 It's not solving a problem that I think very many people actually have.

20:39 I wouldn't have to use it and see it in other people's code because from the snippet you posted I find it a bit confusing.

20:46 It might just be me having to get used to it, but it's not the kind of syntax I'm used to from Python, right?

20:54 It's more readable.

20:55 What do you guys think?

20:57 To me, it's like one step down the slippery slope.

21:02 I mentioned C# and the let stuff.

21:04 I feel like the C# language used to be pretty nice and it's just a complete train wreck.

21:08 Now there's all these little three or four ways of doing the same thing and they're all getting a few characters shorter, but now there's five of them.

21:15 They're all more or less equivalent.

21:16 It's just like, "Whoa, why do we keep adding these things to this language?

21:20 They're just unneeded." I feel like this is in that category of stuff.

21:24 Python's nowhere near as bad.

21:25 But might not be the PEP itself, but more about the, we start to introduce and what other syntax changes that might be coming, right.

21:34 If you open it up for this.

21:36 Yeah.

21:37 So what I want to sort of point people out for this is the tweet by Raymond Hedinger and then the Twitter conversation that follows it, which is really interesting.

21:47 Like you don't normally see this much conversation in a thread on Twitter, but There's like 44 messages in the, out to this one tweet.

21:55 And it's, it's pretty interesting.

21:57 Some people like it, some people don't, but I think it's worth reading through.

22:01 Nice.

22:02 So anyway, new PEP in town and it's approved, right?

22:04 This is not proposed.

22:05 My understanding is this is approved.

22:07 So it's, it's now going to be part of maybe three Python three eight.

22:10 Yeah.

22:11 I'll totally use it, but it is, it did one was one thing that made Python different was that like conceive you've, if you tried to do check for equality and you accidentally did assignment instead, it's wrong.

22:25 And I guess that's why the syntax is specific, so that it's not gonna be an accident.

22:30 - Right, and C, you could say something to the effect of like, if match equals pattern.match, do such and such, right?

22:39 And you actually meant if those two things were actually equal, right?

22:42 So there's a way, the syntax does make that sort of fall through error case, not actually.

22:47 You can't omit an equal on accident and actually get assignment.

22:50 - Yeah, you have to explicitly put the colon in there.

22:53 It's colon equal.

22:54 So, okay.

22:55 - All right, well, there it is.

22:57 PEP 572, the most controversial accepted PEP I've heard in a little while.

23:01 Pretty interesting.

23:02 All right, well, that's it for our news items, you guys.

23:04 Brian, you got anything else you wanna share with the world these days?

23:07 What's going on?

23:08 - Well, actually, kind of nice tie in with this PEP 572.

23:13 I wanted to just talk with somebody about all the new stuff that's proposed, actually proposed and accepted already for 3.8 and I think just for 3.8.

23:24 So that's kind of what Anthony Shaw has been up to lately.

23:27 So I'm going to have Anthony on testing code and we're going to talk about that.

23:31 >> Oh, that's going to be great.

23:32 >> I've also realized that doing this podcast with you, it's pretty easy to do it once a week because you're waiting for me to do it once a week.

23:41 So I'm trying to rope in some other people and Anthony is one of the people that agreed to try to do testing code more regularly to try to get more of those out.

23:50 Oh, yeah, that's awesome.

23:51 Yeah, that's really cool.

23:52 How about you, Michael?

23:53 Well, if we're going to give shout outs to Anthony, I just had him on Talk Python as well about his security article.

23:59 He's really killing it.

24:00 Him and the real Python guys, they're definitely cranking out the content.

24:03 That's great.

24:04 But the thing that I guess is the biggest news that has me getting up early and staying up late is I'm working on a new about 10 hour course for data driven apps in Pyramid with of SQLAlchemy and like production migrations and all sorts of stuff.

24:18 That is almost done.

24:19 It should be out next week.

24:21 Just finalizing a lot of the videos this week and it'll be rolling.

24:25 - Nice. - Nice.

24:26 - Yeah, so that's what I'm up to.

24:28 Bob, how about you?

24:29 You wanna tell people about PyBytes real quick, what that is?

24:32 - Yeah, PyBytes started out as a blog with articles and we quickly moved into a blog code challenges and that got quite some traction.

24:41 So we build out a code challenge platform which you can find on codechallenge.es.

24:46 So codechallenge is spelled as codechallenges together, but ending in .es.

24:52 And you can log in with GitHub and code in the browser.

24:56 So there are exercises with automated pytest.

25:00 Yes, Brian, it's pytest.

25:01 And you can code in a browser and verify those exercises.

25:06 And yeah, the feedback is great.

25:07 I mean, people are learning a lot of Python that way because it's so practical.

25:13 And so we're growing that we have a Slack community behind it where a lot of people are joining and we have very good Pythonic discussions.

25:21 So that's, yeah, I'm excited about this.

25:24 - Yeah, right on.

25:25 Yeah, it's a cool platform.

25:26 Happy to see you doing it.

25:28 And thank you for being on the show, Brian.

25:30 - Thank you.

25:31 - Thank you, as always.

25:32 I'm happy to hear you're doing more testing code as well.

25:34 That's pretty sweet.

25:35 - Yeah.

25:36 - All right, catch you later.

25:37 - Bye.

25:37 - Thank you for listening to Python Bytes.

25:40 Follow the show on Twitter via @pythonbytes, that's Python Bytes as in B-Y-T-E-S.

25:46 And get the full show notes at pythonbytes.fm.

25:50 If you have a news item you want featured, just visit pythonbytes.fm and send it our way.

25:54 We're always on the lookout for sharing something cool.

25:57 On behalf of myself and Brian Auchin, this is Michael Kennedy.

26:00 Thank you for listening and sharing this podcast with your friends and colleagues.