« Return to show page
Transcript for Episode #86:
Make your NoSQL async and await-able with uMongo
00:00 KENNEDY: Hello and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds. This is Episode 86, recorded July 11th, 2018. I'm Michael Kennedy.
00:09 OKKEN: And I'm Brian Okken.
00:11 KENNEDY: And Brian, we have a new special guest, Bob Belderbos.
00:14 OKKEN: Yeah, pretty excited.
00:14 KENNEDY: Yeah, welcome, Bob.
00:16 BELDERBOS: Hey, nice to be here, thanks.
00:18 KENNEDY: Yeah, so people may recognize Bob from his PyBites, with an i, code challenge stuff that he does, as well as the 100 Days of Code course that we did together, right?
00:27 BELDERBOS: Yeah, we did a major course on 100 Days of Code in Python, was very fun.
00:34 KENNEDY: Yeah, it was only what, 20 hours of video and 80 hours of exercise?
00:37 OKKEN: Yeah, that was awesome.
00:39 KENNEDY: So, welcome to the show. We're super excited to have you here. Before we get into everything, let's just say thank you to Digital Ocean, our customer and they sponsor our show as well, so check them out at pythonbytes.fm/digitalocean. You'll get 100 dollars free credit if you're a new user, that's pretty awesome. Brian, if I had to guess, if I were just going to like, pick out of like, thin air, what kind of topic you might have, it might have something to do with testing.
01:03 OKKEN: Yeah, I do like testing quite a bit.
01:05 KENNEDY: So what do you got for us?
01:05 OKKEN: Well, we have responses, and there's a funny story around it, but this is, responses is a utility library for mocking out requests Python library, and it's from getsentry, so it's under the getsentry GitHub account and it looks pretty cool. We'll include a small snippet in our show notes, but it's a decorator approach and other ways to be able to test your pretty much mock out if you have an application that uses requests to access really any end point on the internet, you an use this to mock out those and it has some dynamicness to it and it looks actually pretty complete and pretty nice.
01:52 KENNEDY: That's pretty sweet, yeah. You put your attribute on your function and during that one, any call to requests, like requests.get or the response.json or whatever is going to be basically overridden as if you had done that with patch, but it's a little more focused on just requests, right?
02:09 OKKEN: Yeah, it's focused on requests and it's also designed to go well with the pytest framework, so it cleans up after itself real easily.
02:19 KENNEDY: Nice, how does it integrate with pytest? What's special about that?
02:22 OKKEN: Yeah, the decorator you put on your test is like, responses.activate, so then you're going to have it, you register an endpoint to the mock and what response it should give. And then all of that gets cleaned up at the end of the test. You don't have to undo it or anything like that.
02:40 KENNEDY: Yeah, that's awesome. Very nice.
02:42 OKKEN: The funny thing is, is this is exactly what Anthony and I were considering building together. We just didn't know this one existed. And so I wanted to point it out, because you know, responses is a clever name to go with requests, I get it, but if I'm trying to find something that goes well with pytest and goes well with requests, I might not know to search for responses, so that's why I'm highlighting it out. So I think anybody that's trying to do something like this should check it out, it looks pretty complete.
03:13 BELDERBOS: Cool, it looks very useful. So it's not making any call over the network, right? That's the whole point of patching it out, right?
03:20 OKKEN: Well, it'll patch out the endpoints you specify. I don't really know what the behavior is for other ones. It probably lets everything else go through.
03:30 KENNEDY: Yeah, that's one of the things I like about it. As you say, if I request this http endpoint then do this rather than I'm just replacing requests.get with everything.
03:41 BELDERBOS: Right.
03:41 KENNEDY: Right. Oh yeah, pretty nice. I do see your point about SEO and discoverability for responses, because response and responses is just such a common word on so many API libraries, like, how are you going to find the testing one?
03:56 OKKEN: Yeah. I was looking, it says in its description that it's Python 2.7 or newer. Well, so it isn't obvious to me that it was 3.x, but it looks like in the tox.ini file, it's tested up through 3.5 at least, so it's probably 3.6 and 3.7 compatible, too. Haven't tried it, but...
04:15 KENNEDY: Yeah, probably. Yeah, very nice. Alright, Bob, it's time for yours. What one did you bring?
04:21 BELDERBOS: Yeah, I found an interesting decision tree article, 29 Common Beginner Python Errors On One Page, and I found this interesting, because these kind of decision trees or graphics, you get a lot of information in one page you can print out or study in detail and it's like a flow.
04:42 KENNEDY: It's like an infographic for like, getting my code to work. It's amazing.
04:46 BELDERBOS: Yeah, I didn't land directly on the article, actually. It was Microsoft Desk that tweeted it out and that's how I found it on the Python hashtag. And we go through this whole flow like, what kind of error do you get? And then it flows into these common Python errors like attribute errors, syntax error, type errors, and for each one it gives like, a couple of plausible causes and yeah, I mean, if you have more experience, some might be obvious, but some are very subtle and that was like, a great reminder comparing 3 int with a 3 string that returns False. For example, when I was going through it, I also could relate to similar things like when you exhaust out the file that also can happen with a generator, right? A generator, you go through it once, and then if you do that again, then hey, it's like an empty list because the generator you can only consume once, right? So it was a nice, I guess, exercise to take a lot of info in from one infographic. And I think for a beginner, it's very useful to go through that at least once. And I think the author made the infograph after having teached Python for a long time in the biology space, so that was also inspiring for me, like teaching others, yeah, how you could summarize that information at some point.
06:09 KENNEDY: I'll try to give a quick summary just so people can get the visual. So, it's like an infographic and it says, "Start here." Do you get an error when you run your code? Yes. What type of error do you get? And there's a whole branch there. No. Does your code use loops or if statements? If you use an if statement, go check this. You know, just walks you through some pretty basic stuff. I kind of like it. Brian, what do you think?
06:27 OKKEN: Yeah, actually this is one that I've run across before and when our team started using Python for testing, once of the other engineers actually printed this out and posted it on the side of their cubicle so that people that came and asked them questions could just look at that first.
06:42 KENNEDY: That's pretty cool.
06:45 BELDERBOS: I like the fact that it's coming from Python for biologists.com, so we're seeing Python being taught in a lot of different domains, which is awesome.
06:53 KENNEDY: Yeah. That actually is a really good point. It's pretty awesome. So the next one that I have for us is this thing called uMongo, which somehow I have not heard of previously, but Brian, you know I go on and on about async programming stuff and parallelisms and things like that, right? Yeah, so I do know that you kind of like MongoDB also. This is like I get to put them together, it's amazing. So, one of the main challenges of this async await stuff actually becoming super useful and super powerful, there's two fundamental problems limiting the serious adoption of this. One is that web frameworks don't deeply support it. So I can't have an async Flask method. I have to switch to something like Quart or use Vibora or some other framework. Like the main Django, Pyramid, and Flask, they don't support the async, so that's thing one, but we're seeing some movement there, but even if you could get that to work, a lot of the ORM's don't have great ways to take advantage of their parallelism or their potential parallelism, why they're waiting in the network talking to the database, 'cause usually, that's what your web app is doing and that's why you would want the async. It's like you could let it do other processing while you're waiting on a database response. So, what you need is if you're using an ORM or something equivalent, called on ODM for Mongo, is the ability to interact with those databases in an async way, and so this uMongo is a very lightweight and small, as you might imagine from the name, Object Document Mapper for MongoDB and Python that brings this ability to do async operations against MongoDB. Alright, so if you wanted to use, say Quart or Vibora, and you wanted to use MongoDB as the back end, you have to use something like this, which is pretty sweet.
08:34 BELDERBOS: Very interesting. I need to go onto async and train myself. So maybe this is a good opportunity to learn it.
08:40 KENNEDY: Definitely.
08:40 OKKEN: So it's not that, I mean, Mongo itself, you don't need, that's on the other end, so that's already asyncronous or can be, but it's the document mapper that's the problem?
08:53 KENNEDY: You need the library that talks to the database to have an async option. So for example, let's take something that people know more about probably, SQLAlchemy. If you go to SQLAlchemy and you create a query and you say order by or filter or whatever, you need SQLAlchemy to be rewritten so that filter is async, is an async method or it's useless in terms, right? So it's like the place you want to get to is interacting with your database asyncronously, but if the ODM or ORM that you're working with in the middle doesn't support that, there's zero work around, you're done. Alright, you cannot do that, whereas most of the things you're doing is waiting on either web services or you're waiting on databases and that's where you would get the massive scalability. Another thing that's really cool here is there is something called Motor, which is another async way of programming against MongoDB, but this one will let you choose the foundational bits. So you can say, I don't care about asyncronous programming, I just want the base to be PyMongo. Or you can have TX Mongo or you can use Motor async, which is something from Jesse Davis, I believe, at MongoDB, or you'll like this, Brian, you can even say the driver, the foundational database access part of this ODM is Mongo Mock, so if you want, you can just say, actually replace the internals with this mocking database layer for testing.
10:13 OKKEN: Oh, I actually think that's pretty cool, okay, neat.
10:17 KENNEDY: So I haven't used it, I generally use MongoEngine, but this is really looking quite promising and powerful, so definitely want to give a shout out to that. Speaking of shout outs, let's give a shout out to Digital Ocean. So, Digital Ocean is the sponsor of this episode and they have a great, very affordable, very reliable service over at digitalocean.com. So, our stuff runs on Digital Ocean and we have a couple servers doing all the magic, including one doing MongoDB back there somewhere in New York City, I think it is. Anyway, they're super great. Go over there and create a virtual machine. Get an awesome machine for $5 a month. Get it set up in 30 seconds and off you go. If you want to create something more pre-configured, they have a bunch of one click apps that say set up like a Ghost blog for five bucks a month for one click, things like that. So check them out at pythonbytes.fm/digitalocean to get 100 dollars credit for a new user. And you'll let 'em know you're supporting the show and show thanks for them supporting the show. So Brian, what's up with this statistic stuff here? I've not done statistics for a really long time.
11:21 OKKEN: I'm one for saying that I've been known to oppose the stringent math requirements in a lot of computer science and other science fields, but regardless, there's a lot of people that end up having to learn statistics. And I think statistics is a good thing to learn, at least some of the basics.
11:40 KENNEDY: Yeah, I think we often come into, even going through computer science, you might not even take statistics, you probably took calculus and differential equations and other stuff that you never ever use again.
11:49 OKKEN: Right, yeah, exactly, yeah. Let's not go down that tangent too far. However...
11:55 KENNEDY: We'll all start crying about remembering how much work we put into learning that stuff, right?
11:58 OKKEN: I know I have never needed to factor a polynomial ever in real life.
12:03 KENNEDY: How about computing the inverse of non singular matrix? Sorry, let's keep going on statistics.
12:09 OKKEN: However, sometimes I do need to know like, averages and medians and stuff like that, so I'm highlighting this article that's called The Basic Statistics in Python: Descriptive Statistics. And I like it, it's just using some simple Python to demonstrate how you would build up some simple statistics concepts to teach the concepts, like min, max, mean, median, mode, some of the others like standard deviation. It's just a hand full of 'em, but those are kind of hard things to get your head around when you're first looking at it, so being able to play with the numbers with a computer I think is a neat thing. So they give some descriptions on how you would implement some of these functions in Python.
12:53 KENNEDY: Yeah, that's nice.
12:53 OKKEN: It does a nice job. Like for instance, if you wanted to figure out the average, you can take the sum of all of everything and take how many scores and it's like the sum divided by how many. Those are good concepts to look at. The thing that was missing out of it is the little bit at the end I think that says, "By the way, don't ever do this if you're actually writing Python code." Because all this stuff's already built in.
13:19 KENNEDY: Exactly. Yeah, you've got the various numerical libraries for it, right?
13:23 OKKEN: Didn't this stuff already built in? And you know, min and max are already built in just normally and some of the others, but I actually went and looked and found a, I didn't know about the statistics module that's built in, I think as of 3.4 or 3.5 or something.
13:40 KENNEDY: That's right. It's quite new. I don't remember, but I think it might be 3.5 actually.
13:44 BELDERBOS: 3.4.
13:44 KENNEDY: 3.4, thank you, Bob.
13:47 OKKEN: But I threw in some examples. But if you're just using that, it doesn't help you learn what statistics is, so that's why I think the article's still good. I was looking at the statistics and I noticed that it had both standard deviation variants and population standard deviation and population variants. I don't know enough statistics to know what the difference in those are.
14:11 KENNEDY: I know P goes in front of the method name. No, that's awesome that that's in there. Yeah, I think it's interesting that they added that module to the standard library in 3.4, it's cool.
14:23 OKKEN: Yeah, I guess maybe they were tired of people implementing them in crazy hand coded ways.
14:28 KENNEDY: Yeah, it's got to be faster written in C to compute the variants than to do it in a Python loop.
14:33 BELDERBOS: That's a good point. Yeah and it again goes to say that always, keep up with the standard library, because there's so much stuff in there that can save you writing code and probably gain you performance.
14:44 KENNEDY: Yeah, absolutely. Alright, Bob, what's the next one you got here?
14:48 BELDERBOS: This is a big piece. Strings and Character Data in Python by a Real Python.
14:54 KENNEDY: If it's by a Real Python, you know it's kind of like a novella.
14:56 BELDERBOS: Yeah, exactly. 'Cause I want to compare it with the aiding tools article you featured a couple of weeks ago that was like this long piece. It took me an hour to read, but I got so much out of that, and it's the same here. I honestly didn't finish it yet. I'm like 60, 70 percent in and already got so much value out of it, because it does this great detailed tour of strings in Python and all the methods you can use on them and I put some snippets in the show notes, because for a beginner it's great, because you will be working with strings from day one, but even if you know Python, there's a lot of these little tricks in there that can just make you faster in Python. If a string is digit, you can use isdigit on the string, which returns boolean.
15:45 KENNEDY: Rather than try to convert it to an int, catch the exception if not an int.
15:51 BELDERBOS: Exactly.
15:52 KENNEDY: Yeah, that kind of thing.
15:52 BELDERBOS: Yeah, use that method. Or for example, you want to look for white space, a space, a tap, or a new line. There's also a space method that does that for you. Or commonly with terminal apps, you want to have this banner with the text centered. Well, you can just do string, center, and then the width, and then the filler character. So there's a lot of good stuff in there, and I think it's worth to go and spend that hour and learn all that stuff, because it will just shave off time when you're actually coding. And another thing I also have found, kind of the linking unit you do when you read such an article, for example, polymorphism, right? If you have the index method, works on a string, but also works on a list. So I put a snippet in the show notes that I use index on a string and a list and you see that they even a similar a way, the same goes for count, so I found that also an interesting point to highlight.
16:53 KENNEDY: Hell yeah, this is really interesting. I just through that into InstaPaper and InstaPaper says reading time 33 minutes. Yeah, this is definitely, it's like a book chapter almost on string characteristics.
17:03 BELDERBOS: I might be a slow reader, but...
17:05 KENNEDY: Well, I'm sure I would, too. And you know, that's not counting, that's like code. Code you've got to analyze, right?
17:11 BELDERBOS: Yeah, you want to try some things.
17:12 KENNEDY: Exactly, exactly, this is a great find. It's really cool.
17:17 OKKEN: And people coming from different languages, I know I was like this, I did not expect that strings would have all of these cool operations on them.
17:26 KENNEDY: Yeah, especially if you're coming from C where strings are really just character pointer array type things. They're basically just memory, right?
17:34 BELDERBOS: Yeah, but imagine you have to do leading zeroes to a number, and it will take you a couple lines of code. In Python it's just number strings that fill and then the number of zeroes you want to pretend and it just makes for shorter code.
17:48 KENNEDY: Yeah, it's wonderful.
17:48 OKKEN: Zed fill who's he
17:51 BELDERBOS: Yeah, so you want to have 42, for example, and it needs to be a width of five, you can just set fill it with zeroes.
18:00 KENNEDY: Nice. So I'm going to close this out with something that is surprisingly controversial. I don't know, I'll see what your all thoughts are. So, there's a new PEP in town, PEP 572, and whenever I think about these things, I always think someone has worked super hard on this and that's really awesome they're making a contribution. And then to see people react in just, I don't know, not totally excited ways, I don't know, I didn't know how to feel about it, let's say, but there's a new piece of Python syntax that will allow you to, in a single expression, create a variable and assign to it. So often you have to do something to the effect of I'm going to create a variable, set a value, and then if that value is something I'm going to do a thing or I'm going to go do something else. So the new syntax let's you put a := to define that and put it all in one line. So instead of saying like, match = pattern.search, if match is not None, et cetera, et cetera, you can just say if match := pattern search is not None all in one.
19:11 OKKEN: I'm actually even confused about your example. I thought that the point did that. New operator was explicitly to check against none.
19:20 KENNEDY: The point of the thing is to allow you to both create the variable and set its initial value and check it at the same time, whatever you're checking it for.
19:29 OKKEN: Oh, right.
20:39 BELDERBOS: I would have to use it and see it in other people's code, because from the snippet you posted, I find it a bit confusing. It might just be me having to get used to it, but it's not the kind of syntax I'm used to from Python, right? It's more readable. What do you guys think?
20:57 KENNEDY: To me, it's like one step down the slippery slope. I mentioned C# and the let stuff and I feel like the C# language used to be pretty nice and it's just a complete train wreck now. THere's all these little three or four ways of doing the same thing and they're all getting like, a few characters shorter, but now there's five of them. They're all more or less equivalent. It's just like, whoa, why do we keep adding these things to this language? They're just unneeded and I feel like this is sort of in that category of stuff. Python's nowhere near as bad, but...
21:25 BELDERBOS: It may not be the PEP itself, but more about the start to introduce what other syntax changes that might be coming, right? If you open it up for this.
21:34 KENNEDY: Yeah, so what I want to sort of point people at for this is the tweet by Raymond Hedinger and then the Twitter conversation that follows it, which is really interesting. You don't normally see this much conversation in a thread on Twitter, but there's like 44 messages all to this one tweet and it's pretty interesting. Some people like it, some people don't, but I think it's worth reading through.
22:01 BELDERBOS: Nice.
22:01 KENNEDY: So anyway, new PEP in town and it's approved, right, this is not proposed. My understanding is this is approved, so it's now going to be part of maybe Python 3.8.
22:10 OKKEN: I'll totally use it. There was one thing that made Python different was that like in C if you tried to do check free quality and you accidentally did assignment instead, it's wrong and I guess that's why the syntax is specific so that it's not going to be an accident?
22:30 KENNEDY: Right, in C you could say something to the effect of like if match = pattern.match, do such and such, right? And you actually meant if those two things were actually equal, right? So there's a way, the syntax does make that sort of fall through error case not actually, you can't omit an equal on accident and actually get assignment.
22:49 OKKEN: Yeah, you have to explicitly put the colon in there. It's :=, so okay. Alright, well, there it is, PEP 572, the most controversial accepted Pep I've heard in a little while, pretty interesting. Alright, well that's it for our news items, you guys. Brian, you got anything else you want to share with the world these days? What's going on? Well, actually kind of nice tie in with this PEP 572. I wanted to just talk with somebody about all the new stuff that's actually proposed and accepted already for 3.8, and I think just for 3.8, so that's kind of what Anthony Shaw's been up to lately, so I'm going to have Anthony on Testing Code and we're going to talk about that.
23:31 KENNEDY: Oh, that's going to be great.
23:31 OKKEN: I'm also realized that doing this podcast with you is pretty easy to do it once a week, because you're waiting for me to do it once a week. So I'm trying to rope in some other people and Anthony's one of the people that agreed to try to do Testing Code more regularly to try to get more of those out.
23:50 KENNEDY: Oh yeah, that's awesome. Yeah, that's really nice.
23:53 OKKEN: How about you, Michael?
23:54 KENNEDY: Well, if we want to give shout outs to Anthony, I just had him on Talk Python as well about his security article. He's really killing it. Him and the Real Python guys, they're definitely cranking out the content, that's great, but the thing that I guess is the biggest news that has me getting up early and staying up late is I'm working on new about 10 hour course for data driven apps in Pyramid with SQLAlchemy and production migrations and all sorts of stuff. That is almost done, it should be out next week. I'm just finalizing a lot of the videos this week and it'll be rolling.
24:23 OKKEN: Nice.
24:25 BELDERBOS: Nice, nice.
24:25 KENNEDY: So that's what I'm up to. Bob, how about you? Want to tell people about PyBites right now, what that is?
24:32 BELDERBOS: PyBites started out as some blog with articles and we quickly moved into block code challenges and got quite some traction, so we built up a code challenge platform, which you can find on codechalleng.es, so code challenge is spelled as code challenges together, but ending in .es. And you can log in with GitHub and code in the browser. So there are exercises with automated pytests. Yes, Brian, it's pytest. And you can code in the browser and verify those exercises and yeah, the feedback is great. People are learning a lot of Python that way, because it's so practical and so we're growing that, we have a Slack community behind it where a lot of people are joining and we have very good Pythoninc discussions. So yeah, I'm excited about this.
25:24 KENNEDY: Yeah, right on. Yeah, it's a cool platform. Happy to see you doing it. And thank you for being on the show. Brian.
25:30 OKKEN: Thank you.
25:31 KENNEDY: Thank you, as always. I'm happy to hear you doing more Testing Code as well. That's pretty sweet. Alright, catch ya later.
25:35 OKKEN: Bye.
25:38 KENNEDY: Thank you for listening to Python Bytes. Follow the show on Twitter via @PythonBytes, that's Python Bytes as in B Y T E S and get the full show notes at pythonbytes.fm. If you have a news item you want featured just visit PythonBytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy, thank you for listening and sharing this podcast with your friends and colleagues.