Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


« Return to show page

Transcript for Episode #147:
Mocking out AWS APIs

Recorded on Wednesday, Sep 4, 2019.

00:00 KENNEDY: Hello, and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is Episode 147, recorded September 4th, 2019. I'm Michael Kennedy.

00:10 OKKEN: I am Brian Okken.

00:11 KENNEDY: And this episode is brought to you by DigitalOcean. Tell you more about some cool stuff they got coming up later. Right now, I want to talk about tables.

00:19 OKKEN: Okay, and really quick, what's 147 in Hex? No, I just, I couldn't do that, I don't expect you. Anyway, no, but one of the things I do, like, need to do occasionally, I need to do really fast, is generate tables. And I know it's not something that everybody needs to do, but if I've got a whole bunch of data, like, in this case, some dictionaries, a list, and I want to be able to just print that stuff out. They're all the same type of dictionary that just holds some data. I want to print them out. How do I do that quickly and for test reports or whatever? I found this called rapidtables, and it's really kind of awesome.

00:56 KENNEDY: Oh, that is so cool. Wow!

01:00 OKKEN: Yeah, so you just basically, you've got a list of dictionaries, and then you want to print them out, and you want it to look like a good ASCII table. And it shows you how to do it, and it also shows you how to do it with a thing called termcolor.colored. I didn't know this was around. So you can do colored tables if you want. I probably won't do that. It also converts stuff to markdown and restructured text. That's kind of neat. I might use that. But right away I just need the just really easy tables. And it says it's super fast in speed. Like speed wise, it goes fast. But what I like is that it's not a lot of extra code gunk in your code. You just kind of say I want this table and you can print it out easily.

01:41 KENNEDY: Yeah, that's super cool because a lot of times this kind of formatting it's a lot of zfill and pad these strings and lpad, just like trying to get the spacing or you're putting tabs, but every now and then the tab like the word is too big and so the tab will shoot it to the next column. It's a lot of work and this is not you doing that work.

02:00 OKKEN: Right, so for all the columns you have to go through each of the elements to find the longest one to figure out how long and how wide you need to make the column and it's, and yeah. There's other libraries to do this but this one I think is pretty cool.

02:12 KENNEDY: Yeah, it's pretty cool. I also like the markdown and restructured text output option. Yeah, very cool. So if you think about what is one of the most popular packages, what would you say? Take some guesses.

02:24 OKKEN: Requests.

02:25 KENNEDY: It's got to be requests. I don't know if it's the most used one, but it's certainly the one that has people's attention and imagination. So what would you think if I had told you here is a competitor to request called httpx?

02:39 OKKEN: Ooh, I like the X.

02:40 KENNEDY: It's extra cool, absolutely. What's really interesting about httpx though is it's not a from scratch, throw away the ideas of requests. It's compatible with the requests API. So what you can do with requests, you can do with this. So you can kind of drop it in. But it has all these other cool features. And in fact, it comes from this project called Encode. E-N-C-O-D-E, Encode. And over there they have a ton of cool things including Django REST framework, Starlette, Uvicorn, httpx, and also some other async stuff. And a lot of the things happen around here. Uvicorn databases, ORM, httpx is all about adding the asynchronous capabilities to these various things, right? And that's one of the shortcomings of requests, is it doesn't support an async option but this httpx it does a bunch of cool things around the async space to kind of let you gradually go up the stack of complexity. So you can just be straight synchronous stuff doing what requests does. It has this concept of parallel requests, so you can issue a bunch of requests in parallel but you don't have to think about actually creating the async event loop and stuff. You just say, "I want to run these things all in parallel." and you can even ask it, say, "I want to process the responses as they come in." So I can say while parallel.has_pending_responses, the response is parallel.next_response(). So I can kick off like 20 requests and just process them as they come back in like four lines of code or something.

04:10 OKKEN: Oh, that's so cool. I like it.

04:12 KENNEDY: Yeah, or if you really want, if you're in something like Starlette where it has actual async support for the method, you do want that to be async so you can actually flip into like a async mode where those happen in the same way I describe but also on the asyncio event loop. So it's really cool. It also has some other neat features. So, like I said, a requests compatible API, it does HTTP2 which I could do a request for a page and as I pull down say five CSS style sheets and a JavaScript and an image, you can go get those all in one request. So it has support for that, which is cool. It has a standard synchronous interface, but also async in a way if you want it. And Brian, you'll like this one. It allows you to make direct request to WSGI or ASGI, the asynchronous variant of that apps. So image I'm creating a Flask app and I want to test it, I can actually inside of my flask app provide httpx the app, right? The flask app object and then I can issue requests that don't go through the network layer. They just go through this test app instance that I've created.

05:16 OKKEN: Oh, that's cool.

05:17 KENNEDY: Isn't that cool? So if I want to mock out some behavior or like interact with it as if I were external, I don't have to start a server and talk to it. I literally just use the same API here but directly skipping all the infrastructure.

05:31 OKKEN: Oh, that sounds fun. Yeah. In like, you know, a nerdy kind of fun.

05:34 KENNEDY: Yeah, that sounds like really fun but yeah. It is also fully type annotated, so it's very like Python 3 friendly, has 100% test coverage, and yeah. It has support for mocking out things. A lot of cool, nice features and the fact you get to built, all this is built on top of requests means if you're already using requests, it's like you can adopt it pretty easily because you don't have to change anything. But if you want to bring in some of these cool new features, it's just, you know, you start using them.

06:03 OKKEN: Nice.

06:04 KENNEDY: Now one caveat, it does say that this is in some form of alpha. It's like pretty early days for this project, but yeah. It says this project should be considered an alpha release. It's substantially API complete but some areas needs work. Well people start using it, people start contributing to it. All of a sudden it becomes not alpha. So anyway, it's out there for people to use. This is sort of the same umbrella area as Tom Christie and the Django REST framework area so I think it's a pretty good place.

06:30 OKKEN: Cool, okay, neat.

06:32 KENNEDY: Speaking of stuff under the Encode GitHub project.

06:35 OKKEN: So Starlette is part of that also? Okay. So interesting that we did not plan this.

06:41 KENNEDY: Nope.

06:41 OKKEN: It's like we, yeah.

06:42 KENNEDY: The universe planned it to line the parts.

06:45 OKKEN: Oh my gosh, we're wearing the same colored shirt also.

06:47 KENNEDY: Oh my gosh!

06:50 OKKEN: It's black so it's not that weird a coincidence, but anyway. So rhe article I wanted to highlight was a quick and dirty mock services with Starlette. This is by a friend of the show, Matt Layman. Hey Matt, it's kind of a neat thing. So one of the ideas, I don't know if we talked about this on the show, but the idea of testing. So if you've got something that part of your application or the thing you want to test talks to a third party service, sometimes you might not actually want to call that third party service. You can mock or stub out the calls to the service or one of the ideas is to create a fake service to talk to. And so that's what this article's about, is how to create a fake service so that you can use it for testing. And Matt writes a really quick one using Starlette and I think you could use other frameworks too but I like how simple this is. One of the needs he had was that he didn't want the service to just come back right away. He wanted some delay 'cause there was some of the service calls that were taking up to a minute or longer and he wanted to be able to simulate that within his fake service so that you can test the code under test to make sure it is working correct with a big delay. Anyway, I like there's a really easy to read write up, easy way to do that and I know that that's a recommended practice for a lot of instances. But there aren't very many write ups on how to do it and so congrats Matt. I like it.

08:16 KENNEDY: Yeah, that's really cool. When you think of dependencies, you think of well my database layer or Stripe API or something. But time is one of those dependencies that's really tricky.

08:27 OKKEN: Most of the time one of the reasons why we try to do a mock out service is to eliminate that time. You know us want to make sure that the flow of the data goes quickly but you don't want to wait for the service during your test. But at the same time, you kind of want to make sure that, and in most test cases maybe that's the right thing, but you also want to make sure that your application deals with long latencies if that's the normal situation, so.

08:52 KENNEDY: Yeah, for sure. You might want to shrink it from five minutes to five seconds or something so the tests don't run super long, but yeah. Or even like I want to start something and then see in an hour if it's done. You could like, you know, have to fake out like datetime.now. So the next time you ask it, it's like two hours from now, but it's one millisecond, you know?

09:13 OKKEN: Yeah. Well make sure your parametrization of that test doesn't make it so that you're running 50 different versions of the hour long wait.

09:22 KENNEDY: Yeah, for sure. All right, now before we get to the next item, let me tell you about some new services from DigitalOcean. So they've traditionally had recently added Postgres as a service. So now host to database, which is really cool, but maybe you don't use Postgres. Maybe you want to use Redis, so we just talked about queuing stuff. So Redis is now a service that you can get over at DigitalOcean, Redis as a service, and also MySQL as a service. So check that out over at pythonbytes.fm/digitalocean. Get $50 credit for a new user. They've got all sort of good stuff coming your way and, of course, all of our infrastructure runs on it as well, which is fun and very good. All right, the next thing I want to talk about is that other larger cloud provider. Have you heard of something? Some people started using this, Brian. It's called AWS. Have you heard of this?

10:13 OKKEN: I think so, yeah.

10:14 KENNEDY: Yeah, I think like a quarter of the internet at least runs on it, so. The Python API, the Python package, that is used to talk to things like S3 and EC2 and AMI and all the stuff over at AWS is called Boto, yeah? So there's a project brought to our attention by Giuseppe Cunsolo, thank you for that, Giuseppe, called mocking out AWS APIs with Moto. So if you're going to mock out BOTO, what would you call it? Moto, so.

10:46 OKKEN: Oh, nice.

10:47 KENNEDY: Now it lets you do all sorts of stuff like if you say want to mock out an S3 bucket, right? So S3 is like file block storage. You can mock it out and then save quote save a file to it and then see if it's there or preload it with data and then interact with it. And it's got all sorts of interesting stuff. But if you start to dig through this library, it turns out it's got both some really interesting ways to test stuff and it's also super comprehensive. So, for example, if I want to test working with S3, I can say from moto import mock_s3 and then I just put an @mock_s3 on my method and my test method and now all of my true S3 calls with Boto actually don't go to Boto. They go to this Moto mock version.

11:32 OKKEN: Oh, that's neat.

11:33 KENNEDY: Yeah, but apparently it doesn't really end there because if I wanted to work with EC2, I could import mock_ec2 and put that on there and say boto dot working with, you know, do all the boto client EC2 whatever. It's like I don't know why boto's so terribly complicated. It's like super overloaded. But anyway, that's a different time for a different story. You can go and ask for your reserved instances and it'll use your mocked out ones and things like that. So just like if you were to go flip through all the stuff that it supports, ACM, API Gateway, Auto Scaling, CloudFormation, CloudWatch, CloudWatch Events, Cognito Identity, Data Pipeline, DynamoDB, et cetera, et cetera, and we're only into the Ds, right? It goes on and on and on of all the different AWS things that it mocks out. So if you got to test AWS, this is pretty cool. Like you can mock out Glacier, IOT, Kinesis, whatever. You name it.

12:26 OKKEN: One of the notes you added was that it can run as a standalone server mode so that you can even test non Python code.

12:33 KENNEDY: That's right, that's right. So apparently I can, yeah. There's a way to run it as a server and then talk to it as if it were some of these services as well, which is pretty wild.

12:44 OKKEN: It is incredible and I guess good that Boto is all of the interface so that if it's one

12:50 KENNEDY: That it can mock.

12:50 OKKEN: Point to mock it, yeah.

12:53 KENNEDY: Yeah Boto, the reason I don't like it as you say, Boto and you create a client and the you pass it a string. The string could be like EC2 or S3 and then how you interact with it has to do with what factory that string drove it to to create a different, like it's really hard to discover what can I do with this thing because it's like a generic blob that can be converted into something more real. I don't know, it's funky. But yes, in this regard, it's definitely very nice.

13:19 OKKEN: Apparently your life is more complicated than mine.

13:21 KENNEDY: I guess I've worked with S3 and Elastic Transcoder and all that stuff too much.

13:26 OKKEN: Yeah. Okay, cool.

13:28 KENNEDY: What's next?

13:29 OKKEN: I was just thinking that maybe we should have done a Testing Code episode with all this testing stuff. But one of the things that we have, I think we've talked about MongoEngine before. And I know that you've used MongoEngine in some of your courses, right?

13:42 KENNEDY: Yeah, for sure, and on the sites.

13:44 OKKEN: Yeah, and it's like a mappers...

13:47 KENNEDY: They don't say ORM 'cause Object Relational Mapping. 'Cause it's not really the R so they say it ODM for Object Document Mapper. That's like the... The adapted acronym.

13:56 OKKEN: So I didn't know that there were, so I knew about MongoEngine because of you, but I didn't know about others and here's a project called umongo or umongo, not sure.

14:05 KENNEDY: Yeah, I think umongo is the way you would say it. I think that's the Greek mew, yeah.

14:09 OKKEN: It's a Python MongoDB ODM and it comes because the people who made it had two needs. They wanted an asynchronous ODM and for some reason they felt like it was difficult to do serialization and unserialization of documents with existing ODMs. And so this one does it. So it's a asynchronous sort of thing. It works directly with some of the popular Mongo drivers like PyMongo, TxMongo that's for twisted, Motor_asyncio, and mongomock. And I'm kind of excited about mongomock.

14:45 KENNEDY: Yeah, for sure. I got to say, this is probably the most interesting aspect of this because something like MongoEngine, it just works with PyMongo. But this one says you know what? We actually can work with all these different foundations which gives it like fundamentally different behaviors and features, right? If you point it at the Twisted one, it can now integrate with twisted's asynchronous model. Motor_asyncio now lets it plug into true asyncio stuff. Like you could plug it into Starlette like you discussed before and those async methods there. And then the mocking as well. We're just going to take the foundation and make it a mock version. All those are super cool.

15:20 OKKEN: Yeah, and then also because of this I learned about mongomock which is a cool way to mock out your Mongo database, so. Kind of neat. Definitely want to try this sometime.

15:31 KENNEDY: Yeah, it definitely looks interesting. I don't remember exactly when this got created. Do you know what GitHub needs? Is it needs a this project was created on this day, right? Because you can go up after the commits but if there's thousands of commits, you've got to infinite scroll your way back to the origins. But anyway, yeah it's really cool. I think if I had known about this at first I might have even chosen this over MongoEngine, but yeah. It's really nice. It's nice it has a 90% plus test coverage. I really like the different foundations which means you learn like one ODM style programming but then you can use it in these different situations. Yeah, super cool. I like it. And also I didn't know about mongomock either, but now I'm excited about it. It looks cool. Nice, all right. So let me hit you with an opinionated piece here and you can give me some feedback, let me know what you think about it. Some parts I...

16:24 OKKEN: Okay, I've got my tomatoes ready.

16:26 KENNEDY: Yeah, perfect. Get ready to throw 'em, I'll get ready to dodge 'em. So this article was sent over by Tyler Matteson. Listener, thanks for sending that over. Now it's called "The Single Responsibility Principle in Python" and the Single Responsibility Principle or SRP comes out of this larger group of design principles, coding principles, architecture principles, something like that called SOLID. So S is the Single Responsibility Principle, O is the Open Close Principle, L is Liskov Substitution Principle, and these even come in really interesting de-motivator style posters. I don't know if you're all familiar with de-motivators. You're probably familiar with motivators. Like you will see this probably in the entryway kind of near the CO office. It's like probably got a giant eagle soaring above a cloud. It's like "Together we can fly higher than you ever dreamed" or something stupid like that. Well the de-motivators take pictures like that but put negative captions and connotations. So these are really interesting things. For example the Liskov Substitution one has a duck and a rubber duck in the water and it says, "If it looks like a duck and quacks like a duck but it needs batteries, you probably have the wrong abstraction." Right, so there's all these really cool de-motivator style posters around the SOLID principles and I think the SOLID principles are pretty neat. Like you can go overboard with them but they are pretty powerful. So this article focuses specifically on the Single Responsibility Principle and I think talking about it is really interesting. I think the solution is unexpected and it made me think. And at first I'm like, "Ooh, I don't like it." But after a while I'm like "But it is pretty creative." And things that make me think I like to share with people, so I guess that's why I chose it. And it starts out by saying this article will guide you through the complex process of writing simple code. And it talks about stuff that you would expect, right? So there's some kind of super long function say this is terrible. You can't test it, you can't reason about it because it's doing five things. How do you know whether you're doing too much? You can look at measures like the psychometric complexity, the Halstead complexity thing, the number of arguments, the number of statements, body length, all the kind of stuff. And that'll help you condense down into, say, reasonable classes or functions and this is really mostly focusing on functions. So all right, so let's think of another simple example. I want to calculate the price of a bunch of products given a list of products and it seems all well and good except for that it also happens to use some kind of logger object and log out the price. Now this thing is doing two things. It's one, calculating and logging it out. What if you want to test it or work with it with its computation bits but not the logging bit or something like that? So how do you go around dealing with these types of dependencies? Right, this kind of goes back to the testing side of thing, Brian. How do you deal with things like files and database connections or database connection objects and loggers and whatnot? So this guy takes us through his whole sort of thought process which is really good and says, "You know what we're going to do? "We're going to come up with callable classes.

19:42 OKKEN: Oh dear.

19:42 KENNEDY: And the callable classes will be given the dependencies and then you call them like functions, allowing you to pass different dependencies or other dependencies. And now that seems like a really bad idea, but they're using a lot of, my first thought was okay, that's like a curvy shape, weird hammer that I guess you could hit it with. But it's using a lot of interesting Python features. So if you create a class that has, it's a data class, you set frozen equal to true, slots equal to true. It means all of its variables become internal and private only. So it'd be private or final rather so it can't be derived from and you give it a call function. It turns out to be pretty interesting and then they talk about some dependency injection frameworks that work with these types of classes. In the end, it comes up with a pretty interesting way to create these things and then call 'em. I'm not sure that I love it, but it definitely made me think and I was a little bit surprised by it, so I thought you all might enjoy it.

20:49 OKKEN: Yeah, it's a entertaining read, I'll give you that. Yeah.

20:53 KENNEDY: Look at the very last code fragment in that article, Brian.

20:57 OKKEN: Okay. The last code fragment? That says send_postcards = container.resolve(SendTodaysPostcardsUsecase)?

21:06 KENNEDY: Yes, exactly. And the little bump above it. So it is powerful in that you can create these objects, you can pass in mock objects without actually using mocked up patch and stuff. It's interesting. I don't think that I would go write software using this today, but I might use some of the concepts of it maybe.

21:28 OKKEN: My opinion is that it is possible to do this and it might make sure that you don't get fired 'cause nobody else can edit your code. Okay, so I do like the Single Responsibility part of SOLID. I am not a fan of the Liskov Substitution Principle or the Open Closed or dependency injection is something that frightens me. These are things that are put in place to work around limitations of other languages like Java and C. And I'm going to get tomatoes thrown at me for coming up with saying this, but some problems are complex and you can remove the complexity out of one part of your code but it moves to somewhere else. And maybe it makes your unit test better, but your system tests are just the same.

22:19 KENNEDY: Or it makes writing in regular code super hard because your like where did this database connection come from? I have no idea how I got here, right? Yeah.

22:29 OKKEN: It is a thing, it's an interesting article. I'm not a fan.

22:32 KENNEDY: Yeah, I submit this to everyone as a thought piece, not necessarily as a guidance.

22:37 OKKEN: Yeah, well I'm one to say that it's wrong. It's just my opinion, so.

22:40 KENNEDY: All right, well I think we should leave it there for our main topics. Got anything you want to share with folks this week? I've got one quick one.

22:46 OKKEN: Oh, share it with us then.

22:47 KENNEDY: So do you know Bob and Julian over at PyBites? Of course, right? Well they recently, they have their Code Challenges Platform which is at codechalleng.es, right? The dot E-S extension. Like just Code Challenges, put the period between G and ES. And that's a place where you can go to like practice exercises and it's a cool platform they've created. And they just came out with Byte 220 where they walk people through analyzing the Python Bytes RSS feed.

23:18 OKKEN: Oh cool.

23:19 KENNEDY: Yeah, so you can go in there and it shows you how to use feedparser which is a really cool way to parse RSS feeds, go in there and actually answer questions. What were they looking at? It was things like what is the average duration of an episode in seconds? What are the number of episodes with special guests, like Trey was on a show last week for example. What are the most common domain names we mention like GitHub probably is right up near the top and things like that. So it's pretty cool that there's like this guided challenge to get to on our platform. At least our platform's data.

23:51 OKKEN: Yeah, I'll have to come up with ways in our show notes to break their challenge.

23:58 KENNEDY: Yeah, that sounds good. That's all fun, people can check that out if they're interested. You ready for some jokes?

24:04 OKKEN: Definitely.

24:05 KENNEDY: These come from Web Boss's Dad jokes GitHub repo. So they're pretty short and sweet, but they're in the dad joke style, which is kind of bad jokes that are hopefully funny. So what do you get when you cross a computer and a life guard?

24:19 OKKEN: I don't know, what?

24:20 KENNEDY: A screensaver. What do you get when you cross a computer with an elephant?

24:26 OKKEN: I am not sure.

24:26 KENNEDY: Lots of memory. Lots of memory.

24:27 OKKEN: Oh yeah. Lots of memory, okay.

24:31 KENNEDY: Nice.

24:32 OKKEN: So I'm going to share this because I made it up and I thought is was funny, but it's an anti joke. So a Python developer, a PHP developer, a C# developer, and a Go developer all went to lunch together.

24:42 KENNEDY: Oh my gosh, what happened?

24:43 OKKEN: They had a nice lunch and all got along fine.

24:45 KENNEDY: Perfect. Of course they should. They're all developers.

24:50 OKKEN: Anyway, thanks.

24:52 KENNEDY: Awesome, yeah. Well thanks, thanks as always, Brian. See you later. Thank you for listening to Python Bytes. Follow the show on Twitter via @PythonBytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page