Transcript #147: Mocking out AWS APIs

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.

00:04 This is episode 147, recorded September 4th, 2019.

00:09 I'm Michael Kennedy.

00:10 I am Brian Okken.

00:11 And this episode is brought to you by DigitalOcean.

00:13 Tell you more about some cool stuff they got coming up later.

00:17 Right now, I want to talk about tables.

00:19 Okay. And really quick, what's 147 in hex?

00:22 No, just, I couldn't do that. I don't expect you.

00:25 Anyway.

00:27 No, but one of the things I do like need to do occasionally, I need to do really fast is generate tables.

00:33 And I know it's not something that everybody needs to do, but if I've got a whole bunch of data, like in this case, some dictionaries, a list, and I want to be able to just print that stuff out.

00:44 They're all the same type of dictionary, they just hold some data. I want to print them out.

00:48 How do I do that quickly? And for test reports or whatever. I found this called RapidTables, and it's really kind of awesome.

00:56 >> Oh, that is so cool. Wow.

00:59 >> Yeah. So you just basically you've got a list of dictionaries, and then you want to print them out, and you want it to look like a good ASCII table.

01:07 It shows you how to do it, and it also shows you how to do it with a thing called term color.colored.

01:13 I didn't know this was around.

01:15 So you can do colored tables if you want.

01:17 I probably won't do that.

01:19 It also converts stuff to markdown and restructured text.

01:22 That's neat. I might use that.

01:24 But right away, I just need the just really easy tables.

01:28 And it says it's super fast in speed--

01:30 like speed-wise, it goes fast.

01:32 But what I like is that it's not a lot of extra code gunk in your code.

01:38 You just kind of say, I want this table, and you can print it out easily.

01:41 Yeah, that's super cool, because a lot of times, like this kind of formatting, it's a lot of Z fill, and pad these strings, and L pad--

01:50 just like trying to get the spacing, You're putting tabs, but every now and then the tab, like the word is too big, and so the tab will shoot it to the next column.

01:57 It's a lot of work, and this is not you doing that work.

02:00 Right.

02:01 So for all the columns, you have to go through each of the elements to find the longest one to figure out how wide you need to make the column.

02:08 And yeah, there's other libraries to do this, but this one I think is pretty cool.

02:12 Yeah, it's pretty cool.

02:13 I also like the Markdown and Restructure Text Output option.

02:16 Yeah, very cool.

02:17 So if you think about what is one of the most popular packages, what would you say?

02:23 Take some guesses.

02:24 - Requests.

02:25 - It's gotta be requests.

02:26 I don't know if it's the most used one, but it's certainly the one that has people's attention and imagination.

02:32 So what would you think if I told you here is a competitor to requests called HTTPX?

02:39 - Ooh, I like the X.

02:40 - It's extra cool, absolutely.

02:42 What's really interesting about HTTPX though is it's not a from scratch, like throw away the ideas of requests.

02:49 it's compatible with the request API. So what you can do with requests you can do with this, so you can kind of drop it in. But it has all these other cool features. And in fact, it comes from this project called encode in COD encode. And over there, they have a ton of cool things including Django rest framework, starlet UVA, corn, HTTPS, and also some other async stuff.

03:12 And a lot of the things happening around here UVA, corn databases, orm HTTPS is all about adding the asynchronous capabilities to these various things.

03:22 And that's one of the shortcomings of requests, is it doesn't support an async option.

03:26 But this HTTPX, it does a bunch of cool things around the async space to kind of let you gradually go up the stack of complexity.

03:35 So you can just be straight synchronous stuff doing what request does.

03:40 It has this concept of parallel requests.

03:43 So you can issue a bunch of requests in parallel, but you don't have to think about actually creating the async event loop and stuff, you just say, I want to run these things all in parallel.

03:53 And you could even ask it, say, I want to process the responses as they come in.

03:57 So I can say, while parallel.hasPendingResponses, the response is parallel.nextResponse.

04:04 So I could kick off 20 requests and just process them as they come back in four lines of code or something.

04:10 - Oh, that's so cool.

04:11 I like it.

04:11 - Yeah.

04:11 Or if you really want, if you're in something like Starlet, where it has actual async support for the method, you do want that to be async.

04:18 So you can actually flip into an async mode where those happen the same way I described, but also on the async IO event loop.

04:27 So it's really cool.

04:28 It also has some other neat features.

04:30 So like I said, a request compatible API.

04:32 It does HTTP/2, which I could do a request for a page.

04:37 And as I pull down, say, five CSS style sheets, and a JavaScript, and an image, you can go get those all in one request.

04:45 So it has support for that, which is cool.

04:47 It has a standard synchronous interface, but also async and await if you want it.

04:52 And Brian, you'll like this one.

04:54 It allows you to make direct requests to WSGI or ASGI, the asynchronous variant of that, apps.

05:00 So imagine I'm creating a Flask app, and I want to test it.

05:03 I can actually, inside of my Flask app, provide HTTPX, the app, right, the Flask app object, and then I can issue requests that don't go through the network layer, They just go through this test app instance that I've created.

05:16 Oh, that's cool.

05:17 Isn't that cool?

05:18 So if I want to mock out some behavior or interact with it as if I were externally, I don't have to start a server and talk to it.

05:25 I'd literally just use the same API here, but directly skipping all the infrastructure.

05:31 Oh, that sounds fun.

05:32 Yeah.

05:33 In a nerdy kind of fun.

05:34 Yeah, that sounds really fun, but yeah.

05:39 It is also fully type annotated, so it's It's very Python 3 friendly.

05:44 Has 100% test coverage.

05:46 And yeah, it has support for mocking out things.

05:49 A lot of cool, nice features.

05:50 And the fact that all this is built on top of requests means if you're already using requests, it's like you can adopt it pretty easily because you don't have to change anything.

05:59 But if you want to bring in some of these cool new features, it's just you start using them.

06:03 Nice.

06:03 Now, one caveat.

06:05 I does say that this is in some form of alpha.

06:08 It's like pretty early days for this project.

06:10 But yeah, it says this project should be considered an alpha release.

06:14 It's substantially API complete, but some areas need to work.

06:17 Well, people start using it.

06:19 People start contributing to it.

06:20 All of a sudden, it becomes not alpha.

06:22 So anyway, it's out there for people to use.

06:24 This is sort of the same umbrella area as Tom Christie and the Django REST framework area.

06:29 So I think it's a pretty good place.

06:30 Cool.

06:31 OK.

06:32 Neat.

06:32 Speaking of stuff under the ENCODE GitHub project--

06:35 So Starlet is part of that also?

06:37 Yeah.

06:38 OK.

06:38 So interesting that we did not plan this.

06:43 The universe planned it to line it up for us.

06:44 Oh my gosh, we're wearing the same color shirt also.

06:47 Oh my gosh!

06:50 It's black, so it's not that weird of a coincidence.

06:52 But anyway, so the article I wanted to highlight was Quick and Dirty Mock Services with Starlet.

06:59 This is by a friend of the show, Matt Lehman.

07:01 Hey Matt, it's kind of a neat thing.

07:03 So one of the ideas, I don't know if we've talked about this on the show, but the idea of testing. So if you've got something that you're part of your application or the thing you want to test talks to a third-party service, sometimes you might not actually want to call that third-party service. You can mock or stub out the calls to the service or one of the ideas is to create a fake service to talk to. And so that's what this article is about is how to create a fake service so that you can use it for testing. And Matt uses, writes It's a really quick one using Starlet.

07:38 And I think you could use other frameworks too, but I like how simple this is.

07:43 One of the needs he had was that he didn't want the service to just come back right away.

07:48 He wanted some delay, 'cause there were some of the service calls that were taking up to a minute or longer, and he wanted to be able to simulate that within his fake service so that you can test the code under test to make sure it is working correct with a big delay.

08:02 Anyway, it was a really easy to read, write up, easy way to do that.

08:06 And I know that that's a recommended practice for a lot of instances.

08:11 But there aren't very many write-ups of how to do it.

08:14 And so congrats, Matt.

08:15 I like it.

08:16 - Yeah, that's really cool.

08:17 You know, when you think of dependencies, you think of, well, my database layer or Stripe API or something.

08:24 But time is one of those dependencies that's really tricky.

08:27 - Most of the time, one of the reasons why we try to do a Maka service is to eliminate that time.

08:32 You know, you want to make sure that the flow of the data it goes quickly, but you don't want to wait for the service during your test.

08:39 But at the same time, you kind of want to make sure that you're, in most test cases, maybe that's the right thing, but you also want to make sure that your application deals with long latencies, if that's the normal situation.

08:52 - Yeah, for sure.

08:53 You might want to shrink it from five minutes to five seconds or something, so the tests don't run super long, but yeah.

08:59 Or even like, I want to start something and then see in an hour if it's done.

09:04 you could have to fake out like datetime.now.

09:08 So the next time you ask it, it's like two hours from now, but it's one millisecond.

09:13 - Yeah, well, and make sure your parameterization of that test doesn't make it so that you're running 50 different versions of the hour-long wait.

09:21 - Yeah, for sure.

09:24 All right, now before we get to the next item, let me tell you about some new services from DigitalOcean.

09:28 So they've traditionally had recently added Postgres as a service, so now hosted database, which is really cool, but maybe you don't use Postgres.

09:37 Maybe you wanna use Redis.

09:38 We just talked about queuing and stuff.

09:41 So Redis is now a service that you can get over at DigitalOcean, Redis as a service, and also MySQL as a service.

09:49 So check that out over at pythonbytes.fm/digitalocean.

09:53 Get $50 credit for a new user.

09:55 They've got all sorts of good stuff coming your way, and of course, all of our infrastructure runs on it as well, which is fun and very good.

10:02 All right, the next thing I want to talk about is that other larger cloud provider.

10:09 Have you heard of something?

10:10 Some people started to use this, Brian.

10:12 It's called AWS.

10:12 Have you heard of this?

10:13 I think so, yeah.

10:14 Yeah, I think like a quarter of the internet at least runs on it.

10:18 So the Python API, the Python package that used to talk to things like S3 and EC2 and AMI and all the stuff over at AWS, it's called Bodo.

10:31 So there's a project brought to our attention by Giuseppe Consolo--

10:36 thank you for that, Giuseppe--

10:37 called Mocking Out AWS APIs with Modo.

10:42 So if you're going to mock out Modo, what would you call it?

10:45 Modo.

10:46 So--

10:46 -Oh, nice.

10:47 -Now, it lets you do all sorts of stuff.

10:48 Like if you, say, want to mock out an S3 bucket, right?

10:54 So S3 is like file blob storage.

10:56 You can mock it out and then, like, save a file to it and then see if it's there, or preload it with data, and then interact with it.

11:02 And it's got all sorts of interesting stuff.

11:04 But if you start to dig through this library, it turns out it's got both some really interesting ways to test stuff, and it's also super comprehensive.

11:14 So for example, if I want to test working with S3, I can say from Modo import mock S3, and then I just put an @mock S3 on my method, on my test method.

11:25 And now all of my true S3 calls with Bodo actually don't go to Bodo.

11:30 they go to this Modo mock version.

11:32 - Oh, that's neat.

11:33 - Yeah, but apparently it doesn't really end there because if I wanted to work with like EC2, I could import mock EC2 and put that on there and say Bodo.workingWith, do all the Bodo client EC2, whatever.

11:45 It's like, I don't know why Bodo is so terribly complicated.

11:48 It's like super overloaded.

11:49 But anyway, that's a different time for a different story.

11:52 You can go and like ask for your reserved instances and it'll use your mocked out ones and things like that.

11:58 So just like if you were to go flip through all the stuff that it supports, ACM, API gateway, auto scaling, cloud formation, cloud watch, cloud watch events, Cognito identity, data pipeline, DynamoDB, et cetera, et cetera, and we're only into the Ds, right?

12:14 It just goes on and on and on of all the different AWS things that it mocks out.

12:19 So if you got to test AWS, this is pretty killer.

12:22 Like you can mock out Glacier, IoT, Kinesis, whatever, you name it.

12:26 One of the notes you added was that it can run as a standalone server mode so that you can even test non-Python code.

12:33 That's right.

12:34 That's right.

12:34 So apparently, I can-- yeah, there's a way to run it as a server and then talk to it as if it were some of these services as well, which is pretty wild.

12:44 It is incredible.

12:45 And I guess good that Bodo is all of the interface so that it's one point to mock it.

12:52 Yeah.

12:53 Yeah, Bodo.

12:54 The reason I don't like it is you say Bodo, and you create a client, and you pass it a string.

12:58 The string can be like EC2 or S3.

13:01 And then how you interact with it has to do with what factory that string drove it to to create a different--

13:08 it's really hard to discover what can I do with this thing, because it's like a generic blob that can be converted into something more real.

13:16 I don't know.

13:16 It's funky.

13:17 But yes, in this regard, it's definitely very nice.

13:19 Apparently, your life is more complicated than mine.

13:21 I guess I've worked with S3 and Elastic Transcoder and all that stuff too much.

13:26 - Yeah, okay, cool.

13:28 - What's next?

13:29 - I was just thinking that maybe we should have done a testing code episode with all this testing stuff.

13:32 - Yeah.

13:33 - But one of the things that we have, I think we've talked about Mongo Engine before and I know that you've used Mongo Engine in some of your courses, right?

13:42 - Yeah, for sure, and on the sites.

13:44 - Yeah, and it's like a mappers.

13:47 - They don't say ORM because object relational mapping 'cause it's not really the R, so they say ODM for object document mapper.

13:53 That's like the adapted acronym.

13:56 - So I didn't know that there were other, so I knew about Mongo engine because of you, but I didn't know about others.

14:00 And here's a project called MicroMongo or MewMongo.

14:05 - Yeah, I think MicroMongo is the way you would say it.

14:06 I think that's the Greek, Mew, yeah.

14:09 - It's a Python MongoDB ODM.

14:11 And it comes because the people that made it had two needs.

14:16 They wanted an asynchronous ODM.

14:18 And for some reason, they felt like it was difficult to do serialization and un-serialization of documents with existing ODMs.

14:28 And so this one doesn't.

14:29 So it's an asynchronous sort of thing.

14:32 It works directly with some of the popular Mongo drivers, like PyMongo, TxMongo--

14:38 that's for Twisted--

14:39 MotorAsyncIO, and MongoMock.

14:42 And I'm kind of excited about MongoMock.

14:45 Yeah, for sure.

14:45 I got to say, this is probably the most interesting aspect to this because something like Mongo engine, it just works with PyMongo.

14:51 But this one says, you know what, we actually can work with all these different foundations, which gives it fundamentally different behaviors and features, right?

15:00 If you pointed at the Twisted one, it can now integrate with Twisted's asynchronous model.

15:04 MotorAsyncIO now lets it plug into true AsyncIO stuff.

15:09 Like you could plug it into Scarlett, like you discussed before in those async methods there.

15:15 And then the mocking as well, like we're just gonna take the foundation and make it a mock version.

15:19 All those are super cool.

15:20 - Yeah, and then also because of this, I learned about MongoMock, which is a cool way to mock out your Mongo database.

15:28 So, kind of neat.

15:29 Definitely want to try this sometime.

15:31 - Yeah, it definitely looks interesting.

15:33 I don't remember exactly when this got created.

15:36 Do you know what GitHub needs?

15:37 Is it needs a, this project was created on this day, right?

15:40 Because you can go back to the commits, but if there's thousands of commits, you've got to like infinite scroll your way back to the origins.

15:48 But anyway, yeah, it's really cool.

15:50 I think if I had known about this at first, I might have even chosen this over Mongo engine.

15:54 But yeah, it's really nice.

15:56 It's nice it has 90% plus test coverage.

16:00 I really like the different foundations, which means you learn like one ODM style of programming, but then you can use it in these different situations.

16:08 Yeah, super cool.

16:09 I like it.

16:09 And also I didn't know about MongoMock either, but now I'm excited about it.

16:13 That looks cool.

16:14 Nice.

16:15 So let me hit you with an opinionated piece here, and you can give me some feedback.

16:22 Let me know what you think about it.

16:24 Some parts of it--

16:25 - Okay, I got my tomatoes ready.

16:26 - Yeah, perfect.

16:27 Get ready to throw 'em.

16:28 I'll get ready to dodge 'em.

16:29 So this article was sent over by Tyler Madison, a listener, thanks for sending that over.

16:36 Now, it's called the Single Responsibility Principle in Python, and the Single Responsibility Principle, or SRP comes out of this larger group of design principles, coding principles, architecture principles, something like that, called SOLID.

16:50 So S is the single responsibility principle, O is the open-close principle, L is Liskov substitution principle.

16:58 And these even come in really interesting demotivator style posters.

17:03 I don't know if you're all familiar with demotivators.

17:05 You're probably familiar with motivators.

17:07 Like you will see this probably in the entryway, like kind of near the CEO office, it's like probably got a giant eagle soaring above clouds, like together we can fly higher than you ever dreamed or something stupid like that.

17:20 Well, the demotivators take pictures like that but put like negative captions and connotations.

17:26 So these are really interesting things.

17:28 For example, the Liskov substitution one has a duck and a rubber duck in the water.

17:33 And it says, if it looks like a duck and quacks like a duck, but it needs batteries, you probably have the wrong abstraction, right?

17:39 So there's all these.

17:40 There's all these really cool demotivator style posters around the solid principles.

17:45 And I think the solid principles are pretty neat.

17:47 You can go overboard with them, but they are pretty powerful.

17:50 So this article focuses specifically on the single responsibility principle.

17:56 And I think talking about it is really interesting.

17:59 I think the solution is unexpected.

18:02 And it made me think.

18:03 And at first, I'm like, oh, I don't like it.

18:05 But after a while, I'm like, but it is pretty creative.

18:08 and things that make me think I like to share with people.

18:10 So I guess that's why I chose it.

18:12 And it starts out by saying, this article will guide you through the complex process of writing simple code.

18:18 And it talks about stuff that you would expect, right?

18:21 So there's some kind of super long function, says this is terrible, you can't test it, you can't reason about it because it's doing five things.

18:29 How do you know whether you're doing too much?

18:31 You can look at measures like the cyclomatic complexity, the Halstead complexity thing, the number of arguments, number of statements, body length, all that kind of stuff. And that'll help you condense down into, say, reasonable classes or functions. And this is really mostly focusing on functions. So alright, so let's think of another simple example, I want to calculate the price of a bunch of products, given a list of products. And it seems all well and good, except for that it also happens to use some kind of logger object and log out the price. Well, now, this thing is doing two things, It's one calculating and logging it out.

19:06 What if you want to test it or work with it with its computation bits, but not the logging bit or something like that?

19:12 So how do you go around dealing with these types of dependencies?

19:16 This kind of goes back to the testing side of things, Brian.

19:19 How do you deal with things like files and database connections or database connection objects and loggers and whatnot?

19:30 So this guy takes us through this whole sort of thought process, which is really good, and says, you know what we're going to do?

19:38 We're going to come up with callable classes.

19:42 And the callable classes will be given the dependencies.

19:48 And then you call them like functions, allowing you to pass different dependencies or other dependencies.

19:55 And now that seems like a really bad idea, but he's using, they're using a lot of it.

20:00 My first thought was, okay, well, that's, you know, that's like a curvy shaped weird hammer that I guess you could hit it with.

20:08 But it's using a lot of interesting Python features.

20:10 So if you create a class that has, it's a data class, you set frozen equal to true slots equal to true.

20:18 It means all of its variables become internal and private only said it'd be private or final rather, so it can't be derived from.

20:26 And you give it a call function.

20:28 It turns out to be pretty interesting.

20:30 And then they talk about some dependency injection frameworks that work with these types of classes.

20:36 In the end, it comes up with a pretty interesting way to create these things and then call them.

20:43 I'm not sure that I love it, but it definitely made me think.

20:46 And I was a little bit surprised by it.

20:48 So I thought you all might enjoy it.

20:49 Yeah.

20:50 It's an entertaining read, I'll give you that.

20:54 Look at the very last code fragment in that article, Brian.

20:57 OK.

21:00 The last code fragment that says, like, send postcards equals container dot resolve.

21:06 Send to these--

21:07 Yes, exactly.

21:07 And the little bit above it, yeah.

21:09 Yeah, yeah.

21:10 So it is powerful in that you can create these objects.

21:13 You can pass in mock objects without actually using mock.patch and stuff.

21:18 It's interesting.

21:20 I don't think that I would go write software using this today, but I might use some of the concepts of it, maybe.

21:28 - My opinion is that it is possible to do this, and it might make sure that you don't get fired 'cause nobody else can edit your code.

21:37 - Yeah.

21:38 - Okay, so I do like the single responsibility part of solid.

21:43 I am not a fan of the Liskov substitution principle or the open-closed, or dependency injection is something that frightens me.

21:55 These are things that were put in place to work around limitations of other languages like Java and C.

22:01 And I'm gonna get tomatoes thrown at me for coming up with saying this, but some problems are complex.

22:07 And you can remove the complexity out of one part of your code, but it moves to somewhere else.

22:14 And maybe it makes your unit tests better, but your system tests are just the same.

22:19 - Or it makes writing regular code super hard because you're like, where did this database connection come from?

22:25 I have no idea how I got here, right?

22:29 Yeah.

22:29 It is a thing.

22:30 It's an interesting article.

22:31 I'm not a fan.

22:32 Yeah, I submit this to everyone as a thought piece, not necessarily as guidance.

22:37 Yeah, but I'm not willing to say that it's wrong.

22:39 It's just my opinion.

22:40 All right, well, I think we should leave it there for our main topics.

22:43 Got anything you want to share with folks this week?

22:45 I've got one quick one.

22:46 Oh, you'd share it with us, then.

22:47 So do you know Bob and Julian over at PyBytes?

22:51 - Of course, right?

22:53 Well, they recently have their Code Challenges platform, which is at codechallenges.is, right?

23:00 The .es extension, right?

23:02 Just Code Challenges, but the period between G and ES.

23:05 And that's a place where you can go do practice exercises, and it's a cool platform they've created.

23:11 And they just came out with Byte 220, where they walk people through analyzing the Python Bytes RSS feed.

23:18 - Oh, cool.

23:19 - Yeah, so you can go in there, It shows you how to use Feed Parser, which is a really cool way to parse RSS feeds.

23:25 Go in there and actually answer questions.

23:28 What were they looking at?

23:30 It was things like, what is the average duration of an episode in seconds?

23:34 What are the number of episodes with special guests?

23:37 Like Trey was on the show last week, for example.

23:40 What are the most common domain names we mentioned?

23:42 Like GitHub probably is right up near the top and things like that.

23:46 So it's pretty cool that there's like this guided challenge to get through our platform.

23:50 at least our platform's data.

23:51 - Yeah, I'll have to come up with ways in our show notes to break their challenge.

23:56 No.

23:57 (laughing)

23:58 - Yeah, that sounds good.

23:59 So that's all fun.

24:00 People can check that out if they're interested.

24:03 You ready for some jokes?

24:04 - Definitely.

24:05 - These come from Web Boss's dad jokes GitHub repo.

24:08 So they're pretty short and sweet, but they're in the dad joke style, which is kind of bad jokes that are hopefully funny.

24:16 So what do you get when you cross a computer and a lifeguard?

24:19 - I don't know, what?

24:20 - A screensaver.

24:21 When you cross a computer with an elephant.

24:25 Lots of memory.

24:27 - Oh yeah.

24:28 - Lots of memory.

24:29 - Lots of memory.

24:30 Okay.

24:31 - Nice.

24:32 - So I'm gonna share this because I made it up and I thought it was funny, but it's an anti-joke.

24:36 So a Python developer, a PHP developer, a C# developer, and a Go developer all went to lunch together.

24:42 - Oh my gosh, what happened?

24:43 - They had a nice lunch and all got along fine.

24:46 - Perfect.

24:47 Of course they should.

24:48 developers. Anyway, thanks. Awesome. Yeah, well, thanks. Thanks as always, Brian. See you later.

24:55 Thank you for listening to Python Bytes. Follow the show on Twitter via @PythonBytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm. If you have a news item you want featured, just visit PythonBytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Aukin, this is Michael Kennedy. Thank Thank you for listening and sharing this podcast with your friends and colleagues.