Transcript #31: You should have a change log

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.

00:05 This time it's Python Bytes episode 31, recorded on Tuesday, June 20th, 2017.

00:12 I'm Michael Kennedy.

00:13 And I'm Brian Ockin.

00:14 And we have a bunch of cool things to talk about.

00:17 Some of them are huge, and some of them are kind of tiny.

00:19 Let's start small, huh?

00:20 Yeah, let's start small.

00:21 I really appreciate, it's one of the reasons why I like following Twitter for Python news,

00:27 is that's where I found Tiny Mongo.

00:30 So I saw somebody talking about it last week.

00:34 That's awesome.

00:35 I'm a fan of MongoDB and TinyDB.

00:37 And if they could come together, that'd be even better.

00:40 Right.

00:40 So this is essentially an attempt to put, it's not an exact same interface,

00:47 but it's fairly close to the same interaction you do with Mongo with a single file system.

00:53 So it's a single file database.

00:56 And the, Stephen, the person working on this, did not, and I talked with him and he,

01:02 it wasn't his intent to always be right on top of TinyDB.

01:06 But so far, he's been really happy with TinyDB as the backend for Tiny Mongo.

01:11 And so, yeah, it just sits, it's using TinyDB as the database part, but exposes an interface that's very close to Mongo.

01:20 Yeah, that's super cool.

01:21 So basically, if you have code that talks to MongoDB through the PyMongo API,

01:27 you could more or less adapt that really quickly to Tiny Mongo.

01:32 And TinyDB, the backing store for this thing, more or less is like, let's create a simple document database

01:41 that's really just some JSON files living on your disk.

01:44 It's not a full-on production database, but if you're doing simple stuff,

01:47 like really simple things, like this is actually pretty sweet.

01:50 There's no server, right?

01:51 Right.

01:51 And yeah, there's no server.

01:53 I would say probably the other direction probably works the best.

01:56 So if you were going to, your end goal was to use Mongo, that Tiny Mongo might be a good way to start,

02:03 because it isn't the full set of functionality.

02:07 I don't have a complete list of what's missing.

02:10 I just have the personal experience of, I tried to take a Mongo application and just swap this in,

02:17 and I ran across a few errors and I haven't finished debugging those yet.

02:21 I'm just really excited about it because there's more than one document database

02:28 that I can use in small applications.

02:29 Yeah, that's cool.

02:30 And then also one of the applications for this, when I was talking with the maintainer of it,

02:36 is that he's using it on Raspberry Pis even.

02:40 So having a Mongo-like.

02:42 That is really cool, because you don't want to start up a whole separate server on like a Raspberry Pi,

02:47 but certainly having a little couple of JSON files laying around that you have like a database interface over top of.

02:53 That's cool.

02:53 Yeah, definitely.

02:54 So I was excited about this and I'm going to start using it right away.

02:58 That's sweet.

02:58 Yeah, if people are interested in TinyDB, I back on episode 80 of Talk Python many moons ago,

03:05 I interviewed the guy who created TinyDB and talked about some of the use cases.

03:10 And I think there's some extensions you can get like indexing add-ons and stuff like that.

03:14 So there's a lot of stuff to do with this.

03:15 Pretty cool.

03:16 So that sounds pretty dead simple, right?

03:18 Just fire up TinyDB and off you go?

03:20 Yeah, dead simple.

03:22 You know what else?

03:22 I want some dead simple validation.

03:24 And so the next project I chose is called Validus.

03:28 And Validus is on GitHub and it's described itself as a dead simple Python data validation library.

03:33 And have you ever tried to write a regular expression to match an email or a URL or something like that?

03:41 Oh, yes.

03:41 Yeah.

03:42 That's super fun, right?

03:43 No.

03:43 You think you get it working to someone who mails you like, I have a proper email address, but I can't sign up your system.

03:50 It says my email is invalid.

03:52 You're like, oh, gosh.

03:53 So this Validus thing kind of like solves that for a class of types of data, basically simple input.

03:58 So you can just import this and say Validus.

04:01 Is email and give it a string and it will say yes or no.

04:03 And you can ask it questions like, is it an RGB color?

04:06 Is it a phone number?

04:07 Is it an ISBN?

04:08 Is it an IPv4 or IPv6 address?

04:11 Is it a number?

04:12 Is it a slug?

04:13 Like would it fit at the end of a URL without needing encoding?

04:17 All that kind of stuff.

04:18 That's pretty awesome.

04:19 That's cool.

04:20 I'd say it's dead simple.

04:21 It's even got ismongoID.

04:22 Nice.

04:23 Yeah.

04:24 Yeah.

04:24 That's awesome.

04:25 So you know what else I like about this?

04:28 It's Python only.

04:30 No legacy Python.

04:30 Three, six.

04:32 Three, three.

04:32 Yeah.

04:33 Yeah.

04:33 Yeah.

04:33 Three, three and above.

04:34 So it's only a Python three thing.

04:36 So yet another sweet example of that.

04:38 I have a lot of interesting stuff to say about that at the end of the show.

04:41 Not Validus, but Python versus legacy Python.

04:43 While this works pretty well, we may still need to jump in the debugger, right?

04:47 Yeah, definitely.

04:48 And I'm a command line debugger kind of person.

04:51 Actually, I don't really jump into the debugger too much.

04:55 So you're a last resort, a debugger of last resort type person?

04:57 Yes.

04:58 Yeah, definitely.

04:59 Last resort.

05:00 And so in episode 29, we talked about launching the ability to launch PDB, the Python debugger,

05:08 from a failed pytest.

05:09 Somebody on Twitter, another Twitter person, KidPixo, I think.

05:14 Yeah, KidPixo, he runs the Geek Cookies Italian podcast, which I was a guest on like two and a half years ago.

05:20 He's a great guy.

05:20 He sees lots of good stuff.

05:21 Yeah.

05:22 Well, he passed this along because he said he really loves the PUDB debugger.

05:27 And my first reaction is, oh my God, this thing is ugly.

05:30 Because it does look like you're back in the 80s running on a 386 or something.

05:36 I feel like I've dialed into a VBS.

05:38 But it does have themes.

05:40 So after I played with it for a while, I switched it to a midnight theme and it looks just like I'm in my editor.

05:47 And then it's actually pretty slick.

05:49 And one of the things that you can do with it, it's a lot better than PDB.

05:54 And it's still small and fast.

05:56 And there's some documentation in it for how you can do the same thing that we did with pytest.

06:02 You can launch it just with whenever you hit a pytest failure.

06:06 So that's pretty cool.

06:09 Yeah, it's really nice.

06:10 I mean, you can use it over like SSH and stuff.

06:12 So if you're SSHed into a server, you can debug with this.

06:15 But it actually has like little windows.

06:17 I mean, it really does feel like I'm back on a BBS.

06:19 It's awesome.

06:20 Yeah.

06:20 Like you see your code and you can step through it.

06:22 You've got like a variables window and a stack and breakpoints.

06:25 And like, it's really nice.

06:26 It's like a ASCII curses type thing.

06:28 But the local window of already having your listing up and also all your local variables.

06:35 And that changing when you go up and down the stack, it's usually enough.

06:41 So I like it.

06:42 Yeah.

06:42 Yeah, it definitely hits the sweet spot.

06:44 Like the 80% case for debuggers.

06:47 It's cool.

06:48 All right.

06:49 So I'm definitely going to start using that if I need to debug anything without a windowing

06:54 environment like macOS or Linux or Windows.

06:56 Okay.

06:56 So the next thing that I want to talk about is a really interesting sort of wide ranging study

07:02 that the guys at PyUp.io did.

07:06 So PyUp.io is a cool service.

07:09 I'm actually paying customer of theirs because I really think what they're doing is awesome

07:13 and I use it for my web apps.

07:14 So the idea is you basically point, you give PyUp.io access to your requirements file in your

07:22 public or private GitHub repo.

07:24 And if there's a new version of indie requirement or transitive requirement that you depend upon,

07:29 it will tell you like, hey, there's a new release of the Pyramid Web Framework and here's the

07:34 change log.

07:34 And actually this one's a security update.

07:36 So get in there and fix it quick.

07:37 So it will like basically watch your requirements and tell you if there are any upgrades and things

07:41 like that.

07:42 And it'll issue them as a pull request.

07:43 So really cool.

07:44 So these guys have access to all these requirements files and many other things.

07:47 Right.

07:47 And they studied some Django requirements files on GitHub.

07:52 Now this isn't through their business.

07:53 They were able to use BigQuery to just get a hold of all of the Django requirement files

07:58 that are on GitHub.

07:59 And they found some interesting things.

08:01 And I guess this is not private, not the private repos, probably just the public ones.

08:04 But anyway, they said that Django is the most popular web framework and it's pretty old.

08:11 It's been around for 12 years used on all sorts of different projects.

08:14 So let's look at these requirements files, which specify like all the dependencies you have

08:19 to install and see what we can get from them.

08:22 So the first thing they ask is, do developers pin or freeze their requirements?

08:26 Right.

08:27 That's where in your requirements TXT, you could say, I depend on Django and I depend on SQL

08:31 Alchemy and I depend on requests.

08:32 Or you could say, I depend on Django equal, equal this version, request equal, equal that

08:37 version.

08:37 Right.

08:37 That's pinning or freezing.

08:39 And they said that 64% of Django developers pin their requirements.

08:44 That's interesting.

08:45 And another 20% or so do ranges.

08:49 So like I'm willing to take this range of versions, but not leave it unpinned.

08:56 And then some of them are just like, give me whatever I get when I ask for it.

08:59 So that's interesting.

09:00 Another thing that they said was pretty interesting is that Django 1.8, even though I think 1.10,

09:06 1.11 is the latest.

09:07 Django 1.8 is the most popular of them.

09:11 And that was pretty cool.

09:12 But one of the things I really wanted to point out here is they said that what is more worrisome

09:19 is 1.9, 1.7, and 1.6 are second, third, and fourth most popular on the list.

09:27 Why is that a problem?

09:28 None of them are receiving any security updates at all.

09:32 Oh, weird.

09:33 So what isn't that bad?

09:34 So 1.7 and 1.6 went end of life over two years ago.

09:37 So if you are on the web and your application listens on a socket, you want it to have all

09:43 the security patches, let me tell you.

09:45 That's bad news.

09:46 And here's like, if I add those up really quick, that's something like 40% of Django files they

09:52 found are using these older versions.

09:55 And in fact, he said only 2% of all Django projects they could find are actually on a

10:00 secure release.

10:01 Among all the projects, more than 60% use Django releases with one or more known security vulnerabilities.

10:07 And that's pretty intense, man, that only 2% of them are on a 100% known secure release.

10:14 Well, I mean, clearly it's recommended to go make sure that you're using a secure release.

10:19 But I was curious about the pinning or freezing.

10:23 Is that considered best practice?

10:25 So I think it depends on what you're doing.

10:27 For large, complicated applications, it's definitely considered a best practice.

10:32 The idea is you want to make the upgrade in your dependencies at the time of your choosing.

10:38 Right?

10:39 Like you want to have...

10:40 So if you're going to upgrade from, especially major frameworks like Django, if you're going

10:44 to go from Django 1.8 to 1.9, you don't want that to just happen one day when it gets

10:49 released and you happen to refresh your server because that might have breaking changes.

10:53 So you want to explicitly say, I depend on this one.

10:55 Oh, there's a new one out.

10:56 Let me test the new one.

10:57 And then explicitly change that number and have it like flip it for you.

11:02 Okay.

11:03 And basically that's what the PyUp service does that I use.

11:06 It will automatically upgrade my pyramid web framework from like 1.7 to 1.8 to 1.9.

11:12 But it doesn't flip it immediately.

11:14 It's like I have to...

11:15 It'll tell me and change my requirements files as a PR and I have to accept it basically.

11:19 Okay.

11:19 Yeah.

11:20 Yeah.

11:20 But pretty interesting stats there, especially if you're into Django, check that out.

11:25 Yeah, definitely.

11:25 It's kind of concerning that there's so many.

11:27 And then there's...

11:28 Those are...

11:29 I'm sorry to like hang out on this so much, but this is...

11:33 Was this projects or applications and is there a difference?

11:36 So as far as I can tell from the...

11:38 I don't really know.

11:39 Yanis, I think this guy who wrote it probably could maybe chime in in the comments if he's

11:43 listening.

11:44 But my understanding is basically they went and they studied the public repos that use

11:50 Django.

11:51 Okay.

11:51 So this also may not be quite representative because like companies like Pinterest that

11:57 depend on Django, they're obviously not going to make their code public.

12:00 Right.

12:01 So they may be doing slightly different things, but still it's interesting for you into at

12:05 least the open source side of Django.

12:07 Definitely.

12:08 It's cool.

12:08 Speaking of open source projects, do you think they should have a changelog?

12:11 Well, that's what I was curious about.

12:14 Yeah.

12:15 So I kind of am warming to the idea of changelogs.

12:18 I appreciate other projects with changelogs.

12:21 I actually asked some people back on Twitter again what they thought of them.

12:25 And there's a couple of things I came across, which was a website called Keep a Change Log.

12:31 I really like that site.

12:32 It's so clear and compelling.

12:34 It's great.

12:35 Yeah.

12:35 Well, it's also...

12:36 It talks about that there really isn't a standard...

12:39 If there is a standard format for them, this is probably as close as you can get.

12:45 And it talks about different standards in either REST or in Markdown.

12:50 There's different ways to do it.

12:52 And then when I was talking on Twitter about changelogs, some of the people from the pytest

12:58 project piped up and said that they're using a tool called TownCrier to maintain their changelog.

13:05 That looks really cool, but I've never done anything with it.

13:07 What's TownCrier do?

13:08 So what it does is you keep a separate directory within your project so that you can have it on different...

13:15 If you're using different branches.

13:17 And then different changes go in and you keep the changes in little snippet files so that they're...

13:25 Since they're separate files, they merge easy because they're going to be a new file for each change.

13:30 And then you go through and say, okay, I've pulled all these things in.

13:34 I want to go ahead and use...

13:36 Take everything in the directory and add it to the changelog.

13:39 Oh, I see.

13:39 You can keep a separate file that says, these are the breaking changes, these are the new features or whatever, then it'll build a changelog out of them?

13:46 Yeah.

13:46 Oh, sweet.

13:47 Okay.

13:47 Well, it adds to your existing...

13:48 And it can add to your existing one.

13:50 And one of the things I liked, if you're not doing something like TownCrier, one of the recommendations from Keep a Change Log was to keep at the top unreleased changes so that you...

14:02 Things that you haven't put a label on or done an official supported release yet.

14:07 Because those are things that may, I don't know, maybe you may end up kicking out.

14:11 Yeah, they also have some things that you shouldn't do, like don't just take your get changelog and make that your proper changelog.

14:18 Things like that.

14:19 Yeah.

14:19 And one of the things I saw when I was doing some research for this, I did see some various automated ways to do it.

14:26 But that's the sort of thing is you're going to pull things out of file changes.

14:30 And that's not really what you want.

14:31 You really want a human moderated list of things that went in.

14:37 And that's one of the reasons why I liked TownCrier because it was sort of halfway in between.

14:42 Yep.

14:43 Yeah.

14:43 It's definitely really, really...

14:46 It's like a nice way to sort of manage that human.

14:48 But because you don't want...

14:50 Burged conflict.

14:51 Took PR.

14:52 Accepted this.

14:53 I changed the spelling.

14:54 Like, you know, you don't need all that noise.

14:56 You just want the four things that change.

14:58 Do I want to upgrade to this or not?

14:59 Whatever.

14:59 Let's just move on, right?

15:00 Yeah.

15:01 And then I guess I would lump this in last time we talked about different decisions based on scaling.

15:06 And for projects that I'm just...

15:09 I'm the main maintainer of, I would definitely just keep a file.

15:12 But if we start getting a lot of contributors, then something like TownCrier totally makes sense.

15:18 Yeah.

15:19 I think it's really nice.

15:21 I'm going to definitely look into it.

15:22 All right.

15:23 Last thing I want to talk about is asynchronous programming, which is something that I talk about often because I'm a big fan.

15:30 This is an article called Understanding Asynchronous Programming in Python by Doug Farrell from Dan Bader's site.

15:36 And we've had some of Doug's stuff on before.

15:39 He does good writing.

15:40 He works at Shutterfly doing Python there.

15:42 So he takes some of his experience and puts in this article.

15:45 And it's pretty cool.

15:46 What I would call or sort of describe this as, this is like a very friendly introduction to asynchronous programming.

15:52 So it starts out and says, let's imagine like a web server.

15:57 And could it be synchronous?

16:00 Sure.

16:00 It would be fine if we had a synchronous web server.

16:02 And we could optimize the heck out of it.

16:04 But no matter how much we optimize it, like at some point you're waiting on a thing and you want to go do other stuff.

16:11 For example, just like shipping the HTML back to the browser on a slow network, right?

16:16 Like you want to be processing other requests and do that in the background.

16:18 So he's got something to the effect of like eight or nine examples.

16:23 And to sort of start them off, he says, look, the real world is asynchronous.

16:29 For example, if you're a parent, kids are a long running task with high priority superseding any other task you might be doing,

16:36 like a checkbook balancing or laundry or something like this.

16:39 So he has a lot of like analogies back to real life that are pretty cool.

16:43 Then he says, okay, we're going to go through some examples, like eight examples and build them up.

16:46 Start with like a synchronous sort of job doing program that has a queue.

16:51 You put some work in the queue.

16:52 It does the work.

16:53 And then it says, all right, let's see how we can use generator methods with the yield keyword to instantiate like cooperative multi-threading

17:02 or cooperative concurrency, I guess, between those two methods, which is actually a really cool way to do it where there's no concurrent IO.

17:09 There's no threads.

17:10 There's no multi-processing.

17:11 It's just like let's interweave the work of these two methods or multiple methods using generators,

17:16 which I thought was really a cool way to look at it.

17:19 And it says, okay, well, what if some of that work is slow?

17:22 That's a problem.

17:23 And then he kind of takes you on a tour of different APIs and libraries to make this work.

17:27 So G event, twisted, twisted callbacks.

17:30 And so you can compare all these different ways of doing things.

17:33 And I should throw in there some AIO HTTP type things as well.

17:37 But yeah, very, very cool article if you want a super gentle introduction to asynchronous programming.

17:42 So this doesn't cover the AOA.

17:44 AI.

17:45 AI.

17:46 Yes, exactly.

17:47 Yeah, it doesn't cover basically the 3.5 stuff.

17:50 Okay.

17:50 Yeah.

17:51 So this would work on any version.

17:52 I really like this article because we've been talking about asynchronous for a while.

17:57 And I have to admit, I have my hard time getting my head around how to think about it.

18:02 I've been doing it for so long in C++, but I have a hard time getting my hand around it in Python.

18:08 And this article is really a good starter.

18:10 Yeah, I feel like it's definitely a good starter.

18:13 I was happy to have one of our picks this week.

18:15 All right, so that's all the news that we have that we've kind of found.

18:18 But you have extra credit, don't you?

18:20 Yeah.

18:20 Well, yeah.

18:21 In episode 29, I gave the wrong credit to the wrong person for cluing me into PipCash.

18:29 And I'm sure they appreciated it, though.

18:31 Yeah, but it really was KidPixo, and he reminded me that it was him.

18:35 And so sorry about that.

18:37 And thanks a lot for keeping us informed.

18:40 Yeah, definitely keep...

18:41 We really appreciate these ideas and these notes and these little topics people send us.

18:45 They're very nice.

18:46 And then I just had...

18:47 I couldn't resist.

18:48 This is going to be hard to do over a podcast, but we have a link to a funny comic about Python private methods.

18:57 And if you haven't seen this, check it out.

18:59 It's just...

19:00 It's basically a key under the mat in front of a door.

19:03 I love it.

19:07 I love it.

19:08 That's really awesome.

19:08 Yeah, that's kind of the thing.

19:10 It's like, it's private unless you want to look for it than it's right there.

19:13 Yeah.

19:13 Nice.

19:15 All right.

19:16 So update us on the book.

19:18 The book is coming along and taking almost all of my time.

19:21 The multitasking is a hard thing.

19:24 But yeah, the third beta is coming out, should be out this week with the last chapter, chapter seven.

19:31 And this one is using pytest with other tools like PDB and coverage and mock and talks and Jenkins and things that I get a lot of questions about.

19:41 So I'm really happy to get this chapter out.

19:44 Yeah, that's awesome.

19:45 How about you?

19:45 Yeah, last time we talked, I was recording and recording and recording Talk Python episodes.

19:50 So now I'm kind of finishing up recording courses.

19:52 So I've actually got two eight and nine hour courses that I've finished recording over the last couple of weeks.

19:59 So I've finished recording the RESTful and HTTP services and Pyramid.

20:02 And I've also finished recording and writing and recording the MongoDB for Python developers courses.

20:07 So I'm working on editing the final videos for those and getting those up.

20:11 So I'm really excited to get that out.

20:12 Really fun.

20:13 I'm really excited to take a look at that MongoDB course.

20:16 That sounds very interesting.

20:18 It's a cool hands-on one.

20:19 We build like this database that represents a dealership and it's got like millions of records in it.

20:24 We get it to where it will like do queries in like one millisecond, even with millions of records.

20:28 It's fun.

20:29 Nice.

20:29 Yeah.

20:30 Cool.

20:31 All right.

20:31 Well, that's our news for the week.

20:33 Brian, thank you so much for, as always, sharing it with everyone.

20:37 All right.

20:37 Thank you.

20:37 Yep.

20:38 See you all later.

20:39 Thank you for listening to Python Bytes.

20:42 Follow the show on Twitter via at Python Bytes.

20:44 That's Python Bytes as in B-Y-T-E-S.

20:47 And get the full show notes at pythonbytes.fm.

20:51 If you have a news item you want featured, just visit pythonbytes.fm and send it our way.

20:55 We're always on the lookout for sharing something cool.

20:58 On behalf of myself and Brian Okken, this is Michael Kennedy.

21:01 Thank you for listening and sharing this podcast with your friends and colleagues.