Transcript #72: New versioning: Episode 0.0.7.2 (with 72 releases)

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.

00:05 This is episode 72, recorded April 4th, 2018.

00:09 I'm Michael Kennedy.

00:10 And I'm Brian Okken.

00:11 And we've got some awesome stuff for you.

00:13 Before we get to it, Brian, I want to say thank you to Datadog.

00:15 Check out what they're offering over at talkpython.fm/datadog.

00:20 And they've got a bunch of cool stuff, including a cute little doggy t-shirt.

00:24 I still need to get one of those.

00:25 I know.

00:26 I don't have a shirt either.

00:26 I really need to do that.

00:27 All right.

00:29 I guess the first thing we should talk about is versioning.

00:32 When I normally look at commercial software, they have numbers in the front, like version 6 or 12 or version 3 of this software.

00:43 It's pretty rare to have 3.anything in open source, isn't it?

00:48 I don't know if it's rare, but there sure is a lot of stuff that's still...

00:52 We think of stuff that starts with a zero as being in beta.

00:55 And it's not...

00:57 Well, like, for instance, with semantic versioning, you can...

01:00 Once it's at 1.0, the interface is pretty solid and you can depend on it.

01:05 But there was a website put up this month called ZeroVer to talk about zero-based versioning.

01:13 And it's sort of tongue-in-cheek.

01:16 It's from one of our friends, Mahmoud Hashemi, and some others that started it.

01:22 But they kind of wanted to call out a bunch of Python projects and other projects that are perpetually starting with zero by putting up this sort of mock website to say that you don't need to do anything other than zero-based versioning.

01:37 Yeah.

01:38 It really helps you.

01:38 It gives it...

01:39 Like, it starts out with a down-to-earth demo.

01:41 It's pretty awesome.

01:42 So it has, like, some versions and says, yes, these are good.

01:46 Zero.0.1.

01:47 Zero.1.0.dev.

01:48 Zero.4.0.

01:49 And then, no.

01:50 1.0.

01:51 1.0.0.0.

01:53 1.0.18.0.

01:54 2018.04.0.

01:57 Like, none of these are okay.

01:58 Right.

01:59 And, yeah.

02:01 So if you haven't figured it out, it is a joke.

02:04 But it is, like, for instance, I guess I hadn't realized this.

02:08 Flask is one of the ones that was called out.

02:10 It's currently 0.12.2.

02:13 Come on.

02:14 It's been eight years.

02:16 I think that maybe you can go to a 1.0.

02:19 I have a new solution here.

02:21 A new way to solve this problem.

02:23 Just whenever you print or look at the version number, just strip off all the left zeros and dots.

02:30 So it's, like, 12.2 is basically what Flask is.

02:35 You know, there's some pretty ones that are very dependent on by a bunch of people that, like, if they completely change the interface, the API, that would be bad.

02:44 So they're clear to the point where they could bump to a 1.0.

02:48 Yeah, I think eight years, absolutely, is a timeframe where you could say, you know, we're pretty stable at this point, right?

02:56 Like, also Pandas in here with a 0.23.

02:58 And it's 7.1 years.

03:00 So, and also they count the releases, right?

03:03 Like, 21 releases of Flask, 75 releases of Pandas, and it's still on a zero.

03:07 One thing I would like to point out is if you go and you look at a lot of the sort of manager folks at more commercial-oriented enterprise software groups, so people that use, like, .NET or other non-open source development ecosystems, when they see things like 0.1, they're like, oh, this thing is not ready for us to use.

03:31 We can't use this at our company.

03:33 And I think it actually sends a bit of a message that this thing's not quite ready.

03:38 I mean, I know, obviously, looking at this list, it doesn't mean that.

03:41 But a significant number of people, I think, interpret it that way.

03:45 And so, you know, I think it's worth considering maybe saying, all right, we're actually at a version where we're going to call it 1.0.

03:51 Like, Flask is probably fully ready for 1.0.

03:54 Anything that starts with it, it's just the dots and it's not date-based.

03:58 People kind of assume it's just semantic versioning.

04:01 So, I think semantic versioning is the way to go.

04:03 It's not an easy thing, though.

04:05 And that's part of the reason why they're being a little gentle with it.

04:08 If you check out the About page, it talks about, on here, it talks about really what you should do about it.

04:14 But when you're actually running a project, it's hard to decide what's something that's big enough to go to flip the major digit.

04:22 Django is doing that pretty well, I think, right?

04:24 They had their 1.0, and that was stable for a long time.

04:28 They said, we're going to make a major breaking change, so we're flipping to 2.0, right?

04:32 The DropPython 2 support and all that kind of stuff.

04:34 So, Django is one that's not following Mahmoud's recommendation.

04:37 Recommendation, yeah.

04:39 Yeah, I love it.

04:41 I love it how he branded that, him and all the folks involved.

04:43 It's very cool.

04:45 So, when we build these projects in Python and any open source system, really, you basically layer on a whole bunch of external dependencies and packages and stuff.

04:55 How do you know when something has gone terribly wrong?

04:58 Like, suppose you depend on Vexig in Flask, and there's some huge security vulnerability in that dependency.

05:06 Do you get notified?

05:07 How do you know?

05:08 No, I don't know.

05:09 You don't know.

05:09 Right.

05:10 And so, this is actually a really big problem.

05:12 It's like, when you think about problems or security issues with your application, it's not just what you have.

05:18 It's the stuff you're built upon.

05:20 I mean, the whole Equifax thing was a vulnerability in the, was it Swing?

05:25 Swing?

05:26 I don't know.

05:26 Some foundational library in Java, and they just didn't pass it in time, right?

05:31 So, getting notified of these things is really important.

05:33 And so much of our code lives on GitHub, and GitHub decided they're going to take some responsibility for this and try to help people.

05:39 So, there's a nice article that says, GitHub security alerts detected over 4 million vulnerabilities last year, I think it was in the year?

05:48 Yeah.

05:49 Actually, it's not even the year.

05:50 It's since, like, November of last year or something.

05:52 So, that's pretty insane.

05:54 So, they launched this thing called GitHub security alerts.

05:56 Initially, it's only for Ruby and JavaScript, which is lame, but they have Python support coming, which is why I'm talking about this.

06:04 And what it does is it looks at your GitHub repo, and it says, are you using a certain dependency?

06:09 Is that dependency of a known security vulnerability?

06:12 If it does, then, like, right at the top of your repo, you get this great scary warning that says, your application isn't insecure because it depends on this thing that is insecure.

06:21 Yeah.

06:21 Actually, that's a great idea.

06:22 Yeah.

06:23 I don't know if you get an email notice, but certainly, your repo looks scary when that's the case.

06:28 Like, this happened to one of my courses, and it just came back again because one of my courses, the PyTron course, demonstrates using ElectronJS, editing an ElectronJS app.

06:38 And ElectronJS had some security vulnerability.

06:41 It's not actually used, but, you know, whatever.

06:43 It still says, your app, it depends upon ElectronJS, and it's got this issue.

06:47 So, it's pretty cool.

06:48 And then, there's some good numbers and whatnot here.

06:51 It says, nearly half of all the displayed alerts were responded to within a week, and 30% were fixed within the first seven days.

07:01 Oh, that's great.

07:02 That's good.

07:03 That's a good thing that they're adding that.

07:05 And it does.

07:05 So, you said it's coming for Python, and I see that it is planned for this year for 2018.

07:10 So, that's good.

07:12 Yeah.

07:12 There's not a whole lot of details about exactly when it's coming, but, yeah, that will be great.

07:16 They said if you look at repositories that have had a contribution in the last 90 days, so things that are active, it says 98% of such repositories were patched within fewer than seven days.

07:28 Like, that's insane.

07:28 That's a really big deal.

07:30 Yeah.

07:30 Yeah.

07:30 So, they said they found over half a million repositories that had some kind of security vulnerability and were pretty much fixed up.

07:37 So, anyway, that's all really good.

07:39 I just want to give a shout out to PyUp as well, P-Y-U-P dot I-O.

07:43 I use that for my stuff, and it basically does the same thing and more for Python already.

07:49 So, you link it to your GitHub repo.

07:51 It'll, like, look at your requirements dot TXT.

07:53 If there's a new version, it'll send you a PR to upgrade your dependencies.

07:57 And if there's a security alert, it'll tell you.

07:59 Don't really want to get on this tangent too far, but I started using PyUp for the cards project that I started recently.

08:06 Since I'm sort of doing this project, I can't remember who I read it from,

08:11 but the packages that are intended to be used by other applications probably shouldn't have their versions pegged.

08:18 So, if I unpegged all my versions in a package, then PyUp dot I-O kind of complains about that.

08:25 Yeah, that's a little bit.

08:25 It does require you more or less pin your versions.

08:28 And you can do expressions.

08:29 Like, I want it to be this version or higher, and I think maybe it'll upgrade it.

08:34 I don't know.

08:34 There's a little flexibility.

08:35 It's not perfect.

08:36 But for, like, fixed apps, like my web apps, I'll have the stuff pinned, and it just automatically updates.

08:41 Because nothing depends upon it.

08:43 It's fine.

08:43 Yeah.

08:43 Yeah, pretty nice.

08:44 And it's free for stuff, so for open source.

08:47 Yeah, it's pretty nice.

08:48 Great.

08:48 Speaking of open source, PyPI is the place where it lives, and now you can describe it better, right?

08:53 Yes.

08:54 I'm very excited about this, because, like, the cards project I was working on, I was sort of bummed that I had to put the readme in rest.

09:01 Or not rest, but restructured text.

09:04 And now you don't anymore.

09:06 That's awesome.

09:07 So readme.md and a couple other variants of that extension are now supported on pypi.org.

09:14 And we're linking to a couple articles, one of them basically describing all the steps you have to do.

09:19 There's a little bit of changes you have to do to your setup.py file and a couple other things and update all your tools.

09:27 But for the most part, it just works, and that's awesome.

09:31 And then also, just recently, GitHub-flavored markdown has been added.

09:37 Oh, yeah.

09:37 That's nice.

09:38 GitHub-flavored markdown has a little bit more, I think, from the stuff that I played with.

09:42 Like tables and cross-

09:44 Yes, tables.

09:44 Markthrough and stuff.

09:46 So that's nice.

09:48 And I'm looking forward to changing a couple projects to utilize that.

09:52 And now the old legacy PyPI, which I think maybe they've taken from your legacy Python.

09:59 I love it.

09:59 Yeah.

10:01 It still renders the descriptions as plain text, but they comment, don't worry, it's going

10:06 away real soon.

10:07 Yeah.

10:07 I was just talking to Ernest, who's doing some of the work there.

10:11 And they're really, really close to just saying PyPI.org is where you go and doing a redirect.

10:17 He said, you know that little scary bar across the top?

10:20 He said, you can now dismiss that.

10:21 But I want it to just go away automatically.

10:24 When does this happen?

10:25 PyPI.org is really close to being the thing.

10:28 So maybe this will just hasten the move away from legacy PyPI with the descriptions looking

10:34 funky.

10:35 Yeah.

10:35 So hopefully.

10:36 Yeah.

10:36 Awesome.

10:37 I'm really excited to see PyPI making some progress.

10:39 It felt kind of stale for a little while.

10:40 And it seems like it's really been rocking the last nine months.

10:43 To be fair, even if your markdown gets displayed as plain text on legacy PyPI, that's the point

10:50 of markdown is it's still readable.

10:52 So that's okay.

10:53 Exactly.

10:53 If it were HTML with lots of styles, it would have been different.

10:56 That's right.

10:56 Yeah.

10:56 Nice.

10:57 All right.

10:57 So before we get to the next one, let me tell you about Datadog.

10:59 It's a monitoring solution.

11:01 Provides deep visibility and tracking into your distributed apps.

11:05 So your application, your data layer, your servers, your services, everything.

11:09 So within minutes, you'll be able to investigate bottlenecks and actually see where they are throughout

11:14 your entire distributed app, which is pretty cool to put it together.

11:19 So if you want to visualize your Python performance today, get started with a free trial and also

11:24 get that cool Datadog t-shirt, visit pythonbytes.fm/Datadog.

11:28 Earlier I said talk Python.

11:30 They both work, but pythonbytes.fm/Datadog.

11:32 Speaking of web apps and distributed things and whatnot, I think there's a really interesting

11:38 new web server that people should start paying attention to in the Python space.

11:43 So you probably heard of Nginx, right, Brian?

11:46 I know you don't do a ton of web stuff, but yeah.

11:48 Yeah, definitely.

11:49 Nginx is kind of like the static front-end server and load balancer thing for many web

11:54 apps.

11:54 Like on my sites, I have Nginx hitting, like it takes all the requests, does the SSL stuff,

12:01 any static resources, CSS, JavaScript, images, that just gets sent straight back.

12:05 And only the sort of data-driven stuff makes its way back to the Python web server, which

12:11 in my case is MicroWSGI.

12:13 And MicroWSGI is really nice.

12:14 But the Nginx folks have come up with this thing called NginxUnit.

12:20 And so the thing I want to link to is this performance comparison between NginxUnit and MicroWSGI.

12:27 So MicroWSGI is written in C++.

12:29 It's like one of the best high-performance things that will run and farm out your Python application,

12:35 Pyramid Flask, whatever.

12:37 And it works really well.

12:38 But NginxUnit is a little more flexible.

12:42 And for example, you can configure it over a RESTful API instead of just config files.

12:48 It'll run multiple languages and versions at the same time, improve TLS support, HTTP2, which is cool.

12:56 It'll run Python, multiple versions.

13:00 It'll run Go, Ruby, JavaScript, whatever, right?

13:03 So it'll run all these things in this one server.

13:05 It's not just I'm going to run one flavor of Python.

13:08 So anyway, it's pretty cool.

13:10 And the thing I wanted to look at was this comparison.

13:13 So there's this, I don't know who did it actually, a group that put together sort of a performance analysis

13:19 and said we're going to slowly add more and more traffic, concurrent traffic,

13:24 to both of these things running more or less a Hello World Flask app, right?

13:29 And so pull up the pictures.

13:31 And those of you who are listening, there's a little link.

13:32 You can pull up the pictures.

13:33 And this really tells it all.

13:34 Okay.

13:35 You got the pictures, Brian?

13:36 Yeah.

13:36 So if you look at that, there's like a line that's pretty much flat across its NginxUnit

13:41 as you go from zero to 500 concurrent users doing 10,000 requests per second.

13:46 And it's just kind of like, got it, no problem.

13:49 Microwisgee or with or without threads is sort of a linear slope equals one downward trend of performance

13:57 as you add more and more traffic.

13:58 Like, soon as you get to, you know, a couple hundred users, it just really becomes,

14:04 it goes from handling like 7,500 requests to handling 50 per second.

14:10 I mean, it really falls over.

14:11 So I thought that was pretty interesting.

14:13 This whole NginxUnit thing seems like it might be a really powerful and new way to run some nice backing stuff.

14:20 Okay.

14:20 So the high numbers are better.

14:21 You want to keep...

14:22 Yeah.

14:22 Those are requests per second.

14:23 Okay.

14:24 Basically.

14:24 Yeah.

14:25 So it just, once you do 100,000 requests, it goes to zero on Microwisgee where it's still

14:30 basically flat on NginxUnit.

14:31 So really, really cool.

14:33 I think that's quite promising in terms of making Python faster and scale better, which is, you

14:40 know, super important because people move to other languages, Go or whatever.

14:44 Because like, well, we need this concurrency, like, or you could just run something that

14:47 runs it better.

14:48 Well, so they have a little note that says it's still in beta, not for production.

14:51 Yeah.

14:52 It's pretty new.

14:53 It's not quite ready.

14:54 So my message, my takeaway is I'm going to start paying attention to this thing.

14:58 Maybe switch to it at some point, but yeah, don't switch to it yet.

15:01 I wonder what version number it is.

15:03 It doesn't say.

15:03 It's got to be zero something, right?

15:07 Yeah, I don't know either what version number it is.

15:09 That's a good question.

15:10 Okay.

15:11 Cool.

15:11 Very, very funny.

15:12 All right.

15:13 Awesome.

15:13 You've got something on looping, right?

15:15 Trey Hunter, who was on the show last week.

15:18 Didn't he do last week?

15:19 He was your fill and your stand in, your impersonator last week.

15:22 Well, he's got an article, which is a really good read and I'm going to not do it justice,

15:27 but it's called Loop Better, A Deeper Look at Iteration in Python.

15:32 And, you know, I'm glancing through this.

15:35 I'm thinking, you know, I already know how to loop in Python.

15:37 But the general, he shows a few gotcha examples of generators used in loops.

15:44 And generators are like, for instance, even a list comprehension is a generator.

15:49 You can't loop twice.

15:50 And if you use containment check, like is nine in my generator, it'll work once and then it

15:59 won't work the next time.

16:00 But it's not in there anymore.

16:01 And your collection's half the size.

16:03 And it's a little strange.

16:05 And it just behaves weird.

16:07 I mean, I don't know if I've ever run into these, but I didn't.

16:12 It hurt my head at first trying to figure out.

16:14 I didn't know why these weren't working.

16:16 So then the article goes on to describe in detail, really, the iterator protocol and what iterators,

16:22 iterables, sequences and generators and all that good stuff is.

16:26 And then go back and look at those gotchas again and explain with that information why they behave as they do.

16:33 And I think this is just a well-written article that'll be going to make you a smarter Python programmer to read it.

16:40 Yeah, it's cool.

16:40 Definitely covers a lot.

16:41 Well done, Trey.

16:42 I think this is one of those concepts where if you come from a language that doesn't have generators, this concept of generators, or maybe if you just never really use them, the stuff that comes out of these generators, it looks like you just treat it like a normal collection.

16:59 But you're right, they definitely don't behave like normal collections in a lot of ways.

17:02 And you can find these subtle bugs.

17:04 So nice to have them all covered like that.

17:06 Yeah.

17:06 And one of the things, I guess I'll go a little bit, is that generators, it's this iterator protocol and you keep it internally in a loop.

17:15 Python will call the next operator and then eventually it gets to the end.

17:20 There's not a way to reset them.

17:22 So they...

17:23 Yeah, they're done.

17:23 They're done.

17:24 And you got to generate...

17:25 But you can generate...

17:26 However you generate them, you can generate another one.

17:29 Yeah, pretty cool.

17:29 So the final thing that I want to cover is a little bit like the first one.

17:34 It's a bit of a warning, but this is not an automated system like GitHub saying, hey, there's all these repos.

17:40 We're going to tell you there's this problem.

17:41 It's just something people should be aware of.

17:44 So in Django, there's these configuration files and there's this part where you can set debug true or false.

17:49 And there's like a little comment by it that says, do not set this to be true in production.

17:53 However, do you think everyone goes into it, the big long config file and fixes that before they push it out?

18:00 No.

18:00 No.

18:00 No, they don't.

18:01 So the article is called, misconfigured Django apps are exposing secret API keys and database passwords.

18:08 That sounds bad.

18:10 No.

18:10 No.

18:11 No.

18:11 Yeah.

18:11 So it says, researchers have begun stumbling upon misconfigured Django apps that are exposing information like these API keys.

18:19 Could be your stripe key, whatever.

18:20 In just like a week, they discovered 28,000 Django apps where the admin left the debug mode enabled.

18:28 And then, you know, you see, it'll be like screenshots of pulling up just random apps on the internet.

18:33 Here's the AWS secret key.

18:35 Here's the database password, et cetera, et cetera, just listed in the debug tools.

18:39 So that sounds bad, right?

18:42 Yeah.

18:42 Well, especially you're probably, you probably leave that on while you're developing it so that you can look at all that stuff.

18:48 Yeah.

18:48 It sounds really bad.

18:49 And it pretty much is.

18:50 It says, just skimming through a few servers, researchers found debug mode were exposing extremely sensitive information that would allow a malicious actor full access to the app owner's data.

19:01 But they were really, I like that they were really clear to emphasize this is not a failure on the Django side.

19:07 But in fact, you're just not supposed to do this in production.

19:11 And somebody on Twitter was like, it would be so awesome if there was like a comment or like a little note in Django that said, don't put this in production.

19:18 And then, of course, right under there's a screenshot of never run this in production in debug mode.

19:22 It's not supposedly not Django's fault.

19:25 However, I mean, maybe there needs to be more than just on or off.

19:29 Maybe there needs to be a I'm debugging my app, but I don't want to expose all the API keys mode or something.

19:35 Yeah, for sure.

19:37 I think or maybe just the debug stuff is off by default and you have to turn it on.

19:42 And the act of turning it on, you go to the section, you read that.

19:45 But you might never go and read that part of the config file.

19:47 So you just don't know.

19:48 Right.

19:48 I mean, Django is famous for like getting easy, like just getting stuff up really easy.

19:52 I don't have to be a super developer.

19:54 So maybe you just don't know.

19:55 Right.

19:55 Sort of make things worse.

19:57 Security researcher Victor Givas said some of these apps running Django have already been compromised.

20:04 And he found one server running the Weevily web shell.

20:08 That's bad.

20:09 I mean, they were somehow able to entirely take over the computer and just SSH into it.

20:14 And so he said, I've been notifying server owners about their leaky Django apps.

20:19 At the moment, we've reported 1,822 servers.

20:22 Well, 143 were fixed.

20:24 Not so many, right?

20:27 Yeah.

20:27 Or taken offline.

20:28 Which the taken offline tells me that there's some people out there that just don't know how to do that yet.

20:34 So they'll just take it down.

20:36 Yeah, there's like, you know what?

20:37 My little toy site is not worth getting hacked.

20:39 I'm just taking it off.

20:39 Right.

20:40 Yeah.

20:41 Right.

20:41 Well, so I guess takeaway, if you're running Django site, make sure it's not in debug mode.

20:46 Or you could be a statistic.

20:48 Don't be a statistic.

20:49 Yes.

20:49 Don't be a statistic.

20:51 All right.

20:52 That's it for our official six items.

20:55 Brian, you got anything else?

20:56 Yeah.

20:57 I just got ping today to tell me that in episode 70, we covered Wagtail, which is a CMS written in Python.

21:05 But the Wagtail team is trying to get some new features out.

21:10 And they're running a Kickstarter campaign to try to fund that.

21:14 So I think it's a good thing.

21:16 They're not looking for that much money.

21:18 So if everybody pitches in a little bit, it'd be good.

21:20 So we've got a link.

21:21 Yeah, they're pretty close to their goal, right?

21:24 They've got 10 days left.

21:25 They're about halfway there.

21:27 They should get there, hopefully.

21:28 Yeah.

21:28 Wagtail is one of the really nice CMSs that's based on Django.

21:31 Hopefully, if it's a bug mode, equals false.

21:33 Yeah.

21:34 Pretty nice stuff.

21:34 So yeah, if you care at all about Wagtail or the CMSs, go in there and help them out a bit.

21:40 I wanted to mention I've had a lot of great feedback on testing code.

21:44 I've been doing kind of a series of getting an open source project out.

21:50 And all of the sort of the testing requirements around it and talking about some of the common

21:54 test design patterns.

21:56 And that's been going well.

21:58 And I've actually been learning a lot about running an open source.

22:02 I thought, you know, lately I've just been using GitHub for just like a revision control.

22:07 But actually running an open source project, even if it's just got a couple of contributors,

22:12 you learn a lot.

22:12 So hopefully I'll get some of those learnings written up sometime soon.

22:16 You definitely should.

22:17 That's a really cool project you're doing.

22:18 So keep it up.

22:19 Yeah.

22:20 You got any news?

22:20 No news right now.

22:21 Nothing to report, but I'm always working on new projects.

22:25 I will let you know soon.

22:26 All right.

22:26 Well, thanks a lot for today.

22:28 Yeah, you bet.

22:29 Thank you for listening to Python Bytes.

22:31 Follow the show on Twitter via at Python Bytes.

22:34 That's Python Bytes as in B-Y-T-E-S.

22:37 And get the full show notes at pythonbytes.fm.

22:40 If you have a news item you want featured, just visit pythonbytes.fm and send it our way.

22:44 We're always on the lookout for sharing something cool.

22:47 On behalf of myself and Brian Okken, this is Michael Kennedy.

22:50 Thank you for listening and sharing this podcast with your friends and colleagues.