Transcript #79: 15 Tips to Enhance your Github Flow

00:00 Hello and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds.

00:05 This is episode 79 recorded May 23rd, 2018. I'm Michael Kennedy.

00:09 And I'm Brian Okken.

00:10 Hey Brian, how you doing?

00:10 I'm doing great.

00:11 Nice. I think as always, we've got a bunch of fun stuff to talk about.

00:15 And we wouldn't be doing it without DigitalOcean.

00:18 A couple reasons, but DigitalOcean is sponsoring this episode and a bunch of the upcoming ones.

00:21 So thank you to DigitalOcean.

00:24 get $100 off or $100 credit at Python bytes, FM slash digital ocean for new customers. Tell them tell you more about that later. I would be totally surprised, Ryan, if you wanted to cover something about say, testing or pytest.

00:38 Yeah, yeah. So pytest 3.6.0 just got announced. So 360 for pytest. And this is, this is like a inside baseball kind of release because there's not a lot that if I think 80% of the people using pytest won't see a difference. But however, this was a big deal for the team. Essentially it's a revamp of the implementation of the marker system and the data type that was used to hold the markers. So there is a couple other things that's the big thing that's going on in the 3.6.0 release is a reworking of the markers in it and there's a list on their release notes that of all the different defect that they fixed with this. The takeaway for a lot of people is if you were using, if you're writing a plugin or something or using the plugin features and using get marker to find out which markers are applied to a particular function, the get marker is deprecated. There's a new API, there's inner markers and get closest marker and yeah, I'll link in the show notes to read more on that.

01:45 So most of it's a plugin writers change the API change, but it's it's exciting. And I'm excited for the team to get that out because kind of like the Django to release, it's about maintenance and going forward. And so that's great. There is one more feature that we from our couple other things on the breakpoint, the pytest is supporting the breakpoint functionality in 3.7.

02:10 And that is brought to you by our friend Anthony Shaw. So he put that in.

02:13 Oh, nice. Yeah, he's doing a lot of work on Python 3.7 because rumor is he may be doing a course on Python 3.7.

02:19 So awesome. He was able to bring it over here.

02:22 And a couple of other smaller things like the, apparently, I had never run into this, but if you have an assertion failure on equality and the only thing different is whitespace, it's kind of hard to tell. So they now escape characters too.

02:38 So you can see what the whitespace difference is a little bit better, which is kind of cool. I've never heard of that.

02:44 That's a little hard to print it on. You're like, "Oh, so they're the same." No, no. That's two spaces, not one right there.

02:49 Yeah, so for the main port, I wanted to get this out to as many people as possible.

02:54 So if you are depending on reading markers in your internal code, pay attention to this.

03:00 That's it.

03:01 Yeah, it sounds like a nice cleanup of the internal APIs for extension writers.

03:04 And that's always good because that probably means more extensions.

03:07 Extensions are more likely to be built.

03:09 Yep.

03:10 Sounds good.

03:10 So have we talked about GUIs, Python GUIs on this podcast?

03:14 I'm not sure if we have. We probably should.

03:16 We probably should. Yeah, let's do that.

03:18 So there's a lot of stuff going on.

03:22 Part of the reason I went on that rant is because stuff needs to be happening there, but also because some things are happening.

03:28 Like we had the WX Python Phoenix release, which is kind of a rebirth of WX Python, which is really great.

03:34 Well, we also have the same thing going on for Qt.

03:37 So for a while, there was a sort of a split, there was pi qt, there was pi side, there was pi side to there all these ways they defended on different versions of qt. And it was kind of just generally a mess. So the qt company is now officially making something called qt for Python, which as far as I can tell is more or less a, like a rebirth, pi side to for what that's worth. So it's really nice that the company that makes Qt, the cross-platform GUI framework, is really dedicating itself to Python.

04:12 One of the things that I think is cool about the Qt space is they have the Qt Designer, and I think that's really nice and important for a heavy visual way to design the UI.

04:23 I know you can write code and say the position is 20,20 and it stretches this wide, but that is not the same as draggy-droppy, press the button, you know what I mean?

04:33 So I got a lot of, I'm pretty excited about this, let's say.

04:37 So that's really cool.

04:38 They basically are keeping it super similar to the Qt C++ API, where that makes sense.

04:44 So like if you read documentation about C++, which is the native language for Qt, if you replace the pointer dereference, so the arrow, the dash arrow, to replace that with a dot, that may well be the Python API.

05:00 - Okay. - Which is good.

05:01 But some of the drawbacks, let's say, that are it doesn't necessarily leverage the Pythonic features.

05:06 So maybe you call a function to do a thing rather than put a decorator onto something else, things like that.

05:13 One thing that is nice is a lot of these UI frameworks are super painful to install, right?

05:18 You can install them on the system and then they don't work so well.

05:22 Maybe there's some big long compilation step like WXPython takes forever to pip install it onto Ubuntu, the last time I tried doing that.

05:32 So they're planning on shipping a wheels version of Qt, which before you had to get like some separate installer or something.

05:38 So that'll be pretty sweet, that you'll just be able to pip install your thing and it'll come with the foundational stuff you need.

05:44 - That's exciting.

05:45 - Yeah, that is pretty cool, right?

05:46 So, I mean, I really hope that the company behind Qt putting a big effort into this is gonna mean like finally a polished version.

05:54 So we'll see.

05:57 I think the licensing might still be GPL and LGPL.

06:01 So as a combination, take your pick.

06:04 I'm not sure what the variations are exactly there, but I don't know.

06:08 I'd like to see something more permissive, but who knows?

06:12 Still nice to see some progress here.

06:13 So do you know, I was trying to find it, do you know the projected release date for the official Qt for Python?

06:20 Or is that...

06:21 They're talking about...

06:22 So the article I'm linking to is a blog post calling hello Qt for Python and they say they're working on a technology preview.

06:31 So that's all I've seen.

06:33 But they don't seem to have any further information that I easily found that maybe somewhere else.

06:38 Yes, it'll be does say it'll be available under GPL, LG PL and commercial licenses.

06:46 It talks about when development started and stuff like that, but it doesn't seem to have like a release date.

06:51 So there it is.

06:52 All right.

06:53 Cool.

06:54 Nice.

06:55 Well, speaking of sneak peeks on things, we've got, we've talked about MongoDB, the 4.0 release that's coming.

07:02 We've talked about that before, but now you can play with it.

07:05 So the 4.0, 4.0.0 RCS 0 is now available.

07:11 It's the very 0th version of the RCS.

07:13 Yes, there's a lot of zeros there.

07:15 Yeah, so that is out and ready for testing so people can actually get their hands on and try working the again the big news for this.

07:24 There's a lot of new features, but the big news is ACID transactions and multi-document.

07:29 - Multi-document ACID transactions.

07:32 Yes, that's a pretty big deal.

07:33 - And I actually don't know if this is a big deal.

07:35 Well, there's a lot of things here, but non-blocking secondary reads, I don't even know if I know what that means.

07:41 - So the idea with the non-blocking secondary reads is one of the ways you can set up MongoDB is in a, what's called a replica set.

07:49 So there's like a primary thing that you read and write to.

07:53 And then there are other ones which are constantly just staying in sync with that server.

07:59 There's a couple of benefits to that.

08:01 Like you could put them into say different data centers.

08:04 The primary thing is if like for some reason the main server, the primary server fails, it'll automatically switch to one of the secondary ones.

08:12 So it's kind of like a failover redundancy sort of thing as well.

08:17 but you can configure it in a way that you can say, I would like to read from the non-primary database as a way of like adding read scalability.

08:28 So like if I have five servers in the cluster, if I don't do anything, I can only talk to the primary one as a single server, and I get no boost of concurrency, let's say.

08:39 But if you say, I wanna read from the others, well, then all of a sudden, there's like, you're sort of farming that out across six different servers, primary plus other five or whatever.

08:48 Right, that used to block for consistency reasons and now apparently they found a different way to ensure consistency, maybe because of the transactions.

08:57 - Okay.

08:58 - Anyway, that's a long explanation for what I think that means.

09:01 - That does make sense and that's cool.

09:04 At least I knew, I know that there are a lot of people that choose a SQL database over a document database mostly because of the lack of transactions and so that's one of the reasons why I brought this up 'cause I'm excited about transactions.

09:17 - Yeah, I think that's super exciting as well for the reason you just said.

09:21 What I do think is interesting is as people get to more serious applications, they get to a place where often they give up transactions anyway for sort of concurrency.

09:33 Right, like, you know, if I go to Amazon, it's not like, and I go to order something, it's not just going to lock all of Amazon while I interact with, you know, my order.

09:45 what it's gonna do is like say, we'll place the order and if it happens to be that actually the thing you ordered sort of sold out just at the moment that you pressed it, you'll get like a message or something, right?

09:55 Like, hey, sorry, we couldn't fulfill it or whatever, here's your refund.

09:58 So there's a lot of these sort of compensation things that get put into like high scalable stuff.

10:03 I just grabbed Amazon as an example, I don't really know how they work.

10:07 But you know, there's a lot of these large sites that do sort of don't use full on transactions in the same sense that other ones do.

10:15 So it's pretty interesting.

10:16 It's interesting in that I don't really think transactions are something I'm gonna be using in any of my sites.

10:23 They just don't really seem to be necessary with a few possible exceptions.

10:28 I'll get to what those might be in a little bit.

10:30 But yeah, I think you're right that people, when people feel they need it, or there are a few situations where you really do need it, this is super interesting.

10:37 One other thing that's kind of cool, that's not a 4.0 thing, but it's in a 3.6, which is the one right before this, as far as I know, is actually the streaming API.

10:47 So if I've got like say web sockets or something that I want notification of like push of change to the database, you could like run a query and say, I want to stream new results that hit this query.

10:59 And then as stuff is inserted to the database that matches, it'll get pushed out to you instead of repolling the database.

11:07 So suppose I connect to a chat server and I set up web sockets, you could like literally subscribe to like these change streams on like the conversation record, and you would just get them pushed back down instantly without any polling end to end.

11:20 - Okay. - That's pretty cool.

11:21 It's kind of like RethinkDB's feature, primary feature was.

11:24 - I guess where I would probably use transactions a lot, and it's not really transactions, but because of transactions you can do this, is I believe 4.0 also includes rollback checkpoints.

11:38 For instance, you can grab a replica of a big database or something.

11:44 And like for instance, for like during testing, you can have a starting point, do a whole bunch of transactions on it, query it, and then roll back to a previous state.

11:54 - Yeah, that is pretty cool.

11:55 And I think maybe that secondary non-blocking read stuff has to do with that as well.

11:59 You sort of begin a transaction and you start reading.

12:02 - Yeah, anyway.

12:02 - Yeah, yeah, very cool.

12:03 So I'm glad to see that that's coming along.

12:05 I feel like the NoSQL document database world and the relational world are kind of like merging, like they're getting closer to each other in a lot of ways, right?

12:16 We have Postgres getting JSON stuff, we get MongoDB getting transactions, and they're all kind of sort of growing and intersecting in interesting ways.

12:25 Speaking of interesting, DigitalOcean's pretty interesting.

12:28 They're doing a lot of good stuff for us.

12:29 So like the files that you're getting when you download the podcast, the website, all that stuff is running on DigitalOcean servers, and I'm super, super happy customer of theirs, and they're sponsoring the show as well.

12:42 So one of the things that's cool, maybe I mentioned this a while ago, Brian, is their sort of one-click app server configuration.

12:49 So if I want to create, say, a server with MongoDB all configured, I can go there, say, create me a droplet with this version of Mongo or with this other web framework set up, and it'll automatically create all the server configuration and have everything set up and ready to go within like 60 seconds.

13:08 So really, really nice.

13:09 And probably the biggest thing, if you are not using DigitalOcean, you can get a $100 credit by going to pythonbytes.fm/digitalocean.

13:17 So that's a pretty good deal.

13:18 - Yeah, that's great.

13:19 - Yeah, awesome.

13:20 So if you're looking for a nice, affordable, fair, and very fast server hosting, check them out, pythonbytes.fm/digitalocean.

13:28 - So Michael, have we talked about PipMF in the show before?

13:32 - If I recall correctly, I think we were confused about PipMF.

13:35 I was confused about PipMF.

13:37 Basically, you know, I, or surprised, maybe it's the right word, that PIPMF became sort of the officially recommended way of the packaging authority in Python to manage packages.

13:51 And I'm like, oh, when did that happen?

13:52 That was pretty interesting.

13:53 So there's been a lot of debate, and you said there was kind of a coarse Reddit thread, like imagine Reddit was unkind to people.

14:00 Could you imagine?

14:01 - Yeah, right, yeah.

14:02 - That's unfortunate, but I think it's too bad that kind of stuff happens.

14:07 And maybe we should all just speak up like, hey, like that comment is out of bounds, right?

14:11 Anyway, I'm not gonna link to it.

14:13 I don't wanna encourage it.

14:14 But I do wanna link to this thing called Pip-Inf Review after using it in production.

14:19 So there's this team that used Pip-Inf in production since November, 2017.

14:24 So what is that?

14:25 A little over half a year, maybe almost exactly half a year.

14:28 And this sort of comes, they talk about this is what worked for us.

14:32 This is what wasn't working so well for us.

14:34 And in the end, they're like, at no point did anyone in the team ever mention getting read a pipenv, which actually is a pretty strong statement, apparently.

14:44 So, so like, if no, he said, No, we got to get rid of this.

14:46 It's just like, it's not quite working in some way.

14:49 So here, I'll give you the rundown.

14:51 The article starts off pretty accurately says the current state of Python's packaging is awful.

14:56 I don't think anyone would disagree with that the problem is recognized.

14:59 And there are many attempts to solve the mess.

15:01 And PipMF was the first and it did get a lot of traction, but not everyone loved it.

15:06 And he said, one of the areas where PipMF can be a challenge is for libraries.

15:13 So PipMF is around, is more built for managing the dependency of an application.

15:20 But if you're a library author, that it doesn't necessarily make a lot of sense.

15:24 Yeah, I'm on the fence on that.

15:25 Sure.

15:26 Sorry, I forgot the guy's name.

15:27 that he said this was basically supporting multiple environments goes against pip m's philosophy. Right? So they want a deterministic reproducible application environment. But you know, if you're going to do that for say, pi pi, and Python two, seven, and Python three, six, or whatever, well, then it doesn't really work, potentially, right? Because it's, you know, once exact hashes of the exact libraries and if those don't match then you're out of luck, right?

16:02 So that's a challenge.

16:03 I think that's the primary challenge.

16:04 Yeah, yeah.

16:05 And I agree with that.

16:06 And it's just partly I think it's a miscommunication.

16:09 PipM was never intended to work for every library sort of use because libraries by definition or they don't have their dependencies pinned.

16:20 It's at the application level where you pin your dependencies.

16:24 So you say there's this miscommunication and I definitely think you're right because when I looked at pipenv on GitHub, I really saw that as, you know, the statement, pipenv is the officially recommended tool for managing application dependencies from PyPA as pipenv is the officially recommended tool for managing Python dependencies, where really like the application should have been bolded, underlined in all caps, something to that effect, right?

16:53 So pretty interesting. But yeah, I think, you know, it generally their review of it was was good. So I'll try to give you the quick rundown here. So pip file and pip file lock really are superiorities or requirements.txt by a ton. And the guy said, Hey, I first disliked having flake aid and a security checking tool all built into one thing, but I think it's actually great. Installing from private repositories that works really well. Creating a new pip file is easy.

17:21 new problems, introducing pip and to new users or installing from a mixture of indexes and get repos that was all really good.

17:31 Virtual MF or virtual EMV is much easier to get into and understand.

17:36 Now, see dependencies can install be easily installed into a system like Docker.

17:41 And finally, like I said, no one proposed getting rid of it.

17:45 They were just a few edge cases mostly around the library side of things.

17:49 So yeah, pretty good.

17:51 But if you're thinking about using pipenv in production, check this article out.

17:55 It's kind of got some good discussion and a lot of follow up as well.

17:59 I want to add that I was, for library development, I am going to start, I haven't been using it, but I'm going to start using it not from the standpoint of handling all of the dependencies for the library dependencies, but more because the setup.py does that.

18:19 the transitive dependencies and also mostly the developer dependencies. So Pipenv has a developer feature where you can either create the environment for running or create the environment for development and those can be different. And traditionally we've had a requirements_dev or something like that, but it's just you kind of have to know it's there. So for that reason I'm going to try Pipenv. The other reason is the --run flag to be able to run in the environment without activating the environment is going to be useful for things like Jenkins runs and things like that. I'm going to give it a shot. I don't have a report yet, but I'm going to start using this as well. Yeah, sounds good. You're going to have to give us a report after a while.

18:59 Yeah, definitely. Nice. All right. So you've got some stuff for GitHub flow, the whole sort of working in GitHub, PRs, submitting issues, open source goodness. Yeah.

19:11 I've had a development team that's migrating to both a lot of changes in our development workflow, but one of the things is using Git more and we're using GitLab at work, but this is so a lot of these some of these I use GitHub for open source projects of course, but here's an article called 15 tips to enhance your GitHub workflow and or GitHub flow and they a lot of these apply to both Git and GitHub and GitLab. Some of them are GitHub only, but there's some things that you just sort of need to know about the culture around Git and GitHub and GitLab and everything that you don't actually, it isn't obvious from the start.

19:49 So I like having an article that calls out a lot of these things.

19:52 Like one of the talks about, I'm not using projects yet, but I'd like to try to use projects to prioritize issues and maybe track progress and plan.

20:02 Plan for what's going in which release and stuff.

20:06 Maybe if that's built in, might as well try it. Using tags on issues.

20:10 I've started using that.

20:11 I know we have tags on a lot of open source projects like, what is it, Help Wanted and things like that.

20:19 There's some standard ones. Getting to know those are good.

20:22 Templates are something that really, so a lot of this stuff isn't stuff I know about yet.

20:26 It's stuff I want to start using. Templates are something, like if somebody does a pull request against your project, having some predefined stuff filled in for them to know what to fill in. And the default template is sometimes sometimes kind of lame for certain projects.

20:45 Like I've got a library that it, the default one asks for like operating system.

20:51 Well, I don't really care.

20:52 It's not going to affect the library I'm using.

20:55 If the issue is really hard to reproduce, I'll ask somebody and say, "Hey, this is, I'm trying to reproduce it here and I can't reproduce." Anyway, there's a whole bunch of great things like squat down.

21:05 One of the things I didn't know about at first was squashing pull requests and squashing commits.

21:10 That's something that is totally foreign if you're coming to get from other revision control systems.

21:17 So, there's just a good list of a whole bunch of goodies.

21:20 - Yeah, that's really cool.

21:21 And I like the automated tests and checks on pull requests.

21:26 Like, that's really nice.

21:27 Like, if I do a PR to someone else's repo and my PR automatically gets tested, like flaked or whatever, they're wanting to have checked, right?

21:37 That can tell me right away before they get back to me, "Oh, there's a step I missed, let me fix that." And then resubmit the PR, or just update the PR, and then have it rerunning, okay, now everything's good.

21:49 And I'm sure that on the other side of things, if someone is running a project and it's already passing all that, before they even get to it, they can take it more seriously.

21:57 - Yeah, and that helps you with even, you know, you're splitting up branches, and so you can have tests running on multiple branches, which is nice if you have a long-running development feature.

22:08 And then one of the things I wanna play with here there's a discussion in some about pre-commit hooks and hooking things like black up to your pre-commit hook to make sure the styling is correct.

22:19 - Oh nice, yeah, instead of asking, just change it.

22:21 - Yeah. (laughs)

22:22 - Your styling is wrong, you need to break that line.

22:24 Fine, we did that for you.

22:25 - Yeah.

22:26 - That's pretty cool.

22:28 All right, so the last one I got, Brian, is just a feel-good story, Python versus legacy Python, that type of thing.

22:35 So Pandas goes Python only, no more legacy Python for Pandas.

22:39 Wow, that's cool.

22:40 That's a pretty big deal.

22:41 Like pandas in the data science space is one of the true foundational items.

22:46 Maybe it's more popular than any of the others.

22:49 I feel like people almost always start with pandas and then once they get their data processed, they like move to another library.

22:55 So pandas going Python 3 only is really awesome.

22:58 I got this off of Twitter from Randy Olson.

23:00 Thank you for that.

23:01 And basically they're following NumPy's lead.

23:04 Remember NumPy is going Python 3 only.

23:06 So officially starting January 1st, 2019, which is not that far away, seven months-ish, six months, Pandas will drop support for legacy Python.

23:17 And this includes no backports of security or bug fixes.

23:20 The final release will be the day before, and that one's gonna support Python 2, and we're just gonna leave it there, apparently.

23:28 So I feel like data science has got a little bit of an edge on the Python 3 story for everyone, and partly because they've come into the ecosystem as a large group more recently than say the web developers or the automator folks who have been around for a long time.

23:47 Like the data science stuff has really exploded 2012 and onward, so it was a slightly easier choice I think.

23:53 - Yeah, I think so.

23:54 - Yeah, pretty cool.

23:55 All right, well that's it for our news.

23:57 Anything personal you wanna share?

23:58 - No, I'm just excited to get back to like podcasting and stuff.

24:02 It's been good.

24:03 - It was a lot of fun to do the live one though at PyCon.

24:06 Right, like nobody cheered for us today.

24:08 Not that we heard anyway, right?

24:10 It was so fun to just be in the audience and get the--

24:12 - Yeah, and like nobody laughed at my jokes, yeah.

24:15 - Maybe they did, we'll just never know.

24:17 - Maybe we need like a sound--

24:18 - Yeah, like one of those fake audience tracks.

24:20 Nah, that'll take away from the real ones.

24:22 We'll do some more live ones, we're talking about it, right?

24:24 - Yeah, definitely, it was so fun, we wanna do more.

24:27 - Yeah, maybe we can do some more, we'll figure that out.

24:29 So are you excited?

24:31 It's GDPR Eve.

24:32 - Yeah, the only way, well, yeah.

24:35 I don't really know how that affects me, but I'm telling people that's why I forget their name so quickly is because I'm complying with GDPR.

24:42 - Oh, man, I have very mixed feelings about GDPR.

24:46 I'm a fan of privacy and respecting data stuff.

24:50 I'm not a fan of some of the ways in which they're going about it.

24:54 I mean, it's a tech requirement written by non-technical people, for starters.

25:00 - Do you have to change, for instance, the courses site of yours?

25:05 - We have to, I've been doing nothing but 10 hours a day of GDPR programming all week.

25:10 - Oh, geez.

25:11 - Yeah, and I'm not done.

25:12 I got one or two more days.

25:13 And what drives me crazy about this is I'm an American company, 100% in America, and you know, like, that Europe has these rules that apply to us, which it's not about Europe or America.

25:26 Like, what if India decides later that they have other rules that are inconsistent with what I've done for GDPR, and then Brazil has other, like I just think it's kind of crazy to say like lawmakers in one country can like impose their will on all of the world through these laws.

25:41 So it's kind of funky, but I'm gonna do it because they pretty much have to.

25:45 So if basically the reason I'm throwing this out there is if you run a site where you've got like say a mailing list or people buy stuff or you collect user data, just be sure to be really careful and look into this.

25:59 And also we talked about environments we talked about pipenv and various other bits of packaging. So I just want to give a quick shout out to the XKCD Python environment cartoon which came out a few weeks ago. So that would be xkcd.com/1987. It's just about the sort of madness. So my Python environment has become so degraded that my laptop has been declared a super fun site. It's got homebrew for Python, It's got the OS Python Anaconda.

26:29 It's got PIP, another PIP, easy install.

26:33 Okay.

26:34 It's pretty good, right?

26:35 Yeah.

26:36 Yeah, I think we'll probably try to...

26:38 I'm going to link to the Kenneth Reitz's...

26:41 Reitz's...

26:42 I should just stop trying to pronounce names.

26:46 His PyCon talk, because there was a lot of stuff in there about like the history of packaging that I didn't know about.

26:52 So it's a good, good list.

26:54 Yeah, you should definitely link to that.

26:55 That's awesome.

26:56 All right.

26:57 Well, thank you, Kenneth, for working on pipenv.

27:01 And thank you, Brian, for sharing everything with all of our listeners.

27:04 Thank you.

27:06 Thank you for listening to Python Bytes.

27:08 Follow the show on Twitter via @pythonbytes.

27:10 That's Python Bytes as in B-Y-T-E-S.

27:14 And get the full show notes at pythonbytes.fm.

27:17 If you have a news item you want featured, just visit pythonbytes.fm and send it our way.

27:22 We're always on the lookout for sharing something cool.

27:24 On behalf of myself and Brian Okken, this is Michael Kennedy.

27:28 Thank you for listening and sharing this podcast with your friends and colleagues.