Transcript #75: pypi.org officially launches

00:00 Hello and welcome to Python Bytes where we deliver Python news and headlines directly to your buds.

00:04 This is episode 75 recorded April 25th, 2018.

00:09 I'm Michael Kennedy.

00:10 And I'm Brian Oken.

00:11 And we've got a bunch of awesome stuff lined up for you like always.

00:14 But before we get to it, I want to say thank you to Datadog.

00:17 They're sponsoring this episode like they have many and they're a big part of why the show keeps going.

00:22 So thank you Datadog.

00:23 So Brian, people often refer to the thing that runs your Python code.

00:29 when you type Python as the interpreter.

00:33 But only sometimes does it actually interpret. Sometimes it compiles, sometimes it jet compiles, depending on what underlying thing you're using to run it, right? PyPy versus CPython versus, say, Python.net.

00:46 You found it interesting for one like that for this week, right?

00:49 I had to go look it up. I was surprised we hadn't covered it yet.

00:52 There's a package called Numba and it's built on top of NumPy But it's a fairly easy way to speed up a bunch of your code by adding a just-in-time compiler to it.

01:06 And I've also linked, so I've got a link to the library itself and also to an introduction article that I found that's pretty nice.

01:14 It's pretty easy to use and the article we're linking to has some speed up tests where you just put it like a little JIT decorator on a function that you want to speed up.

01:25 And there's a bunch of flags to it.

01:27 You can try to do it ahead of time or not, but it'll compile down your, some of your code to C code as good as it can.

01:34 And you can specify data types too, if you want to even allow it to be even faster.

01:42 And I'm pretty blown away with how trivial it is to add some of this stuff to your code and speed it up.

01:49 Some of the speed ups are like 78 times faster and stuff, depending on your algorithm.

01:53 That is really awesome.

01:54 And this thing lives in such an interesting place, like when I set the stage.

01:59 It's not CPython, but it's also not a full compiler.

02:04 It's not a JIT compiler either.

02:06 It doesn't attempt to do what, like be a general solution, anything like PyPy, P-Y-P-Y, is a general purpose JIT compiler trying to make all your Python fast.

02:17 So they say that this is a compiler for Python arrays and numerical functions, period.

02:24 That's pretty awesome, right?

02:25 There's a lot of people working with that sort of stuff that a lot of the code doesn't have to be really fast, but then they've got a big blob of data that they've got to deal with and run some algorithm over it. And in those cases with a big array, that's a great place to apply this.

02:40 Yeah, it's really cool. And so if you have tight loops that are slow because of some kind of numpy thing or arrays or other numerical operations, slap a decorator on it, make it go fast.

02:51 Pretty sweet. Yeah, we had a comment from somebody that you didn't know if whether or not we talked about libraries.

02:57 They thought we were just talking about articles and stuff and I do want to include libraries.

03:01 And so I've been trying to make sure going back and some of the ones that people really ought to know about if they don't know already.

03:07 We'll try to get those in here and there. So it's good. Yeah, sure. Absolutely.

03:11 If we haven't talked about it, it's definitely a game for being a cover on the show.

03:14 So speaking of libraries that people care about, what's the most common way to get them?

03:19 - pip install. - pip install.

03:21 Well, as of recently, a couple days ago, now you pip install with pip 10.

03:26 So if you haven't yet, you should pip install dash dash upgrade PIP.

03:32 Maybe put a pip three on the front, depending on what OS you're on.

03:35 But yeah, so there's a brand new version of pip and a major version as well.

03:38 That's pretty cool, right?

03:39 - Yeah, so what do we get new?

03:41 Do you know?

03:42 - Yeah, so we no longer have Python 2.6 support.

03:44 It is out.

03:46 If you're living, I don't know, way, way in the past, like 2006, it was good enough for me, it's good enough for my code, then you should stick on pip 9 because it's no longer going to support it. It supports this new feature that's described by PEP 518, which allows you to specify what packages are required to build from source.

04:07 So that's kind of cool. You have like a, you can say these are requirements to start from source. And these are other requirements. It improves Unicode on Windows, it has a pip config command. I've never used pip config of you. I don't think so. It's I think it's new. So apparently, you can set up like default behaviors and stuff like that. And it can be based on virtual environments, or it can be based on users, or it can be based on machines. So this is probably going to be a nice handy thing to dig into, maybe we should do a whole section on just leveraging pip config. Also, if you pip install dash dash upgrade a thing that used to try to upgrade all the dependencies, if I remember correctly, now the default upgrade strategy will be only to upgrade dependencies if needed.

04:49 Right? So if the thing says I require doc op six and you have doc op four, it'll upgrade it. But if it doesn't say it requires that version, then it'll just leave it alone.

04:56 So that could be for better or worse. I'm not sure.

04:59 And then a bunch of bug fixes.

05:00 Okay, neat.

05:01 Yeah. So if you're out there and you're not living super far in the past, be sure to upgrade your pip. It's nice. And there's even been a point released since then.

05:08 One of the things I really like pip10, this is a minor thing, is that when you type pip list, it lists your packages and it used to in pip like nine, it would give you a warning that there's a table configuration, and you can do the old style or the new style, and that's gone, and in pip 10, it just gives you the table and I like that.

05:29 >> Yeah, that's really awesome. I didn't like that warning there because if you're doing screencasts or videos or anything, it looked like something was wrong, but it wasn't actually wrong.

05:37 It was just like, "Hey, we're changing how this works." So I'm glad that that's over.

05:41 So when I have functions that just have a bunch of positional arguments, right?

05:47 And it's like true, true, false, true.

05:50 Like, what the heck does that mean? Who knows, right?

05:52 So you've got a nice way to fix that.

05:55 That's a modern Python only thing, right?

05:58 I'm chuckling because I'm just leaving Michael hanging on the transitions between articles today.

06:03 But I'll try to work on that.

06:05 Sorry, I got it.

06:06 Anyway, this is actually pretty awesome.

06:08 There's a, I don't know how I missed this.

06:10 There's an article from Trey Hunter called keyword named arguments in Python and how to use them.

06:15 And basically, I think he's right. I was talking to him earlier and a lot of people looked at this article and went, yeah, I know how to use keyword arguments.

06:23 So keyword arguments are very useful and they're things you always have to name your variables that are argument names for a function.

06:30 And the caller of the function can use those. You can specify them.

06:33 And if you don't specify them, you have to send them in order.

06:36 but if you do specify them, you can rearrange the order if you want.

06:40 You can also, one of the things that's often used, how they're often used is, you've got like a whole bunch of default values for arguments, and a caller just needs to override like one or two, you can just name it that way.

06:53 But one of the things in Python 3 is the different use of the asterisks in all of this.

07:00 and there's a way to kind of separate your positional arguments from your named only arguments.

07:07 And if you stick a star in as one of the variables, it forces everything to the right to be only named.

07:15 So a caller has to provide those with names.

07:18 And I didn't, that's something that I completely missed and that's pretty cool. I like it.

07:23 I really like this feature.

07:25 And I implore people who are using **kwargs just, and that's what you pass to the function, maybe a *args thrown in there for additional pain and suffering, to look at this and think about how you might be able to restructure your code like this.

07:43 So a lot of things, you'll have a function and you want to take a variable amount of things.

07:48 And so the way they do it is they say **kwargs.

07:52 You look at that signature and you're like, "What do I do here? I have no idea. I sure hope the documentation tells me." Because the function says, "I take anything, you make up a name, I'll take it." But there's usually like a handful of things it actually takes.

08:05 There's a bounded set of like, "These are the five things you can pass." They're not all required. Put them in as keyword arguments.

08:11 But if it's **kwargs, you're just saying to everybody, "Good luck. Go hunting. Hopefully I documented this." Or you find something on Stack Overflow.

08:21 So you could say star, comma, and then keyword, the argument name equals the default value, argument name equals default value, and then it has exactly the same behavior, except you can see both in your tooling editor as well as in looking at the function, what it takes, and you call it exactly the same way.

08:38 So I really love this feature.

08:40 I love it too, and I like your explanation of other reasons to use it.

08:44 The, I'm with you, the star star, the KWARGs, It just makes it so the user of that function has no idea what they're supposed to pass in.

08:53 The worst offender of this is the AWS API.

08:57 It's super frustrating.

08:58 Like sometimes it's dictionaries of dictionaries, sometimes they're not.

09:02 It's just, don't get me started.

09:04 And the documentation doesn't cover it, right?

09:06 The documentation will say, well, this sub part is this name, but it's this type, which is really a dictionary.

09:10 But there's no mention or description of what goes in that sub dictionary.

09:14 It's just, they can be totally avoided by doing cool stuff like this, which has the same effect.

09:19 Other stuff that's cool, by the way, is Datadog.

09:22 So if you have an app that spans multiple processes, maybe is using different services, and it's a pieced together larger app, you should definitely check out Datadog.

09:32 It's a monitoring solution for providing deep visibility, and it tracks down issues across distributed Python apps.

09:40 So it's pretty awesome. Within just a few minutes, you'll be able to investigate bottlenecks in your code by exploring graphs and rich dashboards.

09:47 So if you want to check that out, if you think having better insight into how your overall app is working, visualize your Python performance today.

09:54 Get started with Datadog, do a free trial, and they'll give you a free Datadog t-shirt.

10:00 It's very cute and gray and white, and I like it.

10:02 So check it out over at pythonbytes.fm/datadog.

10:06 One of the things that we haven't talked about for a while is how to install packages.

10:11 There's a really great new way with pip10 to pip install them.

10:17 But there's more to it than that, right?

10:19 It's not just pip install the thing and the client side behavior is different.

10:24 For the first time in a very long time, ever maybe, I don't know, it goes farther back than I know the history of it.

10:32 That means something different on the server side.

10:34 So finally, finally, finally, pypi.org officially launches.

10:38 Yay!

10:39 More importantly, legacy PyPI, the one at pypi.python.org/pypi, that one, is now over at legacy.pypi.org.

10:52 And that's like an old stick-around version, and they're actually shutting it down.

10:57 So the pypi.python.org will redirect over to what's called warehouse.

11:02 That's the code name for the new implementation running at pypi.org.

11:06 That's nice. I like it.

11:07 Yeah, it's super awesome.

11:08 This is no longer based on a sort of prototype code that grew before the web frameworks even existed.

11:14 This is based on Pyramid.

11:16 It's based on a lot of the common programming APIs and styles that we know today.

11:22 It even uses Kubernetes and Docker and all sorts of amazing stuff.

11:26 It uses Elasticsearch, maybe Postgres, I'm not entirely sure about that.

11:31 So yeah, if people are thinking about adding features to PyPI and then they've thought, oh, but this is some pretty gnarly code, I'm not going in there.

11:39 Well, the new version is the official version and it's much easier to add features to.

11:42 - You talked with some of the people involved with this, right?

11:45 - I did.

11:46 So I talked with three of the folks that worked on it and we talked about a lot of the stories and when it launched and stuff and that's coming out on Talk Python this week.

11:56 So it's a race.

11:57 Does Python Bytes beat Talk Python episode 159?

12:00 I'm not sure, but one of them, it's gonna be just about the same time.

12:04 So if you're interested in this, go listen to it.

12:06 You'll hear a lot of interesting stories.

12:08 Like, Brian, do you remember when it used to be there was pypi.io?

12:11 - Yeah.

12:12 - Do you remember that in the early days?

12:13 And then later it became pypi.org.

12:16 I figured out what the story was around that.

12:19 I thought it was indecision, right?

12:21 Oh, we'll do IO.

12:22 No, no, actually we should make it more organizational.

12:25 We'll do .org, right?

12:25 It turned out that what had happened was pypi.org was owned by someone else.

12:31 The PSF didn't own it.

12:33 And they had to try to get ahold of it.

12:35 So in the meantime, they used pypi.io, but their ultimate goal was always to have pypi.org.

12:40 It just took them a while to actually acquire the domain.

12:43 So if stories like that are interesting, we got like a whole hour of it on DocPython.

12:47 So people can check that out.

12:48 So packages in Python.

12:51 - We got a theme going on here.

12:53 - I think we do have a theme.

12:55 I got one more package after this, by the way.

12:57 - So one of the things I forgot about is when you're kind of coming into, there's a lot of people that, as we know, come into Python from other languages, and they'll rush in and figure out how to do something quickly.

13:09 And then they think, "Yeah, I'm a Python expert because I wrote a script." It isn't people are trying to become experts or anything.

13:15 But one of the things that tripped me up right at first is getting my head around what is a module, what is a package, really how do users make those, and dealing with Dunder and NITs and Dunder and NIT files and on where do they go and all that stuff.

13:34 And so there's a nice tutorial now.

13:37 I think it's at real Python, Python modules and packages and introduction.

13:42 And I just really liked this.

13:44 It's a good rundown. It's pretty simple.

13:46 And this is something I'm bookmarking.

13:49 So a lot of the people I work with that are new to Python and trying to figure this stuff out, I can point them to this.

13:55 It's a good introduction.

13:56 Yeah, it's a great introduction.

13:57 Yeah, it's definitely that kind of stuff that when you're new, it's a little bit weird to figure out the difference, right?

14:03 like use packages and modules kind of the same, but they're not the same, what's the story?

14:08 And especially learning how to package up your project to make it accessible to other programs is tricky.

14:14 - This isn't going into packaging itself, which is, it's a bummer that we have those as two different aims.

14:22 Really, this is talking about a directory with a dunder init in it and subdirectories and stuff.

14:28 We're talking packages and sub packages.

14:30 But it is the initial part.

14:32 Once you have this down, then you can jump into packaging and distributing code to share.

14:38 But this is a good starter. So it's a good learning place.

14:41 Yeah, it's definitely a good learning place.

14:43 So remember last week we spoke about the joint project that the PyCharm team and the Python Software Foundation did with their 2017 Python survey?

14:55 Yeah.

14:55 Right. One of the interesting pieces of data there was the people who are data scientists do more Python 3, modern Python that is, than as a percentage of their projects than say web developers or other types of Python developers.

15:11 It's partly because they're starting from scratch, more of them, it's like a more new type of thing.

15:17 So why start with the old?

15:18 Well, the big news today is that Pandas, one of the major most important foundational data science libraries, is going modern Python only at the end of the year.

15:30 That's a big thing. Yeah, definitely.

15:32 Yeah, I would say this is as big of news as Django 2 going Python 3 only on similar timescales.

15:40 Well, they're already out with their new stuff.

15:42 So I guess just in terms of their old support ones.

15:44 So that's pretty cool, right?

15:46 One more major, major building block that it used to be, well, if I move to Python 3, I won't be able to get my libraries.

15:53 It's now the story is if you don't move to Python 3, you don't get the best newest libraries.

15:57 So pretty awesome.

15:58 And it isn't just getting the newest, it's also security updates and things like that.

16:02 Yeah, like part of their announcement said, on December 31st, 2018, pandas will drop support for Python 2.7, presumably stuff before that's not even supported.

16:11 This includes no backports for security or bug fixes.

16:14 Period.

16:15 They are open to letting somebody take that to become their job, but they're like, "We're not doing that, so we're moving on." Is pandas like a volunteer thing?

16:24 Is it open source, just run by open source people, do you know?

16:27 I think so. I think it might be part of the high side stuff.

16:31 I probably sorry, part of the support thing is, I mean, you can kind of expect some companies to have back like, you know, security and bug fixes on old releases, for a while, but you know, open source projects, these are just people volunteering their time. I think it's completely reasonable for people to say, Hey, we're done trying to support two seven. Let's move on.

16:51 Yeah, it makes sense. So I looked while you're talking and pandas is a numb focus sponsored a project. So I think it's sort of donation supported, but through a more formal science organization. But still, those people are contributing features. They can either focus on bug fixes and making Python two stuff work or just moving forward and getting the latest greatest stuff going. So that's pretty cool. Yeah, it's a good focus.

17:17 So Brian, that's it for this week for items. Got anything in particular you want to talk about with everyone? I've got a couple episodes coming out on Talk Python and I'm getting ready for PyCon. I'm looking forward to that.

17:31 Yeah, you have a talk going there, right? That's gonna be pretty awesome.

17:34 Yeah, I'm getting a little nervous for that, but it'll go.

17:37 Don't worry, it's not like it's recorded and then also there's a bunch of people there.

17:41 Oh yeah, it's gonna be forever. No stress. Just kidding.

17:44 No stress. Thanks for that.

17:46 No, I'm just teasing you because I know you'll do great. We also have a booth there. You and I and and a few others so people can come visit us at the booth like the last year.

17:54 >> I'd like people to try to reach out if they want to just come on and if there's something they want to record, I think I'm going to take some recording equipment so we can do some short recordings.

18:04 >> That'd be awesome. Yeah, I'm definitely bringing my mic and going to do some stuff like that as well.

18:08 >> What's up with you?

18:09 >> Well, remember last time Matt Harrison was on the show and we were joking about trying to get some course or something other done?

18:15 >> Yeah.

18:15 >> Well, it's taken me a ton of work this past seven days, But nonetheless, we now have a brand new course to announce.

18:22 Python 3, an illustrated tour.

18:24 Nice.

18:25 So the idea is, it basically only covers the features that were new to Python as of Python 3.

18:32 So it says, here are a couple of peps that talk about, say, type annotations, and here's some graphics and other sort of walkthrough of what that means for you.

18:40 So if you feel like you are not using all of the features of Python 3, or you're coming especially from Python 2, and you're like, I need a quick show me the new stuff sort of in a practical way, then check out the Python 3 and Illustrator tour.

18:54 That's done by Matt Harrison and it came out really well.

18:57 So check it out at talkpython.fm/illustrated.

19:00 - Okay, I'll check that out.

19:01 - Yeah, yeah, I learned a lot.

19:03 It's really, really fun.

19:04 We keep learning things.

19:05 Like for example, you talked about the star at the beginning of the function arguments, right?

19:09 That kind of stuff is in there.

19:10 And I didn't know about that until too recently either.

19:12 - Okay, cool.

19:12 - All right.

19:13 - It's been a good talk today, thanks.

19:15 - Yeah, you bet, very nice one.

19:16 We really dug into the packages this time, didn't we?

19:18 And we don't plan this.

19:20 It just happens.

19:21 It just happens.

19:22 That's right.

19:23 All right.

19:24 Well, thanks, Brian, and talk to you all later.

19:25 Bye.

19:26 Thank you for listening to Python Bytes.

19:28 Follow the show on Twitter via @PythonBytes.

19:31 That's Python Bytes as in B-Y-T-E-S.

19:34 And get the full show notes at PythonBytes.fm.

19:37 If you have a news item you want featured, just visit PythonBytes.fm and send it our way.

19:42 We're always on the lookout for sharing something cool.

19:44 On behalf of myself and Brian Okken, this is Michael Kennedy.

19:48 Thank you for listening and sharing this podcast with your friends and colleagues.