« Return to show page
Transcript for Episode #15:
Digging into Python packaging
00:00 Python Bytes Transcript
00:00 Episode #15: Digging into Python Packaging
00:00 Michael KENNEDY: This is Python Bytes. Python headlines and news delivered directly to your earbuds. I’s Episode #15, recorded Monday, February 27th, 2017.
00:00 This is Michael Kennedy. I’m here with my co-host, Brian Okken. Hey, Brian. Happy Monday. Happy Python News Day.
00:00 Brian OKKEN: (Laughing) Yes.
00:00 KENNEDY: (Laughing) Today we’re going to have a couple of really interesting items. We’re going to start off with a theme. The theme is, people should probably understand Python packaging distribution better, right?
00:00 OKKEN: Yes, definitely. It jumps up on you as a surprise, I think, as soon as you try to share some Python code with somebody. Even if you’re just sharing it with your own workgroup, dealing with packages. There’s a blogpost out, I think it was on medium.com, by Jie Feng (@flyfengjie). “A Simple Guide for Python Packaging.” It’s actually a really quick read. It’s a really nice introduction to talk about where to put the _init_ file and basically, what is a package and how to deal with it. Even to the extent of pushing something up to PyPI. Yeah, it’s a good article.
00:00 KENNEDY: Yeah, I thought it was really nice. It’s like 5 minutes or less to read, so definitely jump in. One of the things that I was reminded of while I was reading it, it was on the top of my mind, is the difference between modules and packages. Many people know the differences but if you haven’t worked on Python packaging, maybe you don’t. I feel like people often use module and package interchangeably, like these are synonyms and we just swap them. You _import_module or _import_package or whatever. There’s a difference, right? A module is a single Python file where technically what defines a package is a directory with a dunder_init_.py and one or more Python files in it. I guess it could be empty and just does nothing but it’s more. Python treats those differently if it finds a dunder_init_.py.
00:00 OKKEN: It’s very important when sharing code. One of the nice things, at the bottom of the article there was a link to another article. Actually, an entire set of instructions called “How to Package Your Python Code” by Scott Torborg (@storborg). He’s a Portland person.
00:00 KENNEDY: Oh, right on.
00:00 OKKEN: Yeah, go Portland! It’s a longer introduction but I read it in a half an hour, the entire site. It’s a fairly detailed look at packaging. I think this documentation should be included in the PyPA documentation because it’s really good.
00:00 KENNEDY: You think it’s that good? That’s awesome. Way to go, Scott.
00:00 The next item that I want to talk about is also related to this. I feel like this is the next layer up. Once you get into packaging and start packaging your code, one of the reasons you might do that is if you’re building a package for the world. Submit that to PyPI and everybody can play with it. Another reason is, you just want to reuse that code and maybe this code is proprietary code and you don’t want to reuse it with the world, you want to reuse it only within your company. So, one of the cool things you can do it set up your own PyPI server. Then that is private, behind your firewall/authentication or whatever and you can put your code there across your different apps, across your different teams, across continuous integration as along as it has access to this. You can basically put additional private stuff and could load in your proprietary libraries in there. I think that’s really cool.
00:00 The thing I brought up was an interesting take on this idea called, “Elastic PyPI” by Kyle Hornberg. The idea is, we’re going to take and we’re going to use Elastic Search and AWS Lambda for a server-less deployment of your own private PyPI. The AWS Lambda stuff is like, ‘Here I can give you a Python function and I’ll give it to AWS’ and it lives at the end of the URL. Just basically, you make a request, it runs it. You don’t have to set up a server. It’s even a little bit less than platform as a service, like really lightweight. So, it’s a combination of this and Elastic Search which I thought was kind of cool.
00:00 OKKEN: Yeah, it’s pretty neat. I’m still looking for the right solution for internal to accompany inside of a firewall. I haven’t quite stumbled upon the right thing yet.
00:00 KENNEDY: Yeah, I know. So, I actually listed 3 other ones that I know about that are in this realm. This runs on Elastic Search and AWS Lambda which isn’t behind your firewall but it private. There’s also PyPI server, there’s DevPy. Those are interesting, you can basically pip install those and set them up and run them behind the scenes. I still feel like you’re on to something, that there’s something more to do here.
00:00 OKKEN: There might be something less and I’m just missing it. Do I even need it or can I throw a directory somewhere?
00:00 KENNEDY: Yeah, I feel like there’s GitHub Enterprise and things like that. Here’s a really cool box that is GitHub for your enterprise. Maybe there’s something like that for PyPI. These are good enough. It seems like an interesting area, right?
00:00 OKKEN: Definitely.
00:00 KENNEDY: So, we got an email from a listener who said, ‘Look, I’m really enjoying our podcast. All the news that you find is great but I’m kind of living in an RSS world. I feel like RSS is great, but it’s not really where all the news comes from. Where do you guys get your news from?’
00:00 Brian, where do we get our new from?
00:00 OKKEN: Well, where we get it from… Where you guys get it from I think should be PythonBytes.fm, this podcast. Honestly, I took a look and I pay attention to Twitter. I pay attention to newsletters and Planet Python.
00:00 Let’s go through these. We put up in the show notes, the link to mine and Michael’s Twitter accounts. We don’t really put out a lot of news on the Python Bytes, maybe we should.
00:00 KENNEDY: Yeah, we probably should. I feel like we should probably tweet out at least our 6 items every week, right?
00:00 OKKEN: Yeah, definitely. That’s a good idea.
00:00 KENNEDY: We should get on that.
00:00 OKKEN: Let’s do that. But there’s a feed aggregator called planetpython.org, if you don’t know about it already. It has a few different websites and it does everything new that day. I think it’s updated several times a day there. My RSS reader just points to that. I just pay attention to that.
00:00 KENNEDY: Yeah, and there’s a lot of great stuff there.
00:00 OKKEN: We didn’t really order them in any particular order but there’s a few Python-related newsletters. There’s Awesome Python, there’s Python Weekly, Pycoders Weekly and Import Python. That’s 4. I don’t know if I have a preference, I just like scanning through all of them.
00:00 KENNEDY: These guys all have great stuff. Every one of them has at least one unique thing each week so they’re all worth subscribing to.
00:00 OKKEN: And I also like the second opinion because if there’s an article or new project or something that shows up on all 4 of them or 3 out of 4, I probably should go read it.
00:00 And then Reddit is used quite a lot by Python people. And there’s a Python sub-Reddit that’s pretty good.
00:00 KENNEDY: Yep, and I threw in trending on GitHub, so if you go to GitHub/trending?l=python, it will actually show you all the trending repositories that are primarily Python. That’s a pretty good trick for things people haven’t discovered yet.
00:00 OKKEN: Yeah. I haven’t used that; that’s neat. I’m not really hanging out on Reddit a lot, so one of the things that I think is good to know about Reddit, is that there is a couple of filters at the top. When you’re looking at the interface, it seems like a constant stream of information but there’s ways to say, ‘What’s the interesting ones, the top ones in the last week or the last month?’ It’s helpful to filter some of this stuff.
00:00 Well, let’s move off that news and onto news.
00:00 KENNEDY: Yes. We talked a lot about asynchronous programming, the concurrency story in Python and as we’ve said, this is for Python 3.4 and 3.5, these are some of the really big changes to the newer version of Python. So, what I wanted to highlight was something done by David Beazley (@dabeaz). He’s done a bunch of great public speaking and teaching and stuff. He’s created this project called, “Curio: The Coroutine Concurrency Library for Python.” This is a small, little library built for performing concurrent IO and common system things such as launching sub-processes and farming out work on threads and process pools. It’s built from the ground up to basically be taking full advantage of the programming paradigms, things like async and await in Python 3.5. It doesn’t have a lot of the baggage that comes from using Tornado or Twisted or something like that. Pretty cool.
00:00 OKKEN. Okay. I haven’t looked at this but I did want to see some good example of a production library that uses async and await to try to understand that better.
00:00 KENNEDY: Yeah, it’s got a really cool example of writing basically a socket-based server that can communicate with people and handle thousands and thousands of concurrent requests using a single thread. The mental model for how it works should probably be a little bit like node.js. You sort of subscribe to some work and you get a callback when that work is done so you get a message from a client or something. But because it uses async and await, it looks basically like standard procedural non-concurrent code, which is the way that code should look.
00:00 OKKEN: Yeah, okay, cool.
00:00 KENNEDY: Check that out if you are wanting to look at a take on async and await.
00:00 OKKEN: My last item, I got this through Reddit. Just the other day I was looking through the ‘Hot Things from the Last Week on Reddit in Python.’ This item was Pandas switching to use pytest as its test framework. I thought that was really cool because pytest is something I like.
00:00 KENNEDY: You should write a book on it.
00:00 OKKEN: Yeah, I should. (Laughs)
00:00 KENNEDY: That’s really cool. I think it’s an interesting look at the challenges of moving a very large, highly visible, highly used project from one testing framework to the other. There were comments, like in this GitHub thread, things like, ‘Well, pretty much everything moved over okay except for these 69 tests. They’re really hard and we need somebody to champion taking on these 69 tests that won’t move.’
00:00 OKKEN: Just 69? That seems like a lot of work.
00:00 KENNEDY: Yeah, they said many of the ones that are challenges are because of external resources like databases or other funky things like that.
00:00 OKKEN: Actually, external resources is the reason I chose pytest in the first place because I depend heavily on resources. And it handles fixtures the best.
00:00 KENNEDY: Awesome. Cool example.
00:00 So, my last one is, I kind of cheated and picked something that I did. But I think this one is noteworthy, given the amount of attention the community gave it.
00:00 Last week I had the honor of having Guido van Rossum on Talk Python. We spent a little over an hour talking about how he got into programming, what were the influences on the Python language. There’s a language called ABC that he was actually working on before Python. Why did ABC fail, and go away, why did Python succeed? How are we doing on the move to Python 3.x? What are his favorite features? And even what he’s up to at Dropbox these days. Really interesting conversation looking backwards and forwards with the creator of Python.
00:00 OKKEN: I really enjoyed that conversation, it’s one of my favorite episodes now. I really liked the discussion of the 2.x versus 3.x, the reasons why they did it and also the difficulty. And a lot of people dragging their feet and not wanting to switch and there are real world reasons that just sort of hit them; they didn’t think about those. It was a pretty interesting take; I liked it.
00:00 KENNEDY: Yeah, I think people will learn a lot listening to Guido. I wanted to highlight that because I think Guido doesn’t do a lot of those kind of open, ask-me-anything-type things. He does a lot of public speaking and stuff. I thought this was a great look inside. He was a great guest and it was a lot of fun to talk with him. So, check that out.
00:00 OKKEN: Well, that’s our 6 picks for today.
00:00 KENNEDY: Yeah, so that’s the news for Monday, February 27th. What else is going on? You had a great episode with Mahmoud, friend of the show, Mahmoud Hashemi.
00:00 OKKEN: That I finally got it all edited and I put this out last Sunday. It’s Episode #27. I really enjoyed the discussion with him. We started off with the excuse of talking about the different levels of a testing, like unit versus system and stuff like that. We just talked about all sorts of stuff testing-related and Python-related.
00:00 The other excitement is I released it on a new website so I have testandcode.com now running everything. It will be on both websites for a while until I get everything switched over.
00:00 KENNEDY: That’s great. I’m looking forward to listening to that episode with Mahmoud because he has great things to say. I had him on to talk about Enterprise Python. He works at Paypal and does a bunch of cool stuff with Python there. I’m sure that’s really interesting, that’s great.
00:00 OKKEN: Well, he used to work at Paypal. You’ll have to listen to Episode #27.
00:00 KENNEDY: Now I must listen to it. Oh, my gosh, I’m out of date. That sounds very exciting.
00:00 So, testandcode.com, you finally unified the domain and the rename that you did a few months ago?
00:00 OKKEN: Yeah, testandcode.com domain and the podcast have the same name. And the pystest book is out to reviewers. I’ve got 3 chapters left to write but we’re chugging ahead like a large freight train.
00:00 KENNEDY: Yeah, it looks like you’re ahead. I know you were like, ‘Geez, Can I get this done before PyCon?’ which is the end of May, but it feels like you’re making good progress.
00:00 OKKEN: Yeah, to actually get an in-print book would be a miracle at this point but we are hoping to get a beta e-book out by PyCon.
00:00 KENNEDY: Awesome. Well, congratulations. I’ve seen some of the covers and they’re both very cool so I’m looking forward to seeing which one you go with.
00:00 OKKEN: So, how about you? What’s up with you?
00:00 KENNEDY: Not a whole lot is going on with me right now. I released my “Consuming HTTP Services” course and I’m just working on 2 or 3 more courses. I’m actually writing a Python article for Code Magazine so I’m supposed to submit that this week. It’s about celebrating the 3,000th day of Python 3.x because just a few days ago, Python 3.x was 3,000 days old.
00:00 OKKEN: So, Code Magazine, is that a print thing?
00:00 KENNEDY: Yeah, it’s print.
00:00 OKKEN: Cool. I didn’t know there were print code magazines anymore. (Laughs)
00:00 KENNEDY: Some people still read them I guess. We’ll see.
00:00 OKKEN: Well, awesome. It was fun talking with you today.
00:00 KENNEDY: Thanks for sharing all the news. Now people can go out and package their Python code up and hear Guido talk about it.
00:00 Good to see you, thanks everyone listening in and we will catch you next week.
00:00 OKKEN: Thanks a lot.
00:00 KENNEDY: Bye.
00:00 Thank you for listening to Python Bytes. Follow the show on Twitter via @pythonbytes and get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbyes.fm and send it our way. We’re always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.