Transcript #22: PYTHONPATH considered harmful

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.

00:07 It's April 18th, 2017. I'm Michael Kennedy.

00:11 And I'm Brian Okken.

00:12 And we have a bunch of cool stuff to share with you. But before we do, Brian, we have to say thank you to Advanced Digital.

00:16 Oh, yeah. Thank you to Advanced Digital.

00:18 They're a brand new sponsor of the show, and they have some really cool Python job opportunities. We'll talk about that later.

00:24 Let's talk about the Python path now.

00:26 I like this. Somebody came out with a, I should write names down. Let me look it up just a second. Okay. Moshe Zedka came out with an article called Python Path Considered Harmful.

00:38 And this is a good article. It's very short. And I guess I had forgotten about the Python path because fairly early in my Python career, realized it's not useful. It's only bad. Ignore that it's there.

00:52 So what the Python path is, it's an environmental variable that you can put directories in so that you add to the places where Python will look for import modules, or modules or packages to import.

01:06 Every time I've ever thought maybe I had a good use for it, I've regretted it later.

01:13 And one of the classic examples is that if you had a package, you would, I guess, maybe want to add that top directory for a local package to the path.

01:25 But there's a whole bunch of stuff in the top directory that's not appropriate to be imported, like the setup.py and so forth.

01:32 But anyway, just, I guess, a public service announcement.

01:37 Don't use Python path.

01:38 Is there a good use for it?

01:39 Yeah, I don't know.

01:40 I mean, we have virtual directories.

01:42 Those work pretty well.

01:43 You can Python and set up py, develop if you want to kind of inject something into the path.

01:50 Like there's a lot of options that don't involve that necessarily.

01:52 - Yeah, and also if you're like, one of the reasons might be if you're developing a local package and you want to be able to import from it, like from your test directory, but that's still the best recommendation is to do a pip install -e to as install editable is the best way to do that.

02:13 - Yep, exactly, exactly.

02:15 So the one that the first one I want to talk about are actually algorithms.

02:21 There's a really interesting article about how I think we talked about this one and I'm not sure if I'm blending in my mind with another article I read, but basically that we can use languages like C and stuff to try to make our code faster by trying to get closer to the metal.

02:37 But a lot of times, a way to make your code faster is actually to have better understanding and visibility the algorithms.

02:44 Right.

02:45 You could try to make the bubble sort really, really fast and see, or you could just use quicksort in pretty much any language, right?

02:51 And it's a way, way faster.

02:52 So I ran across this GitHub repo that is just a bunch of minimal examples of data structures and algorithms in Python.

03:00 And it's keon, you know, github.com/keon, K E O N slash algorithms.

03:06 And there's just tons of really cool examples.

03:09 So there's there are different topics like arrays and graphs and trees.

03:12 So like in the arrays, there might be like the ability to flatten an array or to merge intervals on arrays or in graphs, you can clone it or traverse it or find paths through it.

03:24 All those types of algorithms that are like, "Wow, that actually solves exactly what I need and here's a little example.

03:29 I thought it was really cool." I think it helps professional programmers, but if you're looking for a job, there's a lot of interview type questions around algorithms and data structure.

03:39 So it might also be worth studying there.

03:41 Oh, that'd be a good place.

03:42 Yeah.

03:43 So I'm not sure how many there are, maybe 50 data structures and algorithms, but pretty cool.

03:47 That's neat.

03:48 Cool.

03:48 Yeah, so if you want to brush up on your algorithms, then there's a place to do it.

03:52 Well, speaking of data structures, in episode 11, we talked about a package called Atters, A-T-T-R-S.

03:59 Atters has a lot of fans.

04:00 People love that library.

04:01 Yeah, it's a cool package to be able to make classes easily, and just the attributes of classes are easy to define.

04:10 Right, kind of make them complete as well, right?

04:12 Give them the equivalency tests and all the various things, not just, oh, here are the fields, right?

04:17 Right.

04:17 And this is an older article from last year.

04:22 I guess it's not that old.

04:23 It's from Glyph.

04:25 And it's just a really good article about-- the article is titled "The One Python Library Everyone Needs." And he talks about the annoying bit of having to create your own classes and make sure that the copies and less than all work.

04:39 and the sorting and just letting that actors work for you.

04:44 He also discusses briefly some problems with named tuple, but I'm a huge fan of named tuple still.

04:51 And I don't know if I buy his slamming on it too much, but.

04:55 - It's been all good for me, I think.

04:58 But now this is really cool.

04:59 And I need to be trying out adders.

05:02 I haven't done it yet, but it seems like, you know, it really does add a lot.

05:06 I should just make it a habit.

05:07 - Anyway, it's good article.

05:08 You know where I would like to try out some new stuff?

05:10 Where?

05:11 At a new job, building cool stuff with Python.

05:13 I would love that.

05:14 Yeah, so our sponsor, Advanced Digital, they sponsored this episode of the podcast because they're looking to work with you.

05:21 Everyone out there who knows Python and wants to build cool stuff with Python.

05:24 So Advanced Digital, I'd never heard of them, but they actually run one of the 10 largest news sites in the US by traffic.

05:33 So they're really high scale Python shop.

05:36 They're located in Jersey City just across the river from from Manhattan.

05:41 So you know, beautiful view, see the Manhattan at night and take the path over there.

05:46 They fund employee development and conferences, you go to PyCon, things like that.

05:50 And they do mostly Python, but they also run other things.

05:53 So if you want to work in an environment like that, and do Python for your job, check out And those guys will hook you up.

06:02 That's great.

06:03 Yeah, sounds fun, right?

06:04 Cool.

06:05 So one of the things that makes high traffic websites run well is concurrency, right?

06:11 And we kind of beat this drum often, but because it's an awesome drum.

06:16 And last week I had David Beasley on Talk Python to Me to talk about a project he created called Curio.

06:22 You heard of Curio?

06:23 I have.

06:24 I haven't played with it though.

06:25 So Curio is it's an interesting project in that it's kind of like halfway between a framework that you can just grab and use and half like really low level building blocks.

06:35 But the idea is, we have async IO in Python sense three, four, and it's got this event loop and allows a sort of asynchronous programming through callbacks.

06:47 But David said, Well, what if we actually started from scratch, and we had async in a way and available.

06:53 We had these asynchronous coroutines as the primary concept and an async library for Python.

07:00 What would that look like?

07:01 Well, that's Curio.

07:03 So it's a library for performing concurrent IO operations and system programming tasks like launching subprocesses or threads or whatever.

07:12 And it's solely concerned with the execution of async coroutines.

07:15 It's cool, right?

07:16 Yeah, very.

07:17 And it's got all these really neat data structures.

07:20 So it has this thing called a universal queue.

07:23 And cues are one of the primary ways to communicate between threads without locking.

07:27 So you don't have you're not sharing the data, you make a copy, put on the queue.

07:31 So you're not worried about race conditions and things like that.

07:33 But one of the problems is the threading model that works for say, cues between threads is not the same that goes between async IO execution.

07:43 And that's also not the same that goes between curio.

07:45 So he added this thing called a universal universal queue that spans all three of those worlds and lets them inter communicate with each other.

07:52 has async threads for sort of managing computational execution and threads as if they were asynchronous coroutines and all sorts of stuff like that.

08:03 And you really learn a lot about async if you dig into this thing.

08:06 Okay.

08:07 And you said it is a sort of halfway between something low level and high level?

08:12 Yeah, yeah, exactly.

08:13 So if you want to make say like a TCP/IP game server with TCP or UDP or something like it's It's actually got constructs to say launch TCP server that's async and plug in the callbacks and things like that.

08:24 Wow.

08:25 Yeah, but there's no there's no HTTP layer, right?

08:28 So it's not like a web framework.

08:30 It's not like Sanic or Gepronto or one of those things.

08:33 But it's not super, super low level.

08:35 It has all these building blocks.

08:36 So it's like, if I was going to build a framework thing of some sort that did a bunch of asynchronous and I wanted it asynchronous at its core, maybe using Curio as the core of your project to build that framework on top of might be perfect, but it's not itself a framework really, not yet anyway.

08:54 - Okay, interesting, I'll have to keep an eye on it.

08:56 - Yeah, it's the built-in concepts for tasks and threads and queues and whatnot, very, very neat.

09:02 - Neat. - Yep.

09:03 All right, you have some package sorcery.

09:05 - Yeah, I was playing with the SRC and trying to figure out if this section should be package sorcery or use the source loop or to source or not to source, anyway.

09:18 These are all great questions.

09:20 Well, speaking of great questions, in episode 15, we talked about Python packaging.

09:27 And one of the listeners, Christoph, had a question last week.

09:32 And the question really was that he was reading some other articles that were in conflict and actually starting with the pytest documentation and pointing to an article by Yono--

09:46 I think it's Yonal, called Packaging in Python Library.

09:49 And the conflict really is whether or not a distribution package for, like, if you're going to push it up onto PyPI, should it have all of the module or the source packages, should they be top-level directories, or should they be under a SRC directory?

10:08 And the recommendation-- some people recommend, and pytest did as well, of using this SRC directory instead.

10:16 And the argument that Yonel puts forth is referenced in lots of places.

10:22 And it actually all sounds good.

10:25 Some of the problems are based on using TOX and other testing tools to be able to install things a lot and uninstall easily and not muck with the namespace too much.

10:38 So I went out and tried to find some examples that used this.

10:41 And there's-- actually had difficulty finding some.

10:45 If this is such a good idea, I guess my question is, why isn't this not promoted more by the Python Package Authority and their documentation?

10:55 And pytest itself, even though it recommends this, that package doesn't use it.

11:01 So I guess that's my question to the Python community is, should we be using a SRC directory or not?

11:08 OK, yeah.

11:09 Well, this is a good place to point out At the bottom of every episode, we have a discuss section and people come in and they ask questions and give us feedback on the various episodes.

11:20 And this one came out of one of those, right?

11:22 - Yes. - Yeah, this conversation.

11:23 So if you want to comment on one of these shows, like this is episode 22, pythonbytes.fm/22, boom, go to the bottom and pick it up, right?

11:30 So feel free to jump in.

11:32 And that's probably on episode 21 where this conversation was happening.

11:36 - Yeah, and I get why a lot of existing projects like requests don't have this in it because they didn't before.

11:43 But if there's a lot of people like Christoph out there that are trying to come up with some new code to share with people, and they want to do it the right way.

11:52 And it's a legitimate question is what is the right way?

11:54 - Yeah, absolutely.

11:55 Yeah, yeah, great, great topic.

11:57 Now, this last one I have is a bit of a downer, but I just want to, I wanted to cover it because I think it's an important topic and we've talked about it a few times.

12:05 And I'm going to say two companies names.

12:07 But before you think anything negative about these companies, These are the two companies that were funding a thing that nobody else was funding.

12:14 So thank you to them for doing it.

12:16 But it turns out that Intel is pulling funding from its OpenStack effort, like a sort of an initiative that it started with Rackspace.

12:26 So Intel and Rackspace were collaborating on a project called OpenStack Innovation Center, and that started back in 2015.

12:34 And it was supposed to be funded through 2018, but actually it got pulled out early, right?

12:40 There was a lot of good things that got done there. They said the objective was to create the world's largest open stack developer cloud and developer enterprise capabilities with open stack and it quickly accomplished the first goal and made a great progress towards the second.

12:53 So that's all good that that was done. But it turns out there's 30 Rackspace employees who have been working on this now have two weeks to find another job within the company.

13:04 And so I guess the story here is just, you know, we all need to be vigilant and careful about how we fund our open source work.

13:13 Maybe a little bit of diversification, not in terms of what people are doing, but in terms of the companies that we have supporting us.

13:20 So the more companies contributing smaller amounts, I feel is probably a safer place to be than a few companies contributing huge amounts.

13:28 Like we have the same problem with PyPA.

13:29 Yeah, definitely.

13:30 and the packaging authority and all those guys, right?

13:33 Like that whole set of projects.

13:35 - Yes.

13:36 - So I don't know, I'm gonna try to talk about this a little bit at PyCon this year.

13:40 I'll try to do an open session or something if I can pull it together.

13:43 - There's not really a good time of the year to try to find another job in two weeks.

13:48 - No.

13:50 - Anyway, sorry for you guys.

13:51 - Definitely not.

13:51 Yeah, that's a real bummer and OpenStack is awesome.

13:54 This is not, they're pulling their support from OpenStack.

13:57 This is this initiative that was on top of it, but still.

13:59 - Okay, so it's not all of OpenStack, it's just one part of it?

14:02 - No, this was an initiative specifically that they were doing, trying to bring people together around OpenStack and some other stuff.

14:10 - Okay. - Yeah.

14:11 All right, well, that's our news for the week, Brian.

14:13 You got anything in particular?

14:15 - No, I don't.

14:15 - Yeah, no worries.

14:16 We're still awaiting that book release.

14:19 That's gonna be a good day.

14:19 - Well, I'm frantically in the middle of edits and there's some incredible, I'm finally working through a lot of the feedback I got from people And I'm just still very humbled by the help I've received by the community.

14:33 It's great.

14:34 - Yeah, that's awesome.

14:35 So I have a quick piece of news for everyone.

14:37 If you are in Europe or you would like to spend a little time in Europe, europython@europython.eu, be sure to check that out and get your tickets because they've already sold out the early bird tickets and the main tickets are on sale.

14:53 I've already heard from some people who were hoping to go to PyCon this year in the US, not going, tickets are sold out.

15:00 So if you're in Europe and you wanna go, don't wait, these things sell out and then you'll be sad.

15:04 - Are you gonna go, Michael?

15:05 - No, not this year.

15:06 I was supposed to go last year, I really wanted to, but we were moving back to the US within like a few days of it running, so just didn't work out.

15:14 Maybe next time, maybe next time.

15:15 - Yeah, maybe we could both go.

15:16 - That would be awesome.

15:18 All right, well, Brian, thank you for sharing everything with us.

15:22 - Thank you.

15:23 - Yep, it was fun.

15:24 Thank you for listening to Python Bytes. Follow the show on Twitter via @PythonBytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at PythonBytes.fm. If you have a news item you want featured, just visit PythonBytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Aukin, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.