« Return to show page
Transcript for Episode #118:
Better Python executable management with pipx
00:00 KENNEDY: Hello, and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds. This is Episode 118, recorded February 19th, 2019. I'm Michael Kennedy.
00:11 OKKEN: And I'm Brian Okken.
00:12 KENNEDY: And this episode is brought to you by DigitalOcean, check them out at pythonbytes.fm/digitalocean. More on that later. Right now, Brian, man, it's cold outside.
00:22 OKKEN: It's not bad.
00:24 KENNEDY: I would say it's frozen.
01:06 KENNEDY: Frozen-Flask looks so cool.
01:08 OKKEN: Yeah it really does, and then I also stumbled across a tutorial that's called Dead Easy Yet Powerful Static Website Generator with Flask. And it talks about Frozen-Flask and it also talks about an extension to Flask called Flask-FlatPages which allows you to have a bunch of pages generated for mark down.
01:31 KENNEDY: Now that's pretty cool. Yeah I haven't heard of that, that's nice.
01:34 OKKEN: And it's pretty fun, I got to warn ya, I tried to go through this tutorial. It's still got a lot of great information, it's well written, it's just a look, since it was from 2012 it's a little out of date, some of the imports don't quite work so you'll have to do some Googling. But the Frozen-Flask that we also link to has some really cool, it's some pretty easy tutorials on there too. So it looks fun.
01:55 KENNEDY: Yeah, that looks really cool. There's always this tension between having like a data driven website, where maybe I've got a bunch of podcast episodes or something, and they all have notes but I want to have like a whole bunch of stuff around them and have that generated, and of course that would be like a regular Flask data driven app, right? But if I'm going to do that with a static thing, how do I do that? Do I create a bunch of static markdown, how do I mix it all together? It can be painful, right? So Frozen-Flask is like well you create the dynamic thing but then you host it, you don't have to deal with the infrastructure hosting, you just put HTML and CFS somewhere and it runs great right?
02:30 OKKEN: Yeah, like for instance I was thinking that it'd be a neat way to do this. I always potentially looking for ways I can use something also, like if I've got a something internal for our test team for instance and we want to look at all of our test data and analytics and all that stuff, having a more dynamic site would be really neat. But then also a periodic generation of like a version report or something as a static site. This would be useful for something like that.
02:59 KENNEDY: Yeah, and some place that I just want to throw out is like an extra for this part that I have no experience with but seems super cool, and I think you also are familiar with this is Netlify. Yeah so Netlify is a place specifically for hosting static sites. So it says build, deploy and manage modern web projects. And it gives you a global CDN, continuous integration, and automatic free HTTPS, I think even your own domain name, 100% for free. There are ways to pay for it but so much of it is for free that this seems like a really good option.
03:32 OKKEN: Yeah, and they're not sponsoring this episode, it's just cool.
03:36 KENNEDY: Nope, that just something that's cool, that's right. DigitalOcean, they are sponsoring this episode. So last week I think it was, we spoke about the new PEP that is for __pypackages__, and you said that's awesome except.
03:50 OKKEN: Yeah, .
03:51 KENNEDY: That's awesome but, and one of the challenges you said, was well if you just want to have packages and put them in there and run your app, that's totally good, it'll pick it up and run. But the big, the thing that was missing was to run the command line binary apps, like pytest, or tox, or pserv, or like uWSGI, all those things that would then get stuffed into that place that is really nearly impossible to put into you path, right?
04:18 OKKEN: Yeah, yeah, well not impossible but...
04:20 KENNEDY: Not impossible but it's a super pain like every time you create a virtual environment to add it. Two of the paths right? So Chad Smith, who had worked on that before, well we mentioned he was part of this python-loc for like a local python project to let you do like a preview interaction with that, sent us a message and said, "Hey, actually you should know about my other project called pipx." I guess for pip executable or something like that. And the idea is it will execute binary python packages in isolated environments. And the closest analogy to this is probably pipsi, which we have spoken about before.
04:56 OKKEN: Oh yeah, okay.
04:58 KENNEDY: So the idea is, yeah you can create these, you can basically instead of pip install a thing, you can pipx install a thing and it will automatically create a virtual environment for you, install it, and it's dependencies, and it's even nice, it makes sure your pip is up to date, which is almost never the case when you create a new virtual environment. And it just installs it there and then it puts it in the path, it puts it in a location on your system that when you install pipx is in the path but is not in the restricted part of your operating system. So you can pip install anything you need to run so you can run it from the command line, in your path, without doing sudo.
05:36 OKKEN: Okay, so the entry points are going to all go from a bunch of different virtual environments and their all going to go to the same place, I guess.
05:42 KENNEDY: Yeah, it does a bunch of like linking back. Some place that's in the path back to the various virtual environments it manages.
05:49 OKKEN: Okay, this is a very cool, now I'm excited.
05:51 KENNEDY: Yeah and I'm even more excited. Yeah this is super cool, you can safely install these packages into virtual environments which is where they belong, globally but without messing up other things, or put them all into one virtual environment that's in your path. You can do things like list the stuff that's installed, so you can see which of the, it's kind of like Brew, you can say what have I installed here through pipx which is pretty cool. I'd also run binaries out of the __pypackages__ directory as well. So it's kind of a companion tool to python-loc as well.
06:21 OKKEN: Okay, nice. Being a Gen-Xer I'm a sucker for anything with X at the end.
06:25 KENNEDY: I know, it's amazing, it's the best. So basically you just pip install pipx and then from then on you can say pip install like ptpython which is an awesome thing you could pip install pytests. You could pip install all sorts of things, glances. There's a couple of ways in which it's sort of like pipsi but it's also better so one of the things I really like, you know I made the analogy to home brew before is you can go and say pipx upgrade all. So everything that you've installed throughout the last six months or whatever pipx, it'll go and find those and see if there's any updates for any of your little Python utilities you've installed.
07:04 OKKEN: Oh, cool.
07:05 KENNEDY: That's cool, right? Then, there's also a little comparison matrix thing, by matrix I mean one dimensional, single line. Anyway, there's a comparison to pipsi and it talks about how pipx is under active development and pipsi isn't and a bunch of other stuff. It's based on Python 3.6 and so on. It uses venv, not virtual environment and there's just a bunch of little niceties and one that made me smile is it prints out emojis when it's doing stuff, it prints out little happy face. Like little indicator emojis on the terminal which you know makes me happy,
07:37 OKKEN: Yeah, what's the other thing that does that? Pipenv does that, right?
07:40 KENNEDY: Yeah, some of the Kubernetes tools do as well. It's a trend I've seen come along here, but there's a lot of smiley, happy little things on the terminal, which doesn't hurt. Anyway, what's the next thing you've got for us?
07:52 OKKEN: This is an article from Vicki Boykis that's called data science is different now. It's a fairly lengthy article, but it's some pretty good information. So here's the situation we've got. We've been talking about data science, everybody's been talking about data science. There's just a lot of buzz around it, so there's a lot of people with just a little bit of training in it going through boot camps or moocs or is that how you pronounce moocs?
08:20 KENNEDY: I think so, yeah. The massive online courses.
08:25 OKKEN: Yeah, okay. They're either self taught or they get a remote degree or some other training and they want to get a data science job, so there's a whole bunch of people trying to get into data science and this partly can become a problem. She notes that the market can be very, very discouraging for this flood of beginners because since there's a lot of beginners, there's a lot of people, the jobs are becoming more advanced. They have more specific requirements and also the job description is sort of vague, so what is it you're actually going to be doing? And when people are actually doing it in the workplace, it seems a lot more like engineering than this shiny, new thing that we've been talking about.
09:11 KENNEDY: That's right.
09:13 OKKEN: It's just kind of a good view of reality check for data science people interested in getting into it. She also ends with some advice, not just a complaining about it but her advice is for people that want to get into it. Her first bullet point is don't get into data science. I actually laughed out loud. I'm like that's awesome advice. But no, basically she said just don't follow the crowd, because following what everyone else is doing is not going to differentiate you and she recommends a back door approach like starting as a junior developer or in dev opps or project manager or something like that and learn a lot of data science stuff on the job and then build that up into a position and that you can work at the company as a data scientist.
10:00 KENNEDY: That sounds like pretty good advice because I feel like data scientists are a little more on their own than software developers. Like at a company there will be five or ten software developers working on one web app or one other automation project. How often do you have five to ten data scientists working on the same problem? I feel they're all doing their own thing a lot of times, which as a beginner can make it more difficult. So maybe get some of these engineering skills through the approach that you we're just talking about and then branch out into that more solo side of things.
10:36 OKKEN: And then she has a whole list of specific skills for let's say you do get a junior developer job or you got some other job which skills should you beef up on? So she's got a list of a bunch of skills to try to beef up on.
10:50 KENNEDY: What are they?
10:51 OKKEN: Creating Python packages, putting R into production which I don't like that one. You should use Python, but anyway. What do I know?
10:58 KENNEDY: We'll talk more about that later.
10:59 OKKEN: Optimizing spark jobs so they run more efficiently, I don't even know what that mean. Version controlling data. That sounds neat. Making data models and reproducible. Version controlling SQL. Cleaning and building.
11:11 KENNEDY: That sounds hard.
11:13 OKKEN: Clean data in data lakes. I love data lakes and data swamps. Just funny. Tooling for time series, forecasting at scale, sharing on Jupyter notebooks, thinking about systems for clean data and lots of JSON.
11:28 KENNEDY: Yeah, that's cool. That sounds good, like good advice. Speaking of good stuff and sponsored episodes, this one is brought to you by Digital Ocean and they, you know it's February 19 as we're recording this Brian and not long ago, five days ago was Valentine's Day right? Digital Ocean had a Valentine's Day present for the world and their present was database as a service. Until now you would go over to Digital Ocean and create a server and you would have to put your own database on there or some other external service like object rocket or something like that, so now you can get managed Postgres databases on Digital Ocean which is pretty sweet if you're going to work with PostgreSQL. You just check the box and they handle all the operational bits for you. It's pretty cool, right?
12:16 OKKEN: Yeah, that is cool. Neat.
12:17 KENNEDY: Yeah definitely. So, check them out at pythonbites.fm/digitalocean. New users get $100 credit to play around with Postgres or whatever else you want to play with. They're a very good service and definitely can recommend them. This next one is pretty interesting, it's not super actionable yet, Brian, but it starts to approach something that I'm really passionate about. The thing I want to tell you about is Rust Python. We have the language Rust, we have the language Python and somehow they come together. The Rust doesn't replace the Python, it replaces the C in CPython. The idea is like heres a new interpreter, a new run time built on rust instead of on C. That feels like a more modern type of language to use than just C to build a foundation and that might be a little bit nicer if they could get that working really well. It's Python 3 only, so 3.5 and above and basically the interpreter is written in Rust. This comes to us from Frederick Arapil, so thanks for sending this over. This looks pretty interesting, I would say it's definitely under active development. When I was getting ready for this recording I looked at the site and Github said last commit an hour ago, so feels like people are working on it. You always wonder is this thing real or are people doing anything? I say that because they're not really ready to like do much, like you an create REPL and you can create some basic Python and stuff but like there's not a great standard library written with Rust foundation and things like that. So there's a lot of work to be done, but the goals are to have a full Python 3 environment entirely in Rust with no CPython bindings or no C bindings really and the clean implementation without compatibility hacks. So if you want to practice some Rust and play around, there's a lot of things that you could do here that mind be fun. I think people who have played with this probably understand regular CPython better as well, right? Obviously, if you reimplement it in another language would be good, but they have a bunch of tasks on the issue trackers and they're labeled very nicely like good first issue and stuff. When people want to get started, they can do that.
14:19 OKKEN: Also can be a fun getting in in the early stages sort of a thing to jump in and help out.
14:24 KENNEDY: Exactly, like in five years if people keep working on this, this will be really hard to contribute to but right now it'd be really easy, relatively speaking. Why do I think this is super cool? Rust comes from Mozilla, and Rust has good support for compiling to WebAssembly, right? WebAssembly is the much better, faster, stronger way of running something in the browser. If this thing got off the ground and it actually worked in a meaningful way and it could be compiled to WebAssembly, well then all of a sudden we have a super awesome Python implementation in the browser on the client side.
15:03 OKKEN: Python really full stack on both ends, the server and the client.
15:09 KENNEDY: Yeah, so it could be super cool and this is one possible path down that. I don't know that that's the goal, but they do talk about WebAssembly being an option in sort of a toy sense because it's not quite ready yet. There's some stuff there. So anyway that was what really got my interest. Another implementation at Rust, kind of interesting I'm like oh wait WebAssembly support. That would be a big deal.
15:31 OKKEN: Maybe in a few years we'll see something like a book called PythonScript, just the good parts.
15:39 KENNEDY: Very cool, alright well what's the next one?
15:42 OKKEN: This is an article, kind of a team up between Mike Driscoll and Real Python so Mike Driscoll wrote an article on Real Python called Jupyter Notebook: An Introduction. Of course, we've talked about Jupyter notebooks before and a lot of the stuff around data and around that. However, this is actually a tutorial that I think was kind of missing and it's not all the cool things you can do with it, it's I know I want to use it. No, really. How do I start? It really starts from installing it of course and then kind of walks through some of the real basics like once you get it up and loaded you've got a blank notebook or blank interface and you can create a new notebook in there and then they even have something simple like I didn't know that it was so easy to change the name just by clicking the name of the untitled notebook that you get right away and you can rename it something else it talks about how to run cells because I think I played with notebooks quite a while before I learned the shortcut of shift enter. It goes through some of the menu options and some of the neat features that might not be obvious if you didn't explore them before. Like, file has a save and checkpoint as you're working in notebook. It's kind of like version control thing built into it, that you can just save at different checkpoints and move on.
17:07 KENNEDY: That's cool.
17:08 OKKEN: The edit menu has, I wouldn't have thought to look there for things like splitting and merging cells or moving cells around. It's in there. The cell menu has a bunch of cool ways to run, I knew you could run a cell or run everything but there's also things like run a subset or run everything above this cell or below the cell or something.
17:30 KENNEDY: Excellent.
17:31 OKKEN: At first when I was reading this, there was a couple questions that I have right away when I start with the notebooks that wasn't covered so I'm going to go ahead push promote these practices right away is when you're installing Jupyter, or if you're installing it with pip do it in a virtual environment and then all the installs within your notebooks will go in that environment and you can keep those separate from the rest of everything else. Once you have a notebook up you can pip install within there now with the % pip install that works now thanks to I think Jake Vanderplast publicized it, not sure if he did the work or not. That's kind of what I really wanted to talk about. This is a good article to send to people if you're working in notebooks and you want to share something with somebody else, they've never experienced them before just send them this also.
18:21 KENNEDY: Yeah, it's a good resource and very comprehensive. Definitely a good one that Mike Driscoll put up there. Now I want to round out the show talking about the Python Developer Survey results from 2018, this is the end of 2018 so this is pretty fresh from the PSF and Jetbrains. We talked about their stuff a year ago when they did it for 2017. This is the next year. Have you seen this yet, Brian.
18:47 OKKEN: I only briefly looked through it.
18:50 KENNEDY: There's a couple of interesting things that I pulled out. One of them is that Python usage as a main language one of the main questions they say is so you're here because you use Python, is that because you use it as a secondary thing, like maybe you're a C++ developer but you use it for dev ops or something like that or do you use it as your main language. The answer of Python is my main language is up 5% from last year.
19:15 OKKEN: That's awesome.
19:16 KENNEDY: Yeah, so now it's 84% of the people and it was 79% last year when the PSF conducted the survey. The other thing to talk about maybe is what do you use Python for? And the number one thing is data analysis and then web development and then machine learning. That's a lot of data science and a lot of Jupyter stuff happening. Between those top three, right? One of the things that's cool about this survey is now that they've been doing it a few years is they have a comparison, like a trend predictor, is the derivative positive or negative on this trend here? I put those numbers down for 2018 and 2017. So for data analysis it's 59% of the people now use that for data analysis. That's up 8% and it's above web development, so you might think it's a zero sum game like more people are not doing web and now they're doing data sites but no web development's up 2% and machine learning is up 7% to our comment and all that right? Pretty interesting. And the other thing they let you do is they let you ask four people who answered this question, lets partition their answers whether Python is their main language or it's just kind of a secondary thing they do, right? So the main takeaway of this section is that web development is the only category with a large gap separating main users of Python verses secondary ones, like 56 verses 36%, 20% difference depending on how you use Python. All the other ones are pretty similar.
20:43 OKKEN: Okay. I want to point out a couple other things. In the survey, 93% say that they're involved with the software testing or writing automated tests.
20:55 KENNEDY: That's pretty cool, right?
20:56 OKKEN: Awesome. Or at least maybe that professional verses hobby. I don't know. Anyway.
21:02 KENNEDY: Yeah, but it's quite high and they said they ask you which test framework you're using? Fot test frameworks, pytest is number one. Of course, that's pretty cool. Another one that's interesting is kind of like last year they said sort of scientific computing, data oriented like data analysis, machine learning and what not. How does that compare to web development and it's now 27% web development, 28% scientific development they call it. It's like data science plus machine learning. Also Python verses Legacy Python, super good news here of the surveyed folks, 84% are using Python 3 verses 16 for Legacy Python.
21:41 OKKEN: Yeah, that's awesome.
21:43 KENNEDY: That's really good and that's also growing very rapidly, so like 9% more are using modern Python which is great. Web frameworks Flask is now the most popular web framework on the survey, DJango was last year.
21:54 OKKEN: Yeah, that's interesting but they're still really close.
21:57 KENNEDY: They're neck and neck for sure. Then it goes Tornado and then Pyramid and then databases Postgres, MySQL, SQLite. Those are the top three relational and then MongoDB is tailing along in the nosql bit. Then ORMs, SQLAlchemy and Django ORM are tied for that. Anyway that's just giving you the surface, there's a ton of cool, little interactive bits you can go explore the survey with.
22:18 OKKEN: Yeah, neat.
22:19 KENNEDY: So good stuff and yeah, people should go and check it out of course the links in the show notes. All right, Brian you got anything else you want to share? That's not one of our main topics?
22:28 OKKEN: Yeah, Nina mentioned to us you can pronounce her name, but I have trouble with this.
22:33 KENNEDY: Sacaranco?
22:35 OKKEN: Yeah, she hit us up and said if we could plug this it would be great. Mentored sprints for diverse beginners or something that's at PyCon I don't know if they've always been a thing but that's something that's available this year. It's newcomers introduction to contributing to an open source project and we have a link in the show notes.
22:53 KENNEDY: Yeah, cool.
22:54 OKKEN: There's a call for applications both for projects and for participants that's open until March 14th. And a highlight if you're wondering if this event is for you, it definitely is and we'd love to have you take part in it. So that's a neat thing.
23:08 KENNEDY: Yeah, it will take place on Saturday, May 4th from 2:30 to 6:30 at the main PyCon so its a great things and I'm glad we get a chance to highlight it. Alright, you got a joke for us?
23:20 OKKEN: I don't have a joke.
23:21 KENNEDY: Alright, I grabbed one. This one plays a little better in cartoon form but I'm going to try it in audio so if you have some pseudo code and like sample.txt or something like that what's the easiest way to convert that to python?
23:35 OKKEN: I don't know.
23:36 KENNEDY: You change the extensions to .py. For some reason the psuedo code is very similarly written to Python and of course it speaks to Python's readability, right? So maybe you got to throw a colon in there somewhere but yeah pretty much. I have an extra, I actually have two extras for you. The first one is this Python song and it takes a bunch of like modern songs and it converts them into like learning chapters so you learn things like you learn about Python basics like looping and variables. You learn about classes, you learn about web design, so there's a little different song for each one, somewhere like Sweden or Norway or somewhere like that. I can't remember, maybe Denmark and they put together this really funny like six and a half minute set of little songs all put together into chapters and it's all about Python.
24:26 OKKEN: I'm totally going to subject my daughter to this.
24:28 KENNEDY: Yeah, yeah, it's pretty funny. And then I'll throw one more in there for ya real quick. Speaking of daughters, my daughter literally sent this in about an hour ago and speaking of all the data analysis and data science folks I think this one might resonate. I'll let you be the judge. So an infinite number of mathematicians walk into a bar. The first one tells the bartender he wants a beer. The second one tells the bartender he wants half a beer. The third one says she want a fourth of a beer. The bartender interrupts and sets two beers on the table and says people, you really need to learn your limits.
25:01 OKKEN: Okay, that's funny.
25:03 KENNEDY: Pretty good, right? Alright, super. Well, Brian these joke segments always make me smile.
25:10 OKKEN: Yeah, definitely.
25:13 KENNEDY: Thanks for doing the episode, as always.
25:15 OKKEN: Alright. Thank you.
25:16 KENNEDY: I guess we will see everyone at PyCascades, right?
25:18 OKKEN: Oh yeah. That's this week.
25:20 KENNEDY: Yeah, we head out in like three days or something like that for the conference. If you're going to be there, hopefully we'll get the chance to meet you. Otherwise, see you some other place.
25:28 OKKEN: Alright, bye.
25:29 KENNEDY: Bye. Thank you for listening to Python Bytes. Follow the show on twitter via @pythonbytes that's pythonbytes as in b-y-t-e-s and get the full show notes at pythonbytes.fm. If you have a news item you want featured just visit pythonbytes.fm and send it our way. Were always on the lookout of sharing something cool. On behalf of myself and Brian Okkin this is Michael Kennedy, thank you for listening and sharing this podcast with your friends and colleagues.