Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


« Return to show page

Transcript for Episode #26:
How have you automated your life, or CLI, with Python?

Recorded on Tuesday, May 16, 2017.

00:00 Michael KENNEDY: Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is Episode #26, recorded May 16th, 2017. I’m Michael Kennedy.

00:00 Brian OKKEN: And I’m Brian Okken.

00:00 KENNEDY: We're here to deliver a bunch of cool news around the Python community. But before we do, I'd like to say, “Thanks,” Brian. I’d like to say, “Thanks” to Rollbar; once again they’re sponsoring this episode.

00:00 OKKEN: Yay, Rollbar.

00:00 KENNEDY: Yay, Rollbar. They're going to be at PyCon, so be sure to drop by their booth and tell them that we sent you.

00:00 Alright, let's talk about the terminal.

00:00 OKKEN: Yeah, when I think of command line interface – I mean use like the REPL and other terminal interfaces – but most of the time when I'm writing applications that I think of as command line, you can launch them from the command line with flags and stuff like that. But if you wanted to do like, an interactive terminal application, like a REPL or something, I don't know how to do that. I ran across this two part series and the first part was just actually talking about (this is on open source.com) and the first article talks about four different applications that have good interface characteristics. It started with the MySQL interface and the Python REPL, which were good, but both could be better. The author talked about bpython as an adds auto completion to Python.

00:00 KENNEDY: Bpython, interesting. I've never heard of bpython. I use ptpython.

00:00 OKKEN: The article also mentions ptpython, so I was going to try both of them out and see which one I liked better.

00:00 KENNEDY: Okay. You have to give us a report.

00:00 OKKEN: Yeah, there's apparently for MySQL, there's a mycli that adds context aware completion. I don't spend enough time in MySQL to worry about that, but it’s in a couple others. If you're a postgress person, there's something called pgcli that adds fuzzy search. Then he talked about bash like we spend our time on the bash command line, and the replacement is called fish, that supposedly has better search history.

00:00 KENNEDY: Oh, nice. Yeah, I’ve heard fish is good. I use Oh My Zsh and also Conch, which is a Python-based one, which is cool as well for the shell.

00:00 OKKEN: Yeah. So, that was the first part. In the second part of this series, it talks about how to do this sort of stuff with Python. He talks about actually taking four different libraries. There's one called the prompt tool kit. You can use that to add a REPL-like interface to your application. You can add things like command history and auto suggestion and auto completion, which sounds cool, actually.

00:00 KENNEDY: Yeah, that actually does sound pretty cool.

00:00 OKKEN: And click; I'm familiar with using click for options and flags, but apparently, it also has a pager and the ability to launch an editor outside of it, which would be cool.

00:00 KENNEDY: That sounds cool, yeah.

00:00 OKKEN: Then a couple more. Pygments for syntax highlighting and fuzzy finder, which is a way to make fuzzy suggestions on whatever somebody types in.

00:00 KENNEDY: That’s cool.

00:00 OKKEN: And the nice thing is, there's an example on how to hook up fuzzy finder and prompt tool kit to make the completion look nice. It is a well written series and when I add interactive stuff to an application, maybe I'll check this out.

00:00 KENNEDY: Yeah, it definitely looks cool.

00:00 You know, CLIs are great for automating things, right? So, I found a Reddit conversation. It's not linking to an article but it's a conversation that I thought was outrageous for some of the things and interesting and inspiring for others. It's called, “How Have You Automated Your Life with Python.” So, this person throws this out as a general discussion on Reddit and some people come and say, ‘You know, there’s something magical about writing code that interacts with the physical world.’ Some of them are totally straightforward. There’s somebody that said, ‘Hey, I take the train to work (or something) so I run a script every five minutes between 5 and 5:30 which scrapes the train web sites and sends me a notification if my train is on time or delayed or canceled or something like that.

00:00 OKKEN: That's pretty cool.

00:00 KENNEDY: To your Slack channel you’re like, ‘You better get going, your train’s showing up early or it’s late or whatever.’

00:00 So, that was pretty standard, right? There's another one that's little more playful. A person wrote a script to check if a nearby ice cream shop is stocking their favorite flavor by screen scraping their menu.

00:00 OKKEN: Oh, nice.

00:00 KENNEDY: (Laughs) ‘Is my ice cream in stock?’ ‘Yes.’ ‘Alright, I'm going to get some.’ That person is passionate about their ice cream.

00:00 OKKEN: Yeah, well have you been to Salt & Straw?

00:00 KENNEDY: Yes.

00:00 OKKEN: I'd like something like that for Salt & Straw. That’d be great.

00:00 KENNEDY: Yeah, for sure. I wouldn't want the ice cream flavor, I would want the line length.

00:00 OKKEN: Oh, yeah, definitely. Those of you who are coming here for PyCon, you’ve probably heard of Voodoo Donuts, which is like an iconic donut place Portland. You'll see people with these pink boxes all over the place. Salt & Straw is like the ice cream equivalent. Even in the rain, there will be like a 30-person line outside. It’s crazy.

00:00 OKKEN: Yeah, and Blue Star’s better than Voodoo.

00:00 KENNEDY: Woah. You’re going to start some fights here. Let’s just move on. (Laughs)

00:00 So, these are all pretty standard. There was somebody that said, ‘I have a script that generates weekly status emails off my git commit messages and creates time sheets in Harvest based on that.’ Harvest is time tracking, like, ‘I spent 3 hours on this.’ That's pretty cool.

00:00 The one that like made me laugh the most, and I don't think this is meant to be all creepy but it could come across that way, ‘I recently wrote a quick Python script that tells me when my girlfriend comes home.’ It says it sniffs the DHCP traffic on the wifi network and looks for her WiFi ID. So, her phone gets home and it auto joins the WiFi and it sends him a note. Like, ‘Hey, girlfriend’s home. Clean up the sink before you get yelled at.’ Something like that.

00:00 OKKEN: A few weeks ago, we did one on notifying you when your friends order pizza.

00:00 KENNEDY: Yes, exactly. This is kind of like that. That's pretty interesting, but also kind of weird. (Laughs)

00:00 There's a bunch more, I only grabbed some of them. That's pretty interesting. If you've automated something amazing, go and stick it up there. Throw it into the the chat, the discuss, on this episode page.

00:00 OKKEN: Speaking of automating your life a little bit, I ran across this article and it’s by the PyBites people. It's called, “Building a Simple Birthday App with Flask-SQLAlchemy. The idea, which I loved, behind it is, they were tired of relying on Facebook to know when all the birthdays are coming up and I totally agree with that. Also, it's just kind of a fun example for building a Flask application, which actually might be useful for people. I used to have a phone app that did this and I don't know what happened to that, so I think I might actually check this out to build. I've never played with SQLAlchemy, maybe I'll try this out.

00:00 KENNEDY: Yeah, that sounds really fun. Nice.

00:00 That definitely is another way to automate your life. What we have coming up is also a maybe an automated spelling, you might say. But before we do let me tell you guys about Rollbar.

00:00 As you all know, I use Rollbar for all of our sites. Rollbar is constantly watching. If something crashes on the site, I get an immediate notification, maybe in Slack, maybe my phone; all sorts of stuff. So, if you run web applications or even apps that you distribute, you owe it to yourself to work with Rollbar. Go to Rollbar.com/pythonbytes and you can sign up. It's super easy to add like Pyramid, Flask, Django. They have quick starts for all of them, it's like really, really no work. And those guys are going to be at PyCon, just like Brian and me, and they are inviting you to stop by their booth, get some swag, check them out, get a demo… things like that. I'll check out their booth at PyCon, how about you?

00:00 OKKEN: I definitely will. I’ll try to bring them some of my stickers.

00:00 KENNEDY: I’ll bring a sticker over there, as well. We're coming, Mike, were coming! (Laughs)

00:00 Alright, so the thing that I have next is this blog post that got sent to us and it's really interesting. First, I thought, okay, this is like kind of a somewhat interesting project that might be worth talking about, but it's probably just something really basics, someone learning a program. But in fact, I think it has a lot of interesting lessons contained within it.

00:00 The article is, “Spelling with Elemental Symbols.” So, we know that the periodic table has a bunch of one to two character words or letters representing elements, right? Like, Na for sodium, H for hydrogen, and so on. The idea is, can you take a regular word and spell it atomically? You know, NaH, right? Something like that. Can you take the stuff that's on the periodic table and make words out of it, right? That's basically what this does. It sounds kind of interesting, but there's a couple of things that are worth looking at. One is, you can look at a word and you can break it up into this piece and say, ‘Okay, maybe it's a 4-letter word that consists of 1,1,1,1 possible letter arrangements, or it could be 2,1,1 or 1,1,1,2, right? There's different ways in which you might pull those letters into the chemical elements. But if you look at the length of a word, say there's 10 characters in a word, how many possible combinations of that are there? Would you guess?

00:00 OKKEN: Yeah, I'm really bad at math.

00:00 KENNEDY: Well, not so much the number but the formula for it, like the concept for it. It turns out that the number of these arrangements, based on the number of characters, is the Fibonacci Sequence.

00:00 Isn't that crazy? So, if you have like a 5-letter word. there's whatever the sixth value of the Fibonacci numbers is, that's how many possible options are; 10 letters of the 11th Fibonacci and so on. It's like N+1, basically. Isn’t that crazy? I want randomly grab stuff off the periodic table, Fibonacci. Okay, so that's pretty interesting, but it turns out the guy was doing a bunch of work to sort of break these apart. The really computational part is given a word. How many of these Fibonacci-numbered partitionings are there? That's where most of the work was, so he said, ‘Okay, well, I try to optimize it in a couple of ways.’ And he does a lot of interesting performance profiling. So, you can learn how to profile your code there, which is really cool. It turns out it was really slow doing that, so he said, ‘Well, what else can I do? Maybe I can use Memoization.’ Have you heard of Memoization?

00:00 OKKEN: Don’t quiz me on it, but yes.

00:00 KENNEDY: So, if you have a function that is stateless, it just works on its input parameters and then it gives you back a number or a value based solely on the input, you can use this idea called Memoization to say, basically, cash input arguments and then the potentially expensive output. So, if you get a word of link 7, well, you know what the 7 Fibonacci arrangement of these things is, and you don't ever have to compute it again. So, he's like, ‘Let’s try this to make it faster’ and it turns out that it did make it faster, but like 30%. Whatever his data set was, it went from 22 minutes to 16 minutes, or something to this effect. That was better.

00:00 Then he said, ‘Let’s rethink this algorithm,’ switch to a directed acyclic graph using recursion and the complexity went from O(2^n) to O(n) and the time to compute these weird spellings, if you will, went 16 minutes to 10 seconds, which is really cool. He takes this pretty easy to understand problem and works through it and does Profiling, Complexity analysis, Memoization, Recursion, Graphs. There's just a lot of stuff to learn. If you're kind of getting started with some of these ideas, this is a very approachable way to get at some of it.

00:00 OKKEN: That's what I love about this article, it's not really that you have this need for this application. It's this kind of a problem and how do you approach it from a computer science standpoint and an engineering standpoint of faster is better and getting down to it. It is pretty cool.

00:00 KENNEDY: Yeah, it is really cool and is there something super rewarding about writing code that takes 22 minutes to run and getting that down to 10 seconds. You’re just like, ‘Yes, I beat you. I beat you, problem.’ Now to just got to get it run on GPUs and get it up sub-second.

00:00 OKKEN: Sub-second, yeah.

00:00 KENNEDY: So, speaking of helping people learn and get started, you've got something cool for us.

00:00 OKKEN: Yeah, well actually, I'm not sure if it's cool or not. I ran across another Reddit comment thread. This one is a discussion about an IDE (Integrated Development Environment) called, I think it's ‘Tony,’ ‘Thani’? THONNY. Any guesses on how to pronounce that?

00:00 KENNEDY: I'm going with Tho-nee.

00:00 OKKEN: I'm not sure. There was a discussion about the merit of this particular IDE, and I haven't tried it, I have to be real honest, but I have mixed feelings about encouraging beginner IDEs. I guess mostly I don't know if it's a good idea or not. I think learning how to do use a decent editor is kind of part of learning how to be a programmer. Actually, I was just curious about what you thought about beginner editors?

00:00 KENNEDY: I did look at this and I also had mixed emotions about it. It’s interesting, but I'm not entirely sure that I would recommend it. I don't really know why, but I definitely think it's quite interesting to see what it does. There’s a little video and stuff, you guys can click at the link and check it out. But imagine an IDE where you have a Python file and you type in it and then to the right of it, like a little tab that says, ‘These are all the variables that are in your code.’ That's kind of cool that you can actually see the variables that are at work while you're writing your code. Now, obviously, we don't need that these days, but when you're new, those things get lost, right?

00:00 OKKEN: Yeah.

00:00 KENNEDY: So, that's really nice. It has like a single button debugging experience, which is pretty cool. And then it has wire debugging and it has a shell. You can type stuff in the shell, like a REPL, basically, and it will I let you explore the state of the program, I believe, if I understand that correctly. So, those are all pretty interesting things.

00:00 OKKEN: Okay, maybe I'm changing my mind because full-blown IDE like visual studio or something, or even PyCharm, might be a little overwhelming for somebody just starting out. But learning the concepts that an IDE might be able to give you, maybe that's a good thing.

00:00 KENNEDY: Yeah, maybe it is. One thing that's cool about it is, it comes with Python 3.6 built in and it is a Windows app. So, you just install it and you open it up and then boom, you have Python 3.6. You don't have to think about, ‘Oh my God. Vcvar is up. Where is the path? Why is this not working?’ All that that grief just kind of goes away and that's pretty cool.

00:00 OKKEN: Yeah. It probably doesn't have the vim key bindings, though.

00:00 KENNEDY: No, probably just emacs. (Laughs)

00:00 OKKEN: Emacs? You think so? (Laughs)

00:00 KENNEDY: I don't know.

00:00 OKKEN: Probably not and it's probably just a GUI thing.

00:00 KENNEDY: Yeah, probably. I'm not going to do anything with it, but it definitely looks interesting if you are trying to teach people, especially maybe kids. I'm not entirely sure, but I think it might be good for that group.

00:00 OKKEN: And the IDEL is kind of in that same category, but I guess I brought this up here because I'd like to hear some comments from our listeners as to whether whether they encourage people to use beginner IDEs when they're teaching programming or not.

00:00 KENNEDY: Yeah, for sure. Leave a note on the end of the show notes or the show page, pythonbytes/26 for this one.

00:00 Alright, last thing. I have some salvation for the PDF people out there. So, especially if you are in data science or doing web scrapping and you’re trying to get data, often it’s bundled up in these reports which are in PDFs. The support for actually going into these PDFs and getting them out in Python has not been good. I’ve had a lot of people say, ‘I've tried this, I've tried this, I've tried this, none of them work. Do you have any ideas, Michael?’ And I say, ‘Nope, no ideas.’ So, there's recommendation on Twitter that I thought was really good, from Dan Shorstein. He said, ‘Hey, you should check out this thing called PDF Plumber.’ You can go in and you can extract graphs, you can debug your processing of the PDF file visually. So, if you're trying to extract a table, you can go and say, ‘Highlight the thing I'm looking for like this’ and it will literally show you a PNG (Portable Network Graphic) with like rectangles around the pieces you're getting, which is pretty cool. So, you can extract a bunch of tables, you can work with the shapes, you can group stuff together. Somebody did a master's thesis on this and a lot of those ideas are brought into it.

00:00 If you're out there doing data science and or something, where the data is wrapped up in PDFs and you've been having a hard time getting it out, check out PDF Plumber.

00:00 OKKEN: Great. Awesome.

00:00 KENNEDY: Yeah, very good. So, that's our news for this week. You've got, what like a day, to finish the book? How’s that?

00:00 OKKEN: (Laughs) No, I turned it in already.

00:00 KENNEDY: You did? Man, congratulations. How does that feel?

00:00 OKKEN: It's not done yet, so the first five chapters will be released, unofficially, in stealth mode, on the 17th.

00:00 KENNEDY: Awesome and that's tomorrow. Just in time for PyCon.

00:00 OKKEN: Or either that, or today, or yesterday, depending on when people get this. (Laughs)

00:00 KENNEDY: Could have been 3 weeks ago, if you listen to a late. (Laughs)

00:00 OKKEN: Yeah, but definitely by time I'm hanging out of PyCon, it will be available. So, this is the Python testing with Pytest, even as is first five chapters are like ten times better than anything I've ever read before.

00:00 KENNEDY: That’s awesome. Congrats.

00:00 OKKEN: How about you? What's going on?

00:00 KENNEDY: Well, I am really looking forward to being at PyCon. I'm gonna be there Thursday night. We have our booth there and Thursday, Friday, Saturday, Sunday, we will have our booth set up. People can come hang out with us, come talk to us and will be doing some open sessions and things like that, so it will be a lot of fun.

00:00 OKKEN: Yeah.

00:00 KENNEDY: I'm definitely looking forward to that, being there with you and all the other podcasters. I actually just finished recording my “RESTful/HTTP Services in Pyramid” course. That's all totally done, I'm just waiting on the final edits to publish that to the world. I'm now working on my “MongoDB for Python Developers” course so, I'm frantically writing as fast as I can before PyCon overwhelms me.

00:00 OKKEN: Oh, nice.

00:00 KENNEDY: Alright, that’s all I’ve got. Well, welcome back to the U.S. and I will see you and everyone else at PyCon.

00:00 OKKEN: Yeah, see you in a couple days.

00:00 KENNEDY: Bye.

00:00 Thank you for listening to Python Bytes. Follow the show on Twitter via @pythonbytes and get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbyes.fm and send it our way. We’re always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page