Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


« Return to show page

Transcript for Episode #56:
The pendulum of time swings beautifully in PyPI

Recorded on Wednesday, Dec 13, 2017.

00:00 KENNEDY: Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is Episode #56, recorded December 12, 2017. I’m Michael Kennedy.

00:00 OKKEN: And I’m Brian Okken.

00:00 We’ve got a bunch of awesome stuff, like always, lined up for you. Before we get to it, Brian, let’s just say thanks to Rollbar.

00:00 Thanks, Rollbar.

00:00 Yeah, thank you, Rollbar for supporting the show. I’ll tell you more. About that they’ve got going on later.

00:00 want to hear the tick tick of the pendulum, if you will.

00:00 I was reading an article called, “Dealing with Datetimes Like a Pro in Python.”

00:00 Which can be way harder than it should be, right?

00:00 Yes, and I think that there’s things that like arrow and maya so you don’t have to deal with datetimes directly, but this is a nice article on dealing with datetimes if you want to use the library. However, at the end of it they mention Pendulum, which I hadn’t run across before. So, checking it out, there’s a link here for “Pendulum Datetimes Made Easy.” The website is beautiful. Also, the interface to it, it looks really nice. You can do things like import Pendulum then Pendulum Now in Europe or Paris or something. Then you can do things like add dates together and subtract dates, the normal things you have to do with datetimes. The model looks nice. I like it.

00:00 Yeah, it looks really nice and website is quite stunning. To me, on the one hand, it’s documentation. On the other, it really shows the people working on this really care about the project and it’s not just some random thing thrown on GitHub. So, yeah, it’s really cool. You say pendulum.now and you give it a time zone like Europe/Paris. And then you subtract weeks from it. One thing I like to do is put times relative to humans, like a week ago, five seconds ago, ten minutes. So, it just has a diff for humans, that’s really nice.

00:00 I’m going to include in the show notes, links to arrow and maya. I think it's a matter for developer taste really and what looks more natural to you, so check them all out and see what’s nicer for you.

00:00 Yeah, it’s really nice. The other one to throw in here while we’re thinking about all this is parsing datetimes, which I think there’s 700 different formats for datetimes in texts, which is insane. I really like to use Python dateutil. So, pip install Python-dateutil, but just import dateutil. It can parse lots and lots of stuff. You just go, ‘Here. Load that into the time.’ That’s one more thing, but this is really slick. It has more time delta things, how many weeks, how many days, how many hours. As opposed to just the total seconds from the built in one. Really nice. I like it. Good find.

00:00 the next thing that I want to talk about… We’re going to talk about some web stuff, aren’t we? The next one I want to do is sort of a web thing but it doesn’t really have to be. It’s any kind of service or long-running thing. One of the things that can be a challenge with web applications is long running tasks or tasks that are slow but you don’t really need to wait on their outcome. I’ll give you an example.

00:00 my training website, training.talkpython.fm, there’s a capability to email everyone in a class. It turns out, if you, in the same request, actually email thousands of people one at a time, like through Amazon SES or something, that request times out. Then you’ve emailed some people and you don’t know how far you got and you’ve got to go figure out how to rewrite that some other way. That’s a problem. There are certain things like email, like logging, like other sort of ‘kick this job off and forget about it’ that would be much better served in some sort of asynchronous way. One of the best things you can do to add scalability to your app is to ‘push those kind of things onto some background and get to it when you can’ style and probably some formal way is to use some kind of queuing.

00:00 thing I want to talk about now is an article called, “Flask Asynchronous Background Tasks with Celery and Redis.” Although it has nothing really to do with Flask, it uses Flask as an example. It could be Django, it could be Pyramid, but it’s the Celery bit that’s interesting. So, you take Celery as the worker. It’s going to run back there and it’s going to look at the Redis message bus and what you do is you just kick jobs off to it, and it will start pulling them off and working on them. So, what I could have done is said, ‘Oh, I want to email 3,000 people.’ Go to the database, get the 3,000 people and just put, ‘3,000 email this person jobs’ onto a Celery queue and been done straight away. It would do it over the next minute or two.

00:00 Actually, that’s pretty cool. The wheels are turning in my head thinking about things I want to use Celery for.

00:00 I know, I feel the same way. It’s a really nice write up of how you walk through these things. It talks about using Celery client to connect Flask to Celery. The Celery worker process, the message broker that works with the Redis back end. And the other thing that’s interesting about this, so often the Python examples – especially the Python web examples – are done on MacOS or Linux, and this is all on Windows.

00:00 Wow.

00:00 And I don’t think Redis is even supported on Windows. Or Celery, one of them. I think it’s Redis. There’s an unofficial build or something. I think it’s extra interesting to show this other angle and show people how to do it where it might not be so obvious.

00:00 Also, something like taking on Celery, it’s kind of nice to look at an example project to see how it’s been used because just jumping right into the entry of the documentation can be a little scary sometimes.

00:00 Here’s a nice, little, simple example. Take it and do what you will with it.

00:00 Yeah, well we have another web example that I liked and this is from Bob Belderbos of PyBytes. He wrote a guest article on RealPython called, “Building a Simple Web App with Bottle, SQLAlchemy, and the Twitter API.” The reason why this caught my attention is because I was curious about having some way to interact with Twitter and Python. I think I’m going to walk through this a little bit. From start to finish, it’s a project that isn’t pushing things into Twitter, it’s reading Twitter and then displaying them on a little Bottle website that they pushed up to Heroku. It’s kind of cool. They document everything from start to finish for this. It’s neat.

00:00 Yeah, it has the deployment bit as well, pushing it to Heroku and making it work there. Which is also kind of cool because your web apps are fun but they belong on the web, not your laptop.

00:00 It’s on my Celery queue of ‘things to learn’ because I’m busy learning Flask at this time, but maybe I’ll check this out next.

00:00 Yeah, it sounds really interesting. Nice job, Bob.

00:00 before we go on to talking about something that’s actually not on the web but has to do with Python, let me tell you a quick story of yesterday. Yesterday, I decided, ‘Let me see how far I can push the performance of MicroWSGI and Nginx on the Talk Python sites.’ Luckily for you, Brian, I didn’t do this on pythonbytes.fm. ‘Alright, I think I can push the parallelism out better. I can crank up the number of threads. I can crack open the process affinity and sister affinity. So, get the maximum parallelization out of MicroWSGI, which is one of the better application-tier, worker process web servers for Python. That seemed like a good idea, right?

00:00 Yeah.

00:00 Turns out, something about the threading is bad and I don’t know what it is. I don’t really care to look into it, but I started getting Rollbar notifications saying that the template files could not be found for the website, which has been largely unchanged for months. And then, it couldn't convert this template to a string and all sorts of random errors. I’m like, ‘What is going on?’ And there’s some kind of corruption timing problem in this whole series. It happened like 1 out of 100,000 times. Really rare. But it started coming in a couple of times yesterday, so I’m like, ‘I guess we’re going to turn that back down.’ I turned it back down, errors went away. Without Rollbar, I never would have known it.

00:00 you guys want to make sure your apps are working, go check out Rollbar. Just pythonbytes.fm/Rollbar and sign up. You, too, may start getting weird messages that you need to go look into. It will be awesome. (Laughs)

00:00 (Laughs) Yeah, nice.

00:00 So, that was yesterday. Today, I want to move on and talk about something I actually covered way back on Talk Python, about 40 episodes ago. Not quite a year, 40 weeks ago. That was the Visual Studio Code extension, written by a guy named Don Jayamanne. He wrote a really cool Python extension to basically add Python capabilities to VSCode. And it turned out to be the first or second most popular thing in Visual Studio Code, and it had millions of downloads. That’s pretty cool, right?

00:00 That’s very cool.

00:00 One of the things I looked at when we spoke, it’s pretty interesting, he took maybe ten different packages – Jedi, Flake8 and those types of things – and wove those together into an extension for VSCode. He didn’t really build a huge amount but he definitely put it together in a really usable way. So, that was pretty cool.

00:00 this week, the news is that he’s now officially hired by Microsoft because of his Open Source project.

00:00 That’s nice, really cool.

00:00 Yeah. So, here’s a guy who started an Open Source project and got this pretty influential job just because he had a really successful plugin for this Open Source thing. Pretty Cool.

00:00 doesn’t really mean many changes, there’s just a few more people working on the Python extension for VSCode, which if you’re into VSCode, it’s a lot like Atom or Sublime. It’s quite nice if that’s your style editor. It does a lot of cool stuff. Basically, there will be more people working on it, fewer bugs, faster turnaround.

00:00 other big news is that they’re also hiring for Visual Studio Code Python developers. If you’re looking for a job and that sounds pretty awesome, there’s some really cool guys over there. There’s Brett Cannon, Steve Dour, some of the big hitters on the Python core dev team, in addition to Don. So, that might be worth looking into if that interests you.

00:00 feel like you’re doing this kind of web crash course, getting into a lot of different things on the web, right?

00:00 I’m trying to embrace the web a little bit more.

00:00 Nice. What did you find?

00:00 One of the things we talk about a lot is mechanics of how to get websites up with Flask and Bottle and Django an all that. Sometimes the design of your website as it turns out at the end, leaves a little bit lacking.

00:00 It looks like a developer designed it.

00:00 Yeah, it definitely looks like a developer designed it. Which is usually better than when it looks like a database engineer designed it, but still.

00:00 an article called, “A Comprehensive Guide to Web Design” on Smashing Magazine. It’s a long article but it’s all one page. It’s a good overview of the basics of ‘Make sure you have this stuff down to make sure your website doesn’t suck to bad.’

00:00 That’s really cool. I feel like design is one of those things that feels like a barrier to developers. ‘I’m not a designer. I can’t do stuff that’s pretty. I can’t build these types of things.’ But if you put in just a little effort, you can turn that barrier into a superpower because most people don’t have it. And it’s actually not that hard to sit down and focus and try to study it for a month or something.

00:00 Especially a lot of times there’s a lot of projects that get done – like internal work projects or different tools for a small group of people – that you don’t have the money to hire a front end designer. So, something like this to go through and make sure that you do your breadcrumbs appropriately and you put your search box in a definable place. Some of these page structure overviews are good to make it more usable. This doesn’t add a lot f work to your workflow, it actually takes away some of the stress of designing a web page if you’re not a designer.

00:00 Yeah and it’s a cool framework to think through of stuff you have to do. It’s cool.

00:00 item, what would you name the baby of Requests and Selenium, coming together to create some new library?

00:00 (Laughs) Well, I know the answer.

00:00 (Laughs) Requestium maybe one of the candidates you’d throw out there, right?

00:00 thing I want to talk about is Requestium. We have Requests, which everybody knows is a super popular library for downloading HTML. And we have Selenium, which fewer people know, but it’s basically a headless, Chrome browser. There’s other options as well, let’s say Chrome for now. And you control it from Python. So, use them in different situations. Requests is super lightweight, takes very few dependencies; it doesn’t depend on say, Chrome. It just depends on a few packages. And you can download the HTML and do what you will with it. But if you happen to run into, let’s say, an angular JS, or VJS, or any other JS site – it’s a front end JavaScript framework that basically requires the JavaScript to execute to get the content – you end up with a bunch of curly braces ({}) that have no meaning when you hit it with Requests.

00:00 I didn’t know that.

00:00 Yeah, you just get the HTML that’s basically, ‘Here’s where JavaScript will fill out the holes. Good luck with that.’ So, that’s kind of useless. You can use Selenium, but Selenium is way more heavyweight.

00:00 basically merges the power of Requestium and Selenium and Parsel, which is like BeautifulSoup, Scrapy. It understand HTML and let's you ask questions of it. The idea is that it’s written for mostly when you’re doing Requests but that one time or to when you’re like, ‘This login sequence has to happen in a richer, JavaScript way.’ It lets you do crazy things like automatically switch between Requests sessions and Selenium sessions. It lets you basically share cookies across them and all sorts of interesting stuff like that. So, if you need a little bit heavier weight than Requests but not full on Selenium, here’s a nice marriage of the two.

00:00 That’s nice.

00:00 Yeah, so very good for either testing or web scraping or some other types of things. If you need to test something that’s JavaScript heavy, this is a pretty good option.

00:00 Nice.

00:00 Well, that’s out news, Brian. At least the ones we’ve selected for everybody else. But how about you? What do you have going on?

00:00 Well, I finally got a Testing Code episode out, not too long ago. I don’t know if I brought it up already.

00:00 I don’t think so. Not here.

00:00 It was a nice interview with Katharine Jarmul, or Kjam, about testing and data science. It’s Testing Code Episode #33 and it’s been very well received so far. A lot of people gets some value out of it.

00:00 That’s great. Katharine’s done a lot of cool work. I’ll definitely go over there and check it out.

00:00 And the last thing I want to mention that I had asked people to leave Amazon reviews for books that they like, including mine, hopefully. So far, nine people left reviews on Python Testing with pytest, so thanks, a lot.

00:00 That’s great.

00:00 How about you? Any news?

00:00 Not a whole lot of news. I’m working hard but I’m getting things ready to talk about. Nothing to talk about yet but someday soon. The big thing I have coming up, I just recorded some nice Talk Python episodes. One on optimizing web servers and then later this week I’m recording one which is a panel session on machine

00:00 and artificial intelligence at the Large Hadron Collider and particle physics.

00:00 That sounds interesting. The last few episodes you put out – you always have good episodes – but I’ve really enjoyed the last few quite a bit.

00:00 Thanks, same for you.

00:00 man. Well, thank you for sharing this stuff and everyone, thank you for listening.

00:00 Thank you.

00:00 Thank you for listening to Python Bytes. Follow the show on Twitter via @pythonbytes and get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We’re always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page