#56: The pendulum of time swings beautifully in PyPI
Sponsored by Rollbar! pythonbytes.fm/rollbar
Brian #1: Pendulum for datetimes
Michael #2: Flask asynchronous background tasks with Celery and Redis
- Easiest way to a significant scalability to your app: queuing
- What is Celery: Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well
- How Celery works:
- Celery client: This will be connect your Flask application to the Celery task
- Celery worker: A process that runs a background task
- Message broker: The Celery client communicates to the Celery worker through a message broker (redis in this case)
- All examples on Windows
Brian #3: Building a Simple Web App With Bottle, SQLAlchemy, and the Twitter API
- Guest article on RealPython, by Bob Belderbos of PyBytes
- Fun full project start to finish using Tweepy to load tweets.
- Ends with a bottle app running on Heroku
Michael #4: Python extension for VSCode updated, now brought to you by Microsoft
- Don Jayamanne, creator of the Python extension for Visual Studio Code, joins Microsoft
- Full announcement: https://blogs.msdn.microsoft.com/pythonengineering/2017/11/09/don-jayamanne-joins-microsoft/
- Had Don on Talk Python back on episode 101.
- What does Microsoft Python team publishing the extension mean?
- For all practical purposes the transition should be transparent to you. Additionally:
- The extension will remain open source and free
- Development will continue to be on GitHub, under the existing license
- More dev resources means (generally) faster turnaround on bug fixes and new features
- Microsoft is hiring for Visual Studio Code / Python! They are hiring devs immediately to continue and expand work on our Python support for Visual Studio Code. If you are passionate about developer tools and productivity, this could be an ideal endeavor!
Brian #5: A Comprehensive Guide To Web Design
- Crash course in web design principles, not the mechanics
Michael #6: Requestium
- Integration layer between Requests and Selenium for automation of web actions.
- Merges the power of Requests, Selenium, and Parsel into a single integrated tool for automatizing web actions.
- The library was created for writing web automation scripts that are written using mostly Requests but that are able to seamlessly switch to Selenium for the JavaScript heavy parts of the website, while maintaining the session.
- Features
- Enables switching between a Requests' Session and a Selenium webdriver while maintaining the current web session.
- Integrates Parsel's parser into the library, making xpath, css, and regex much cleaner to write.
- Improves Selenium's handling of dynamically loading elements.
- Makes cookie handling more flexible in Selenium.
- Makes clicking elements in Selenium more reliable.
- Supports Chrome and PhantomJS.
Our news
- Test & Code 33: Testing in Data Science with Kathrine Jarmul
- Thanks to the 9 folks to that have left an Amazon review for Python Testing with pytest.
Episode Transcript
Collapse transcript
00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to
00:05 your earbuds. This is episode 56, recorded December 12th, 2017. I'm Michael Kennedy.
00:11 And I'm Brian Okken.
00:12 And we got a bunch of awesome stuff like always lined up for you. Before we get to it, Brian,
00:16 let's just say thanks to Rollbar.
00:17 Thanks, Rollbar.
00:17 Yeah, thank you, Rollbar, for supporting the show, tell you more about what they got going
00:21 on later. I want to hear the tick-tock of the pendulum, if you will.
00:26 I was reading an article called Dealing with Date Times Like a Pro in Python.
00:30 Which can be way harder than it should be, right?
00:32 Yes. And I think that there's things like Arrow and Maya that try to get so that, you know,
00:38 I really have to deal with date times directly. But this is actually a nice article on dealing
00:43 with date times if you want to use the direct library. However, at the end of it, they mentioned
00:48 Pendulum, which I hadn't run across before. So checking it out, there's a link here for the
00:53 Pendulum Python Date Times Made Easy. And the website is beautiful. But also the interface
01:01 to it, it looks really nice. I mean, you can do things like just import Pendulum and then
01:08 Pendulum Now in Europe, Paris, or in America, or something. And then you can do things like
01:15 add dates together and do, you know, subtract dates. And the normal things you have to do with
01:21 date times. But the just the model looks nice. I like it.
01:24 Yeah, it looks really, really nice. And the website is quite stunning. And you know, to me,
01:30 like on one hand, like, okay, whatever, it's documentation. On the other, it really shows the
01:34 people working on this actually care about this project. And it's not just some random thing
01:38 thrown on GitHub. So yeah, it's really cool. So you say like Pendulum.now and you give it a time
01:42 zone, say like Europe slash Paris. And then you can subtract weeks from it. Or you can,
01:47 one thing I like to do is put times relative to humans like a week ago, five seconds ago in 10
01:53 minutes. And so it just has a diff for humans, you know, and that's really nice.
01:58 I'm going to include in the show notes links to Arrow and Maya. And I think it's, it's a matter of
02:03 developer taste really, and what looks more natural to you. So check them all out and see what's nicer
02:09 for you.
02:09 Yeah, it's really nice. The other one to throw in here while we're thinking about all this is parsing
02:13 date times, which I think there's 700 different formats for date times and text, which is insane.
02:19 But I really like to use Python date util. So pip install Python dash date util, but just import
02:25 date util. And it compars like lots and lots of stuff. So you can just go here, load that into time,
02:31 you know, and that's, that's one more thing. But this is really, really slick. It has more
02:35 sort of time delta things like how many weeks, how many days, how many hours, as opposed to just like
02:40 the seconds, total seconds, the built in one really nice. I like it. Good find. So the next thing that
02:45 I want to talk about, we're going to talk about some web stuff, aren't we? So the next one I want
02:48 to do is, it's sort of a web thing, but it doesn't really have to be it's any kind of service or long
02:53 running thing. So one of the things that can be a challenge with web applications is long, long
03:00 running tasks or tasks that are slow, but you don't really need to like wait on their outcome.
03:04 So I'll give you an example. On my training website, training.talkpython.fm on that site,
03:10 there's a capability to email everyone in a class. And it turns out, if you in the same request,
03:19 actually email thousands of people, one at a time, like through Amazon SES or something that
03:25 request times out, and then you've emailed some people and you don't know how far you got,
03:31 you've got to go figure out how to rewrite that some other way, right? That's a problem.
03:35 Yeah. So there's certain things like email, like logging, like other sort of kick this job off and
03:41 forget about it, that would be much better served in some sort of asynchronous way. One of the best
03:46 things you can do to add scalability to your app is to push those kinds of things onto some background,
03:50 sort of get to it when you can style. And probably the most formal way is to use some kind of queuing.
03:57 So this, the thing I want to talk about now is this article called Flask, asynchronous background
04:01 tasks with Celery and Redis. Although it has nothing really to do with Flask, although it just
04:06 uses Flask as an example. It could be Django, it could be Pyramid, like really it's the Celery bit
04:10 that's interesting. So you take Celery as the work, the worker, right? It's going to run back there.
04:15 It's going to look at the Redis message bus. And what you do is you just kick jobs off to it,
04:21 and it'll just start pulling them off and working on them. So what I could have done is said,
04:24 I want to email, you know, 3000 people. So go to the database, get the 3000 people and just put 3000
04:30 email this person jobs onto a Celery queue and been done straight away. And it would just, you know,
04:36 do it over the next minute or two. Actually, that's pretty cool. Yeah, I've got my the wheels are
04:40 turning in my head thinking things that I want to use Celery for. I know, I feel the same way. It's
04:44 really nice write up of how you like walk through these things. So it talks about using the Celery client to
04:48 connect Flask to Celery, the Celery worker process, the message broker that works with the
04:54 Redis back end. And the other thing that's interesting about this is so often these Python
04:59 examples, these especially Python web examples are done on macOS or Linux, and this is all on Windows.
05:05 And I don't think Redis is even supported on Windows or Celery, one of them. I think it's Redis. It's not,
05:11 there's like a, an unofficial build or something. So I think it's kind of extra interesting to show
05:16 this other angle and show people how to do it where it might not be so obvious. Also something
05:21 like taking on Celery, it's kind of nice to look at an example project to see how it's been used,
05:26 because just jumping right into the, the entry of the documentation can be a little scary sometimes.
05:31 Here's a nice little simple example and take it and do what you will with it. Yeah. Well,
05:35 we have another web example that I liked in it. This is from Bob Belderboss of PyBytes. He wrote a
05:43 guest article on real Python called building a simple web app with bottle SQLAlchemy and the
05:49 Twitter API. And the reason why this caught my attention is because I was curious about having some
05:55 way to interact with Twitter and Python. So I think I'm going to walk through this a little bit. And
06:00 from start to finish, it's a project that pulls, it isn't pushing things into Twitter,
06:06 it's reading Twitter and then displaying them on a, a little bottle website that they pushed up to
06:12 Heroku. And that's kind of cool that they document everything from start to finish for this.
06:17 Yeah. And it has the deployment bit as well, like pushing it to Heroku and making it work there,
06:22 which is also kind of cool because your web apps are fun, but they belong on the web, not your laptop.
06:27 It's on my queue of things to, on my salary queue of things to learn because I'm, I'm busy learning
06:33 Flask at the time, but maybe I'll check this out next. Yeah. It sounds really interesting. Definitely.
06:38 Nice job, Bob. Now, before we get on to talking about something that's actually not on the web,
06:45 but to do with Python, let me just tell you a quick story of yesterday. Yesterday I decided,
06:52 let me see how far I can push the performance of micro WSGI and NGINX on the talk Python sites.
06:59 Luckily for you, Brian, I didn't do this on pythonbytes.fm, but like, all right, I think I can push
07:06 the parallelism out better. I could crank up the number of threads. I can crack open the process
07:11 affinity, processor affinity. So like get the maximum parallelization out of micro WSGI, which is one of
07:18 the better sort of application tier worker process web servers for Python. Okay. So that seemed like a
07:26 good idea, right? Yeah. Turns out something about the threading was bad and I don't know what it is.
07:32 I don't really care to look into it, but I started getting roll bar notifications saying that the template
07:40 files could not be found for the website, which has been basically largely unchanged for months.
07:48 And then it couldn't convert this template to a string and all sorts of random errors. And I'm like,
07:54 what is going on? And there's some kind of corruption timing problem in this whole series.
07:59 And I would have, it happened like one out of a hundred thousand times. I mean, really,
08:04 really rare, but it started coming in like a couple of times yesterday. So I'm like, all right, well,
08:09 I guess we're going to turn that back down and turn it back down. Errors went away. And without roll bar,
08:13 I never would have known it. So if you guys want to make sure that your apps are working,
08:17 go check out roll bar, just pythonbytes.fm/rollbar and you sign up and you too may start getting
08:24 weird messages that you need to go look into. It'd be awesome. Yeah. Nice.
08:27 Yeah. So that was yesterday, but today I want to move on and talk about something that actually
08:34 covered way back on Talk Python, let's say about a 40 episodes ago. So not quite a year, 40 weeks ago.
08:41 And that was the Visual Studio Code extension written by a guy named Don Jayamana.
08:48 John Amani. Sorry about that. I'm sure I'm messing up your name. And he wrote a really cool Python
08:55 extension to basically add Python capabilities to VS Code. And it turned out to be the first or second
09:00 most popular thing in Visual Studio Code. It happened like millions of downloads. That's pretty cool,
09:05 right? That's very cool. Yeah. Yeah. And so one of the things I looked at, you know, when we spoke,
09:10 it's pretty interesting. He took, I'd say maybe 10 different packages, Jedi and other, you know,
09:16 Flake 8 and those types of things. And he basically wove those together into an extension for VS Code.
09:22 He didn't really build a huge amount, but he definitely put it together in a really, really usable way.
09:27 So that was, that was pretty cool. So today, this week, the news is that he is now officially hired by
09:34 Microsoft because of his open source project. That's nice. Really cool. Yeah. So here's a guy who started
09:39 an open source project and got this pretty influential job just because he had a really successful
09:45 plugin for this open source thing. So pretty, pretty cool. You know, it doesn't really mean many changes.
09:50 I just, there's a few more people working on the Python extension for VS Code, which if you're into VS Code,
09:56 it's a lot like Adam or a sublime. It's quite, quite nice. If that's your style of editor, it does a lot of
10:03 cool stuff. And so basically there'll be more people working on it, fewer bugs, faster turnaround.
10:07 The other big news is they're also hiring for a Visual Studio Code Python developer. So if you're
10:12 looking for a job and that sounds pretty awesome, you know, there's some really cool guys over there.
10:16 There's Brett Cannon, there's Steve Dower, some of the big hitters on the Python core dev team,
10:21 in addition to Don. So that might be worth looking into if that interests you.
10:25 Nice. So I feel like you're kind of doing this web crash course and getting into a lot of different
10:30 things on the web, right?
10:32 I'm trying to embrace the web a little bit more.
10:34 Nice. So what'd you find?
10:35 Yeah. So one of the things with, we talk about a lot of mechanics of how to get websites up
10:41 with Flask and Bottle and Django and all that. But sometimes the design of your website that,
10:49 as it turns out at the end, is like leaves a little bit lacking.
10:53 It looks like a developer designed it.
10:55 Yeah, it definitely looks like a developer designed it, which is usually better than when it looks like
11:01 like a database engineer designed it. But still, there's an article called a comprehensive,
11:07 it's on Smashing Magazine called a comprehensive guide to web design. It's a long article, but it's
11:13 all one page. And it's a, it's a nice, I think a good overview of the basically the basics of
11:20 make sure you have this stuff down and to make sure your website doesn't suck too bad.
11:26 That's really cool. I feel like design is one of those things that feels like a barrier to developers.
11:31 You're like, I'm not a designer. I can't do stuff that's pretty. I can't build these types of things.
11:35 But if you put in just a little effort, you can turn that barrier into a superpower because most
11:39 people don't have it. And it's actually not that hard if you just sit down and focus and try to study it
11:45 for a month or something. Especially a lot of times, like there's a lot of projects that get done, like
11:49 internal work projects or, or different tools for a small group of people that you don't have the money
11:56 to hire a front end designer. So something like this to go through and make sure that you do your breadcrumbs
12:02 appropriately and you put your search box in a findable place and some of these page structure
12:09 are reviews that are good to just make, make it more usable. And it, this doesn't add a lot of work to your
12:14 workflow. It actually, I think it takes away some of the stress of designing a web page if you're not a designer.
12:21 Yeah, it's cool. It gives you a framework to think through and stuff you got to do anyway. That's cool.
12:24 All right. Final item. What would you name the baby of requests and Selenium coming together to create some new library?
12:33 Well, I know the answer, but.
12:35 Requestium.
12:36 Requestium.
12:37 Maybe, maybe one of the candidates you throw out there, right?
12:40 Yeah.
12:40 So the thing I want to talk about is this thing called Requestium. And we have requests,
12:44 which everybody knows, super, super popular library for just downloading HTML.
12:49 And we have Selenium, which fewer people know, but is basically a headless Chrome browser.
12:55 There's other options as well, but let's say Chrome for now. And you control it from Python.
12:59 And so use them in different situations. Requests is super lightweight, takes no, very,
13:04 very few dependencies. I mean, it doesn't depend on say Chrome. It just depends on a few packages.
13:08 And you can download the HTML and do what you will with it. But if you happen to run into,
13:13 let's say an AngularJS or VueJS or any other JS site that has like a front end JavaScript framework
13:20 that basically requires the JavaScript to execute, to get the content, you end up with just a bunch
13:26 of curly braces that have no meaning when you hit it with requests, right?
13:29 Yeah. I didn't know that.
13:30 Yeah. You just get the HTML that's basically, here's where JavaScript will fill out the holes.
13:34 Good luck with that. You know what I mean? And so that's kind of useless. So you might use Selenium,
13:39 but you know, Selenium is way more heavyweight. So that Requestium basically merges the power of
13:44 requests, Selenium, and this thing called Parcel, which is like beautiful soup or Scrapey. It
13:49 understands HTML and lets you ask questions of it. So the idea is like, it's written for mostly when
13:55 you're doing requests, but that one time or two that you're like, oh, you need this login sequence
14:00 has to happen in a richer JavaScript way. And it lets you do crazy things like automatically switch
14:06 between request sessions and Selenium sessions. And it lets you basically, I think, share cookies
14:13 across them and all sorts of interesting stuff like that. So if you need like a little bit heavier weight,
14:18 then request, but not full on Selenium, here's a nice combo sort of marriage of the two.
14:23 Oh, that's nice. Cool.
14:24 Yeah. So very good for either testing or web scraping or some other, other types of things.
14:30 But if you need to test something that's JavaScript heavy, this is a pretty good,
14:33 pretty good option here.
14:34 Nice.
14:34 All right. Well, that's our news, Brian, at least the ones we've selected for everyone else. But how
14:38 about yourself? What do you got going on?
14:39 Well, I finally got a, a testing code episode out this not too long ago. I don't know if I brought it
14:45 up already.
14:46 I don't think so. Not here.
14:47 It was a nice interview with Catherine Jarmal or Kjam about testing and data science. And it's a
14:55 testing code episode 33, and it's been very well received so far. So a lot of people get some value
15:00 out of it.
15:01 Yeah, that's great. Catherine's done a bunch of cool work. So definitely go over there and check it out.
15:04 And the last thing I wanted to mention was I had asked people to leave Amazon reviews for books
15:10 that they like, including mine, hopefully. And so far, nine people have left reviews on Python testing
15:17 with pytest. So thanks a lot.
15:19 Yeah, that's great.
15:19 How about you? Any news?
15:20 Not a whole lot of news. I'm working hard, but I'm getting things ready to talk about. Nothing
15:26 to talk about yet, but someday soon. I guess the big thing I have coming up, I just recorded
15:31 some nice talk Python episodes. One actually on optimizing web servers. And then later this week,
15:38 I'm recording one, which is a panel session on machine learning and artificial intelligence at
15:44 the Large Hadron Collider and in particle physics.
15:46 Oh, that sounds interesting. And the last few episodes that you've put out, I mean, you always
15:52 have good episodes, but I've really enjoyed the last few quite a bit.
15:54 Thanks a bunch. Yeah, same for you. All right, man. Well, thank you for sharing this stuff. And
16:00 everyone, thank you for listening.
16:01 Thank you.
16:02 Thank you for listening to Python Bytes. Follow the show on Twitter via @pythonbytes.
16:07 That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm.
16:14 If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're
16:18 always on the lookout for sharing something cool. On behalf of myself and Brian Okken,
16:23 this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.