Transcript #56: The pendulum of time swings beautifully in PyPI

00:00 Hello and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds.

00:06 This is episode 56 recorded December 12th, 2017. I'm Michael Kennedy.

00:11 And I'm Brian Okken.

00:12 And we got a bunch of awesome stuff like always lined up for you.

00:14 Before we get to it, Brian, let's just say thanks to Rollbar.

00:17 Thanks Rollbar.

00:17 Yeah, thank you Rollbar for supporting the show, tell you more about what they got going on later.

00:22 I want to hear the tick tock of the pendulum, if you will.

00:26 I was reading an article called "Dealing with Date Times Like a Pro in Python." Which can be way harder than it should be, right?

00:32 Yes, and I think that there's things like Arrow and Maya that try to get...

00:36 So that you don't really have to deal with date times directly. But this is actually a nice article on dealing with date times if you want to use the direct library. However, at the end of it, they mentioned Pendulum, which I hadn't run across before. So checking it out, there's a link here for the pendulum Python date times made easy and the website is beautiful but also the interface to it it looks really nice I mean you can think do things like just import pendulum and then pendulum now in Europe Paris or in America or something and then you can do things like add dates together and do you know subtract dates and the normal things you have to do with date times but the just the model looks nice. I like it.

01:24 >> Yeah. It looks really, really nice and the website is quite stunning.

01:29 To me, on one hand, whatever, it's documentation.

01:32 On the other, it really shows the people working on this actually care about this project, and it's not just some random thing thrown on GitHub.

01:39 Yeah, it's really cool. You say like pendulum.now and you give it a time zone, say like Europe/Paris, and then you can subtract weeks from it or you can.

01:47 One thing I like to do is put times relative to humans, like a week ago, five seconds ago, in 10 minutes, and so it just has a diff for humans, you know, and that's really nice.

01:58 - I'm gonna include in the show notes links to Arrow and Maya and I think it's a matter of developer taste, really, and what looks more natural to you, so check 'em all out and see what's nicer for you.

02:09 - Yeah, it's really nice.

02:10 The other one to throw in here while we're thinking about all this is parsing datetimes, which I think there's 700 different formats for datetimes and text, which is insane, but I really like to use Python dateutil, So pip install Python dash dateUtil, but just import dateUtil.

02:26 And it can parse like lots and lots of stuff.

02:29 So you can just go here, load that into time, you know, and that's one more thing.

02:33 But this is really, really slick.

02:34 It has more sort of time delta things, like how many weeks, how many days, how many hours, as opposed to just like the seconds, total seconds, the built-in one.

02:42 Really nice, I like it, good find.

02:44 So the next thing that I wanna talk about, we're gonna talk about some web stuff, aren't we?

02:47 So the next one I wanna do is, it's sort of a web thing, but it doesn't really have to be.

02:52 it's any kind of service or long running thing.

02:54 So one of the things that can be a challenge with web applications is long running tasks or tasks that are slow, but you don't really need to like wait on their outcome.

03:05 So I'll give you an example.

03:06 On my training website, training.talkpython.fm, on that site, there's a capability to email everyone in a class.

03:15 And it turns out, if you in the same request, actually email thousands of people one at a time, like through Amazon SES or something, that request times out.

03:28 And then you've emailed some people and you don't know how far you got, you've got to go figure out how to rewrite that some other way, right?

03:34 That's a problem.

03:35 Yeah, so there's certain things like email, like logging, like other sort of kick this job off and forget about it, that would be much better served in some sort of asynchronous way.

03:45 One of the best things you can do to add scalability to your app is to push those kinds of things onto some background, sort of get to it when you can style.

03:53 And probably the most formal way is to use some kind of queuing.

03:57 So the thing I wanna talk about now is this article called Flask Asynchronous Background Tasks with Celery and Redis.

04:04 Although it has nothing really to do with Flask, although it uses Flask as an example.

04:07 It could be Django, it could be Pyramid, like really it's the Celery bit that's interesting.

04:12 So you take Celery as the worker, right?

04:15 It's gonna run back there it's gonna look at the Redis message bus.

04:19 And what you do is you just kick jobs off to it, and it'll just start pulling them off and working on them.

04:23 So what I could have done is said, oh, I wanna email 3,000 people.

04:27 So go to the database, get the 3,000 people, and just put 3,000 email this person jobs onto a Celery queue and been done straight away.

04:35 And it would just do it over the next minute or two.

04:37 - Actually, that's pretty cool.

04:38 - Yeah.

04:39 - I've got my, the wheels are turning in my head thinking things that I wanna use Celery for.

04:43 - I know, I feel the same way.

04:44 It's a really nice write up of how you like walk through these things.

04:47 So it talks about using the Celery client to connect Flask to Celery, the Celery worker process, the message broker that works with the Redis backend.

04:55 And the other thing that's interesting about this is so often these Python examples, these especially Python web examples are done on macOS or Linux.

05:04 And this is all on Windows.

05:06 And I don't think Redis is even supported on Windows or Celery, one of them.

05:10 I think it's Redis.

05:11 It's not, there's like an unofficial build or something.

05:14 I think it's kind of extra interesting to show this other angle and show people how to do it where it might not be so obvious.

05:20 Also, something like taking on Celery, it's kind of nice to look at an example project to see how it's been used because just jumping right into the entry of the documentation can be a little scary sometimes.

05:32 Here's a nice little simple example. Take it and do what you will with it.

05:35 Yeah, well, we have another web example that I liked and this is from Bob Belderboss of PyBytes.

05:41 He wrote a guest article on RealPython called "Building a Simple Web App with Bottle, SQLAlchemy, and the Twitter API." The reason why this caught my attention is because I was curious about having some way to interact with Twitter and Python.

05:58 I think I'm going to walk through this a little bit.

06:01 From start to finish, it's a project that isn't pushing things into Twitter.

06:06 reading Twitter and then displaying them on a little bottle website that they pushed up to Heroku.

06:13 And it's kind of cool that they document everything from start to finish for this.

06:17 It's neat.

06:18 Yeah, it has the deployment bit as well, like pushing it to Heroku and making it work there, which is also kind of cool because web apps are fun, but they belong on the web, not your laptop.

06:27 It's on my queue of things to, on my celery queue of things to learn because I'm a busy learning Flask at the time, but maybe I'll check this out next.

06:36 Yeah, that sounds really interesting. Definitely. Nice job, Bob.

06:39 Now, before we get on to talking about something that's actually not on the web, but has to do with Python, let me just tell you a quick story of yesterday.

06:50 Yesterday, I decided, let me see how far I can push the performance of MicroWZGI and Nginx on the Talk Python sites.

06:59 Luckily for you, Brian, I didn't do this on pythonbytes.fm.

07:03 But like, alright, I think I can push the parallelism out better.

07:07 I could crank up the number of threads.

07:09 I can crack open the process affinity, processor affinity.

07:13 So like get the maximum parallelization out of MicroWZG, which is one of the better sort of application tier worker process web servers for Python.

07:24 Okay, so that seemed like a good idea, right?

07:26 Turns out, something about the threading was bad.

07:31 And I don't know what it is.

07:33 I don't really care to look into it.

07:35 But I started getting roll bar notifications saying that the template files could not be found for the website, which has been basically largely unchanged for months.

07:48 And then it couldn't convert this template to a string and all sorts of random errors.

07:54 And I'm like, what is going on?

07:55 there's some kind of corruption timing problem in this whole series.

08:00 And I would have, it happened like one out of 100,000 times.

08:04 I mean, really, really rare.

08:05 But it started coming in like a couple of times yesterday.

08:08 So I'm like, all right, well, I guess we're gonna turn that back down.

08:12 Turn it back down, errors went away.

08:13 And without Rollbar, I never would have known it.

08:14 So if you guys wanna make sure that your apps are working, go check out Rollbar, just pythonbytes.fm/rollbar and sign up and you too may start getting weird messages that you need to go look into, it'd be awesome.

08:27 - Yeah, nice.

08:27 - Yeah, so that was yesterday.

08:30 But today I want to move on and talk about something that I actually covered way back on Talk Python, let's say about 40 episodes ago, so not quite a year, 40 weeks ago.

08:41 And that was the Visual Studio Code extension written by a guy named Don Jayamana, Jayamani, sorry about that, I'm sure I'm messing up your name.

08:52 And he wrote a really cool Python extension to basically add Python capabilities to VS Code.

08:58 And it turned out to be the first or second most popular thing in Visual Studio Code.

09:03 It had like millions of downloads.

09:04 That's pretty cool, right?

09:05 - That's very cool, yeah.

09:06 - Yeah, and so one of the things I looked at, you know, when we spoke, it's pretty interesting.

09:10 He took, I'd say maybe 10 different packages, Jedi and other, you know, Flake 8, and those types of things.

09:18 And he basically wove those together into an extension for VS Code, he didn't really build a huge amount, but he definitely put it together in a really, really usable way.

09:28 So that was pretty cool.

09:30 So today, this week, the news is that he is now officially hired by Microsoft because of his open source project.

09:36 - That's nice, really cool.

09:37 - Yeah, so here's a guy who started an open source project and got this pretty influential job just because he had a really successful plugin for this open source thing.

09:47 So pretty, pretty cool.

09:49 It doesn't really mean many changes, just there's a few more people working on the Python extension for VS Code, which if you're into VS Code, it's a lot like Atom or Sublime.

09:58 It's quite nice if that's your style of editor.

10:02 It does a lot of cool stuff.

10:03 Basically, there'll be more people working on it, fewer bugs, faster turnaround.

10:07 The other big news is they're also hiring for Visual Studio Code Python developers.

10:12 If you're looking for a job and that sounds pretty awesome, there's some really cool guys over there.

10:16 There's Brett Cannon, there's Steve Dower.

10:18 some of the big hitters on the Python core dev team in addition to Don.

10:22 So that might be worth looking into if that interests you.

10:25 Nice.

10:26 So I feel like you're kind of doing this web crash course, getting into a lot of different things on the web, right?

10:32 - I'm trying to embrace the web a little bit more.

10:34 - Nice.

10:35 So what'd you find?

10:36 - Yeah.

10:37 So one of the things with, we talk about a lot of mechanics of how to get websites up with Flask and Bottle and Django and all that.

10:44 But sometimes the design of your website, as it turns out, at the end, is like leaves a little bit lacking.

10:53 - It looks like a developer designed it.

10:55 - Yeah, definitely looks like a developer designed it, which is usually better than when it looks like a database engineer designed it, but still.

11:04 There's an article called a comprehensive, it's on Smashing Magazine, called "A Comprehensive Guide to Web Design." It's a long article, but it's all one page, And it's a nice, I think a good overview of the, basically the basics of make sure you have this stuff down and to make sure your website doesn't suck too bad.

11:26 - That's really cool.

11:27 I feel like design is one of those things that feels like a barrier to developers.

11:31 You're like, I'm not a designer, I can't do stuff that's pretty, I can't build these types of things.

11:35 But if you put in just a little effort, you can turn that barrier into a superpower because most people don't have it.

11:40 And it's actually not that hard if you just sit down and focus and try to study it for a month or something?

11:46 - Especially a lot of times, there's a lot of projects that get done, like internal work projects or different tools for a small group of people that you don't have the money to hire a front-end designer.

11:58 So something like this to go through and make sure that you do your breadcrumbs appropriately and you put your search box in a findable place and some of these page structure overviews that are good to just make it more usable.

12:12 And this doesn't add a lot of work to your workflow.

12:15 It actually, I think it takes away some of the stress of designing a webpage if you're not a designer.

12:21 - Yeah, it's cool.

12:22 It gives you a framework to think through and stuff you gotta do anyway.

12:24 That's cool.

12:25 All right, final item.

12:27 What would you name the baby of requests and Selenium coming together to create some new library?

12:33 - Well, I know the answer, but--

12:36 - Requestium. - Requestium.

12:37 - Maybe one of the candidates you throw out there, right?

12:40 - Yeah.

12:41 - So the thing I want to talk about is this thing called Requestium.

12:43 And we have Requests, which everybody knows, super, super popular library for just downloading HTML.

12:49 And we have Selenium, which fewer people know, but it's basically a headless Chrome browser.

12:55 There's other options as well, but let's say Chrome for now.

12:58 And you control it from Python.

12:59 And so use them in different situations.

13:01 Requests is super lightweight, takes no, very, very few dependencies.

13:05 I mean, it doesn't depend on, say, Chrome.

13:07 It just depends on a few packages.

13:09 And you can download the HTML and do what you will with it.

13:12 But if you happen to run into, let's say, an AngularJS or Vue.js or any other JS site that has like a front-end JavaScript framework that basically requires the JavaScript to execute to get the content, you end up with just a bunch of curly braces that have no meaning when you hit it with request, right?

13:29 - Oh, I didn't know that.

13:30 - Yeah, you just get the HTML that's basically, here's where JavaScript will fill out the holes.

13:34 Good luck with that.

13:35 You know what I mean?

13:36 And so that's kind of useless.

13:38 So you might use Selenium, but Selenium is way more heavyweight.

13:42 So the Requestium basically merges the power of requests, Selenium, and this thing called Parcel, which is like BeautifulSoup or Scrapy.

13:49 It understands HTML and lets you ask questions of it.

13:53 So the idea is like, it's written for mostly when you're doing requests, but that one time or two that you're like, oh, you need this login sequence has to happen in a richer JavaScript way.

14:03 And it lets you do crazy things like automatically switch between request sessions and Selenium sessions.

14:10 And it lets you basically, I think, share cookies across them and all sorts of interesting stuff like that.

14:16 So if you need like a little bit heavier weight than request but not full on Selenium, here's a nice combo sort of marriage of the two.

14:24 - Oh, that's nice, cool.

14:25 - Yeah, so very good for either testing or web scraping or some other types of things.

14:30 But if you need to test something that's JavaScript heavy, this is a pretty good option here.

14:34 - Nice.

14:35 - All right, well that's our news, Brian.

14:36 At least the ones we've selected for everyone else, but how about yourself?

14:39 What do you got going on?

14:39 - Well, I finally got a Test and Code episode out this not too long ago.

14:45 I don't know if I brought it up already.

14:46 - I don't think so.

14:47 Not here.

14:48 - It was a nice interview with Katherine Jarmal, or KJAM, about testing and data science, and it's Test and Code episode 33, and it's been very well received so far, so a lot of people get some value out of it.

15:01 - Yeah, that's great.

15:02 Katherine's done a bunch of cool work, So definitely go over there and check it out.

15:04 - And the last thing I wanted to mention was I had asked people to leave Amazon reviews for books that they like, including mine, hopefully.

15:13 And so far, nine people have left reviews on Python testing with pytest.

15:18 So thanks a lot.

15:19 - Yeah, that's great.

15:20 - How about you, any news?

15:21 - Not a whole lot of news.

15:21 I'm working hard, but I'm getting things ready to talk about, nothing to talk about yet.

15:26 But someday soon, I guess the big thing I have coming up, I just recorded some nice talk Python episodes, one actually on optimizing web servers.

15:36 And then later this week, I'm recording one, which is a panel session on machine learning and artificial intelligence at the Large Hadron Collider and in particle physics.

15:47 - Oh, that sounds interesting.

15:48 And the last few episodes that you've put out, I mean, you always have good episodes, but I've really enjoyed the last few quite a bit.

15:55 - Thanks a bunch.

15:55 Yeah, same for you.

15:58 All right, man. Well, thank you for sharing this stuff and everyone, thank you for listening.

16:02 Thank you.

16:03 Thank you for listening to Python Bytes. Follow the show on Twitter via @pythonbytes. That's Python Bytes as in B-Y-T-E-S. And get the full show notes at Python Bytes.FM. If you have a news item you want featured, just visit Python Bytes.FM and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends