Transcript #45: A really small web API and OS-level machine learning

00:00 Hello and welcome to Python bites where we deliver Python news and headlines directly to your earbuds.

00:05 This is episode 45 recorded September 27 2017. I'm Michael Kennedy, and I'm Brian Okken. And as usual, we got a bunch of cool news items lined up for you. So Brian, let's first say thanks to roll bar. Yeah, thanks, Robar. Yeah, thanks for sponsoring this episode. We'll tell you guys more about roll bar if you don't know about them later. But let's start with something super small. Like I don't want to start anything big.

00:26 - This was recommended by a listener, Ivan, I'm not going to try his last name, but thanks Ivan, a little micro framework called Pico.

00:36 And there was a lightning talk given at EuroPython 2017 and we have a link to it.

00:43 But this is just a, it's a very simple, very, well, I don't know how simple the code is, I haven't looked, but it's simple to use.

00:49 It's a little web framework that you can use for actual web pages.

00:54 It does have some CSS and JavaScript serving, I think.

00:59 But the main idea of it is a very simple, easy to use web framework for people that are not web developers.

01:06 So let's say, I think it was developed in a scientific community.

01:09 So people that can just hook up, you really, it's really hook up a web endpoint with just a decorator that says pico.expose and you got a function.

01:19 And there you've got a service, a web service you can use.

01:24 So it's pretty amazingly simple.

01:26 - Yeah, it is quite simple.

01:27 And one of the things that is unique about it, well, relatively unique, is that it comes with a JavaScript client that automatically generates a proxy for objects described in your API.

01:42 And that's pretty trick.

01:42 - Oh, wow, I missed that.

01:44 That's cool.

01:44 - Yeah, isn't that cool?

01:45 So instead of having to define the REST call and then actually just do that, like with direct AJAX calls, whatever framework you're using, how you do that.

01:55 If you have something that has like a hello function and takes a string, you can create one of these, one of these clients and just say .hello and pass it a string and then it gives you like a promise, which is really cool, I think that's kind of unique.

02:09 - Yeah, it's one of the simplest, like very little boilerplate you have to throw in some code.

02:14 I was looking at this because if I had some services at work trying to pull out some database objects, I think non-developers could maintain it fairly well, okay.

02:24 I mean, probably not non-developers, but not web developers.

02:27 - Right, right.

02:28 So I think it's pretty interesting.

02:30 I actually haven't heard of it, so I don't know how durable it is, how good it is for building rich applications.

02:35 I have lots of requirements, but it looks pretty cool to me.

02:38 It's definitely worth checking it out.

02:39 And it's small and easy to get started with, so that's always nice.

02:42 There's not a lot of mental overhead to use the thing, right?

02:45 - Yeah, and the link we have, which thank Ivan for this also, to the exact part of the lightning talk.

02:52 So it's just a few minutes of one of the maintainers talking about this project and it's a really good overview.

03:00 - That's cool, you always hear like the why did I build it?

03:02 Yes, I know Django REST framework and other things exist.

03:05 I still built it, things like that, right?

03:07 - Yeah, and it isn't, it's specifically not REST compliant, but for a lot of cases you don't really care.

03:13 - Yeah, it's interesting, it's almost more like traditional XML web service proxy type looking thing.

03:18 Anyway, very cool. Definitely check that out if that sounds interesting to you.

03:22 So I had a question for you and everyone.

03:25 Brian, have you installed macOS High Sierra?

03:27 >> No.

03:28 >> It came out yesterday. I already installed it.

03:30 It was a bit of a risk, but I had stuff backed up, so why not give it a shot, right?

03:33 >> We're talking today, so apparently it went okay.

03:35 >> We're talking on the same computer I installed it on.

03:37 So it all went okay and it was all pretty smooth and seamless.

03:40 So super excited to have a new macOS.

03:43 But this one is actually more like a foundational release.

03:46 So there's a bunch of underlying systems and things that have been changed to make it able to build more cool stuff.

03:52 So like one of the popular things people will talk about maybe is APFS, a new Apple file system that is like a modern built in 2017 type file system, not like, you know, 30 year old files.

04:03 It's a really, really cool type stuff like that.

04:05 But one thing, the reason we're talking about on this show is it comes with something that I think is actually a kind of a big deal.

04:11 It comes with something called core ML.

04:13 So that's cores, like all the systems like, you know, core storage, core, whatever, right?

04:18 Core ML is core machine learning.

04:21 So here with the latest Apple operating system, maybe the first major OS to come with like built in machine learning.

04:28 Wow.

04:29 Is that crazy or what?

04:30 Well, the core ML is a set of API's that you can use.

04:33 And basically, it packages up a lot of the stuff that they're already doing anything like photos where the photos can like identify, you know, people so you can say show me all the pictures of Brian and it would just like find those magically and all my photos, Siri, text to speech, all those types of things, right?

04:51 So they want to make it possible for you to use some of those.

04:54 So basically, it's core ml and it comes with pre built machine learning models, you can create your own models and then package it up with your app and send it on so you could train it to do whatever.

05:05 And they even offer some default ones.

05:07 It's pretty cool.

05:08 Yeah.

05:09 Yeah.

05:10 So another thing that's pretty sweet about it is it will use basically on any of the Macs from 2012 or later, it will use a mix of CPU processing and GPGPU processing depending on the task.

05:20 And it'll just figure that out for you.

05:22 So this whole, do I use...

05:23 I'm guessing that makes it slicker.

05:25 Well, how many cores does your Mac have?

05:28 I have no idea.

05:29 Probably four with hyper-threading, right?

05:31 Probably.

05:32 So it's either two or four plus hyper-threading, which would double that, right?

05:35 Some of the GPUs have like thousands of cores, thousands.

05:39 So if you want to do something in parallel, which a lot of machine learning is like if you have either eight cores or 2000 cores, that's a big difference.

05:47 So it's really cool that it that's built in.

05:48 Yeah, anyway, so I think this might be the first major OS to like come with machine learning built in.

05:53 It's just a sign of the times, right?

05:54 All right, you probably got to log your code and figure out what's happening when your machine learning models don't do what you want, right?

06:00 I don't have a list, but we've covered several simple logging modules on the on the show so far. But this right now we're just talking about plain old logging. The built in logging library. Am I getting that right? I think it's just logging.

06:14 Yep, just the logger.

06:16 Yep. Logger, import logger. Yeah. The reason why I haven't really used it too much before, to be honest, is I have had trouble getting my head wrapped around all the little pieces.

06:27 And it's a fairly complex module. And for good reason, it does a lot. But this is the This is the first, I'm pointing to a blog article called A Guide to Logging in Python, and it walks through using logging very simple and then adding on, changing our mental model to include all the different pieces like logging file handlers and memory handlers and filters and all that stuff.

06:55 And it's the first time I've read about logging where I, from start to finish, I wasn't lost the entire time.

07:00 So it's a good introduction.

07:03 That's cool and it talks about like why not just do print.

07:05 There's all sorts of things like multi-threading support, categorization, and different login levels, time rotating files, all kinds of stuff better than just print.

07:16 So yeah, this is really cool.

07:17 I do feel like there's a lot of configuration and stuff in the built-in logging module that kind of tries to do everything.

07:23 So I can make it tricky and this is nice.

07:25 Yeah, and there's some things that it does that I didn't even know it did.

07:28 I didn't know it does like automatic file rotation just built in.

07:31 That's cool.

07:32 - That's really nice.

07:33 - Anyway, if you're trying to figure out the built-in logging module, check this one out.

07:37 - I can tell you that time-based rotating file is important when your website generates gigabytes of log files.

07:43 You don't want that to be one file.

07:45 - Yeah.

07:46 - Speaking of websites, it kind of sucks when your websites crash for your users, right?

07:49 - Yeah.

07:50 - They don't like it, but they might not tell you that might just go away.

07:52 So that's why you want to get Rollbar, right?

07:54 So like we said, Rollbar's supporting the show.

07:56 Visit them at pythonbytes.fm/rollbar, And you can install it in just a few minutes.

08:03 pip install rule bar, few lines of configuration, and all the errors in your website are captured with tons of detail.

08:10 Things like local parameters, arguments passed to methods when it crashed, all that kind of stuff.

08:15 Notifications, Slack, email, whatever.

08:18 It's beautiful.

08:19 So definitely install that if you're running a web app based on Python.

08:22 So speaking of web apps, you might care about memory, right?

08:28 A lot of times one of the things that puts a lot of pressure on your web apps is not the CPU, but it's actually memory.

08:32 And I'd say that's true for, certainly for my web apps, it seems like memory is more of a pressure than CPU by quite a bit.

08:40 So one of the things that I thought was interesting is somebody wrote an article called Let Me Introduce Dunder Slots.

08:47 So slots are alternative backing store for class data, I guess is maybe the simplest way to put it.

08:56 Have you played with these, Brian?

08:57 - No, I haven't.

08:58 - This is really crazy.

08:59 You can, like when you create a regular Python class and you implement a dunder init, and then there you say self.name equals something passed in, self.email equals some email address passed in, that goes into dunder dict, right?

09:15 Like each instance of that class has a dictionary that has the name, email, the name, name, and then the two values that you passed in.

09:23 And every instance of the class gets a separate instance of the dictionary, they're one-to-one.

09:27 it be super easy to do lookups, right, or one, it's super easy to make it dynamic. Like if you just interact with a class and you try to add new stuff to it, it just goes into that dictionary. So that's cool. But what's not so cool is if I have 10 million instances of that class, I have 10 million copies of that dictionary, which has 10 million strings, each one that says email, and another 10 million strings that say name, why do I need to store those? I probably don't, right? If I'm really not going to be dynamic, I probably don't. So you can use this thing called dunder slots. And you would say, the slots of this class are name and email. And then that that slot is stored on the type, not the instance. So instead of having 10 million names, and 10 million emails, in terms of the field name, right, you just have the two. And otherwise, they're just stored like in like an array, and a positional thing. So super good for performance, And it's like the test they did in this article, 57% less memory usage just by adding that one line.

10:28 And it's a little bit faster for access, but it's definitely better on memory.

10:31 Can you use both?

10:32 No.

10:33 Well, you can still do the self dot whatever and assign to it.

10:35 But basically, if you try to assign to something that's not declared in the slot, it'll say it doesn't have that property.

10:42 It wasn't pre allocated in the type basically, or predefined in the type.

10:46 So yeah, it's pretty cool.

10:48 I actually go into this in depth in my write Python code course and you'll see that it's this is even better in terms of memory than an unnamed tuple.

10:58 You wouldn't think you could do better than an unnamed tuple for like space but like this is actually even better and you get all the type class inheritance behavior that you'd expect.

11:07 Very cool.

11:08 >> Seems like more of the mental model of classes I have in the first place.

11:12 >> Yeah, yeah for sure.

11:13 It's very much like C++, C#, traditional, like these are the things that are in here and they never change.

11:20 A static language type of thought of a class, yeah.

11:22 - Well, I'm definitely gonna have to go and re-watch your seasoned developer course and do these again.

11:29 - Yeah, it's pretty cool.

11:30 Yeah, it's super easy.

11:31 You shouldn't use it all the time, but when it makes sense, it can save you tons of memory.

11:35 - Well, that's cool.

11:36 Hey, a long time ago in episode 11, we covered pipenv from Kenneth, and I always get his name wrong, so you say it.

11:43 - Kenneth Wright.

11:44 - Kenneth Wright, okay.

11:46 - Maybe that was one of like the 10 things he did that week, I don't know.

11:48 - Yeah, so he's been doing a lot, but one of the things that, the first time I took a look at this, I gave it a honest college try, and it just, to just be honest, it was, I didn't know why I needed it.

12:02 - You're like, I already got this covered, whatever.

12:04 - Yeah, I already got this covered.

12:06 But one of the things that changed my mind is not too long ago, he put up a video, And so if you go to docs.pipenv.org, there's a four minute screencast of him just using it.

12:18 And that video got me convinced.

12:21 I'm like, oh, wow, this is really a lot easier than I've done before.

12:26 And actually I've been doing a lot more virtual environments than I used to.

12:31 And I kind of lose track of which ones are where.

12:34 So this helps.

12:35 And so PIPenv, if you haven't listened to episode 11, is something that deals with your virtual environments and pip and install and all that for you.

12:45 And it's just a way of working that if you give it a try, you might like it.

12:49 So the video is one thing that's new that convinced me, but there's also a bunch of other stuff that he's done recently.

12:55 He also included security checks.

12:58 So our scare from last week of whether or not you're gonna install a problem package, this pipen will look through with pipen check.

13:08 You can look through all your dependencies make sure that you don't have any security vulnerabilities installed in any of them.

13:14 That's awesome. And that's not like you have an old version of Django, so it has a security vulnerability. That's like somebody called it Django without the D and put a virus in it, right? That type of thing, right? Yeah. And the other thing that one of the things that it has that it's had from the start is a lot of packages. So packages have these hash values to compare your actual install from what's been published.

13:40 Pipenv deals with that and checks those, which is hard to do manually.

13:45 Then one of the things it does recently is, also it allows multiple package indexes.

13:52 So you could have PyPI of course, but also maybe a company index and a group index, maybe even one for your project.

13:59 >> That's really cool.

14:00 >> The features are piling up and he recently said that it's nine months old, but it's had 192 releases.

14:07 So he's not sleeping a lot, I don't think.

14:10 >> No, probably not. Yeah, that's really cool.

14:12 My favorite thing is pipenv lock-r will generate a requirements.txt file. That's cool.

14:17 >> Right. That's actually the thing that turned me off the first time.

14:21 It's because pipenv uses a thing called pip file and pip file.lock, which I don't really follow why I need those, but I know sometimes I need a requirements.

14:32 allows you, you can use this and still get your requirements files. So yeah, very cool. Very cool. Alright, so the final thing I want to talk about is a little bit, a little bit of a softer, more squishy concept, right? Not just an API or something. But stack overflow, they're doing some interesting data science, I think they actually have like full time data scientists that are just like mining these and like generating reports and analysis on the industry. So that's, that's pretty cool. And what I'm actually pointing to for this is not Stack Overflow, but to an Ars Technica article, which is a follow-up to this kind of unfortunate article they did called "Tabs and Spaces, Who Gets Paid More?" or something like that.

15:16 And they made the claim that like, well, people who are uninformed use spaces, and for some reason, they get paid more than people who use tabs.

15:24 Don't know why.

15:25 That was something they found in the survey.

15:27 Well, the reason why is because those are Python developers, right?

15:31 Whereas the other ones aren't.

15:34 So that's an interesting thing in and of itself.

15:36 But this is like a follow up to say like, let's look at not programming languages, but like just different locations.

15:42 So if you live in New York versus you just live in a random place in the US versus Germany or France, basically the US versus Europe, well, US and Europe all compared against each other.

15:54 So it talks about like in these different places, if you're a DevOps or a data scientist, you earn really well, probably using Python.

16:00 Surprisingly, if you do graphics programming, like OpenGL or something, you're not paid very well, even though that's super hard to do.

16:07 The reason is, I think, and they sort of hinted this as well, is you're probably working in a game company, and there's a lot of young people working at game companies who are just, they wanna work on games, period.

16:18 It doesn't matter if they have to work 80 hours a week and get paid a little for it, right?

16:22 - Okay. - Yeah, so that's pretty tough.

16:23 I have heard that the game industry is a pretty hard place to work, but that's sort of one part of it, right?

16:30 You don't get paid tons.

16:31 But the most surprising fact was really that in the US, developer pay is significantly more than in Europe.

16:40 And it's not like 10% more or something like that.

16:43 It's like, I don't know, close to double.

16:46 It's really like quite a bit more.

16:48 So they say things like, hey, people in the US have substantially higher median income, even regardless of experience.

16:56 So they say, for example, a median salary of a developer in the US is comparable to somebody with 20 years experience in Canada or Germany, and it isn't even quite higher than people in France or the UK with 20 years experience.

17:08 Like a new, like, "Hey, I just graduated.

17:09 What can I do now?" sort of job.

17:11 So pretty interesting.

17:14 The comments are also super interesting because people coming from all over the place and they're thinking about like, "Well, okay, salary is not everything.

17:21 There's cost of living.

17:22 There's cost of health care.

17:23 There's social support." There's a lot of stuff, so this is partly interesting for the article, but also partly interesting for the way people are analyzing it.

17:32 - Yeah, well, actually, it's kinda nice to have some good news for being an American.

17:37 (Luke laughs)

17:38 - Yeah, it's been a little sketchy lately, but hooray.

17:42 - We've got the weirdest president, the highest healthcare costs, but hey, we get paid a lot.

17:46 - Yeah, and healthcare actually makes a big part of the conversation, and like, hey, well, you guys pay so much more for healthcare, maybe the salary doesn't offset it, but we don't pay like half our salary yet in healthcare.

17:56 So it doesn't offset it yet.

17:58 Anyway, pretty interesting.

17:59 So if you're thinking about this kind of stuff, here's an article with a lot of data to back it.

18:05 All right, that's our news items, Brian.

18:07 Got anything else you wanna share with the folks?

18:09 - No.

18:10 - My gosh, you're not doing anything, right?

18:12 You're just like chilling now that the book is done and you're just kicking back?

18:17 - I think people have, some people have already received the book, although I haven't.

18:21 I'm waiting for my box to show up this afternoon.

18:24 - Oh, how exciting.

18:24 Yeah, I've seen a lot of Twitter messages, people posting that they've shipped and things like that.

18:28 That's great, congrats.

18:29 - Thank you.

18:30 How about you?

18:31 - Not too much going on right now.

18:32 I'm working on a free MongoDB course, and that is super close to done.

18:38 So I'm hoping to have some announcements soon, but I'm not there yet.

18:41 - One of the things, I'm gonna try to start some new projects and not talk about the book so much every episode, but I'd really love to hear from people when they get them and what they think, go ahead and send me a shout out on Twitter @brianhawkin and say, "Hey, I got your book," and that'd be cool to hear from people.

18:58 - Yeah, that'd be awesome.

18:59 Yeah, it's really cool.

19:00 People are excited about it.

19:01 I've been watching from the sidelines.

19:02 - All right.

19:03 - All right, well, thanks for joining me for another one of these chats.

19:05 - Thank you.

19:06 - Yep. - Talk to you later.

19:07 - You bet, bye.

19:08 Thank you for listening to Python Bytes.

19:10 Follow the show on Twitter via @pythonbytes.

19:13 That's Python Bytes as in B-Y-T-E-S.

19:16 And get the full show notes at pythonbytes.fm.

19:19 If you have a news item you want featured, just visit pythonbytes.fm and send it our way.

19:24 We're always on the lookout for sharing something cool.

19:27 On behalf of myself and Brian Okken, this is Michael Kennedy.

19:30 Thank you for listening and sharing this podcast with your friends and colleagues.