Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


« Return to show page

Transcript for Episode #103:
Getting to 10x (results for developers)

Recorded on Tuesday, Nov 6, 2018.

00:00 KENNEDY: Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is Episode 103, recorded November 6th, 2018. I'm Michael Kennedy

00:11 OKKEN: And I'm Brian Okken.

00:12 KENNEDY: Hey Brian, how're you doin'?

00:13 OKKEN: I'm great today.

00:14 KENNEDY: Yeah, it's another wonderful day, a lot of cool news in the Python space. I have something that I think you all will really like. I'm looking forward to sharing that.

00:22 OKKEN: Cool.

00:23 KENNEDY: Yeah and I know you got some good ones, so I'm looking forward to talking about those as well. Before we do, let's say thank you to Digital Ocean. Check them out at pythonbytes.fm/digitalocean. Get a $100 free credit for new users. I'll tell you more about them later. Right now, you've got, kind of, this magical C++ Python combination cued up for us, right?

00:42 OKKEN: I do and we found out about this because a listener, I think it's Sebastian Shrubard, wait, Sebastian Shrizard, Brizzard. Sorry about your last name.

00:57 KENNEDY: Thank you Sebastian for writing, for sending that in. Brian, if anyone else wants to send in some news, you can also butcher their name, in honor of them sending that in, right?

01:05 OKKEN: Yeah, that's tradition.

01:07 KENNEDY: As I would as well, I do a lot too, that's right. It's becoming a tradition. Thank you Sebastian. All right, tell us about this thing.

01:13 OKKEN: Well I think, because of the flames in the logo, they actually intended to be pronounced Phoenix, it's F-E-N-I-C-S. It's a project. I'm glad that Sebastian sort of translated this for us. It's an open source computing platform for solving partial differential equations. This is actually really cool and I'm going to quote right from their site, "Fenics enables users to quickly translate scientific models into an efficient, finite element code with a high level Python and C++ interfaces to help you get started. It's got powerful capabilities for experienced programmers, but it's easy to get started." And it runs on multiple platforms from laptops to high performance clusters and it actually looks really pretty cool for anybody dealing with partial differential equations. It's a NumFOCUS backed project so there's money behind it which is cool.

02:09 KENNEDY: NumFOCUS seems like it's everywhere these days.

02:11 OKKEN: Yeah, well especially in, and it's good to have to like highlight that to say that this is a backed project because it's a, these sorts of things you wouldn't want to depend on and then have them go away.

02:23 KENNEDY: Right, some open source projects, like depending upon them are kind of like getting a puppy, right? Like they're cute and fun, but then, then you got to walk 'em and stuff like that, if they get dropped or don't work the way you want.

02:35 OKKEN: Yeah, and Sebastian said right off the bat, it looks cool, but there's some features inside that you might not know about right off the bat. So, I'm going to quote an email that he sent to us. He said FENICS is in fact a C++ project with a full featured Python interface. The library itself generates C++ code on the fly and be called on the fly from Python. It's almost magical. Under the hood it uses Swig and recently moved to PyBind 11. I guess the architecture that was set up to achieve this level of automation might be useful for other situations.

03:13 KENNEDY: Yeah, so that's crazy, you write Python code, this thing writes C++ code and then calls it all dynamically at runtime.

03:20 OKKEN: Yeah, that's amazing. And from the project website, being able to develop the algorithm locally on whatever computer you're on even a laptop or desktop, then deploying the same code to run in parallel on thousands of processes, that's just awesome.

03:41 KENNEDY: Yeah, I think there's a lot of cool stuff happening here. I mean not everyone is solving using finite element methods to solve PDEs, right, I understand that's a limited group, but there's a lot of, a lot of projects that may find what this project is doing interesting from a performance perspective in a dynamic meets compiled language perspective.

03:59 OKKEN: Yeah, and it's also, it's also one more example of problems being solved in Python that you wouldn't have thought you could solve with Python because they're just too, take too much high performance computing.

04:12 KENNEDY: That's a super interesting point. So often I hear people who are not that familiar with Python say, well Python is slow. I'm like, hold on. What do you mean that Python is slow? Like you have to say doing this operation in Python is slow because there's so many variations. I mean it can be well Python is slow in CPython so you can use PyPy, but it could be way more interesting like well yes, but you would actually use say, some kind of library that has C level compilation elements like SQLAlchemy or NumPy or something so when you actually talk about that, like you're doing C, you're not doing Python at the hotspots and then you get you know, way out there with things like Dask and like this and so on, and it's, it's pretty awesome.

04:55 OKKEN: Yeah, definitely, it's a neat to see.

04:57 KENNEDY: You're kind of a fan of Regular Expressions, are you?

04:59 OKKEN: Yes, I like Regular Expressions.

05:01 KENNEDY: Are you a fan of cursive language? Like the fancy calligraphy type stuff, like do you write that way often?

05:07 OKKEN: Well I like it, I don't write that way, but.

05:10 KENNEDY: I don't write that way either, but this next project I want to talk about is its goal I think primarily is to make regular expressions more easy to indicate their intent and easier to maintain. I like the joke that Regular Expressions are kind of a write only language, you know, you write them but then you can't read 'em anymore but they're magic and they do their thing, right. So we got this project called cursive_re and re is of course the Regular Expression module in Python, right? This comes from Chris Patty of Podcast Internet fame and is actually created by Bogdan Popa, my turn to hopefully take a shot at getting the name not too wrong, but this is a library for doing Regular Expressions in modern Python, so 3.6 and above, and the idea is instead of writing in the string symbols like you normally would you know, bracket, 0-9., closed bracket, dot, that sort of thing, you write in this higher level language of combinators they call them, and then that overrides the operators in Python to generate a regular expression. So this is like all sounds kind of wonky and crazy, but if you see an example super clear. So you can go and define say like a hex color and the way you define it is you would say I would like to create a you say beginning of line plus this hash symbol plus group repeated hex digit, or repeated other hex digit exactly three times plus end of line, and you write it in these things that are symbolic of what regular expressions do and they can be you know, sort of ordered in together and added together and so on. And then if you call the string representation of them you get the actually regular expression.

07:05 OKKEN: Okay, so it's a library to build regular expressions.

07:09 KENNEDY: Yeah, it's a library to build regular expressions. The way Bogdan describes it is he says, it's a tiny Python library made up of these combinators to help you write regular expressions that you can read and modify six months down the line.

07:23 OKKEN: Yeah, yeah definitely one of the problems with regular expressions is they're terse and that's good and bad, you, they're too terse sometimes.

07:33 KENNEDY: Yeah, sometimes. And then the other problem is there's a lot of special symbol, like regular expressions are basically nothing but symbols, right? They're like a symbol like exploded and its guts came out all over the text, right? But the problem is some of those symbols are have to be escaped if you actually want to search for them, but it's always kind of hard to know well which ones do I have to escape, so it also does thing like if you tell it I'm looking for the text like of square bracket, it'll escape properly in regular expression format the text representation of that right, because bracket normally means something else, like it's a set of characters or just something like that. So I really like that it's sort of a safe way as well, like it's a more, you talk about what you want and if it has to be escaped or whatever, it does. Yeah, it's a cool example.

08:21 OKKEN: I like this, this is great.

08:22 KENNEDY: Yeah, I do too, I can certainly see myself using this if I'm writing regular expressions. It's great. Speaking of seeing yourself.

08:30 OKKEN: Seeing yourself, yes, actually for a long time I've been following and paying attention to what Adrian Rosebrock, this is even an easy one and I even massacred this. Adrian Rosebrock, he has a site called pyimagesearch and essentially Adrian is teaching people about OpenCV and Python and actually a lot of AI stuff and doing some really cool things with Python and like cameras and webcams and stuff, and even on Raspberry Pis and stuff and doing lots of neat things. And I don't have a particular article to point to, we just haven't covered it before I don't think, and people should know about it. So Adrian has both paid and free resources to teach people all about computer vision and I think he's doing a cool job.

09:18 KENNEDY: Yeah, he's doing a real cool job and there's so many great examples over there, I think OpenCV is great and this is probably the best resource for OpenCV and Python intersected, right? I get emails every one and then with boxes detecting things running around on videos or something like that, it's great. So if you got to do anything to do with Computer Vision and you want to use Python to do it and why wouldn't you want that, then this is a cool place, right?

09:42 OKKEN: Yeah, and he has some, like tons of cool projects that he's done over the years of you know, hooking up a Raspberry Pi with a camera to detect people coming to your door and stuff like that, and it's, it's some cool stuff, so. And then you brought up that he had one of the most successful Kickstarters ever.

10:00 KENNEDY: Yeah, he did a Kickstarter called Deep Learning for Computer Vision with Python. And yeah, it's definitely one of the most successful Kickstarters ever. So if you want to check out that book, that's really good. I think he has some videos that are coming along with it, but I linked to the Kickstarter as well. Yeah, it did, it did okay. All right, so speaking of doing very, very well, I want to tell everyone about Digital Ocean before we move on and I decided it's time to think about this a little bit differently, think about Digital Ocean and your hosting stuff a little bit differently. So you know, most of us, I think you brought this up, you're not Netflix, you're not Google, you're not Facebook, right, remember that, you're not LinkedIn, you don't need these crazy architectures, and yet some of the most popular hosting platforms out there you know, like AWS or Azure, they are built with thousands and thousands of knobs so that you could be Netflix with 50,000 servers running continuous chaos experiments and all that crazy stuff, right, but if you who are actually just building what 99.9% of us are, it's more standard applications, then all that stuff is just overhead and complexity and cost, so you can join companies like Slack, WeWork, Docker, GitLab, and of course, us over at Digital Ocean and pay like five times less than what you would for AWS or Azure, right? So for example, bandwidth is one cent instead of nine per gigabyte, servers are five bucks instead of 50, or whatever a reserved instance is over at AWS. It's not just about price, keep it simple, use what you need, don't have all these crazy, crazy features that you probably don't actually need 'cause you're not building Netflix or Facebook or LinkedIn. Anyway, try them at pythonbytes.fm./digitalocean, get a $100 credit for new users, and see why we all love it for their infrastructure.

11:48 OKKEN: Awesome, thanks.

11:48 KENNEDY: Indeed. So, Brian, this next one, have you, have you watched this video here that I'm about to talk about?

11:57 OKKEN: I don't think I have, maybe I have.

11:59 KENNEDY: All right, so while we're talking click this and open it up and I'll tell people about it and maybe your mute YouTube that's going.

12:07 OKKEN: Oh okay.

12:07 KENNEDY: Okay. So this thing that I want to talk about is a visualization of Python development from original origin way back in the day up to 2012 and it's using this underlying system called Gource, G-O-U-R-C-E, Gource I'm guessing. And the idea is Gource is a visualization library that visualizes trees and stuff like this, like graph trees, not real trees, and what you can do is you can point it at a source repository that can be SVN or even support CVS, but Git and Mercurial, and you can point it there and it will do an animated over time visualization of that source library, so not just Python, any of these repositories you can point at. And it'll all have the files as they grow, the sizes of the repository, and then this little like animated characters come in and start editing files and interacting with it. So if you play this video that we're linking to, this is a Gource animation, a Python deveopment up until 2012. You can see it starts out and Guido's like cruising around adding little bits, then a few more people come, and then if you start skipping further and further ahead it gets just like madness at the end, like people are swirling all over the place, and it's just a really great way to like see the growth of Python visually through animation, at least in terms of who is participating and building it. What do you think?

13:29 OKKEN: Yeah, it is cool, and one of the fun things to see is that there's sometimes some people that just sort of sit around one area which makes sense and then the other people that fly around and edit all sorts of stuff.

13:42 KENNEDY: Yeah, it's really interesting, right? And you can see people appear and then they'll fade away, they'll come and make some contributions and then they'll like leave the scene. So anyway, I think this is really cool and it's, you know, there's not a lot to take away from it other than it's just nice to appreciate it. So I would say watch the first minute and then just like skip minute by minute and watch a little bit 'cause it's 14 minutes, you don't really want to watch the whole thing.

14:02 OKKEN: I think I do, yeah.

14:04 KENNEDY: You can just leave it running for the rest of the show. So here's what I, here's what I think. The call out to the audience. One, who wants to build this for 2012 to present on Python on the, the mod, you know, again. And I think this would make an amazing lightning talk, if you built that video and then you like went up there and just did a four minute animation, wouldn't that be cool?

14:25 OKKEN: It would be cool just to have that going on in the background while you did some other talk.

14:29 KENNEDY: Yeah, or run this at like between sessions at a conference, I think it would be great. Anyway, I think people will appreciate checking it out.

14:37 OKKEN: Cool.

14:38 KENNEDY: So one of the things Brian that we hear a lot in software development is that there's often a wide range of skills and productivity between developers, and I'm, I've done a lot of training and interacted with you know, literally thousands of people in person, and I think it's something of a myth, but I think largely there's a lot to this. There's some people that just fly and they're just focused and others that just kind of bounce around the keyboard randomly, what do you think?

15:04 OKKEN: So I know that it's a bit controversial, there is this notion of a 10x developer, and often there's a backlash against it also, but I think people think of it as like somebody that's 10 times better than the average good developer and I don't think that's it at all. I think that it's just a notion that there is sometimes orders of magnitude between the most effective person in a group, organization, and the least effective. And I don't know how you argue against it. And if you've ever been at large organizations it's just is, that at least maybe it's not 10 times, but there's definitely, there's a lot to it.

15:43 KENNEDY: It's a multiplicative factor in there I would say for sure

15:45 OKKEN: Yeah, and so this is, regardless of what you want to take away there's some good advice in this article. There's an article I'm linking to that's what any developer can learn from the best. And I think these are good things. So one of the things, the idea around it is just this isn't magical and it isn't something that is only, it isn't just about skills and hard skills, it's other stuff too, and there's a clear path to excellence, people are not born great developers, they get there through focus, deliberate practice. And here's a few traits, they just listed some traits of things that they see in good developers versus not so great. So great developers are, a few of the traits are problem solving, they're skilled at what they're doing, they're mentors and teachers, they're excellent learners, and passionate about stuff. So the problem solver bit I think is really interesting because often the 10x or the multiplier doesn't come in from that they do the same work faster, it's sometime is they can just look at things differently because of experience, because of playing around with lots of different things and say oh, let's just solve this problem differently, and it just gets done faster because they take that, take a different approach. One of the, I was talking about with somebody about databases recently and there's some, for instance, there's some problems that can be solved with graph databases easily that are almost impossible with a relational database, it's just using the right tool for the right job sometimes.

17:21 KENNEDY: Yeah, and I think a lot of that is that it's not that knowing that means that you're 10 times smarter or 10 times better, that means you were curious enough to keep looking and to continuously be on the lookout for well okay, I now totally know all about relational databases, what else can I learn, how do I compare it? And I think that's one of the biggest traits that I've noticed among people that have you know, some number of multiplier of efficiency or skills is not that they are necessarily more skilled or have more natural talent, but they are just continuously learning and really passionate about it and they're just you know, they're just always picking up these little things that help you at each step, right?

18:04 OKKEN: Yeah, and there's an honestly of just like this isn't just like about developers, I think it's in every field. For instance, somebody that's you know, working with like tools and stuff, somebody that needs to hammer in a lot of nails might be smart enough to go, hey, I should go get a nail gun, I could do this a lot faster with a nail gun.

18:24 KENNEDY: You know what that makes me think of? Like you could say I need to have the rings on my engine or you know, my transmission fixed on my car, and you take it to a regular place, they're like great, we'll have it back in three weeks. You go watch something like Indie Car Formula One, they'll pull in and they'll like change the transmission in two laps. Right, those are both mechanics. Those are not the same.

18:47 OKKEN: Right, exactly. The deliberate practice is important and I think some people forget that the difference between being an average or a below average developer and being an above average developer is mostly just deciding that you're going to do that and setting aside some of your time in your life to pick something you want to improve and then go and do it, figure out what you want to learn and go do it, so anyway.

19:11 KENNEDY: Yeah. I guess what I would say to take away from this article is it's really interesting, it's numerically based, right, there's a lot of, it's based on a survey the guy did with like 1000 folks or something, and most importantly, it's about a growth mindset, right? It's not to say that well there's these people and they're just smarter than you or they're smarter than other people or whatever, it's here is how those people got that way and you can do it too. And I think that's, that's the right message.

19:39 OKKEN: Right, and the person that wrote the article said this is from a lot of, he's taught a lot of people and there's a lot of people that want to be better, but they don't know how to, so this is just sort of some direction on what things to work on, so.

19:52 KENNEDY: Yeah, that's a good article. Yep, a nice find. Let's close this out with a bit of chaos, huh?

19:57 OKKEN: Sure, why change tradition, right?

20:00 KENNEDY: That's right. So you've heard of, I'm sure you've heard, I'm pretty sure you even had on your show if I remember correctly, you've talked about the Chaos Monkey and things like that, right?

20:11 OKKEN: Yeah, Chase runs with...

20:12 KENNEDY: Well tell people what the, yeah, he's from Netflix, right?

20:14 OKKEN: He used to be, yes.

20:17 KENNEDY: Okay, so I think the idea of the Chaos Monkey originated there, tell us what the Chaos Monkey is.

20:20 OKKEN: Well I'm probably going to get this wrong, but there's this notion of taking parts of your system and intentionality breaking parts or shutting down especially in a distributed system, taking some nodes and just killing them every once in a while and seeing how your system recovers from it.

20:37 KENNEDY: Yeah, exactly. So the idea is if you build an architecture both in infrastructure and software that is supposed to take durability, it's supposed to work if this part of your cloud goes down, or it's supposed to work if one of your database nodes goes down, that's the theory, but then there's the reality of how does it actually behave if one of your nodes go down, one of your machines reboots, random stuff like that, right? So the Chaos Monkey runs around in production killing off processes, servers, et cetera, and then in production it does it, and then you know, you just have to build, so 'cause you know the Chaos Monkey is coming and it means like well a standard failure is like nothing 'cause the Chaos Monkey is the madman, right, he's running around all the time. So, this is not a bad philosophy for large organizations or large bits of software, however, you're not Netflix probably, people listening, so how are you going to build that up, right, how are you going to create these things? So there's this cool thing I found called Chaos Toolkit, so Chaos Toolkit is a library built in Python that will help create these Chaos Monkey like things.

21:46 OKKEN: Cool.

21:47 KENNEDY: So Chaos Engineering is what they call it, it's the discipline of experimenting on distributed systems in order to build confidence in the system's capability to withstand turbulent conditions in production. All right, so we talked about the Chaos Monkey and the friends, right, there's other types of chaos things at Netflix, but here's a way that you can easily build those types of experiments and systems and it integrates with Kubernetes, AWS, Google Cloud, Microsoft Azure, some other, other things like that, right? So just to give you a sense of like what it can do, like if you look at the AWS API it'll say you can do things like go to AWS, lambda, and call delete function concurrency that removes the concurrency limit on any specific lambda, or you can just call stop instance on any EC2 instance, or you know, whatever you want, right? And presumably it's going to put that back, I'm not entirely sure, but I guess you probably got to call start instance again on it, or set, you know, add function concurrence, or set, something like that, right? But there's just this infrastructure to help you change these types of settings, these types of things around your cloud providers and you know, make sure your system can take it.

22:54 OKKEN: Nice, yeah. You cannot plan for the best so you plan for what you can for.

22:58 KENNEDY: That's right. Yeah, so you build it so that it doesn't have to be perfect, and then you're in pretty good shape. The stuff that the folks at Netflix are doing is insane though, like they take it to another level.

23:09 OKKEN: Well yeah, and one of the things I remember talking with them about is they, the reason why they test and do all this in production is because their system is so large you can't, it's essentially the world, you cannot, you can't have a test bed that's, that's similar enough to the real world environment, so they don't have that luxury, so.

23:32 KENNEDY: Right, so they just it in production. And I've not seen Netflix really go down so I'd say they're doing it. All right, that's it for our main news, I have a couple of quick ones. You got anything you want to share Brian?

23:43 OKKEN: Yeah, just on Python Bytes Episode 100 one of the things we talked about was pyproject.toml and I wanted to take a deep dive, so the last episode of Testing Code is me talking to Bret Cannon talking about that for almost an hour, so.

23:58 KENNEDY: Oh nice, I definitely want to check that, that's a good one.

23:58 OKKEN: So, how about you?

23:58 KENNEDY: I have two other things that I don't think justify a whole segment, but I just want to throw them out there. Remember a while back we talked about that there was potentially some experiments or something where some college researcher had put some malicious, potential malicious libraries on PyPI. Well it turns out recently someone actually put malicious libraries on PyPI, so this is October 27, 2018 and apparently 12 packages were discovered with various levels of vulnerabilities and stuff, so I'm linking to that and people should check that out. If you don't know how to spell Django, you're going to have a bad time by the way. So it's a lot of this type of squatting like oh I forgot the J so now I have a virus, something like that.

23:58 OKKEN: Yeah, that's sort of lame.

23:58 KENNEDY: Yeah, it's upsetting, at least they've done some work over at PyPI to block, properly spell things that are not actually packages, like re for example, right, the built and stuff.

23:58 OKKEN: Well yeah, that's one of the things that's yeah, was an issue is people can trying to pip install things that are in the Standard Library, you don't have to do that.

23:58 KENNEDY: Yes, exactly, exactly, don't do that. Then the other one was just a quick little Twitter message that someone sent my way and it's pretty cool. This guy on, not the one who sent it to me, sorry, I don't remember who sent it and I should've written it down, but this person called XTrek, something to that effect on Twitter decided to go scanning the Standard Library source code for interesting things like what is the longest class name, what is the longest function name and so on? So that they found out that actually in, in CPython the longest class name is 200 characters, just the letter A 200 times. Which is some kind of test case, but for the real ones there's one called TestMutuallyExclusiveOptionalAndPositionsMixedParent as the longest class name is there, and the longest function name they believe is test_parcer_regression_special_character_in parameter_column_adoption_first_line, which is 84 characters long. And then there's other, other examples. Someone said there's actually test C types that has 33 million characters in it. Anyway, it's an interesting thread if you're just wondering what the really long names are.

23:58 OKKEN: Yeah, it's almost competitive, it almost could compete with Java names.

23:58 KENNEDY: Yeah, exactly, standard types right there. Awesome, all right, well definitely fun to share all these news with you Brian. As always, thanks for doing it.

23:58 OKKEN: All right, thank you, bye.

23:58 KENNEDY: You bet, bye. Thank you for listening to Python Bytes, follow the show on Twitter via @PythonBytes, that's Python Bytes as in B-Y-T-E-S and get the full show notes at pythonbytes.fm If you have a news item you want featured just visit pythonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Okken this is Michael Kennedy, thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page