Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


« Return to show page

Transcript for Episode #65:
Speed of your import statements affecting performance?

Recorded on Wednesday, Feb 7, 2018.

00:00 Michael KENNEDY: Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is Episode #65, recorded February 7, 2018. I am Michael Kennedy.

00:00 Brian OKKEN: And I’m Brian Okken.

00:00 KENNEDY: And we’ve got a bunch of cool stuff. I’m really excited to share a couple of these things with you and I’m excited to have Rollbar as a sponsor. You’ve probably heard about Rollbar from other episodes. If you want to check them out, we’ve got a special offer at pythonbytes.fm/Rollbar. We’ll tell you more about that later.

00:00 I kinda want to know about some charts.

00:00 OKKEN: Have you ever seen the test output on Jenkins?

00:00 KENNEDY: No, I haven’t played with Jenkins.

00:00 OKKEN: Well, in Jenkins Builds, there this thing where, after you run your tests, you can have it display your passed and failed tests in this little chart. It’s an are a chart. I wanted one of those but I wanted something that would show five of those charts, like a whole bunch of them. It doesn’t do it automatically. I probably could have hunted into Jenkins or something, but I wanted to build my own thing. I want to do it in Flask and I’ve been trying to figure this out. There was a great tutorial by – this is going to be a little story – Matt McKay, from Full Stack, that showed how to do this in Bokeh.

00:00 KENNEDY: Very nice. That’s a cool project.

00:00 OKKEN: But Bokeh, the current version of Bokeh, dropped the charting feature.

00:00 KENNEDY: That’s not helpful. (Laughs) There goes your solution you just found.

00:00 OKKEN: I’m trying to do it and I gave up and asked Twitter. What came back, one of these options was pygal. It’s very simple; in 20 minutes I had a Flask app running with these charts up. It’s not as hugely featured as some of the other graphing applications, but it’s very cool and it also can export SVG (Scalable Vector Graphics), so these are small. It’s pretty small in when you’re displaying.

00:00 KENNEDY: That’s cool. The SVGs can have infinite, various high levels of zoo ability, right? Because they’re scalable. That’s their first name. They also do PNGs (Portable Network Graphics), like you said; that’s cool. They have a bunch of great looking charts. These are really nice. You can do bar charts, pie or the radar ones; all sorts of good stuff.

00:00 OKKEN: Though I still want to get something like OK or something like that to work so I can make them interactive, because these aren’t interactive things. You build them and display them. But for my needs right now, that works and it’s very nice.

00:00 KENNEDY: Very nice.

00:00 OKKEN: And kudos, to the people writing the documentation because they documentation in Django, Flask, pretty much any HTML you want to through this in, how to do that.

00:00 KENNEDY: Oh, yeah. That’s super cool. And they have nice pictures, which is always really powerful when you’re talking about graphs or UI or things like this. You can just admire the pictures, and go, ‘I want this. Now I’ll pay attention. Really nice.

00:00 So, a lot of people send us messages and they ask for advice on, ‘What sort of career path?’ Or ‘I’m thinking about this technology’ or ‘How do I get started in programming? Or something like that. There was this really interesting post over on Reddit under “Learn Programming.” It’s not technically a Python thing but a bunch of Python people are hanging out there, so that’s kind of cool. A person who posted sent a message and said, ‘Hey, I’m 31 days into a self-study program on Python and I’m loving every minute of it. A couple of questions. Could you all fill this out, just so we can share our experiences?’

00:00 “What were you doing before you began self-studying programming?” So, it’s for people who are self-studying.

00:00 “What made you want to study programming on your own?”

00:00 “How did you get started?”

00:00 “How long did it take for you to feel confident enough in your skills and knowledge to think that you could be employed?”

00:00 “What else do you do besides self-study that helped you?”

00:00 “What’s next?”

00:00 And this thread just blew up on Reddit. There's all these really interesting conversations. So, anyone out there who is in the early stages of self-study, maybe they’re doing a “100 Days of Code” type thing or they’re at boot camp or they’re taking online courses. I know a few places that have good ones of those. I think going through here and reading this will be really, really valuable to have that shared experience. Isn’t that cool?

00:00 OKKEN: Oh, that’s great.

00:00 KENNEDY: So, there’s another story that’s linked from there so this one doesn’t technically could as an item but people may also find it as a thread to keep following. Someone said, ‘There are people who like 30 and 40 and they’re like, hey, I’m learning code in my 40s and you guys have really inspired me to think it’s possible and it’s not too late for me. So, there's this article called, “Stories from 300 Developers Who Got Their First Tech Job in Their 30s, 40s and 50s” linked from there.

00:00 OKKEN: That’s cool. I have to go read that.

00:00 KENNEDY: Yeah, and I think that might be real helpful to some folks as well. So, if you’re in the space and you want some shared experiences and to connect with some people, check out this thread we’re linking to. I think it’s helpful.

00:00 OKKEN: On that topic of learning, actually, there’s a couple episodes of Test and Code that will be good. I just did one recently about extended learning through universities, which I hadn’t considered before. Also, there’s upcoming, I’m talking with Stephanie Hulburt on talking with mentors and opening up to be a mentor yourself as

00:00 well.

00:00 KENNEDY: Very cool. I look forward to checking those out.

00:00 OKKEN: This is one of those well-planned awesome transitions.

00:00 KENNEDY: Amazing. So, one of the things that you probably don’t care about when you’re getting started is how fast your imports run. You probably don’t even know what an import is in the beginning.

00:00 OKKEN: Exactly. But how long it takes to import your different modules is part of the painful startup process of any Python application. And I didn’t really know how to debug that, actually. I never really thought about it before but in Python 3.7 coming up, there is -X import time that allows you to use a flag that you can run that allows you to investigate and profile all of the time for importing different libraries. Which sounds really cool, actually.

00:00 KENNEDY: Yeah, that is really quite cool. So, you can just basically run it and it will tell you, ‘This library took that long to import. This other library took that long to import.’ And so on. I didn’t realize how long some of these actually took to load.

00:00 OKKEN: Like any optimization, it’s never really where you think it’s going to be. It’s probably someplace else, so having it profile is great. Sometimes you just have to import things but some parts your system may have too many imports initially and they could be possibly delayed until later.

00:00 KENNEDY: Yeah, you could possible do it conditionally, right? If you only know in certain circumstances, that code is going to run, you could maybe somehow delay the import until you actually need it. So, a lot of cool things.

00:00 OKKEN: Let’s say you have an editor application or something. All of the stuff that you would need to convert your format for saving, you don’t really need that all the time. You could load it when somebody’s trying to save.

00:00 KENNEDY: Right. Hit the ‘command S’ control or something like that.

00:00 Yeah, so very cool. This is a feature of Python 3.7 in the new fancy version?

00:00 OKKEN: Yeah, it’s the new fancy version. But if you’ve got an application working in 3.6 right now, you’re probably thinking about making sure it’s going to work in 3.7 by

00:00 now and you can probably use the 3.7 to optimize current code anyway.

00:00 KENNEDY: Because you want data classes and why not be able to time your imports anyway. (Laughs) Nice. So, this just came out in beta last week and we already talked about that. Here’s one more thing you can do if you’ve got some library that’s taking a while to basically import. I think that article you mentioned, I think the conclusion says that, so I can reduce my time for pip and --version from 800 milliseconds to 500 milliseconds. That still sounds like a long time, but that’s actually 60%, 30%, depending on which angle you measure it. Not quite, but it’s a big improvement.

00:00 OKKEN: And it’s down into the pieces where it makes it less noticeable.

00:00 KENNEDY: Yeah, quite cool. Alright, so before we move onto the next one, let me tell you about Rollbar.

00:00 Rollbar is a thing you integrate into your web applications. Whether you’re using Django, Flask, Pyramid; super easy to integrate. Basically, to integrate it into Pyramid, you just put a few things in your config file and that’s it. I don’t even think you have to touch your code if you want to directly interact with the Rollbar API, which you don’t often have to do. You can even get it for JavaScript and some other frameworks as well. It will basically be there when you’re app is running. Any time there’s an error, it will capture all the details. Stacktrace, the URL, all the requests and response values. Even the user who has logged in potentially. It will send that off and give you notification over in Slack or email, or all sorts of places. You want to make sure you’re not missing errors in your web app and if you’re not monitoring it, I bet there are errors in your web app that you don’t know about. Check them out at pythonbytes.fm/Rollbar.

00:00 Brian, speaking of web apps, this one we’re going to talk about now, it’s not the kind of web app you might have originally thought of. I didn’t mention it there in that Rollbar ad. AnPyLar. I’m not really sure of the pronunciation. You want to take a shot at it?

00:00 OKKEN: AnPyLar. I don’t know. It’s kind of fun to try.

00:00 KENNEDY: It’s fun to try and I think the ‘An’ is kind of like angular. The ‘Py,’ obviously Python. The ‘Lar,’ what a great thing to put at the end of a word. But what it is is, it’s a web framework for building front-end rich, client spa-type applications. It’s a Python framework. You write in it and it runs on the client side, in the browser.

00:00 OKKEN: Oh.

00:00 KENNEDY: Isn’t that interesting?

00:00 OKKEN: Yeah.

00:00 KENNEDY: So, it’s basically, create web apps with the elegance, simplicity and full power of Python. And you get these reusable components. It’s very much like AngularJS in that regard; it has a routing engine so you don’t actually navigate, but you move around the different views of the app. You write all the stuff in Python. It’s pretty interesting. So, it’s got a reactive programming model. It has Promises, standard Python formatting, reusable components, the scope styling, integrated routing engine, that’s the stuff I was talking about. So, it’s pretty cool, actually. I don’t know what the underlying technology is for making the Python run. If they built their own thing, if they’re using something like Brython or Sculpt, or something else. But yeah, it’s pretty amazing.

00:00 OKKEN: Yeah, it sounds like it would be fun to play with.

00:00 KENNEDY: Yeah, definitely fun to play with. The one caveat is if you’re going to embed Python compiled to JavaScript in a JavaScript file downloaded, the JavaScript file may be bigger than standard JavaScript file sizes. I checked and I think the minified version is 3.6 megs. Which is…

00:00 OKKEN: ...a little large. (Laughs)

00:00 KENNEDY: It depends on each case. So, imagine this. On one hand, if you’re building a really high traffic website, people use it for mobile or low speed connections, it’s totally unacceptable to put your front end in this. However, if what you’re doing is you’re building ElectronJS apps, you’re building Ionic and Cordova apps. The sort of, offline, ‘Here are my files,’ run them as if it was a web app, it doesn’t matter if the thing is big, right? It just starts up off the local disk anyway.

00:00 OKKEN: Yeah, that’s true. Those are great use case.

00:00 KENNEDY: Like VS code, for example, is ElectronJS. Ionic has a bunch of cool stuff. Under that circumstance, who cares if you’re uncompressed shipping size is another 3 megs. It’s already 100 megs, what's 100 versus 103? Nobody cares, right? Something like that. For those situations, where it’s kind of this offline, rich client, web experience, it’s cool. I like it.

00:00 OKKEN: It’s a good idea.

00:00 KENNEDY: There’s some nice ways to play around with it. They’ve got little demo apps, so down the bottom there’s a little grid of buttons – like eight or ten or something – you can click around there and play with it a bit. So, yeah, it’s pretty cool. If you want to give this a try, and your use case, it makes a lot of sense. Somehow getting that JavaScript there is not a big problem. Definitely check it out it’s

00:00 cool.

00:00 The other thing to point out is WebAssembly is coming. WebAssembly is going to be pretty sweet. If somebody can get a really nice version of Python in WebAssembly, that may make a big difference on a lot of fronts. Do you know WebAssembly?

00:00 OKKEN: I do not. So, WebAssembly basically is a new standard for the browsers. I think it’s already partially supported where instead of shipping JavaScript, you ship binary executable equivalents. So, you got the problem of running, you download a text version of a big thing and then you parse it. You do all the runtime stuff to get it executable and then you execute it. You get it into bi code or whatever JavaScript does. So, I think WebAssembly is more or less, ‘We’re going to get it to that last pre-process step of what JavaScript would do and will give it to you as a binary thing.’ Sort of like shipping .pyc files in Python. It think it should be a lot smaller because it’s small and tokenized and stuff, but also start up quicker. Who knows, maybe WebAssembly makes this better for somebody sneaking Python through the JavaScript straw.

00:00 OKKEN: Running random binaries off of any website on the Internet. What could go wrong?

00:00 KENNEDY: I think it’s only the equivalent of running JavaScript. It’s not like fully executing data sites.

00:00 OKKEN: Yeah, okay.

00:00 KENNEDY: I think it’s no worse than JavaScript, but yeah. I don’t know more than that.

00:00 OKKEN: Okay. I’ll have to keep track.

00:00 KENNEDY: I think we should round out this entire, the rest of this episode on just Python 3 stuff, what do you think?

00:00 OKKEN: I don’t think, on this podcast, we’ve pushed the migration to Python 3 enough.

00:00 KENNEDY: No. We have talked about how bad Legacy Python is. (Laughs) Go ahead. You go first on this Python thing.

00:00 OKKEN: This is actually just a little GitHub repo that is called, “Migrating to Python 3 with Pleasure: A Short Guide on Features of Python 3 for Data Scientists.” It’s a pretty quick read but it’s pretty thorough. It actually gets pretty exciting. It starts off with a quick tutorial with examples of how to use pathlib so that you can easily – if people haven’t played with it yet – you can define paths as these little strings with division marks, like you would in a browser.

00:00 KENNEDY: It’s kind of crazy, right? Pathlib. You put the slashes outside the strings and the division operator becomes OS.path.join type of things.

00:00 OKKEN: Yeah, definitely. But it works great and it makes your code really readable. So, there’s that and it goes on to talk about type hinting and how cool it works with editors like PyCharm and stuff to help you see what you’re doing. Then one of the things that I hadn’t heard of before I came about a few things in here, is how to enforce types at runtime. I didn’t know you could do that, but there’s a package called Enforce that you can put a decorator on a particular function and it will throw an editor if somebody tries to put a pass in the wrong type.

00:00 KENNEDY: I had never heard about that either. That is really awesome. I don’t know of I necessarily want it all the time but it could be fun to turn on. Especially if you’re doing a refactoring or you’re doing some major upgrades. Like, ‘Let’s turn this on and just see if it’s doing what we think it’s doing.’

00:00 OKKEN: Yeah, especially if it’s a testing phase. It would be fun.

00:00 And then this kind of hurt my head. Some function annotations for units, there’s an example of how astropy does it. And it’s things like, you can attach to variables and units, like kilograms or something like that, which just seems cool. I’ve got to play with that a little more.

00:00 KENNEDY: Oh, it looks like… There’s a library of things called Pint.

00:00 OKKEN: Oh, right, yeah.

00:00 KENNEDY: That you multiply say, this thing times a foot, that thing times an inch, and then add them together. Yeah, that’s it. And this look like a little bit of the same type of story, but you might multiply it by a kilogram or a degree. It’s sort of the physicist or astrophysicist equivalent of that. It’s cool.

00:00 OKKEN: Their example showed dividing a meter in speeds in, I don’t know what it is, but coming up with terahertz when you didn’t start with terahertz, but it was something else. I’m just making up words now, I think.

00:00 KENNEDY: (Laughs) Pretty cool, pretty cool. You also have matrix multiplication in there.

00:00 OKKEN: Yeah, which I don’t use a lot but it’s very cool for people that need matrix multiplication.

00:00 KENNEDY: Matrix multiplication is one of these things that’s held up as one of the best examples of the whole PEP process and bringing a new feature into the language. So, from the time the PEP was proposed until it was done, was like eight days. It has to do with how well the people who propose the idea have already done market research amongst all the scientific computation people. They had a really clear vision and a strong displayed need, so it’s really interesting that that’s in here as well.

00:00 OKKEN: And Globbing with ** which is something that I hadn’t used. Z shell does that. Recursive subdirectories with 2 asterix; that’s kind of neat. We threw a whole bunch of other things like f-strings and ‘real division now.’ The rest of it is things that if you’ve been living in Python 3 for awhile, you just kind of take this stuff for granted. But it’s a fairly complete list. It would be good if you are in the science community and you’re trying to convince somebody to switch to Python 3; this article would be a good one.

00:00 KENNEDY: Yeah, definitely a good one. It sounds really interesting. It really brings up that these are the benefits that you get from making this move, rather than just the stake of, it’s going to be expired or you don’t go out of maintenance in 2020. It’s like, ‘Here’s all the good stuff that you’re missing that you don’t even know you’re missing.’

00:00 OKKEN: I didn’t even list it but it includes dictionary ordering and stuff, which I love that.

00:00 KENNEDY: Right, yeah. That is really, really nice. I think Enforce might be the big news out of that. I hadn’t heard of that. That’s pretty cool.

00:00 OKKEN: Yeah, I’m definitely going to go play with that tomorrow. That’s neat.

00:00 KENNEDY: Cool. So, let’s talk about moving to Python 3.

00:00 OKKEN: Okay. Completely new topic.

00:00 KENNEDY: (Laughs) Exactly. I want to give you something like this but from a web development perspective. It’s pretty interesting. There’s this place called ticketea.com. And they said how we migrated from ticketea.com to Python 3 in two weeks. It’s apparently a pretty big project that they have running that runs their whole fulfillment ecommerce type thing and so on. You’ll hear the word Aphrodite. That’s their code name word for their project. So, they do a bunch of stuff with Docker and their first step was to say, ‘We’re going to change Aphrodite’s base Docker image to Python: 3.6 slim from 2.7 fat. I don’t know. (Laughs) Something like this. And when they did that, they just tried to run their tests and a bunch of stuff went crazy. Outdated libraries didn’t work, base string had to be moved to string, urlparse had to be moved to urllib.urlparse and other stuff like that. ‘So, the first thing we’re going to do is run two to three.’ This is the utility that can manually fix some of things. ‘Every time you see base string, that’s str now. We’re just going to do that,’ for example. They ran that against it and they also looked for patterns like, ‘Hey, we’re using map reduce and map-type functions and so on or filtering functions plus Lambda to make it work.’ These could really just be list comprehension skills. They took better advantage of the new language features that were then available to them. So, that’s pretty cool.

00:00 And then, that’s the low hanging fruit. Now what? And they had to run their tests to find some more problems. They took the opportunity to upgrade, at least get ready to upgrade, to Django 2 because they are running on Django, so that’s cool. They said, ‘A couple lessons we learned. Code coverage was 75%, 70%. The more tests the better. It’s good to keep the Python 3 branch updated with the master branch until you switch.’ That’s one way of doing it.

00:00 Remember the Instagram Keynote at PyCon 2017?

00:00 OKKEN: Yeah.

00:00 KENNEDY: That is a whole other level of upgrading to Python 3, right? They upgraded all of Instagram, Python 3, without branching. At all. That’s crazy. They were checking in all the time. So, that’s another level. You can check out that as well. I suspect this is more common that you kind of branch. I’m going to try to keep these going.

00:00 OKKEN: It’s a good story to see somebody doing it the other way.

00:00 KENNEDY: Yeah, for sure. So, they ran Flake8, consulted stuff to make sure everything was working right. I don’t know, I think they had to modify around 200 files to make Flake8 happy again with Python 3, but yeah, all good.

00:00 And the final thing I thought was really cool, they’re running on Google App Engine. I don’t do anything with Google App Engine. But one of the features that these platforms of service have that’s really pretty cool is something called traffic splitting. So, there’s basically like a little slider and you say, ‘This version of the app is going to get 60% of the traffic and this version of the app is going to get 40% of the traffic.’ And you can add in different versions. You can say, ‘Let’s just send in 1% of our traffic over towards these new versions and see what happens.’ So, you don’t completely take the site down, you just break it for a very small group. Then you can even do that, I think, by IP address and stuff. You can say, ‘Only the people that work here get the new version. People on the outside, they get the old version.’ And slowly migrate it along. They call these ‘canary releases’ which I think is actually a pretty cool way to think of it. There’s some neat lessons in here.

00:00 OKKEN: I didn’t know Google App Engine did that. That’s neat.

00:00 KENNEDY: Yeah, I didn't either, but I definitely like it a little bit better now. I don’t dislike it, but it’s definitely a feature that would draw me into it. I think this is really interesting. Yours was on data science and mine was a sort of web story. They both are compelling for moving to Python 3. The things that you get and the steps to get there.

00:00 OKKEN: Definitely. And we’ll keep pounding on it until Python 2.7 is gone.

00:00 KENNEDY: That’s right. We will see Legacy Python fade away.

00:00 So, you have a webcast coming up, don’t you?

00:00 OKKEN: Yeah. We brought it up last week, but I just wanted to remind people that I’m doing a webcast on pytest and PyCharm with the PyCharm guys. That’s coming up on February 22nd, so we’ll leave a link in the show notes. Go register. Hope to see everybody there.

00:00 KENNEDY: Yeah, you should definitely go register for it. There’s a couple reasons. One, Brian’s going to be there and it’s going to be awesome. Two, it’s free, so you’ll get what you pay for, for sure. I promise you that. (Laughs) Unless you count your time, then I don’t know. And then is you also can’t make it, you’ll get the recording notification. If you register and you don’t show up, then you’ll get a message saying, ‘You didn’t show up but hey, you can still get it.’

00:00 OKKEN: You should check out, there’s a whole bunch of old webinars there. It’s not just about trying to push PyCharm stuff. It’s just people in the community showing some interesting things to do. It’s a good thing that they’re doing.

00:00 KENNEDY: I had this project that I wanted to play with. Not so much to build the app, but to play with the foundational bits of the app. I have a program that demonstrates using cookiecutter from an API mode. So, cookiecutter, normally you type cookiecutter (space) some template name then stuff happens and you have a project. Like a starter project, a scaffold type thing for all sorts of various things.

00:00 There’s also an API for cookiecutter. You can put fancy front ends, or if you’ve got any kind of application that needs to generate a sub-project that’s going to be messed with by somebody else, if you’re doing developer tools or you’re building something for your company. You’re like, ‘We’re always going to start this way.’ Here’s the CLI, it asks you the questions and boom, you’ve got this kind of standard starter app that’s integrated into your infrastructure. So, these are all good uses. So, I said, ‘Alright, I want to play with this project called Gooey.’ And we talked about Gooey a while ago. So I took my little cookiecutter app, which would would with all the Pyramid-based cookiecutter templates and put a friendly front end on the front of that, and then I put a Gooey on top of that CLI app.

00:00 OKKEN: Nice.

00:00 KENNEDY: Yeah, it was really fun. It really took 20 minutes to get it 99% working. So, I put a link to the GitHub repository that has that up there so, people can download it. You just clone it, create a virtual environment, pip install -r, the Requirements file, then you can just run the program and you have a really sweet Gooey. It’s based on

00:00 wxPython Phoenix and Gooey and it’s quite cool.

00:00 The one little hang up is I’m having a problem getting it to run if I package it so I can literally hand out a .app or .exe. There’s some kind of hang up and I think it’s a bug with Gooey. I can’t tell entirely but maybe by the time you hear this, I’ll have it figured out. We’ll see.

00:00 Either way, it’s a great example. If you can forget the packaging, it’s actually an awesome example of a really simple Python GUI app that looks professional.

00:00 OKKEN: If you can forget packaging, Python is awesome. (Laughs)

00:00 KENNEDY: The packaging works fine. Gooey works fine. A packaged Gooey app sometimes works fine. (Laughs) It’s the intersection of all of these things that are cool, there’s a small bruise on the shininess of it. But it’s all good. Check that out.

00:00 OKKEN: There’s tons of people working on all these features to make it more seamless in the future.

00:00 KENNEDY: Absolutely. Well, good job. Thanks so much. Brian thanks for sharing everything with all of our listeners and with me.

00:00 OKKEN: Thank you. I’ll talk to you next week.

00:00 KENNEDY: You bet. Bye.

00:00 Thank you for listening to Python Bytes. Follow the show on Twitter via @python bytes and get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We’re always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page