« Return to show page
Transcript for Episode #85:
Visually debugging your Jupyter notebook
0:00 KENNEDY: Hello and welcome to Python Bites where we deliver python news and headlines directly to your earbuds, it's Episode 85, recorded on July 3rd, 2018. I'm Michael Kennedy.
0:10 OKKEN: And I'm Brian Okken.
0:11 KENNEDY: Hey Brian, how ya doing?
0:12 OKKEN: I am doing great today!
0:13 KENNEDY: I know it's another day in paradise, here in the Pacific Northwest, isn't it?
0:16 OKKEN: Yeah, as long you're not driving, this is the third of July!
0:20 KENNEDY: That can be problematic, we do have our traffic issues, don't we? Yeah, well, yeah before we get to our items, I want to say thank you to DigitalOcean, they've been supporting the show in a major way, like I've said, our stuff runs on DigitalOcean, which is really great, if you want yours to run on it, go to pythonbytes.fm/digitalocean, get a hundred dollar free credit for new users. I find myself, Brian, loving more and more the concept of limited use of type hints, in just the right places, I was just typing that 15 minutes ago.
0:49 OKKEN: Were ya? Yeah, yeah, that's nice. The more I use them, the more I'm more warming to them, and there's a couple resources that came by, just recently, that I really enjoy. So there's an article called the "The state of type hints in Python" and this is a fairly thorough article, but it pretty much like, okay now it's 2018, what is it like to use type hints now, and the article talks about if you want to use type hints in 2.7, and prior to 3.6 sort of stuff, and that's doable, but I still think that you get the best experience if you're using Python 3, and especially the Python 3.6, 'cause you can use commented type annotations in 2.7, I believe, because they're just comments, but adding the little arrows for the types and the colons, and everything, it's more concise and it looks better, and my personal feeling is unless you really are trying to convert a large code base, which I know some people are, I would switch to 3 first, and then add type hints, if it was me, if you have that option.
2:02 KENNEDY: Yeah, and 3.6 is even better than 3.5, 'cause you can put types on variables.
2:07 OKKEN: You can, although I find I'm not really doing that, when I'm using type hints, I'm just mostly doing it for interface, to document interfaces.
2:14 KENNEDY: You know where I end up doing that? Is where there are core functions that I can't control. So I need to call a function, and I know it returns something rich, that I know what the type is, and I would like to be able to get better editor experiences, better auto complete, but it's a package function, not something I've created. So then you can say this variable is a SQLAlchemy session, for example, and then it's like, oh I see, now you've got to add and delete, and all this stuff, and it's amazing right?
2:47 OKKEN: Yeah, and you know one of the things I like about the article, and I agree with, is a good reason to use type hints isn't, I mean it is good to be able to catch more bugs and incompatibilities, but there's a lot of reasons why it makes your code maintainable, and more readable because you just have that option, to be able to document it right there, to say, hey this is the type that this is the type that this function needs, so that, there's so many things, and the article walks through a few of them, like refactoring is easier, helping when you come back and read it later, it makes it more obvious. I mean, when you're writing, yes there are some great polymorphic functions, that could take multiple types of type, multiple types! But you know, most of the code I write, really does expect one type, it expects a list, or maybe the most generic things is it expects an iterable, and I'm actually still not sure how to write that in, the type shed stuff, I'm sure there's a way. But yeah, this article does talk about things like unions, I ran mypy on some of my code, and I realized that I had some default values set to None, which I wanted to be okay, but if they're not none, they're strings or their integers, or something. And the type system allows you to add a union type, so you can say, well it's either None, or it's a string, and it's pretty interesting, but this article talks about a lot of stuff, that I really had forgotten about actually, like interface stub files, and I haven't worked with those at all yet, and I know that the type shed is around, that has these interface stub files for the standard library, and a bunch of other popular libraries, but I haven't really used it yet, or maybe I have, but I just don't know about it, but I don't know, this is good.
4:37 KENNEDY: Yeah, you have a interesting quote in here, that type hints should be used whenever unit tests are worth writing, what's the back story on that?
4:44 OKKEN: It's just one of the, in the conclusion of the article, it talked about that, it said okay all of this information, but when should you use it, I mean if you're going to just write a little utility function for, I don't know, for your own use or something, do you need type hints then? And they just came up with this rule of thumb, is if you're going to, I quoted it just as is, it says unit test, I personally would just say automated tests, if the code you're writing is worth automated tests, it's worth putting, trying to think about where you're going to put type. The other thing I wanted to add was, I was listening to, I got halfway through it so far, but there's a recent seminar that Guido presented at Stanford, called Optional Static Typing for Python, and part of that talks about mypy, but it talks about some of the beginnings of this, like where this came from, and how it came to be what it is now, and so for how are we dealing with it now? So it's about an hour and a half though, so it's a chunk of time to watch, but...
5:42 KENNEDY: It's a commitment yeah, but it sounds interesting.
5:44 OKKEN: Yeah, but I think I recommend even like the first 20 minutes, for some of the backstories, is interesting.
5:50 KENNEDY: Okay, yeah that's really cool, I'm definitely a fan of it, and I'm glad you're bringing it up, there's some nice stuff in there.
5:55 OKKEN: So I have no idea how to transition from type hints to MongoDB, so I guess that's a reasonable transition.
6:02 KENNEDY: I'd say so, I've actually found a pretty interesting thing on MongoDB, people know that I'm a fan, of course, and one of the big decision points, I think, when you are trying to decide, do I go with Django or do I go with one of these smaller web frameworks, like Pyramid, Flask, many of the other micro-frameworks, is the micro-frameworks, you can pick the little pieces, I want SQLAlchemy here, and maybe this parts going to be Mongo, and this parts going to be this other aspect, that I would like to bring into this framework for authentication or something, whereas Django is more like here are your big building blocks, click them together, right? And a big challenge there can be like, the Django ORM, and the admin stuff, it's all tied in to that, and well if you don't want to use that, then you're kind of out of luck. Right, there's a lot of challenges, of sort of bringing your own thing, you lose some of the value, at least out of Django, traditionally.
6:53 OKKEN: So the Django ORM it depends on not a particular SQL database, but it does assume a SQL database, is that correct?
7:01 KENNEDY: Yeah, I believe so, I believe that's right. And things like the Django admin exception, which is really valuable to some people, depends upon the Django ORM, right? So like there's these layers of stuff, and so if you pull off bottom brick, there's problems. But actually this guy named Robin, sent me a heads up about this thing on Twitter, that I had never heard of, but it's pretty awesome. It's called Django MongoDB connector.
7:26 OKKEN: Cool.
7:27 KENNEDY: So the idea is, you continue to use the Django ORM, just like you normally would, except it's now had it's backend swapped out to be MongoDB, And that's pretty awesome. And at first you're like okay well how much value am I get in that, right? Like if I continue to use what is effectively, a relational ORM, like you can't use the hierarchal documents, right? But what they've done, a little bit like Mongo engine, is the parameters that go into your query, can be more interesting, right. So you would say, imagine if you had a blog, you'd say blog.objects().filter(), and you would pass like name equals something, and in traditional relational stuff, but in Mongo, you might want to traverse, like into a blog object, and then go find the name, and then find that the name is contained in a set, and so you just pass different arguments to the filter parameter, and boom, it becomes Mongo hierarchical enabled, it's beautiful.
8:28 OKKEN: Wow, that's actually pretty cool.
8:30 KENNEDY: Isn't that cool? It's so simple. And it says you can also use it to connect third party apps like Django REST framework, and Viewflow, and other stuff, which heavily depend upon these Django models, they can now super easily be integrated from MongoDB as well.
8:43 OKKEN: It has a hilarious name.
8:44 KENNEDY: Yeah, what is it?
8:46 OKKEN: Jongo!
8:47 KENNEDY: Yeah, jongo or something like Jongo, the official Django app, you put in the settings file was Jongo.
8:52 OKKEN: That's great, yeah cause for, I get it, Django, Mongo.
8:57 KENNEDY: Django, Mongo, smushed together, yeah exactly. Another thing that's interesting, is this is a Python 3.6 or higher only, and Mongo 3.4 or higher, which is like a year, a year and a half old, something like that, so it's not quite so new, but Python 3.6 is really quite new for the store.
9:14 OKKEN: Yeah, but I mean it also seems like something you would start a project with, you probably wouldn't like rip apart an old project to...
9:21 KENNEDY: Probably not, I mean you may have an app you're trying to make better, and migrate, but yeah I would think you probably wouldn't really uh, yeah I threw that more in there, just like it even has like a pretty high requirement for MongoDB, just so if somebody's got like an older something or other, you can't just jam it in there, you got to make sure you got the new goodness.
9:39 OKKEN: Okay, I was actually, so this is sort of related, I was talking to someone about Mongo before, and they asked me about migrations, from if you're going to changed the database, does Mongo have that sort of concept?
9:53 KENNEDY: It does have that concept, you do it a lot less often. Much of the time what your migration is, is something like I'm adding an additional column here, I'm adding another table there, like it's not that common that you're, say, deleting a column, or you're renaming a column, it happens, but it's way less often than new column, new table. And in MongoDB, new column or new table is automatic, you don't do anything, it just adapts, right, as you use it. So you don't actually run any scripts or migrations to do the common transformations, but if you're going to, say, like delete a column, or reorganize your data, you would do the same thing, and you would run a script against it. I don't think there's something like a Alembic, where there's like a framework for migrations, but the concept of like, a migration with the scripts, definitely exists, but it's not very common.
10:41 OKKEN: Alright, and I brought this up hopefully, cause if we're wrong about this, someone can hopefully get ahold of us and tell us stuff.
10:47 KENNEDY: Yeah if there's some cool framework for doing that, let us know. Awesome, so what's this deal with Python Idioms you got going on here?
10:55 OKKEN: This was a tiny little article, which sometimes I like that, just a small article, of like, "Hey I learned this thing, and it's helpful, and here's a small blog post about it." And this was Amir Rachum, why do I try to pronounce people's names, anyway, multiline strings. So the idiom he's talking about is multiline strings, so if I've got, and we know these are like a triple quote, you can have strings, strings can be multiple lines, if you use triple quote, and we usually see these with docstrings, but it can be really anywhere you're going to use a string. But if you're using it like in the middle of a function, it's awkward because it either, everything but the first line has to be, is like over on the left side of your page. And it's not indented the right place, and if you do indent over and over, it's including all of those spaces in your string. But the way to get around it is to use a standard library function, the module or the package is textwrap, and the function is dedent, so it's textwrap.dedent, and that will strip out, what it does is it takes, it looks at all of the lines of the multiline string, and takes takes the common spacing at the beginning of it, and just rips that off. So it's just automatic, and I use this enough with generating tools and generating other things, and I can't think of a good example right now where I've used it, but I have used it enough times, and I've looked this up, that I want people to know about it, it's a easy trick, so it's good.
12:33 KENNEDY: Oh I love it, I didn't know about it, thank you. This is cool, so I've definitely had multiline text strings, because I've got some big formatted piece of text, and I'm like alright, instead of trying to break this apart, and turn it into something I can store in code, if I just, you know, triple quote it, and put it in here, it's going to be golden. But you do get that weird, you've got to like, indent it to the left outside of your function, and it's all bizarre-o. So this let's you keep your code looking really nice, and it obviously puts the string to, like, visually into the block that it belongs in, it's really nice, I love it.
13:04 OKKEN: Yeah and I've also like tried to punt, and like put a defining global variable, or at least a file global variable, so that I can get around it, but this is cleaner, and it's good, it's a good thing.
13:16 KENNEDY: Yeah, I love it, that's awesome! Other good things include DigitalOcean, so definitely want to tell people about DigitalOcean, because they're sponsoring the show in addition to just us being happy customers there. So you can go from zero to a customized server in sixty seconds, I've talked about how we've done cool stuff with virtual machines, how they've got spaces, how they've got other various things that are pretty awesome, like the one-click containers, for like machine learning. One thing I haven't talked about yet is container distributions, so if you want to set up your own like docker container, you can go click over there and create a Core Os server, or even a Rancher OS, which I've never used, but it's got a cool icon. You can just fire one of those up, and it's like all ready to go to be your container host, for all sorts of cool docker stuff, and Kebernetes. So definitely check them out over at pythonbytes.fm/digitalocean, get a free hundred dollar credit to play around with those kinds of things.
14:12 OKKEN: That's actually pretty cool, I like it.
14:14 KENNEDY: Yeah I like it too.
14:15 OKKEN: And I had a DigitalOcean accident the other day.
14:18 KENNEDY: Oh no, what happened?
14:19 OKKEN: Well I wore my favorite DigitalOcean t-shirt to work and I spilled coffee on it, but luckily it was a gray shirt, and it didn't show up as a stain, so it's all good.
14:29 KENNEDY: It's pretty fail safe, unless you're painting. So how do you feel about your design skills, Brian, like are you a fan of shuffling around CSS and HTML, and that kind of stuff?
14:40 OKKEN: No, CSS is definitely one of those things where I either rely on a framework, or I try to hire somebody.
14:46 KENNEDY: Nice, so something that you might be able to use, if you're using Flask, and I know you've played around with Flask recently, is this thing called Flaskerizer. Okay I don't know what the name stands for, I guess to make Flask-ish, or something. But the idea is, you can go to one of these places that has Bootstrap themes, so Bootstrap is nice obviously, it's design design framework, pretty well known. But what's really awesome about it are the themes, like people make these pre-made themes, for like ten bucks, or even free a lot of times, you can get like incredible web designs, and you just drop them in right, you just put in your logic into the bits where it goes. So this Flaskerizer thing is, what you can do is you can download the themes from certain locations, that have a known format, and it will convert the Bootstrap, the static Bootstrap theme, into a Flask app.
15:39 OKKEN: Oh that's cool!
15:41 KENNEDY: Isn't that sweet? So instead of worrying about how it all goes together, you just go command line, boom, now we've got like a dynamic running, Bootstrap theme, in Flask,
15:51 OKKEN: I like it.
15:51 KENNEDY: Yeah, so there's a couple of sites that they work with, I don't know how general this is, because I don't know how common Bootstrap themes are, but I think it's pretty cool. And of course if you open up the website, and you look around, you can tell that it's a proper design, this is the Github repo, because they go from just plain white like no design, no logo, to clearly a designer with a black turtleneck.
16:18 OKKEN: Yeah, he's definitely a designer, dark black glasses too, and a beard.
16:24 KENNEDY: Yeah exactly. Oh he's all good, he's got all the boxes checked. Very nice, so if you want to do a Flask site, and you want a design, to go pick a theme, and just start working with that design, check out Flaskerizer, it could get you up and running, really quick.
16:39 OKKEN: Wow interesting, I was already reading down this, this is one of the few applications I've ever seen, or the frameworks, that uses nose2 as it's testing.
16:47 KENNEDY: Oh nice, what's the story with nose2? I only know about nose.
16:50 OKKEN: Well so nose was sort of kind of abandoned, but nose2, well I'll offend to anybody that's still working on nose, but nose2 was kind of trying to reboot it, and redo a lot of the, make it backward comparability break, so it doesn't do some of the things, but it does other neat things and clean up the code base. But still, it started kind of running, getting some steam at the same time PyTest was sort of getting a lot of steam, so I think that there's not a lot of projects that use it. But I mean, it's still reasonable, it's just since there's not that many people working on it, getting fixes and stuff is sometimes an issue.
17:30 KENNEDY: Yeah, I see, well very interesting, very interesting. So are you learning Python now, is this your next project?
17:36 OKKEN: No, but I thought this was fun, because I encourage every youth I meet, and actually everybody I meet, to learn Python.
17:48 KENNEDY: Good advice, good advice.
17:49 OKKEN: I tell them, the reason isn't so that you can become a programmer, it's so that you can, it's a power boost to, what are the other skills you have. So if you're a biology student, also learn Python, cause now you're a biology student that knows Python.
18:05 KENNEDY: You're now the favorite person in the lab, that can actually solve the problem, with the data they got in Excel, or something right?
18:10 OKKEN: Yeah, and it doesn't matter, it's science or non-science, if you're an artist, that also knows how to write code, it's just going to help, there's no down side. And so, a lot of times with code, people kind of have, they start writing code and it's precious to them, and they don't want to throw it away, 'cause they worked really hard on it, and so I wanted to highlight this article, because it's advice, this is, I think it's a Real Python, yeah it's a Real Python article. And it's called Learn Python the Methodical Way, and I'm just going to read the steps. The steps are: make your way through a tutorial or a chapter from what you're learning, a book or something, that teaches you some discreet, 4 to 6 step skill, you're going to work through it, while you're reading it. Now, write down the steps, as distinctly and generically as possible, put the tutorial and chapter, and its solutions away, and build your project from scratch, and peek at the solution and steps, only when you get stuck. Now erase the whole thing, and do it again, and then go back completely, put it away, and then erase everything, and do it again, like a day or two later. And this sort of erasing and redoing it, I think at least a couple times is good for people, because it gets rid of that preciousness, because the second time you do something, it's always faster, anyway I thought this was neat advice, so I thought I'd share.
19:35 KENNEDY: I like it, it's pretty interesting. I definitely feel like when you're new, you feel like "I put a lot of effort into that, and so there's no way I'm throwing that away." But the really valuable lesson is, actually the second time, it's probably better. And the third time, you're like "Whoa this really a much improved version of what I did." and it takes less and less time of course, right?
19:59 OKKEN: Yeah, and also you end up like, it's easier to like, throw away the cruft. And you know, if it's just something you spent an hour working on, even if you had to redo the entire thing, it's only a wasted hour, and you'll realize that you can really do it again, pretty fast. And so the only part I kind of disagree about is, I never code without resources anymore, but I wouldn't say try to do it just from memory, but try to use the normal resources that you would otherwise like Google, or Stack Overflow, or whatever, to look up things, I think it's silly to try to memorize like the order parameters to a function, that's what code completions for, and stuff like that.
20:40 KENNEDY: Exactly, that's why we have type hints, isn't it? Yeah so I totally agree, I don't think these like, completely abstract, in isolation, sorts of things, makes sense, but maybe the idea is like don't just keep following these steps, three times, but like try to recreate it from your mind, and to me recreating it is totally fine, to go up. But how do I actually insert that thing, what is that function called? Like that is a totally reasonable Google search.
21:07 OKKEN: Yeah definitely.
21:08 KENNEDY: So we've talked about debuggers before, we talked about the break point thing last time. One of the areas where doing proper debugging, and by proper debugging I mean visual debugging, not command line debugging, but like visual debugging, where you see all the state of the system and everything. That can be really tricky in Jupyter Notebooks, right 'cause, you know like attached to them, they're running in the notebook, in your web browser, right?
21:33 OKKEN: Yeah, I've never tried.
21:34 KENNEDY: Yeah, you can use PDB, but that's like a command line thing, it just happens to be that command line is Jupyter cell, that's not the same I don't think. However, there's this thing called PixieDebugger. So the idea is, you include this PixieDebugger into your Jupyter Notebook, and then it becomes a visual debugger for each cell that you're working in. It's pretty cool, and they have a really nice video, and they've got some nice screenshots of the thing I linked to, and so it has one of those, I don't know what the right word is, like a magic command for Jupyter, that escapes out of Python and speaks to Jupyter, like the %%. And you can just say %%, debug this function or debug this cell, and then boom, it just drops into a thing, where you can like, step over, step into, view all the states of all the variables, it's just that little break point, there's like a little command prompt REPL underneath, when it's paused, and you can just ask it questions, like type in a variable, and it'll show you the value, or you can change the value of the variable, things like that.
22:33 OKKEN: Yeah, this is neat.
22:34 KENNEDY: Yeah, so if you're working in Jupyter, this is really cool, and it's interesting because, it comes from the video, that the guy did, David Taieb, it was at the Watson Data platform, like so a Jupyter notebook hosts it on IBM Watson, which was pretty interesting. So I didn't even know that was a thing, but apparently the IBM Watson Data platform is another one of these hosted notebooks, and the reason I bring that up is, he's like "Well we needed a really good way to debug these things called PixieApps." and a PixieApp is like, a GUI interactive thing, that you can put in a notebook, that's kind of like a web app, but kind of like a desktop app, but in a notebook, it's kind of funky. So I don't know, maybe I'll cover the PixieApps as well, but that's also kind of an interesting thing, it's kind of like Flask running inside of a notebook, it's bizarre, but cool.
23:28 OKKEN: Yeah, neat.
23:29 KENNEDY: Yeah, so anyway, anyone out there doing notebooks, wishing for a visual debugger, check out the video that I link to at the end, it's only like two or three minutes, and you'll know whether or not this for you, it looks cool to me though.
23:41 OKKEN: Yeah, and having debugger tools available in any, wherever you're working is good.
23:45 KENNEDY: Yeah, I know, I always feel like I don't need a debugger, until I'm like geez, I really want a debugger for this, this is just not...
23:52 OKKEN: Until you really needed a bugger, yeah.
23:53 KENNEDY: Exactly, it's not that often, but when I do need it, I'm like oh my gosh, I'm so happy this is here. Awesome, alright, well Brian, thank you for sharing everything, anything else you want to give a shout out to, while we're talking?
24:04 OKKEN: No, not today.
24:05 KENNEDY: Yeah, nah me either, so, well have a happy 4th of July, it is probably exactly that day when this comes out.
24:11 OKKEN: Oh, you too, thanks.
24:12 KENNEDY: Yeah, thanks, bye everyone.
24:14 OKKEN: Bye.
24:15 KENNEDY: Thank you for listening to Python Bytes, follow the show on twitter, via @pythonbytes, and get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pythonbytes.fm and send it our way, we're always on the lookout for sharing something cool. On behalf of myself, and Brian Okken, this is Michael Kennedy, thank you for listening and sharing this podcast with you friends and colleagues.