Transcript #85: Visually debugging your Jupyter notebook

00:00 Hello and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds. It's episode 85 recorded on July 3rd 2018 I'm Michael Kennedy and I'm Brian Huckin. Hey Brian. How you doing? I am doing great today I know it's another day in paradise here in the Pacific Northwest in there Yeah, as long as you're not driving because it's the 3rd of July that can be problematic. We do have our traffic issues Don't we? Yeah. Yeah. Well, yeah before we get to items. I want to say thank you to digital ocean And they've been supporting the show in a major way.

00:30 And so, you know, like I've said, our stuff runs on DigitalOcean, which is really great.

00:34 You want yours to run on it, you go to pythonbytes.fm/digitalocean, get $100 free credit for new users.

00:40 I find myself, Brian, more and more loving the concept of limited use of type hints in just the right places.

00:47 I was just typing that 15 minutes ago.

00:49 Were you?

00:50 Yeah.

00:51 Yeah.

00:52 Yeah, that's nice.

00:53 And the more I use them, the more I'm warming to them.

00:54 And there's a couple resources that came by just recently that I really enjoyed.

01:00 So there's an article called "The State of Type hints in Python" and this is a fairly thorough article but it pretty much like, okay now it's 2018, what is it like to use type hints now?

01:15 And especially, and I, okay, the article talks about if you want to use type hints in 2.7 prior to 3.6 sort of stuff.

01:25 And that's doable, but I still think that you get the best experience if you're using Python 3, and especially Python 3.6.

01:37 'Cause you can use commented type annotations in 2.7, I believe, because they're just comments.

01:44 But adding the little arrows for the types and the colons and everything, it's more concise and it looks better.

01:50 And I, my personal feeling is, unless you really are trying to convert a large codebase, which I know some people are, I would switch to three first and then add type hints if it was me, if you have that option.

02:02 Yeah.

02:03 And three six is even better than three five because you can put types on variables.

02:07 You can, although I find my I'm not really doing that when I'm using type hints.

02:11 I'm just mostly doing it for interface to document interfaces.

02:15 You know where I end up doing that is where there are core functions that I can't control.

02:21 So I need to call a function and I know it returns something rich that I know what the type is and I would like to be able to get better editor experiences, better autocomplete, but it's a package function, not something I've created.

02:36 So then you can say this variable is a, you know, SQLAlchemy session, for example.

02:42 And then it's like, oh, I see now you've got add and delete and all this stuff.

02:46 And it's amazing, right?

02:47 Yeah.

02:48 And you know, one of the things I like about the article and I agree with is the main point of a good reason to use type ints isn't, I mean, it is good to be able to catch all the bugs, catch more bugs and incompatibilities, but it does, there's a lot of reasons why it makes your code more maintainable and more readable because there's, you just have that option to be able to document it right there to say, "Hey, this is the type that this function needs." There's so many things.

03:16 The article walks through a few of them, like refactoring is easier, helping when you come back and read it later, it makes it more obvious.

03:23 When you're writing, yes, there are some great polymorphic functions that can take multiple types of type, multiple types.

03:31 Most of the code I write really does expect one type.

03:33 it expects a list or maybe the most generic thing is it expects an iterable.

03:38 And I actually still not sure how to write that in the type shed stuff.

03:42 I'm sure there's a way, but yeah, this article does talk about things like unions.

03:46 I ran my pie on my, on some of my code and I realized that I had, some uninitial, some default values set to none, which I wanted to be okay.

03:57 But if they're not none, they're strings or they're integers or something.

04:01 And the type system allows you to add a union type so you can say, well, it's either none or it's a string.

04:08 And it's pretty interesting, but this article talks about a lot of stuff that I really had forgotten about actually is like interface stub files.

04:18 And I haven't worked with those at all yet.

04:20 And I know that the TypeShed is around that has these interface stub files for the standard library and a bunch of other popular libraries, but I haven't really used it yet.

04:32 So, or maybe I have, but I just don't know about it.

04:35 But I don't know, this is good.

04:37 - Yeah, you have an interesting quote in here that type hints should be used whenever unit tests are worth writing.

04:43 What's the backstory on that?

04:44 - It's just one of the, in the conclusion of the article, it talked about that.

04:47 It said, okay, all of this information, but when should you use it?

04:51 I mean, if you're gonna just write a little utility function for, I don't know, for your own use or something, do you need type hints then?

04:58 And they just came up with this rule of thumb is if you're gonna, I quoted it just as is.

05:03 It says unit test.

05:04 I personally would just say automated tests.

05:07 If the code you're writing is worth having automated tests, it's worth putting, trying to think about where you're gonna put type.

05:14 The other thing I wanted to add was I was listening to, I got about halfway through it so far, but there's a recent seminar that Guido presented at Stanford called Optional Static Typing for Python.

05:26 And in part of that, it talks about MyPi, but it talks about some of the beginnings of this, of like where this came from, and how it came to be what it is now.

05:35 And so for, and how are we dealing with it now?

05:38 So it's about an hour and a half, though, so it's a chunk of time to watch, but--

05:42 - It's a commitment.

05:43 Yeah, but it sounds interesting.

05:44 - Yeah, but I think I recommend even like the first 20 minutes for some of the backstory is interesting.

05:50 - Okay, yeah, that's really cool.

05:52 I'm definitely a fan of it, and I'm glad you're bringing it up.

05:54 There's some nice stuff in there.

05:55 So I have no idea how to transition from type hints to MongoDB.

06:00 So I guess that's a reasonable transition.

06:02 That's it.

06:03 So I've actually found a pretty interesting thing on MongoDB.

06:06 People know that I'm a fan, of course.

06:08 And one of the big decision points, I think, when you're trying to decide, do I go with Django or do I go with one of these smaller web frameworks like Pyramid Flask, you know, many of the other micro frameworks, is the micro frameworks, you can pick the little pieces.

06:23 I want SQL can be here, and maybe this part's gonna be Mongo and this part's gonna be this other aspect that I would like to bring into this framework for authentication or something.

06:32 Whereas Django is more like, here are your big building blocks, click them together.

06:38 And a big challenge there can be like the Django ORM and the admin stuff, it's all tied into that.

06:43 And well, if you don't wanna use that, then you're kind of out of luck.

06:47 There's a lot of challenges of sort of bringing your own thing.

06:50 You lose some of the value, at least, out of Django, traditionally.

06:53 So the Django ORM, it depends on a, not a particular SQL database, but it does assume a SQL database, is that correct?

07:01 - Yeah, I believe so, I believe that's right.

07:03 And things like the Django admin section, which is really valuable to some people, depends upon the Django ORM, right?

07:10 So like there's these layers of stuff, and so if you pull out the bottom brick, there's problems.

07:15 But I, actually this guy named Robin sent me a heads up about this thing on Twitter that I had never heard of, but it's pretty awesome, called Django MongoDB Connector.

07:26 - Cool.

07:27 - Yeah, so the idea is you continue to use the Django ORM just like you normally would, except it's now had its backend swapped out to be MongoDB.

07:37 And that's pretty awesome.

07:39 And at first you're like, okay, well, how much value am I gonna get in that, right?

07:43 Like if I give up, if I continue to use what is effectively a relational ORM, like you can't use the hierarchical documents, right?

07:51 But what they've done, a little bit like Mongo engine, is the parameters that go into your query can be more interesting, right?

07:59 So you would say, like imagine if you had a blog, you say blog.objects.filter, and you would pass like name equals something in traditional relational stuff.

08:09 But in Mongo, you might want to traverse like into a blog object and then go find the name and then find that the name is contained in a set.

08:20 And so you just pass different arguments to the filter parameter and boom, it becomes Mongo, Oracle enabled.

08:27 It's beautiful.

08:28 - Wow, that's actually pretty cool.

08:29 - Isn't that cool?

08:30 It's so simple.

08:31 Yeah, and it says you can also use it to connect third party apps like Django REST Framework and ViewFlow and other stuff, which heavily depend upon these Django models.

08:40 They can now super easily be integrated with MongoDB as well.

08:43 - And it has a hilarious name.

08:44 - Yeah, what is his name?

08:46 - Django.

08:47 - Django, yeah, or something like Django.

08:49 - The app, the official Django app you put in the settings file is Django.

08:53 - That's great.

08:54 Yeah, 'cause for, I get it, Django, Mongo.

08:57 - Django, Mongo, smushed together, yeah, exactly.

09:00 Another thing that's interesting is this is a Python 3.6 or higher only, and Mongo 3.4 or higher, which is like, I don't know, a year, year and a half old, something like that, so it's not quite so new.

09:11 But Python 3.6 is really quite new for this store.

09:14 - Yeah, but I mean, it just also seems like something you would start a project with.

09:19 you probably wouldn't like rip apart an old project.

09:21 - Probably not.

09:22 I mean, you may have an app you're trying to make better and migrate, but yeah, I would think you probably wouldn't really.

09:30 Yeah, I'd throw that more in there, just like it even has like a pretty high requirement for MongoDB, just so if somebody's got like an older or something or other, you can't just jam it in there.

09:37 You gotta make sure you got the new goodness.

09:39 - Okay, so I was actually, so this is not sort of related.

09:44 I was talking to somebody about Mongo before and they asked me about migrations from, if you're gonna change the databases.

09:51 Does Mongo have that sort of concept?

09:53 - It does have that concept.

09:55 You do it a lot less often.

09:57 Much of the time at what your migration is, is something like I'm adding an additional column here.

10:02 I'm adding another table there, right?

10:05 Like it's not that common that you're say, deleting a column or you're renaming a column.

10:10 It happens, but it's way less often than new column, new table.

10:14 And in MongoDB, new column or new table is automatic.

10:17 You don't do anything.

10:18 It just adapts, right, as you use it.

10:20 So you don't actually run any scripts or migrations to do the common transformations.

10:25 But if you're going to say, like, delete a column or reorganize your data, you would do the same thing and you would run a script against it.

10:32 I don't think there's something like a limbic where there's like a framework for migrations, but the concept of like a migration with a script definitely exists, but it's not very common.

10:41 - All right, and I brought this up hopefully because if we're wrong about this, somebody can get a hold of us and tell us.

10:47 - Yeah, if there's some cool framework for doing that, let us know.

10:49 - Yeah.

10:50 - Awesome.

10:51 So what's this deal with Python idioms you got going on here?

10:55 - This was a tiny little article, which, you know, sometimes I like that, just a small article of like, hey, I learned this thing and it's helpful, and here's a small blog post about it.

11:05 And this was Amir Rachnam.

11:09 Why do I try to pronounce people's names?

11:11 Anyway, multi-lined strings.

11:13 So the idiom he's talking about is multi-lined strings.

11:16 So if I've got, and we know this is like a triple quote, you can have strings.

11:21 Strings can be multiple lines if you use triple quote.

11:23 And we usually see these with doc strings, but it can be really anywhere you're gonna use a string.

11:30 But if you're using it like in the middle of a function, it's awkward because it either, you have to, everything but the first line has to be, is like over on the left side of your page, and it's not indented the right place.

11:45 And if you do indent it over, it's including all of those spaces in your string.

11:50 But the way to get around it is to use a standard library function, the module or the package is Textwrap and the function is ddent.

12:00 So it's Textwrap.ddent, and that will strip out what it does is it takes, looks at all of the lines of the multi-line string and takes the common spacing at the beginning of it and just rips that off.

12:14 So it's just automatic.

12:16 And I was actually, I use this enough with, actually I use it, you know, generating tools and generating other things.

12:23 And I can't think of a good example right now where I've used it, but I have used it enough times and I've looked this up that I want people to know about it.

12:30 It's an easy trick.

12:31 So it's good.

12:33 - Oh, I love it.

12:33 I didn't know about it.

12:34 Thank you.

12:35 This is cool.

12:36 So I've definitely had multi-line text strings I've got some big formatted piece of text and I'm like, all right, I, instead of trying to break this apart and turn it into something I can store in code, if I just, you know, triple quote it and put it in here, it's going to be golden.

12:49 But you do get that weird, you've got to like, indent it to the left outside of your function and it's all bizarro.

12:56 So this lets you can keep your code looking really nice.

12:59 And it obviously puts the string to like visually into the block that it belongs in.

13:03 It's really nice.

13:04 I love it.

13:05 I've tried to punt and define a global variable or at least a file global variable so that I can get around it.

13:13 But this is cleaner and it's good.

13:14 It's a good thing.

13:15 Yeah, I love it.

13:16 That's awesome.

13:17 Other good things include DigitalOcean.

13:20 So definitely want to tell people about DigitalOcean because they're sponsoring the show in addition to just us being happy customers there.

13:28 So you can go from zero to a customized server in 60 seconds.

13:32 I've talked about how we've done cool stuff with virtual machines, how they've got spaces, how they've got other various things that are pretty awesome, like the one-click containers for like machine learning.

13:42 One thing I haven't talked about yet is container distributions.

13:45 So if you wanna set up your own like Docker container, you can go click over there and create a core OS or Fedora Atomic server, even a Rancher OS, which I've never used, but it's got a cool icon.

13:57 And you can just fire one of those up and it's like all ready to go to be your container host for all sorts of cool Docker stuff and Kubernetes.

14:05 So definitely check them out over at pythonbytes.fm/digitalocean.

14:10 Get a $300 credit to play around with those kinds of things.

14:12 - That's actually pretty cool.

14:14 I like it.

14:15 - Yeah, I like it too.

14:15 - And I had a DigitalOcean accident the other day.

14:18 - Oh no, what happened?

14:19 - Well, I wore my favorite DigitalOcean t-shirt to work and I spilled coffee on it, but luckily it was a gray shirt and it didn't show up as a stain, so it's all good.

14:29 - It's pretty fail safe, unless you're painting.

14:32 So how do you feel about your design skills, Brian?

14:35 Like, are you a fan of shuffling around CSS and HTML and that kind of stuff?

14:40 - No, CSS is definitely those things where I either rely on a framework or I try to hire somebody.

14:47 - Nice.

14:48 So something that you might be able to use if you're using Flask, and I know you've played around Flask recently, is this thing called Flaskerizer.

14:56 Flaskerizer.

14:58 I don't know what the name stands for, I guess to make flask ish or something.

15:03 But the idea is you can go to one of these places that has bootstrap themes.

15:08 So bootstrap is nice.

15:09 Obviously, it's a design front end CSS front end design framework.

15:12 Pretty well known.

15:13 But what's really awesome about it are the themes like people make these pre made themes and for like 10 bucks or even free a lot of times you can get like incredible web designs and you just drop them in right you just put in your logic into the bits where it where There it goes.

15:28 So this Flaskerizer thing is, what you can do is you can download the themes from certain locations that have a known format, and it will convert the bootstrap, the static bootstrap theme into a Flask app.

15:39 - Oh, that's cool.

15:40 - Yes.

15:41 (laughing)

15:42 Isn't that sweet?

15:44 So instead of worrying about how it all goes together, you just go, you know, command line, boom.

15:48 Now I've got like a dynamic running bootstrap theme in Flask.

15:51 - I like it.

15:52 - Yeah, so there's a couple of sites that they work with.

15:55 I don't know how general this is, I don't know how common bootstrap themes are, but I think it's pretty cool.

16:01 And of course, if you open up the website and you look around, you can tell that it's a proper design.

16:07 This is the GitHub repo because they go from just plain white, like no design, no logo to clearly a designer with a black turtleneck.

16:17 Yeah, he's definitely a designer.

16:21 Yeah, definitely.

16:22 Dark black glasses too.

16:23 And a beard.

16:24 Yeah, exactly.

16:25 Oh, he's all good.

16:26 You got all the boxes checked.

16:28 Very nice.

16:29 So if you want to do a Flask site and you want a design to go pick a theme and just start working with that design, check out FlaskRiser.

16:37 It could get you up and running really quick.

16:38 - Wow, interesting.

16:39 I was reading down this.

16:41 This is one of the few applications I've ever seen or the frameworks that uses Nose 2 as its testing.

16:47 - Oh, nice.

16:48 What's the story of Nose 2?

16:49 I only know about Nose.

16:50 - Well, so Nose was sort of kind of abandoned, but Nose 2 was a, well, I'll offend anybody that's still working on Nose.

16:57 But Nose 2 was kind of trying to reboot it and redo a lot of the, make a backward compatibility break so it doesn't do some of the things but it does other neat things and clean up the code base.

17:10 But still, it started kind of running, getting some steam at the same time pytest was sort of getting a lot of steam.

17:18 So I think that there's not a lot of projects that use it, but I mean, it's still reasonable.

17:23 It's just since there's not that many people working on it, getting fixes and stuff is sometimes an issue.

17:30 - Yeah, nice.

17:31 Okay, well, very interesting, very interesting.

17:33 So are you learning Python now?

17:35 Is this your next project?

17:36 (laughing)

17:37 - No, but I thought this was fun because I encourage every youth I meet, and actually everybody I meet, to learn Python.

17:46 And I always tell them--

17:48 - Good advice, good advice.

17:49 - Yeah, I tell them the reason isn't so that you can become a programmer.

17:53 It's so that you can, like, it's a power boost to what are the other skills you have.

17:59 So if you're a biology student, also learn Python 'cause now you're a biology student that knows Python.

18:05 - You're now the favorite person in the lab that can actually solve the problem with the data that they got in Excel or something, right?

18:11 - Yeah, and it doesn't matter.

18:12 It's science or non-science.

18:13 If you're an artist that also knows how to code, it's just gonna help.

18:18 There's no downside.

18:19 And so a lot of times with code, people kind of have, they start writing code and it's precious to them and they don't want to throw it away 'cause they worked really hard on it.

18:27 And so I wanted to highlight this article because of its advice.

18:31 This is, I think it's a real Python, yeah, it's a real Python article.

18:36 And it's called Learn Python the Methodical Way.

18:39 And I'm just gonna read the steps.

18:41 The steps are make your way through a tutorial or a chapter from what you're learning, a book or something that teaches you some discrete four to six step skill.

18:51 You're going to work through it while you're reading it.

18:54 Now, write down the steps as distinctly and generically as possible.

18:58 Put the tutorial and chapter and its solutions away and build your project from scratch and peek at the solution and the steps only when you get stuck.

19:09 Now erase the whole whole thing and do it again and then go back completely, put it away, and then erase everything and do it again like a day or two later.

19:20 And this sort of erasing and redoing it, I think at least a couple times is good for people because it gets rid of that preciousness because the second time you do something, it's always faster.

19:33 Anyway, I just thought this was neat advice, so I thought I'd share it.

19:35 - I like it, it's pretty interesting.

19:37 I definitely feel like when you're new, you feel like I put a lot of effort into that, And so there's no way I'm throwing that away.

19:45 But the really valuable lesson is, actually the second time, it's probably better.

19:51 And the third time, you're like, whoa, this is really a much improved version of what I did.

19:57 And it takes less and less time, of course, right?

20:00 - Yeah, and also you end up, it's easier to throw away the cruft.

20:04 And if it's just something you spend an hour working on, even if you had to redo the entire thing, it's only a wasted hour, and you realize that you can really do it again pretty fast.

20:15 And so the one part I kind of disagree about is, I never code without resources anymore.

20:21 But I wouldn't say try to do it just for memory, but try to use the normal resources that you would otherwise, like Google or Stack Overflow or whatever.

20:30 >>Right, exactly.

20:31 >>To look up things.

20:33 I think it's silly to try to memorize like the order of parameters to a function.

20:37 That's what code completion's for and stuff like that.

20:40 - Exactly, that's why we have typins, isn't it?

20:43 Yeah, so I totally agree.

20:45 I don't think these completely abstract, in isolation sorts of things make sense.

20:52 But maybe the idea is don't just keep following the steps three times, but try to recreate it from your mind, and to me, recreating it is totally fine.

21:01 I go, "Ah, but how do I actually insert that thing?

21:03 "What is the function I call it?" That is a totally reasonable Google search.

21:07 - Yeah, definitely.

21:08 - Awesome.

21:09 So we've talked about debuggers before we talked about the breakpoint thing last time.

21:14 One of the areas where doing proper debugging, and by proper I mean visual debugging, not not command line to me like visual debugging, we see all the state of the system and everything that can be really tricky in Jupyter notebooks, right?

21:28 Because you don't like attach to them, they're running in the notebook in your web browser, right?

21:33 Yeah, I've never tried.

21:34 Yeah.

21:35 So you can use PDB.

21:36 But that's like a command line thing.

21:37 It just happens to be the command line is a Jupyter cell.

21:39 That's not the same, I don't think.

21:42 However, there's this thing called Pixie Debugger.

21:45 So the idea is you include this Pixie Debugger into your Jupyter Notebook, and then it becomes a visual debugger for each cell that you're working in.

21:55 It's pretty cool, and they have a really nice video, and they've got some nice screenshots of the thing I linked to.

21:59 And so it has one of those, I don't know what the right word is, like a magic command for Jupyter that escapes out of Python and speaks to Jupyter, like the percent percent.

22:08 And you can just say percent percent debug this function or debug this cell.

22:13 And then boom, it just drops into a thing where you can like step over, step into view all the the states of all the variables that you can set little breakpoints.

22:21 And there's like a little command prompt rebel underneath when it's paused.

22:26 And you can just ask it questions like type in a variable and I'll show you the value or even change the value of variable things like that.

22:32 Oh, nice.

22:33 Yeah, this is neat.

22:34 So if you're working in Jupyter, this is really cool.

22:37 And it's interesting because it comes from the video that the guy did, David Tayab.

22:44 It was at the Watson Data Platform.

22:48 Like so a Jupyter Notebook hosted on IBM Watson, which was pretty interesting.

22:52 - Oh yeah.

22:53 - So I didn't even know that was a thing, but apparently the IBM Watson Data Platform is another one of these hosted notebooks.

22:59 And the reason I bring that up is he's like, well, we needed a really good way to debug these things called, what are they called?

23:06 Pixie apps, I believe.

23:08 And a Pixie app is like a GUI interactive thing that you can put in a notebook that's kind of like a web app, but kind of like a desktop app, but in a notebook.

23:19 It's kind of funky.

23:20 So I don't know, maybe I'll cover the Pixie apps as well, but that's also kind of an interesting thing.

23:24 It's kind of like flask running inside of a notebook.

23:27 It's bizarre, but cool.

23:28 - Yeah, neat.

23:29 - Yeah, so anyway, anyone out there doing notebooks to wishing for a visual debugger, check out the video that I linked to at the end.

23:37 It's only like two or three minutes and you'll know whether or not this is for you.

23:40 Looks cool to me though.

23:41 - Yeah, and having debugger tools available in any, wherever you're working is good.

23:45 - Yeah, I know.

23:47 I always feel like I don't need a debugger until I'm like, "Geez, I really wanted a debugger for this." This is just not. (laughs)

23:52 - Until you really need a debugger, yeah.

23:54 - Exactly, exactly.

23:55 It's not that often, but when I do need it, I'm like, "Oh my gosh, I'm so happy this is here." Awesome, all right, well, Brian, thanks to you for sharing everything.

24:01 Anything else you want to give a shout out to while we're talking?

24:04 No, not today.

24:05 Yeah, not me either.

24:06 So well, have a happy 4th of July.

24:08 It is probably exactly that day when this comes out.

24:11 Oh, you too.

24:12 Thanks.

24:13 Yeah, thanks.

24:14 Yeah.

24:15 Bye, everyone.

24:16 Thank you for listening to Python Bytes.

24:17 Follow the show on Twitter via @pythonbytes.

24:19 That's Python Bytes as in B-Y-T-E-S.

24:22 And get the full show notes at pythonbytes.fm.

24:26 If you have a news item you want featured, just visit pythonbytes.fm and send it our way.

24:30 the lookout for sharing something cool.

24:33 On behalf of myself and Brian Okken, this is Michael Kennedy.

24:36 Thank you for listening and sharing this podcast with your friends and colleagues.