Brought to you by Michael and Brian - take a Talk Python course or get Brian's pytest book


« Return to show page

Transcript for Episode #93:
Looking like there will be a PyBlazor!

Recorded on Thursday, Aug 30, 2018.

00:00 KENNEDY: Hello and welcome to your Python Bytes, where we deliver Python news and headlines directly to your earbuds. This is episode 93, recorded August 30th, 2018. I'm Michael Kennedy.

00:11 OKKEN: I'm Brian Okken.

00:12 KENNEDY: And Brian, you know last time we had some awesome drop-the-mic stuff. I think, we're going to do it again, aren't we?

00:17 OKKEN: Yeah, I think it's some really interesting things today.

00:21 KENNEDY: Absolutely. So before we get onto them though, I just want to say thanks to Datadog. Datadog's sponsoring this show. Check them out at pythonbytes.fm/datadog. I'll tell you more about that later. Brian, a lot of people get into programming, and kind of in air quotes there, by doing scripting around the shell and stuff, right?

00:39 OKKEN: I think so, I think that's why some people, like in the IT world or...

00:42 KENNEDY: Like Bash or even BAT files.

00:46 OKKEN: Or lots of of DevOps roles.

00:49 KENNEDY: Yeah exactly.

00:50 OKKEN: At that point where I learned Python I was doing a lot of C++ development and, writing little utility scripts in Bash and Perl and stuff like that, and so one of the things that I did, you know I want to try to learn Python by writing as much Python as I can. And so I started doing more and more, more and more of my utility scripts in Python and so I think that was a good way to just try to jump in and solve little problems. I could have written it faster in Bash 'cause I knew that, or in Perl, but learning how to do things in Python is good. So there's, it's a GitHub repo that's called Replacing Bash Scripting with Python. And I think it goes through a lot of stuff that, a lot of the issues that you need to do, that you normally write in the little utilities, and how to do that in Python. So this is fun. Talks about dealing with files, reading and writing files. Dealing with standard in, standard out, standard error, things like that. And writing the command line utilities.

01:53 KENNEDY: Yeah things like, dealing with exit codes. So if you shell out to some other process, you run a command, knowing whether it failed or it worked right? Things like that?

02:02 OKKEN: Yeah and how to pipe between commands, if you're going to pull things together, and then it doesn't jump into a lot of the really cool stuff with Python, but it's a good start. But it does introduce people to the path library and shutil, I don't know how you pronounce that, shutil, s-h-util? Mm-hm. And those are things. And then a lot of times I'll may be using sed and awk and grep and things like that, and how to go about replacing that sort of stuff with regular expressions with Regex. And so this is just fun. I like it, to be able to, if you're writing a little utility and it's up to you as to what language you write it in, this might help you write it in Python.

02:43 KENNEDY: It's cool and I think one of the real key things to actually making programming stick when you're trying to learn it, or even at learning a new programming language, you know some other kind of programming, it's to have what you're doing be relevant. To you right? Like if you work in a DevOps world and you're like, well I'm going to go learn Python so I'm going to do this data science project, that may be totally irrelevant to you. So having this as a resource for people who generally do scripting and automation type stuff, this is really cool.

03:12 OKKEN: Yeah and then also you're going to, the stuff, when you're learning the stuff you start out writing isn't, just to be frank, and it's the same with everybody, it's not that great to start with. So starting small is good so you don't have something huge to rewrite later.

03:27 KENNEDY: True, I guess a lot of people, I know I did one in the earlier days, went through the experience of going back and reading code I wrote six months ago and going, wow that's bad. But then that same person who looked back badly on the old code, six months further down the line you look back and, yep that's, that's bad too, just in a different way right? But eventually you get there but yeah, having the small stuff definitely helps.

03:53 OKKEN: And I've heard a lot of people like in a DevOps world where they just, you start writing one or two of your scripts in Python and then you can just, you add to it, as you learn more stuff you can do more powerful things. And having, like you said, having it be relevant to your job is important for what you're doin' for learning.

04:13 KENNEDY: Yep, it's a really cool find. I'm glad you brought it up and I'm sure it'll help a ton of people.

04:17 OKKEN: Now this next topic I'm pretty excited about.

04:20 KENNEDY: I'm honestly kind of beside myself about this one. This literally could be a game changer and, so this is called PyIodide and I don't know if you've heard of it before. I had not heard of it until just, beginning of this week I think actually. So Iodide is a project by Mozilla and it's sort of, think of Iodide as a competitor to Jupyter Notebooks, where the concept of Jupyter Notebooks is you have this webpage and it's like a view into a Jupyter server, and then when you execute code, that code executes on the server. Which is a little sketchy because if you're letting people run arbitrary Python code on your server, that's not amazing so maybe you're using some sort of docker thing to spin them up and shut them down. It gets complicated right? Well Mozilla wants to make this environment available to everybody but in a distributed fashion. And so the goal is to bring this environment to the browser and when you execute code, it executes on your browser, client side. So you can do whatever you want. Well that's cool but the problem is a lot of people want to do data science in Python and Python, while it's had some JavaScript options for running code in the browser, they've always been like, well you can kind of do this and it's mostly like Python, and so on. Awhile ago, as in two weeks ago, we asked the question of, will there be a PyBlazor? Remember Blazor is the .Net community's attempt to bring the .Net CLR, their equivalent of CPython, into the browser with web assembly and make it run there. And they're making that actually work which was kind of blowing my mind and giving me a lot of hope that maybe, maybe somebody could do it for CPython right? And what do you think, do you think it'll happen?

06:07 OKKEN: Well, it's a nice dream.

06:09 KENNEDY: I agree, it's a nice dream. But, sometimes dreams become reality right? So PyIodide is an implementation, a running version of all of CPython, an exact bit of CPython, same thing, in the browser. In web assembly.

06:28 OKKEN: That's so cool.

06:30 KENNEDY: Is that incredible or what? My mind is blown. So I reached out to one of the contributors, Michael D., and asked him, this is really CPython, running in web assembly, in the browser? And he said yeah, yeah. So Michael works for Mozilla, this is a Mozilla project, and they're trying to bring all of CPython, at least the data science part of it, into the browser. Now there's a few things that it won't do, like you can't open a socket. Obviously TkInter doesn't work 'cause what does that mean in the browser right? There's some parts of CPython that are not there but if it's permitted in the browser, it's identical to CPython and that, I think, is amazing.

07:13 OKKEN: That is amazing. I can't wait to play with this.

07:15 KENNEDY: I can't either and so, they also want to have the data science tooling there and so things like NumPy, Matplotlib, and so on, they have heavy dependencies upon C. So you can't just bring it over and run it in this web assembly version of CPython, so you have to take the C code that supports things like NumPy and Matplotlib, and get them to compile to web assembly themselves. So they've also done that. So you have native NumPy, native Matplotlib, on native CPython, client side, in the browser, with web assembly. And it's fast.

07:49 OKKEN: I'm still trying to get my head around this. How could it be fast?

07:51 KENNEDY: I know it's , it's like binary stuff executing, and it's also not that big right? The download, I noticed using the web tools, when you go and interact with this page, it downloads like cpython.wasm right, the extension for web assembly, and it's 2.5 megs, right? And on my connection that's like one and a half seconds to get that thing up and running. And then it doesn't have to parse it like it does like two megs of JavaScript . Make sense of that right? Effectively compile it. It just takes it and runs it 'cause it's already compiled.

08:23 OKKEN: Okay that's, okay, so that's not nothing, but that's pretty cool.

08:27 KENNEDY: It's not nothing but it's not, it's not that bad and you know you could cache, somebody could setup a CDN for that thing and cache it, like say, Mozilla.

08:34 OKKEN: Yeah. Like, say Mozilla .

08:38 KENNEDY: So I think it's awesome. I want to just say thank you Mozilla and Michael and team for workin' on this 'cause this, this is amazing and this could really, really change, it could really expand where Python is broadly applicable. Now I think it's worth pointing out, this generally is built with the concept of making data science work in the browser, right? Whereas this PyBlazor thing, the Blazor, my imaginary PyBlazor thing that I made up, that was around building single page apps, more of the web side of programming than the data science programming side of programming. But this thing does have extensions to let you work with HTML and the DOM and events and stuff like that. So you could, it's not a long shot to go from this, to having an interactive, rich application for a web app, in Python.

09:33 OKKEN: Yeah. So it's still not quite where we want but, really cool.

09:37 KENNEDY: It's really cool and, I mean it's totally working for what they're trying to do but I'm just thinking, if I want to say replace a react thing, or an AngularJS thing, with Python, there might be a few more steps to get that angle of stuff working. But it shouldn't be that far because we already have the CPython runtime working, which is really what you need, and the integration with HTML.

10:03 OKKEN: Yeah so this is cool. I want a lot of people to grab this and run with it.

10:07 KENNEDY: Yeah I think this really opens up the possibility for some amazing stuff. Once again, thank you Mozilla and also, I just want to reflect a little bit. I think it's really ironic, or interesting coincidence, at PyCon 2018 as we mentioned when we talk about the Blazor thing, I think it was Tim Callahan, I'm still forgetting the name, but somebody from Mozilla came and gave an impassioned presentation as the opening keynote of, please let's find a way to use a web assembly to get Python more capable in the webspace and specifically in the browser, client side. And I think it's just pretty awesome that Mozilla another team in Mozilla, actually is the one that implemented this. So that's really, it's really amazing. So these guys working on PyIodide, they may want to document their stuff right?

11:01 OKKEN: Yeah you're so much better at the transition thing than I am. I'm like, oh, next topic. I personally like, I like Markdown, I write, even when I have to use, to be honest, when I have to use Microsoft Word, I write in Markdown and then I convert it to Word. Anyway I don't why I was talking about Word but, so reStructuredText is not something that comes natural to me. But there's a lot of Python documentation that's written in reStructuredText. So if you're going to contribute to an existing project, you can't say, well I can't do any of the documentation 'cause it's all in reStructuredText text. Convert it to Markdown.

11:35 KENNEDY: Just use Markdown. No we're not rewriting all of this in Markdown.

11:39 OKKEN: Right so whether or not you like it, reStructuredText is part of the Python world so this is a wonderful, there's a wonderful article from Simon Wil, oh I always said his name wrong. I always thought it was Wilson but it's Simon Willison. A subset of reStructuredText worth committing to memory. And that's a noble goal but instead of committing it to memory I'm just going to bookmark this article. But it's, he's got the necessary things for easy documentation. You got paragraphs, headings, bulleted lists, how to do italics and things like that. And then of course for documenting code you need, how to do inline code and code snippets and images and internal references and things like that. And there's a lot covered here but, one of the issues that I always had was, the tutorials on reStructuredText. reStructuredText can do a lot, and it's kind of overwhelming and you don't need all of that to understand enough to help document a Python project. So this is great.

12:43 KENNEDY: Yeah it's cool. These are the five or six things you need headings, links, bulleted lists, numerical lists, images, things like that right?

12:50 OKKEN: Yeah and then how to get your code in there.

12:52 KENNEDY: Yep, yep exactly. That's great I'm going to make use of this, the one that you're sharing here, because I'm workin' on a project where I have to write a lot of reStructuredText and, yeah, I basically have doubled my knowledge of reStructuredText by reading that.

13:05 OKKEN: Yeah and I mean,

13:08 KENNEDY: 'Cause like you, I do Markdown most the time.

13:09 OKKEN: So most, to be honest, most of the time when I needed to do something in reStructuredText I would just find somebody else that did something like that and copy it and not know why I was doing these things. So having a little a, little primer and tutorial is good.

13:25 KENNEDY: Yeah and it's great. Speaking of finding things. Let's talk about Datadog really quick. So if you have issues, performance problems, bugs, things like that with your application, you should check out Datadog. So they're sponsoring this episode. And it's a monitoring platform that brings together metrics, logs, request traces, all in one place. And they have this new feature called Trace Search and Analytics and it lets you break down the usage of your Python app around performance. For things like, show me all of the operations that this customer or this user did, or all the requests that go to this URL, and aggregate the performance of that, which is pretty awesome. So start troubleshooting, optimizing your Python apps today with a free trial and Datadog will send you a free cute little t-shirt. So check them out at pythonbytes.fm/datadog. So I have a theme for the rest of this, rest of the items I've chosen Brian. So, have you ever heard that some software has security problems, people can break into it? You know, even though Python's not compiled, Python code can have it too right?

14:28 OKKEN: I assume so, yeah.

14:29 KENNEDY: Well our friend Anthony Shaw told us about this cool project's, little app you can run, it's really a lyncher basically you run against your program, called Bandit. So Bandit is designed to find common security issues in Python code. So what it does is you point it at your project and it'll load up each file into an abstract syntax tree, run a bunch of analysis over that AST, and then it'll generate a report and it'll tell you things like you're misusing the telnetlib, or you're using eval, or you're doing try-accept-pass and eating errors.

15:07 OKKEN: Oh.

15:08 KENNEDY: Opening a sub-process with Popen with shell equals true. I actually don't know what's wrong with that but I should probably figure that out. But apparently it's bad, shouldn't do it, and that's error 602. So it'll tell you, for example like I wouldn't know that that's necessarily bad. I know I've heard it before, but I don't remember why it's bad. So it's pretty cool and it just has a whole bunch of errors it'll find, security lapses, it'll find and tell you about them.

15:34 OKKEN: So this is something that you, is like a static analysis, it's not running your code? It's looking at.

15:39 KENNEDY: No, it's static. Yeah it's static analysis.

15:42 OKKEN: That's cool.

15:43 KENNEDY: Yeah isn't that nice?

15:44 OKKEN: Yeah, neat.

15:45 KENNEDY: Yeah it's like I see you did a try-except and you just ate that error and that can actually cause other types of problems. Maybe you should have validated something, but instead you just keep going. Things like that.

15:54 OKKEN: So this might be good, even for non, I mean it's definitely important for Python projects that have a interface to the web, or like a web application, but you could use this for a lot of stuff. Not just that.

16:08 KENNEDY: Sure.

16:08 OKKEN: So.

16:09 KENNEDY: Yeah, yeah it definitely lets you know when you're doing things wrong in a different category say than probably Flake8.

16:15 OKKEN: Yeah, neat.

16:16 KENNEDY: Yeah and because it's, you could easily integrate this type of thing into a continuous build, integration pipeline.

16:21 OKKEN: Yeah, I'll definitely have to check this out. Cool.

16:25 KENNEDY: Yeah this is, it even probably has a little testing angle to it somehow.

16:27 OKKEN: Maybe, yeah.

16:28 KENNEDY: Maybe. Awesome, alright what you got next for us?

16:31 OKKEN: Well last week we talked about what Netflix was up to with Jupyter Notebook so I'm, and I mentioned that I've kind of been trying to learn more about them so, one of the, we had somebody submit another, it's another repo, but it's a whole bunch of Notebooks. So it's, what he's put together is, learn Python 3. I think I even have got the wrong link here. Learn Python 3 with a Jupyter Notebook. And so I was actually playing with this, this morning and it was just a blast. So we have, it's got beginner and intermediate Python concepts. And you just sort of like can walk through it, and one of the things I had to learn of course is to, at first is just to, how to do this? How to clone, I cloned the repo and just ran, and then installed Jupyter and then ran the, just say, what is it, you just say Jupyter Notebook, or something and it opens up a web browser. And then you can click different Notebooks and run them and see what they do. And the thing that really intrigued me that, a bone that he put in there for me to try to grab is, there's two topics on testing and I've been asked this before of how, can you run pytest within a Notebook? And I didn't know if you could or not. I heard rumors, but there's two examples in here of how to, two Notebooks that have tests embedded in them so you can run your tests within, so you can define a function or something in a different cell, and then write a test for it in another cell. And then run it.

18:09 KENNEDY: I think that's really cool and I had no idea how to run pytest either in there. And I always sort of saw that as one of the weaknesses of Notebooks. I have lots of respect for them but one of the things I don't really like is it's hard to do like, how do I run tests against this code? How do I get a code coverage report? Of the code, mind you, when a Notebook is doing things like that right? Seem much easier when you have a set of files that you can call directly.

18:35 OKKEN: Yeah and I'm still not, so this is running pytest within the Notebook and I'm not, I think there's also a way that I haven't figure out yet to take a Notebook and run, if there's tests inside of it, run the, run tests against it.

18:48 KENNEDY: Yeah there may be, and the stuff that Netflix talked about last week, allow you to basically take a Notebook and execute it as if it were a function with an input and an output. And that in some sense would open up a testing angle, just by itself right?

19:02 OKKEN: Definitely. And especially if you save the output somewhere, that'd be great. Anyway it was just really fun. It's fun to look around. The other thing that I like, as an example, is there's the examples of how to do pip install within a notebook. It has those at the top 'cause it doesn't assume that you've installed pytest to begin with.

19:23 KENNEDY: Oh yeah that's cool.

19:23 OKKEN: And it shows that. And then it's using another pip installable package called the ipytest which is how it's running pytest within the Notebook. So, it's cool.

19:34 KENNEDY: That is really cool. Nice find. Alright last one, I said I was on a bit of a theme. So we talked about Bandit. And Bandit let's you find code that is being either used in a bad library or is using a bad pattern, like try-except-pass, things like that. This next one takes a different angle on the same sort of problem around security and integrity of your application. And it's called detect-secrets. So their sort of tag line there is, an enterprise-friendly way of detecting and preventing secrets in code. And this is actually released from Yelp. So on one hand, using code badly, that's a problem. But also putting hard-coding passwords and other things like that into your code, can be a problem right?

20:21 OKKEN: Yeah, definitely.

20:23 KENNEDY: And so the enterprise-friendly aspect I think comes from the fact that it says, alright look we assume that your code is just messed up. There's all sorts of badness in here. What do you do if you find yourself in a hole, and things keep getting worse? First thing you do is stop digging, right? Before you get out of the hole, you just stop making it worse. So there's a lot of mechanisms in detect-secrets to say, let's not make this worse. Let's prevent new secrets from going into the code base.

20:51 OKKEN: Oh nice.

20:51 KENNEDY: Now we can start to peel off, start to remove some of these things that are hard coded in there, but let's go and set a baseline and go, alright if we get any new detections of secrets being entered, that's going to fail the build. But the way it is now, we're going to accept it. And then we can make pass after pass to actually take away some of those and reset the baseline.

21:10 OKKEN: Okay interesting.

21:12 KENNEDY: Yeah it's pretty.

21:13 OKKEN: Also there's some things--

21:13 KENNEDY: Pretty interesting right?

21:14 OKKEN: That look like they're secrets that should be detected but you've inspected the code and said no, this one's fine.

21:20 KENNEDY: Yeah exactly. And that baseline thing will let you say, I approve of these, but not anything else.

21:26 OKKEN: Yeah, cool.

21:27 KENNEDY: Which is pretty cool and they also show you how to set it up as a git commit hook. So as you know from the testing space, one of the challenges is, you can have awesome tests, but if no one runs them or worse, one or two people on the team are very passionate about them and a whole bunch of other people hate them and refuse to run them no matter how much you talk about it, the people who care about the tests suffer from the neglect that the people who explicitly ignore it right? Maybe that person enters some problem that'll fail a test but they don't run their test so they don't know. You get it, you try to run your test, and then you're hit with their problems as well right? So this git commit hook here basically means everyone on the team is going to be basically running this as part of a check-in.

22:11 OKKEN: The whole idea around git commit hooks is interesting. There's a lot of interesting things people are doing, like making sure that, like there's a, black has a git commit hook as part of it, or attached to it so people can make sure their code is still black before they check it in.

22:25 KENNEDY: Yeah that's the one I was thinkin' of as well. And it can even automatically reformat your code. To the group's format, as part of the check-in right.

22:33 OKKEN: I would be curious to know if there's any other, I'd like to have people let us know if there's other git commit hooks that we haven't talked about, that are helping their teams out. Maybe they'd help other people out too.

22:43 KENNEDY: Yeah it'd be fun to collect a list of use cases of what people are doing. Yeah, if you're doing something cool like git commit hook check out pythonbytes.fm/93 and put a comment at the bottom. That'll be in the show notes, and sort of blow the show notes for everyone to come see. Alright well, that's all of our items for this week. Brian you got anything personal you want to share? Before we're out of here?

23:04 OKKEN: No, I guess, pytest came out with a new version the other day, which kind of blew me away because I had submitted a very minor bug, but I had submitted a bug to the project. And I only submitted it like five days ago. And it's already now in a, fixed, and in a release. So that's amazing.

23:24 KENNEDY: That's a really good turnaround time. That's awesome.

23:25 OKKEN: Yeah, that's fun. Anything with you? Got any news?

23:29 KENNEDY: Nothing I'm ready to talk about. I'm working on a couple of new projects, one that I'm doing reStructuredText for. Couple of new courses are coming down the pipe, things like that but nothing ready to talk about just yet.

23:41 OKKEN: Alright.

23:42 KENNEDY: So how's that for a long winded no?

23:45 OKKEN: Well it was fun talkin' to ya'.

23:47 KENNEDY: Yeah, you as well. And thanks for everything and see you later.

23:50 OKKEN: Alright bye.

23:50 KENNEDY: Thank you for listening to Python Bytes. Follow the show on Twitter via @pythonbytes. That's pythonbytes as in b-y-t-e-s, and get the full show notes at pythonbytes.fm. If you have a news item you want featured, just visit pytonbytes.fm and send it our way. We're always on the lookout for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.

Back to show page