Hello and welcome to Python Bytes, where we deliver Python news and headlines directly
to your earbuds. This is episode 158, recorded November 20th, 2019. I'm Michael Kennedy.
And I'm Brian Okken.
And this episode is brought to you by DigitalOcean. DigitalOcean's awesome. Check them out at
pythonbytes.fm/DigitalOcean. Tell you more about that later. But Brian, I find that Python
is making its way into all these different areas, not just traditional computer science or maybe
data science. Right. There's an article that I saw that's kind of interesting. I mean,
there's not a lot of details, but essentially it's saying that Python is replacing Excel in
banking and investing. The real title is Python already replaced Excel in banking. But we've got
some interesting quotes from here. So I'm just going to read it out. This is from the article.
If you wanted to prove your mettle as an entry level banker or trader, it used to be the case that you
had to know all about financial modeling in Excel. Not anymore. These days, it's all about Python,
especially on the trading floor. And it goes on to talk about how a lot of different modeling that
used to be done in smaller cases in Excel, but it would take like a few minutes to run the Excel
modifications and analysis. Now they can do even like way more data and have it done in like a second
or two. So it does, it doesn't make sense when in cases where split second decisions are change,
how you react to the market that you'd want to have speed and ease. So Python makes sense to me.
Yeah, that's really interesting. I'm sure it's using a lot of the data science stuff like NumPy and whatnot
to make that fast deep down below. The whole trading, the algorithmic trading, high speed training,
all that kind of stuff. The latency that those folks care about is crazy, right? Like if you could get it from
four milliseconds to three milliseconds, we'd really appreciate that, right? And they'll actually like rent
servers that are nearly co-located to the stock market to reduce the actual latency or set up alternate
direct connections over microwaves. There's all kinds of crazy stuff. And so if you can go from minutes to seconds,
that already seems like it would make a big difference to these folks.
Yeah. And also being able to go to from minutes to seconds and while incorporating more data.
Yeah. Super, super cool.
I'm imagining like walking through the trading floor and seeing some, some guy in a hoodie sitting with
a laptop on the floor. I mean, like, I don't understand this, but yeah, whatever.
Five years ago, that person would have been arrested. Now people are like, Hey, I need some help,
man. Can you give me some advice on this trade?
Yeah.
I have a little personal experience with this Python replacing Excel and banking and trading.
Can't talk about the details, but I did teach a class through a bunch of folks working on the
European stock market and they actually couldn't even take the class during the day because they
had to be there for a while. The market was open. So we had the class in the evening for a week over
there and they were all really into learning Python because they had been trying to analyze how their
day went and do this kind of analysis that you're talking about in Excel. And they're just like,
we can't do this anymore. We have to get like better tools. And Python was the answer for them as well.
Pretty cool.
Oh, that's great. Interesting.
Yeah. Another thing that I think is really, really good news is something that GitHub just announced.
GitHub has announced a ton of things. While you were not with us last week when we recorded in Florida,
we talked about how GitHub has added code navigation to all the source code there, much of the source code.
you go in there and like click on functions and classes and say, go to definition and Python. And
that's pretty awesome. So give it a week and GitHub launches security lab to help secure the open source
ecosystem. Wow.
So you've probably heard about bug bounties and like these bounties paid out to security
researchers before, I would guess. Yeah. Yeah. So it's pretty much like that is my understanding of
it. So it's like a bug bounty program to go and find bugs in open source libraries. But what's kind
of cool is it seems like the folks like paying out that money are not the open source projects,
right? Like Apple might pay out a huge amount of money, like a hundred thousand dollars for finding
a big vulnerability in iOS or Microsoft might or whoever, but who's going to pay to find that
security bug in Flask or wherever it is. Right.
All right.
It seems like that this is to pay for those types of things. So it says organizations as well as
individual security researchers can join a bug bounty program with rewards of up to $3,000 is available
to compensate bug hunters for the time they put into searching for vulnerabilities in open source
projects. Oh, that's neat. Cool, right? Yeah. Yeah. So apparently this has been in beta since
for a little while. When was it exactly? A little while, not very long. Anyway, the founding members
who were part of it have already found, reported, and helped fix more than a hundred security flaws
already across the open source ecosystem. That's pretty cool. Another thing that's interesting is the bug
report in order to count must contain a code QL, like SQL, but code QL or something. I don't know.
Code QL, which is an open source tool that GitHub released at the same time. Remember we talked about
there's semantic code analysis engine. And what it does is basically this is a query that runs against
source code that will uncover the vulnerabilities in dependent projects.
Okay. So if I find a bug in Flask, I don't know there is one, but let's just say I just pick a
random project. I find a bug in Flask and I submit this, I submit a query to GitHub so that they can go
find all the projects that depend on Flask that have out of date versions of Flask that need to also
subsequently receive warnings to get their stuff updated.
So do they then notify all these, the other maintainers or?
Yes. So if you look at that article, there's like some screenshots of what it gets. So they will get,
the actual project will get an automated pull request that fixes the security vulnerability.
Maybe it bumps the requirements pinned version to something where it's fixed or something, right? It gets the
PR to automatically fix it. And then there's also a button where they can publish an advisory
out to, from that repository to dependent repositories. And they could also request a CVE,
which is like a vulnerability official number to be recognized as an actual issue. So GitHub became,
what was the term they used? A CVE numbering authority, a CMA, of course, to, so that they can actually
issue these vulnerability numbers to be understood and like referenced as unique IDs across the security
landscape. Interesting.
Yeah. So all this stuff is integrated into GitHub. So GitHub, the researchers find the issue in the
main project. The main project gets a PR. The main project can then also push out these warnings to
other folks and request CVEs for their projects. That's pretty cool, right?
Yeah. Open source is growing up.
Yeah, it totally is. And it seems like it's, it's pretty solid for, for all the folks working on it.
It doesn't seem like it requires much of the maintainers. It's more like there's this bug bounty program,
from what I can tell. And also they threw in there right at the end of this. GitHub also updated the
token scanning and in-house service that scans for like API keys, like AWS access keys or whatever
that have been accidentally left inside of source code.
Oh, that's good. Yeah.
That's really good.
Yeah. It'd be pretty nice to like, you probably didn't mean this. Click this button to make this go.
Anyway, I think this is really cool. I think this is like, this is just plumbing to make open source
more secure. And I like that.
Yeah. And also just to, to be able to say, to have companies put money at open source projects to keep
them fixed. And it's not necessarily trying to get the, maintain the official maintainer to do it,
but to have some incentive for, for everybody else to watch these things. So that's great.
Absolutely. Yeah. These bug bounty programs have been working really well for the industry and it's cool to see
GitHub putting that in there. Also cool is digital ocean, not just for sponsoring the show, but because
they have awesome infrastructure and awesome product and we use them for our stuff. So let me tell you
about a new thing that they have generally available memory optimized droplets. And if you have a memory
heavy workload, basically this is the best way to get tons of memory in a droplet or a virtual machine.
So you can get eight gigs of Ram for each dedicated CPU. And then it goes from two CPUs all the way up to
enough to get you 256 gigs of Ram, whatever that math works out to be. And it's really good for like
high memory applications, like high performance SQL or no SQL databases and memory caches like Redis or
indexes, some kind of large data analysis runtime, something like that. So check those out at
pythonbytes.fm/digital ocean, really good stuff over there. Lots of cool things coming.
Uh, Brian, what you got next for us?
Well, we have a couple of friends of ours, Bob Belderbos and Julian Sequeira. They run a thing
called PyBytes and PyBytes challenges, not affiliated with Python bytes, just sounds similar.
It's the I versus the Y it's not even close to the same thing.
It's P Y B it I T dot dot. Yes. Anyway, I enjoy it. It's a challenges platform where you can just
sort of, there's a few of them for free, but it is a paid service that they give. It's
one of those things where you, they give you an, like kind of a written assignment and some test code
already there. And it checks to see, and then you have to fill in like the body of a function to make
all the test pass. It's a kind of a brain teaser sort of thing. It's a fun way to keep up, make sure
that you're practicing out of the box Python stuff that you don't normally do. That's what I use it
for. But the news is they just added test coverage. So, or tests testing. So in the past you were,
you didn't write the tests. They wrote them to evaluate your code, but they've added, a few test
challenges where they write the code and you have to write the test code to check that code.
And it's kind of cool, but they were, they actually talked to me about this as well as to
try to pick my ideas, but they came up with it on their own. How do you evaluate if the test code is
good? So if you, you evaluate if your source code is good by running tests, but the other way around
is a little difficult. Yeah. How do you test the tests? Yeah. So they did it a couple of ways.
they're using coverage up high to make sure that you're hitting a hundred percent coverage and,
you know, yes, it's debatable as for a large project of whether you should get a hundred percent
coverage, but for a small function or some small bit of code, it should, you should be able to hit
a hundred percent coverage. That's a nice thing. The other one is mutation testing. So there's a couple
projects we've heard of mut mut and mut pie M U T P Y. And, I think we talked about this earlier,
but, Ned Batchelder did write an article about his experience with mut mut, but,
PyBytes is using mut pie. And what it does is it takes your, the source code and changes something
about it. And mut pie works at the level of the, abstract syntax tree. And it changes like,
for instance, a division operator to a multiplication or, or changes a string to some other string or
something. And then it runs the tests again. And the idea is you want your test
to be able to, it makes a whole bunch of mutants of the code and you want the tests to be able to
kill off all the mutants, except for the original. That's how they're testing. It's kind of a neat
idea, but it's fun to play with. It is an interesting question to ask. How do you test the test? And I
think this is pretty creative. well done, Bob and Julian. I haven't used a mutation testing a lot.
I've tried it out, but I haven't used it like for projects. The idea of using it in a training
situation is a novel thing I haven't heard of before. And I think that's a cool idea to be able
to, to try to test somebody's, test code. Yeah, I agree. And like you said, a hundred percent code
coverage for a project that's real is challenging. I think also maybe a mutation testing for a project
that's real tricky because maybe it changes like, you know, the print statement that shows what the title
the app is and who cares? Like no one's going to check for that. Right. Right. But in this case,
where pretty much it's a very small focus bit of code and you're supposed to test it, like
presumably any changes to that are going to appear in the couple of tests. You're right. Yep. Nice.
Now, speaking of tests, I feel like I stole this one from you, Brian, just out of the universe. I mean,
so I want to talk about pi HTTP test. So this one comes from Florian Dallas or Dallas, sorry. And,
he actually sent in two things for this week, which they were both excellent. So I'm going to cover
them. And this is a command line tool for HTTP tests against restful APIs. Okay. All right. So
the idea is basically I want to test some restful endpoint and instead of going over and say, okay,
I'm going to create, I'm going to get requests. I'm going to do a get, I'm going to get the dictionary.
I'm going to verify like this thing is in the dictionary and so on. What you basically do is you just
write a simple little JSON document for each test that you want to run. Oh, cool. Yeah. So then it
has things like, what is the name of the test? What HTTP verb do you want to use? What is the URL
combination between host and endpoint? The headers you need to pass, a query string you need to pass,
and then you get back a report. It actually gives you a cool report in a like column or style validation
that lets you assert things about it. Yeah. There's a handful of these types of things. And I think it's kind of
neat way to describe API testing. Yeah. It seems really cool. There's a bunch of neat little libraries
that are used as well, like tabulate, which is a cool way to print the tabular data that they're
showing there and things like that. But yeah, I like this project. If your job is to test a bunch of
HTTP endpoints, you know, this is pretty cool. Yeah. Neat. Nice. All right. What else? What's next?
Oh, next. X-Ray. This was suggested by a listener. I think it's Guido Imperial.
Yep. I agree. Thanks, Guido. Sent it in. We haven't covered it before. And actually,
I didn't know about it before. People in the data science community probably do because it seems like
pretty powerful. But the gist of it is it's built, it uses and builds on top of NumPy and Pandas and
ask to offer in-dimensional arrays. You can do in-dimensional arrays in Pandas already,
I believe. But one of the neat things about these is that they've got labels on them. So
they're self-describing and they've got indexes. There's a few data types within it. There's a data,
so there's X-Ray data array. The data array is the in-dimensional array, but it has metadata,
like names and labels for the dimensions. And you can also have coordinates and attributes. And
coordinates are essentially like the tick elements for the different axes. And then attributes,
the data array doesn't really do anything with the attributes, but it's a way to consistently keep data
with data. So if you have to keep track of some extra things like, you know, where was this data
collected or really anything you can, you can add them as an attribute. And then a data set is a
dictionary-like collection of data array elements. I was playing with this and it's, it's pretty darn
cool. The, one of the things, nice things about using it is just keeping all of that, the dimension
names together. So if you have a multi-dimensional array, even just like a three-dimensional array,
it's sometimes hard to keep track of, you know, which axes is which, and this is all together. But it's not
just packaged together. You can also do things like use the label names and the axie names, and even axie
elements at the coordinates, they don't actually need to be numbers. For instance, you could have like the
months of the, months of the year or, or the letters of the alphabet be coordinates. You can use those as
selectors to be able to select rows and columns and those return different data array elements. The data
array elements also can be used in algorithms. They can just be passed directly to pandas algorithms.
So these are pretty cool. Yeah. It looks a little bit like it's taken some of the features from NumPy,
some of the features from pandas, some of the features from Dask, and sort of brings them together
into one package. So when I was going through some of the tutorials, I was to get somebody to talk about
this. It was like a three-dimensional array in, I think it's in pandas, is used to be, is considered
a panel. But when I went to look at the panel information, it looks like panels are being
deprecated for something else. So even in the pandas documentation, it was pointing to this x-ray
project. So... Oh, interesting. I think the people in the pandas community are definitely familiar with it.
But if you're using pandas kind of on the side, and you're not really in it all the time,
this might be helpful. Now, previously you spoke about Bob Bilderbos, and I said we got this item
from Florian Dalits. I'm going to bring those two things together in this next one. So...
Okay.
Bob had introduced us to Carbon. Remember that?
Yeah.
Carbon is like screen, sort of beautiful screenshots for colored code, right? Code, it's like a mock
faux little shell or whatever editor. Like, you don't use screenshots of real editors. You just
create that with carbon at carbon.now.sh. And that's cool, but those are generally static.
So Florian sent in this thing called term to SVG.
And it's a cool way to create animated terminal GIFs. So instead of going all the way to create, like, full-on screencasts of your
screen, you can run this in your terminal, and then you just do whatever you want to do in the terminal, and it captures it
perfectly into SVG, and then you get... convert that out to some kind of animated thing. Like, I guess the SVG itself is animated,
so you just show that in the browser or wherever you want to put it. Isn't that cool?
Yeah.
Very cool.
You basically just type term to SVG. Once you have it installed and it starts recording, you do a bunch of
stuff, and then there's a way to get out of its recording status. So it's pretty cool. It produces, like,
lightweight, clean-looking animations, or you can even do still frames if you want for, like, a project page.
So instead of, like, carbon is cool because I can put in the text and the code I want to show up, but maybe it
doesn't have, here is what the progress bar, and then the install steps with the spinner look like.
Right? It doesn't naturally capture what actually happens when that code or those terminal commands
execute.
So this file, it has color themes, animation controls, all sorts of good stuff. And yeah, it's pretty cool.
So there's a... probably if you want to... if this sounds interesting, you want to check out the examples.
So there's a whole page of examples, and there's a bunch of different stuff happening. You can just look
through there. And I think there's also templates that configure how it records and stuff. So there's a
bunch of predefined templates that you can go play with to get started from.
That'd be really cool for, like, a tutorial site or something to...
Yes, exactly.
Yeah.
Or even... or if you have a project, right? Like, if you're the maintainer of pipx,
it'd be cool to use this to create a way to, like, show how awesome pipx is. Like, this step,
then this step, and then boom. Right? And just put that right in your GitHub readme.
Yeah, I love it when there's little animated things in the readme. So when you go to GitHub,
you just see that.
Yeah. You and I, we spend an inordinate amount of time jumping into new projects and going,
is it interesting? Yes or no? Why is it interesting, right?
And this kind of stuff is the thing that just goes, after 10 seconds, I knew I wanted to learn
about it, right? It really makes a difference, and it's easy.
Yeah. Yeah. Very cool. Definitely check this out.
Yeah, for sure. All right. Yeah. So that's a good one. People can check that out,
Term2SVG. Pretty cool. All right. Well, that's it for our main items. What else you got?
I have one bit of extra news, is that pytest 5.3.0 was released the other day. And it is mostly,
there's some cool features. And if you, you know, pytest nerds, definitely check it out. But I wanted
to bring it up because I think a lot of people that just use pytest and are using it with continuous
integration systems should pay attention to this because the JUnit XML output, they've changed the
default. So the default format, there's an XML output has, there's an old version and a new version.
The new version has some more information, but they wanted to make sure that people know about
this. So if you run it, you'll get a warning and it's not really a warning. It just says,
it's just to make you aware that there's a particular format that's being deprecated.
So eventually in the 5.4 release, they won't support the old format. So if you see this,
encourage anybody using pytest and continuous integration to read the change log and understand
what's going on and make sure they're ready to either pin pytest or change their system.
Yeah. It's a good thing to put on people's radar for sure.
Okay. How about you, Michael? Any extra spits?
Yeah, I got a bunch for you. Actually, a couple of things. PyCon. PyCon's awesome. We love that
each year. And this year it's going to be in Pittsburgh for the first of its two years in that
city. And PyCon registration is now open. You can go and register, get your ticket before it
sells out. Oh, cool. Yeah. That comes to us from Jacqueline Wilson. So thank you very much
for sending that in. And then also I saw, I can't remember where I saw this somewhere. Actually,
I think somewhere funky, like a flip board or something. So Facebook has now decided that
Microsoft's Visual Studio Code is their default development platform. That's a little surprising
to me. Yeah. Interesting. Yeah. That's an article on ZDNet. And they're also helping Microsoft
improve the remote development experience in VS Code. Cats, dogs, all live in the same place.
Okay. Yeah. This is cool. I suspect that things like Vim and Emacs and stuff probably have a strong
representation there. But apparently it's all about Visual Studio Code over there now.
Anything else?
Yes. Two more things. Very exciting. So if the release schedule lines up correctly in the future,
extends as I expect it, this should be Wednesday before Thanksgiving, right? And that would mean
the day or two after that is going to be Black Friday. So I just want to point out that Talk Python
training is going to have a really awesome Black Friday sale. Get a whole bunch of stuff on buying
all of the courses, but also we're doing some special things to support the PSF and other stuff,
some surprises in there that I suspect people won't guess at. And there's no way people are going to
guess what is there. So check it out over at training.talkpython.fm. But you got to act right
away because it's only going to be there for like four days. It's a big deal. So check that out. And also,
we have a new course coming, Python for the .NET developer. So, so many people are coming from C#
and the .NET world over into the Python space. I thought it would be cool to create a course that
kind of gives them a big hug and holds their hand and helps them step over that divide. So it's like,
do you know about ASP.NET? Here's Flask. And here's how you use it in Python. Do you know about any
framework? Here's SeqWalchemy. Here's how you use it in Python. Like all the things that they need or
they love from C# and .NET. Here's the Python equivalent and why it's awesome and how it works.
Is that one that you did or did somebody else do that?
No, no, I did that one.
Because you're like the perfect person for that.
Exactly. I spent so many years doing C# and now I'm all about Python. So exactly. I figured
like, why don't I try to think back to the way it was for me many years ago and like sort of extend
that experience back to other people. It's probably not going to be out yet. It may be out at the time
that people hear this, but it's coming really soon. So I'll just put it out there as that.
That's nice. Hey, speaking of Black Friday, I do not have any insider knowledge,
but Pragmatic Publishers often does a Black Friday sale too. It's usually fairly steep. So if you've
not picked up the pytest book yet, and really, if you're listening to this and you haven't read it yet,
what's going on?
Come on.
If you haven't, maybe check out pragfrog.com and see if there's a sale.
Definitely. I'm sure there will be. It would be surprising.
Yep.
There weren't. Awesome.
How about a joke or two or three?
I like three jokes.
Okay.
It's a good number. So this one, first one is more of just a geeky STEM type of joke,
but I think people will like it. So I love soda drinks, you know, Coca-Cola, Dr. Pepper,
root beer, things like that. So this one, I try to not drink too much, but I do like it. But here's
how that world can clash together with math. What do you get when you put root beer into a square
glass?
I don't know. What?
Beer.
I don't even get it, but it's funny.
If you take root of beer and you square it.
Oh, okay.
Okay.
Right? Like the square root of beer and then you put it in a square glass.
Okay. That was bad.
What's your next one here?
Okay. What do you call an optimistic front end developer?
I don't know. What do you call it?
A stack half full developer.
That is awesome.
Okay. Now I also, I was going to tell a version control joke, but they're only funny if you
get them.
Get G-I-D. Awesome. Those are both good. I like them. Yeah. Great.
Cool. Well, thanks again for having a nice conversation this week.
Yeah. You bet. Thanks as always. See you later, Brian.
Bye.
Thank you for listening to Python Bytes. Follow the show on Twitter via at Python Bytes. That's
Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm. If you have a news
item you want featured, just visit pythonbytes.fm and send it our way. We're always on the lookout
for sharing something cool. On behalf of myself and Brian Okken, this is Michael Kennedy. Thank you
for listening and sharing this podcast with your friends and colleagues.