Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.
This is episode 152, recorded October 9th, 2019. I'm Michael Kennedy.
And I'm Brian Okken.
And this episode is brought to you by DigitalOcean. Check them out at pythonbytes.fm/digitalocean.
Get $50 credit for new users.
Now, we may have touched on this concept of legacy Python before, Brian. Have we covered it?
Yeah, I think we have.
We definitely have.
So we know that there are companies out there that say it's really tricky for us to upgrade to Python 3 because.
And sometimes that's because, I don't know, just they don't put the resources into it, right?
Like they would rather work on features rather than going back and rewriting old code to do the same thing.
But so it's not so old. Things like that.
Other times, it's because they have a ton of Python code.
And we're hearing more and more stories of these companies that have been like head in the sand, waiting until the very, very last minute to make those migrations.
And they're just like, all right, finally, somebody has raised it to the level that like it has to be dealt with, right?
Yeah.
Well, it turns out that banks use a lot of Python code, as we know.
And I've heard of Bank of America using a ton of code and having a lot of people working on some Python projects.
But JPMorgan, JPMorgan Chase, they use maybe even more.
They use a ton of Python.
So there's an article that's based on this presentation by Misha Selman, who is the executive director at JPMorgan Chase about this.
It was given at PyData 2017.
So they've been working on it.
But the problem is they have 35 million lines of Python 2 code.
Oh, that's a lot.
In terms of Python code, that's kind of ridiculous, right?
That's an insane amount.
And so they've got a lot of Python code that has to be converted to Python 3.
And this is from their Athena trading platform, which is at the core of their business operations, right?
So they got a late start to migrating Python 3.
And people are pointing out this could be a security risk for them, right?
Like we saw what happened with Equifax and some outdated things there.
Like who knows what the risks are?
I think it's probably less than something like the web frameworks that were out of date at other places.
But yeah, they have a lot of stuff that has to be migrated.
And internally, they use Python for pricing, trading, risk management, analytics, and even machine learning.
So just to look at some stats from this project, the feature set utilizes 150,000 Python modules, over 500 open source packages, 35 million lines of Python code contributed by 1,500 developers.
Okay.
So they got a big team.
That's a huge scale.
And by the way, I wonder how much JPMorgan Chase is contributing back to those 500 open source projects.
Hopefully some.
All right.
Now, it says they're going to miss the deadline, right?
That most of the strategic elements are going to be in place by Q1 2020.
But they can't do it all.
And I know it's probably like a good roadmap for folks, right?
They don't have to upgrade it all and then release that new thing, right?
They can upgrade elements at a time.
Yeah.
And there's a lot of great stories on how folks have done that.
I think probably the Instagram project was the most awesome one I've seen where they didn't even branch, right?
They just found a way to seamlessly move from Python 2 to 3 while still running on 2 and then finally flipping the switch.
Here's another one I thought you would find interesting, though.
They have some other stats.
You know, on your projects, how often do you commit code?
It's like once a week, once a day, once an hour.
Yeah, several times a day.
Yeah.
I'm kind of the same.
I do that.
And you guys don't really release stuff.
But like say the Python Bytes website or the Talk Python training site, you know, those probably do some form of website release every other day.
Some sort of deploy, restart, like run through the whole deployment process.
So JP Morgan Chase uses continuous delivery with continuous integration, continuous delivery with 10,000 to 15,000 production changes a week.
That's amazing.
It's like mind blowing, isn't it?
Yeah.
So they're on it, I guess.
But it's just such a project of massive scale that it's hard to get your mind around and hard to find analogies.
So I'm sure there's a few other projects like this in the world, but it can't be many.
No.
Well, that's like one a second or faster.
It's constantly deployed.
It's got to be microservices and other stuff, right?
Otherwise, just like how would you go to the website?
How would you use the services?
Anyway, quite incredible.
All right.
Well, what you got for our next one?
This is just kind of a cool little tool called Organize.
And it was suggested from Ariel Barkin on Twitter.
And I took a look at this and I'm going to start using it right away.
So it's a Python-based file management automation tool.
And the idea is people are lazy with how they save files and download files and whatever.
And on my Mac, for example, all the screenshots just show up on the desktop.
And then, you know, occasionally I'll just take everything and lump them into a clutter folder or something.
But this is a tool where you can give it rules.
It's an EAML file.
And say, have it do things like move all your screenshots from the desktop into a screenshots folder.
Or look through all your downloads to look at the incomplete downloads that you canceled or something.
They're still sitting there.
And just trash those if they're older than, you know, a few days old or something.
Like doing things like removing empty files from certain folders like your download or desktop or other places.
One of the examples is to organize your receipts and invoices into date-based folders.
Which is pretty cool because there's macros involved that you can look at the file touch time and figure out what date and extrapolate the dates and stuff.
And yeah, I always, when I'm paying bills or something, I save the receipt to just wherever in the downloads folder or something.
And having this, just running this every once in a while could clean it up and put everything in its place.
It's pretty cool.
It's super cool.
You could just put it on like a cron job that runs every five minutes or every minute or something, right?
It just goes, boop.
It's got to be super quick.
Just looks at the files and a few folders and then does some text matching.
It's one of those like, you know, automate the boring stuff sort of things that somebody thought, you know, everybody has this problem.
So, yeah, it's nice.
Yeah, I like it.
I have the same problem with receipts and stuff.
I'll get them an email or as a PDF attachment or actually it's just an email that I'll print it to PDF so that I can save it for taxes.
And they just like clutter up.
Yeah, it's, I could totally see just using that.
The rules seem like they're rich enough to do that.
So, yeah, it looks really good.
Yeah.
Super cool.
All right.
Speaking of cool, let me just tell you about DigitalOcean.
So all of our services run on DigitalOcean.
Audio you're listening to now somehow flowed through the DigitalOcean servers to get to you.
And they've got all sorts of great options out there.
They're simple but powerful.
There's not knobs to run absolutely every little edge case, right?
You set up the main server that you want to work with.
You have spaces.
You have hosted databases in MySQL and Postgres.
And you even have caching like Redis and things like that.
So super nice.
Check them out at pythonbytes.fm/digitalocean and get $50 credit for new users.
Highly recommended.
Now, this next one is a fun one.
And it took me a minute to realize what this was about, Brian.
So I realized there's this new PEP, PEP 589.
And it allows you to define typed dictionaries.
Like define a type that represents a dictionary.
Well, it turns out there was already a way to do that, which is why I was confused.
Because there's PEP 484, which has been around for a while, which lets you create a dict of K, V.
Which is like, here's a dictionary of arbitrary keys.
And it has maybe integers.
Or it has user objects.
Or whatever, right?
So you can define these uniform dictionaries, which is kind of interesting.
But this new PEP, it lets you go much farther.
It's proposed by Juka Letozalo.
And it's actually sponsored by Guido Van Rossum.
So remember recently we spoke about Guido.
And we had this philosophical debate of like, well, he's all about typing these days.
But originally typing was like explicitly left out of the language.
What's the story?
So here's another typing thing that he's participating in, which I think is interesting.
So this is accepted.
It's scheduled for 3-8.
So all sorts of interesting stuff.
And it's coming down the line, right?
Soon, actually.
So what it lets you do is imagine you have an arbitrary JSON document or an arbitrary Python
dictionary, really.
But it's super easy to think of like, well, somebody sends me a JSON request and I want
to treat it as if I know what's happening here.
It lets you actually specify the shape of those things, both the keys as well as the values and
potentially nested documents, right?
So you might have a JSON object that's got like some values.
One of those values might be a list of other JSON documents.
You can describe that with this type dict thing.
So the way it works kind of caught me off guard at first, but I think I like it.
So what you do instead of just saying, you know, there's a dictionary of like string comma
user, you actually create a class which derives from typed dict, okay?
Okay.
And then it has fields.
It looks a lot like data classes a little bit.
So you might have like a name colon stir and a year colon int in this thing that is not
actually the dictionary, but it is the type that validates the dictionary.
All right.
Oh, okay.
And then you can say it is one of those, right?
So I say the example they give is there's a movie.
So you say movie colon capital M movie is the name of the class.
And then it's just a dictionary, but the dictionary has the name, which is a string value and a
year, which is an integer value and so on.
And then you can actually validate it.
And the valid, the, the like static type checker, like my pie and so on.
Well, if you say movie of director, it'll say, no, no, no, you can't set this value into this
dictionary because it doesn't have a key called director.
Or if you try to set the year to the string 1982 when quotes, it'll say, no, no, then this
is a string at expected an integer.
But the errors come at the type checking time, right?
This is a type checking time.
Although, you know, it's totally reasonable that things like PyCharm and VS Code would add
edit time checking for this as well.
Cause they do for all the other type stuff.
Yeah, but it's not a runtime.
It's not a runtime thing.
Yeah.
All the typing stuff.
Okay.
And this is definitely that way.
So you're not like re-implementing the dictionary.
You're not creating a dictionary type that is like different.
You create a type, which then talks about just a plain dictionary.
So quite interesting actually.
Yeah.
It does take a little while to look at it and go, does this make sense?
But yeah, it does.
Right.
Imagine you're getting, you're writing an API and somebody's submitting like a JSON post to
you and you want to know, is it valid?
Right.
You could use this basically to validate your schema or at least describe the schema you
expect.
Yeah.
Neat.
It is neat.
Speaking of APIs and new web things, your next one is one of those, right?
I got carried down that rabbit hole.
No, that's cool.
The next one, I was just enticed by the name.
So there's a package called gazpacho.
It's just great.
It's fun to say.
It's fun to eat.
But anyway, gazpacho is a web scraping library.
And the goal of it is to replace requests and beautiful soup for most web scraping projects.
And I got to tell you, I was going to do, I have some web scraping projects that I wanted
to do.
And I know that requests and beautiful soup are easy to use and are super powerful.
But that one use case where you're just grabbing, like you're just doing a get, then you parse
it and then you find some stuff in it and separate it out.
That's so common that this is basically it's optimizing for that.
There's an example article that I'll link to also that uses gazpacho to scrape hockey data
for the use of fantasy sport use.
But it's just a really simple interface.
You import from gazpacho, you import get and soup as a class.
And you can use those to grab some HTML and parse it, find some stuff in there.
It's just a handful of lines of code and you've got a web scraper on your hands.
So I like it.
I think I'll give it a shot.
But I tried it out and I wanted to bring this up because I tried it out and I ran into a
problem that I was getting these certificate errors.
Have you ever gotten certificate errors when you're trying to parse things or pull things
down?
Yeah, just once or twice.
And it's the kind of thing where you bounce off the walls of Stack Overflow until you get
it fixed and you forget how to fix them.
But yeah, so what did you do?
I did the same thing, went to Stack Overflow.
And apparently in within the, and I don't know if this is just a Mac thing or not,
but on Macs at least, when you install Python, you also, in the install directory in applications,
Python 3, whatever, there's a file called installcertificates.command.
And you just have to run that.
And then it has the list of certificates or something.
I don't know how certificates work, but it makes it so that you can access SSL stuff from
Python.
So yeah.
Ran into that today.
That's right.
I'm glad you're linking to it.
So now we'll have it for forever.
Yeah.
That's cool.
It's nice.
Spacho is like two to three times faster than Beautiful Soup, which is pretty sweet.
I like that.
It also does a lot less.
So that makes sense.
Yeah, for sure.
It's a more focused thing.
Yeah.
That's like the 80% case though, right?
You just need to go do simple things.
That's what I'm going to use it for.
So the last thing I want to cover for our main items is PIP.
So remember, actually, we spoke about PyDist, P-Y-D-I-S-T?
Yeah.
This is like a private PyPI as a service, I guess.
It's kind of the way I would describe it.
So right now, I think they, before we had talked about this and we're like, well, it just
in beta doesn't seem to have any pricing or anything like that.
So they have pricing and a little bit more details.
They've more or less launched at this point.
And so this article is not about this, but it was written by the folks who run that.
Just that's the connection back to the previous thing.
And it talks about how pip install works.
So for this section, I just want to talk to you real quick about when you say pip install
certify, like it did in that previous article you just mentioned to fix your certificates.
What do you do?
How does it work?
All right.
So it walks you through all the steps and all the decisions and whatnot that pip has to make
when you say pip install some package.
So the first thing it has to decide, well, first, I guess it does the package exist, right?
And then it needs to figure out which distribution of the package to install.
Because we have eggs, we have wheels, we have source.
We have all these different types of distributions.
There are seven different kinds of distributions.
But the most commons are either source distributions or binary wheels.
So focus on those, right?
So source distribution is just, here's your Python code and maybe the C code that comes with it.
And as part of the setup, we're going to run a compiler against the C code to make sure that
that's compiled in your machine, right?
Super easy to write, not so easy to make sure it works everywhere, not just works on my machine,
right?
Because you've got to have compilers and all the platforms.
And oh, yeah, what about that old version of Windows that was a minimal install and doesn't
have GCC or Visual Studio or whatever?
So wheels are a little bit more safe and also faster.
But that means they have compiled C code, which has to be, you have to have multiple ones for
different platforms, right?
So Windows versus macOS or something.
Yeah.
The benefit is stuff installs fast, right?
So like NumPy takes about four minutes to compile from source.
So if you did a source dist of NumPy, pip install might be slower than you would otherwise
expect, right?
So anyway.
Yeah.
The four minute pip install, yes.
Yeah, that's before you even hit the dependencies, right?
That's just the primary thing.
Yeah.
Okay.
So it has to figure out which one of those are.
And there's actually a known URL.
So like pipi.org slash simple slash package name is where you would go.
So you could go to that slash request, for example.
And there is a huge just flat, it's like an, it's a weird API.
It's like HTML list of like a bunch of wheels with platform names and tarballs and all
sorts of stuff.
So it starts out my going there to figure out what is here.
What can I find?
So first it determines what system you're on and what's compatible with things.
So like if you have a binary wheel, there's actually a PEP that talks about how you figure
out which one that is.
And then if it's a source gist, like, well, you just assume it works.
So once it has that, then it'll try to get the best and it prefers wheels.
And then it has to figure out the dependencies.
So for binary wheels, there's a file called metadata that has a list of those.
So that's cool.
You can just look at that.
If it's a source distribution, it figures it out by running the setup.py.
So that's interesting.
So to run setup.py to actually figure out what dependencies it has to install, you know,
go do that.
And then you might have two dependencies.
You might have a thing and you might depend on, let's say, BeautifulSoup.
But you also have some other library that also depends on BeautifulSoup if you follow the
dependency tree.
And they might even specify versions.
So you might wonder, well, what happens if one depends on one version and the other depends
on the other?
Turns out it just installs it anyway.
Let's take the latest.
That's going to be fine, right?
It's different than like a requirements file that has like different dependency, like pinned
versions.
Like there's a slight difference there.
So finally gets it, builds it, installs it.
And then it has to figure out where's the path?
Am I going to install it to a virtual environment?
Am I going to install it into the system or the user path?
Things like that.
So you can look at sys.prefix to figure out which one of those are.
And there's some environment variables.
And finally, it copies it over in the right place.
And your package installed.
Oh, before it considers your package installed.
Also converts the source files into PYC bytecode files.
So they don't have to get parsed again.
Then your package is installed.
Okay.
Yeah.
So anyway.
Simple.
Yeah.
So if you're wondering like what happens as part of the pip install stuff, there's a
lot of details.
And I didn't cover all of it.
But like, you know, as much as I thought made sense.
I was just curious.
I was going to try to find one of those complicated packages.
Yeah.
That I knew had to be compiled.
Because I went to a couple of mine.
And they're just Python codes.
So there's just one per version.
One wheel.
But like NumPy, for instance.
I know it's got some compiled code in it.
It's got like, I lost count.
It's like 15, 16, 17 different wheels for each version.
Yeah.
Requests has got a ton as well.
Yeah.
It's interesting.
It is interesting how that works.
I'm glad it all works.
I don't have to think about it too much.
I don't have to think about it either.
But it turns out there's like a lot of conversation in there about some stuff that is not totally
solved even today.
Right.
about trying to resolve the dependencies in a totally predictable way before you start
installing anything and stuff like that.
So it's worth checking out.
It's a hard problem.
Yep.
For sure.
But want to finish up with a cool trick?
Like a zoo trick?
A zoo animal trick?
Oh, yeah.
I'm just zoning today.
So Kevin Markham, he runs, what's the thing he runs?
Data School.
Plus a super nice guy.
Well, he's doing something neat that's called Daily Pandas Tricks or Tricks and Tips or
something like that.
But anyway, we've got a link to it.
He's sending out a little tip or trick about pandas every day on Twitter.
And the page we're linking to has a whole bunch of them already built in.
And I like the notion of just trying to fit something.
Often they're little screenshots, but they're still pretty small.
A little lesson of how to do something cool.
I just picked out one, which is like, let's say you wanted to rename all of the columns
in a data frame the same way, like to replace all the spaces with underscores or something.
And he just shows you how to do that in a little thing.
I think that's neat, especially for something, for a package like pandas, there's a whole bunch
of stuff you can do with it to have a way to just see a little extra new thing every
day to say, that's something I might use.
I'll keep looking at that later or something.
So I don't think we've talked about it before, and I think it's a cool thing he's doing.
So I wanted to highlight it.
Yeah, it's definitely a cool thing he's doing.
And pandas is one of those things where it's not always obvious.
All the little magic that you can do, right?
Like if you want to go to the columns and do string operations, just dataframe.columns.str.apply
your operation, right?
Like that's, after you use it for a while, it's obvious, but maybe not, not right away.
It definitely isn't to me.
Pandas feels a little like magic to me.
I'm looking at this going, I would not have guessed that.
Exactly.
It's not obvious.
But once you know it, it's like, well, of course that's better than like, there's this saying
that if you find yourself looping over things in like NumPy or pandas, you're probably doing
it wrong.
One of the nice fun things I think is if you get really good at something, you'll start
learning the things that you shouldn't do, but that are fun.
And some of Kevin's tips are, you can do this.
It's sort of fun, but don't because it's confusing to other people.
But anyway, here's the trick.
Nice.
It's neat that he's including those.
It's clever, but too clever sometimes.
Cool.
All right.
So do you have any extras to share?
Oh, not only that we just got finished with our first Python West meetup.
And last night, and it was both exhausting and really fun.
So thanks for helping out with that.
Yeah, you bet.
Good job putting it together.
It came out really well.
Everyone seemed to have a great time.
There was a totally good turnout.
I was blown away that it was actually, you know, basically sold out, not sold out, but booked
out on its very first run, which is crazy.
And people out there listening, if you want to come and give a talk at the meetup and you're
willing to find your way to Portland, shoot a message to Brian or me.
Yeah.
Let us know.
That'd be cool.
Yeah.
Would be cool.
And then before anybody asks, it was not recorded.
So yeah, you have to be here.
Yeah.
How about you?
You got some news to share?
I got all sorts of stuff.
A few really quick things.
One, I upgraded to macOS Catalina yesterday.
And so far, so good.
No major problems.
All the Python things seem to be working.
So if you're wondering, I did hear that someone out there was having trouble with Miniconda.
I don't use Miniconda.
So I have no idea about that.
Maybe do a Google search if that matters to you.
Also, Brian, I switched to working with Adobe Audition.
I've been using Audacity and GarageBand.
Finally broke down and paid the $30 a month for Adobe Audition.
And wow, is it worth it.
It is so good.
What has been wrong with me to not do that?
I just didn't want to learn new software.
It's not so much about the money.
It's just like, I don't want to learn new hotkeys.
I already know the hotkeys.
But it's so super good.
The reason I bring it up on the show instead of after is if you hear like weird artifacts
or something odd in the audio, call our attention to it.
Because there's all these dials and knobs that can like do things like chop off the S's
at the end of words if you turn them too far and stuff like that.
So hopefully things sound better.
If they don't, let us know.
And then the two Python related things.
Really quick, Azure Databricks also is dropping support for Python 2.
So just one more brick to fall for Legacy Python.
A Python death clock continues to toll for those who hang on to their Python 2.
And the folks over on the VS Code team, Rong Liu in particular, just announced that at PyCon
China, they just revealed a cool new Jupyter UI variable explorer and Telesense stuff for basically
running Jupyter's inside of VS Code.
So if you're a VS Code user and you care about Jupyter, check that out.
Very cool.
Yeah, absolutely.
Absolutely.
Well, that's it for the stuff.
I got a story for you, a joke maybe.
Yes, please.
This one comes to us from maybe an unexpected space.
It comes to a person on Twitter, goes by the sarcastic pharmacist, sent us this actually
really good joke and a nice comment.
And the theme is that it's hard to distinguish between what is like super easy in programming
and what is like nearly impossible for people who are not doing the programming themselves.
So this is actually an XKCD article 145.
It's got a programmer, a woman sitting there working at her desk and there's like a manager
type who comes up and is issuing feature requests.
Okay.
The manager, I'm going to think of one of the people from office space maybe.
And it comes over and says, when a user takes a photo with the app, it should check whether
they're in a national park.
And the woman says, sure, easy.
Easy GIS, look up, give me a few hours.
Oh yeah.
And it should also check whether the photo is a bird.
She says, I'll need a research team in five years.
The subtitle is, in CS it can be hard to explain the difference between the easy and the virtually
impossible.
Yeah.
So there you go.
Yeah.
I don't know.
That resonates a lot with me at least.
Yeah.
We'll probably get a bunch of the image people telling us that it's like five minutes now with
all the new image libraries to do a bird.
Yeah.
But that's now, right?
Like we probably should, I should see if there's a date for this just to be fair.
They don't have dates on these.
That's kind of funky.
All right.
Anyway.
Well, there's probably some algorithm that figures out the number of the XKCD and maps
it back to a date.
But yeah.
Yeah.
But that's funny.
Cool.
All right.
Well, great to chat with you as always.
Thank you.
Yep.
Bye.
Thank you for listening to Python Bytes.
Follow the show on Twitter via at Python Bytes.
That's Python Bytes as in B-Y-T-E-S.
And get the full show notes at Pythonbytes.fm.
If you have a news item you want featured, just visit Pythonbytes.fm and send it our way.
We're always on the lookout for sharing something cool.
On behalf of myself and Brian Okken, this is Michael Kennedy.
Thank you for listening and sharing this podcast with your friends and colleagues.