Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.
This is episode 95, recorded September 12th, 2018. I'm Michael Kennedy.
And I'm Brian Huckett.
Hey, Brian. How are you doing this fine, fine Wednesday?
I am excellent.
Nice. It's also excellent that Datadog is sponsoring a show.
So before getting further, tell them thank you. Pythonbytes.fm slash Datadog.
It's a cool shirt if you go there and follow along. We'll talk more about that later.
You know, I feel like summer's coming to an end, and I've been quite lazy all summer.
I'm not sure I'm ready to get back into the main swing of things, but it's upon us.
Yeah. And do you know who else is lazy? Programmers are lazy.
Productively lazy. They put lazy to good use.
Yes.
They make lazy a virtue.
And so this was our segue from nothing into the first item, which is Dataset.
And Dataset is a Python package that bills itself as databases for lazy people.
And this is actually something I totally want to try because it looks fun.
Their premise is programmers are lazy.
Oh, it says, first, I'll just read some of the top.
Although managing data in relational databases has plenty of benefits,
they're rarely used in day-to-day work with small and medium-scale datasets.
But why is that?
It's because people are lazy, and they'll throw it in JSON or CSV instead.
Oh, they say the answer is programs are lazy.
They'll use the easy solution.
And I guess I can't disagree.
I've used JSON format as essentially a local database before.
But this is kind of cool.
This is what it is, is that it's built on top of Alchemy.
So it's built on top of SQLAlchemy, so it can work with any database or a SQL-style database.
And it's just really easy.
It looks kind of like a NoSQL.
It's kind of hard to describe, of course, over the air.
But it's pretty simple and worth checking out, I think.
Yeah, I like it.
It does automatic schema creation, upshirts.
It has query helpers, like distinct and stuff like that.
So if you were to say, I'm just going to use like an in-memory dictionary or other things like that, it's kind of nice that it helps with some of those things.
So you just said a couple terms that I don't even know what those are.
So upcert.
Upcert is I have a record, and I'm going to try to save it to the database.
If it does not exist in the database, an update would fail, right?
But if it already exists, an insert would fail because it would be a duplicate key.
And upcert says, hey, data access layer, take this thing and save it.
If it doesn't exist, put it in there as a new thing.
If it does, make an update and set the values to the new one.
Okay.
And it also deals with sparse stuff.
So one of their initial examples is, let's say you've got people or something.
And in the first person, you give them a name and an age, and you can insert that.
The second person comes along, and you give them a name and age and a gender.
And then you can search easily.
And yeah, like you said, it deals with the schema for you already, and you don't have to deal with that.
Yeah, and the example that you have in the show notes here uses SQLite as the back-end database,
but it uses the memory connection version.
So you can just load it up with data and then query it and work with it.
And then when your app shuts down, it just goes away.
Maybe you output stuff to a JSON file or whatever.
So you don't even have to store the database necessarily.
Yeah, to be able to use some queries on information.
That's interesting.
But you can also just play with it this way and then turn it into a file-stored fileback database as well,
even with SQLite or with something else.
Yeah, absolutely.
Yeah, it's a good find.
It's quite interesting.
I like it.
Yeah.
So I have a question for you, Brian.
Okay.
Why is NumPy, not this thing that we're going to talk about next, but NumPy itself, faster than regular Python?
Do you know?
I think because it's got stuff compiled in C.
Exactly.
Because it's written in C, and it can even do parallelism and stuff.
It could take advantage of cores.
Like, my new laptop is ridiculous.
It has 12 cores.
Nice.
That's a lot, right?
So maybe we could actually take advantage of that with NumPy.
Well, if you have NumPy code, this thing that I'm going to tell you about takes it to another level.
It's called Kupy.
I think it's how you say it, because I think it's based on CUDA.
So Kupy is what I'm going to go with.
And its full name is KupyGPUNumpy.
And the idea is it's an API-compatible library with NumPy.
So all the NumPy features that NumPy has, you can call the same functions on Kupy, I called it.
But instead of running on your, you know, 6, 4, whatever cores you have on your machine, it runs them on the GPU cores, which is insane.
Oh, wow.
Okay.
Isn't that cool?
And I looked, and I did a quick little search, just like, hey, what's like a modern machine learning or data science type GPU you might get?
So pretty standard one might be the GeForce GTX 1080 Ti.
Okay.
These things are getting super expensive because of all the Bitcoin miners and stuff.
But anyway, you get one of these things, and it has 3,584 cores.
3,000.
And you can run your code parallel on all of those.
Wow.
So instead of, you know, like having like, oh my gosh, I can't believe I have 12.
No, you have 3,500.
And all you have to do, the only line of code you have to change is instead of import NumPy as NP, which is the very common thing that people do, you would say import Kupy as NumPy.
That's it.
And now you're running on these CUDA cores doing GPU-backed data science.
Okay.
Do you remember what CUDA is?
I don't know what CUDA stands for.
I bet it's an acronym because it's all capitalized.
Yeah.
There's a lot.
There's like layers upon layers of acronyms here.
I don't actually know what CUDA cores are.
Okay.
Yeah.
It didn't mean to put you on this.
It's the mechanism of parallelism on the GPU, basically.
Okay.
Nice.
But I don't actually know more than that.
Yeah.
Isn't that cool?
I like it.
Yeah, yeah.
So it's really cool.
It has this compatible API.
I threw a little code sample in the show notes there.
And if for some reason you're like, you know, I actually need to customize how my code runs on the GPU, which is a thing sometimes people do.
You can like program against the CUDA cores and CUDA kernels and things like that.
You can actually embed in your Python code, C++ code, and CUDA pie will actually compile that down to a CUDA binary, which is even cooler.
Okay.
I was just curious.
So I'm really not a hardware guy.
So bear with me.
You said you have 12 cores.
Is it on a laptop that you're running?
Yeah.
It's a new MacBook Pro.
So it's an Intel Core i9 maxed out.
And it's really six cores that are each hyperthreaded is how it works.
But the OS sees them as 12.
So are there GPUs on a normal laptop or on your laptop?
Or is this GPUs just something that, okay.
No, no.
There's a pretty high end one on my MacBook.
It's not as high end as this.
Not even close.
But, you know, maybe half or something, I would guess, in terms of performance.
That's a pretty bad estimate because I don't really know.
But yeah, you could run this on a laptop.
Yeah.
I'm just curious if the Kupai would speed up things on just on a laptop or something or if
it would.
I would think so.
I mean, you got to have an algorithm that's like well adapted to GPUs.
But if you did, then I would think so.
Yeah.
Okay.
Well, this is neat.
For the people that really care about it, really care about it.
So this is cool.
Yeah, absolutely.
And I mean, you can go and get like GPU clusters on AWS or on DigitalOcean or things like that.
Yeah.
Okay.
And so you could actually ship your code up there even if you don't have one.
Final note on this one is there was a PyCon 2018 presentation on this.
And so I'm going to link to the presentation as well if people want to watch 30 more minutes
of this.
I think I would.
Yeah, I actually do too.
It looks really interesting.
Yeah.
All right.
I'm feeling a theme coming on.
In episode 84, we did touch on somebody called in or called in.
We actually don't have phone lines.
Somebody contacted us and said, hey, you should cover pre-commit.
And we have.
We did talk about pre-commit in episode 84, but we just sort of talked about what it was.
But today I ran across this fairly cool article called Automate Python Workflow using pre-commit.
I like this kind of an article actually of, okay, here's these cool tools using pre-commit
black and flake.
How do I put that in my day-to-day workflow?
And how does it really work?
And this is from LJ Miranda.
So good job, LJ.
It's got a great graphic at the start with telling you that you've got changes.
When you add something, you go to get add, you go to staging.
And then when you do a commit, what happens is the pre-commit will intercept that part and
it kicks off whatever pre-commit hooks you've got.
Set it up.
And if all of those pass, then it lets the commit happen.
And if it doesn't, it kicks it back.
And then it shows you how to deal with all of the different configuration that is available
with pre-commit.
I like this.
It's a good starter.
If you're still quite not on board with pre-commit, this is a good article to read.
Yeah.
Pre-commit's pretty cool.
And that's a Python package that you install that then manages all the rest, which I think
that's great.
Yeah.
This article, there's a little video.
And I think it's an animated GIF or something.
A little short demo video that runs.
I don't know how to do this.
This is neat.
So it shows it in action.
Yeah.
That's really cool.
I like those little autoplay GIFs that'll animate stuff.
Because sometimes it's like, you know, if you could just see it happening, it would be
so much more easy to grok with little pictures trying to tell me.
Yeah.
And I also don't mind.
Something like that is fine if it has like an actual video to play.
But don't give me a half hour video.
A little couple minute video at most is great.
Yeah.
A half hour GIF, probably not the way to go.
I don't even know if you can do that.
So the way, one way to go that is good though, is to check out Datadog.
So this episode is sponsored by Datadog, as I said, and I really appreciate them supporting
the podcast.
Datadog is a monitoring platform that brings metrics, logs, requests, traces, all that kind
of stuff into one place across different systems and computers and all sorts of stuff.
So you can use their trace search and analysis, which lets you break down Python application
performance using high cardinality attributes like show me what this customer has done across
my application or show me all the behaviors for this URL and really easy to troubleshoot
your app.
So start doing that with your Python apps today with a free trial and Datadog will send you
a free t-shirt, which has a cute little dog on it.
So visit them at pythonbytes.fm/Datadog to get started.
So you were talking earlier about that cool little GIF thing, and I think you can do it with
Camtasia.
Like you can record.
Okay.
So I think you can do it with Camtasia.
You can record basically a screencast and export it as a GIF, which is already pretty
cool.
Oh, okay.
But this next item has a really nice little animated GIF thing going on as well because
it's super good to see it in action.
So have you heard about PySpy?
I have not.
So PySpy is interesting for a couple of reasons.
It's interesting because it's a cool tool that people can use in some places that they
could not previously do so.
It's a Python profiler, so you can hook it up to your Python application and it'll tell
you where your Python app is spending its time, what functions and what it's doing and things
like that.
And it acts kind of like the Unix top command, which will take over your screen and it'll
show you a list that's kind of updating every couple of seconds what's happening.
Okay.
That's pretty cool.
So I can hook up this profiler and it'll live show me sort of the equivalent of like a process
report, like a CPU usage report.
But it'll say right now you have these various functions that have run recently and here
like we'll put the most expensive ones on top, things like that.
Oh, neat.
That's cool, right?
And so you can watch that little graph, that little GIF thing and see it going.
This is written by Ben Fredrickson and it's just taken off.
I think it was started in July or something like that.
It's already got 2000 GitHub stars.
So what's even cooler though is it'll let you visualize your Python's app's time without
restarting or modifying your code in any way.
And it can attach to running processes and then start to profile them.
Oh, nice.
So normally profiling happens by I run a profiler, which runs my code, which does a bunch of stuff.
Or maybe I reverse, I'll write some code, I'll import C profile and I'll call a function,
start profile, save profile, export, et cetera.
Right?
Like you, it's really invasive.
So if you do it from the outside, like the profiler runs your app, you can't do it in
production.
It makes it slow, all sorts of stuff.
If you do it the other way, you're doing all sorts of writing code to change it.
This, you just say, Hey, there's a random Python program.
I'm going to go profile that and it'll just attach to it.
Nice.
Yeah.
You can just give it a process ID.
Yeah, exactly.
Give it a PID.
And what's cool is because of that, that means you can use it in production.
I could log into my web server that's getting pounded on, not responding correctly or whatever.
And I could actually begin to profile it without like wrecking my thing or slowing it down or
restarting it or whatever, or any long running process.
Like while the problem is happening, you can just attach to it and figure out what's wrong.
That's the key thing.
Cause maybe restarting it, rerunning it, it takes like four hours to get into that weird state.
You never know, right?
Yeah.
Oh yeah.
This is cool.
Sweet.
That's pretty trick.
So it's written in Rust actually, but it's pip installable.
So all sorts of cool things.
And then he even goes into, Ben goes into how does it work?
So there's a section on how does PySpy work?
So I'll just read you this and tell me if this sounds like a program you would have written.
It's not what I would have.
PySpy works by directly reading the memory of the Python program using process VM Red V system
call Linux or VM read on OS 10 or read process memory on windows.
And then it just analyzes the memory over and over.
That's crazy, right?
But it knows enough about Python to go, well, that means X.
And it just, you know, off it goes.
So there's a bunch of more details on how he actually makes it work.
I'll link to that section as well.
It's a pretty cool profiler.
And I really like the attach to running processes without affecting them.
That's pretty unique, I think.
And so I wanted to highlight it.
Yeah.
Nice.
And it can do icicle graphs, which I don't know why that would be neat, but it looks neat.
Yeah.
Those are cool.
I get sometimes, you know, just visually some things are really out of whack.
You're like, what is that big bar from?
Oh, that's a radot sort.
Why are we calling that a thousand times?
Yeah.
Things like that.
Let's just sort it once.
All right.
What do you got for us next?
I've got SymPy, which is just sort of fun.
SymPy is a, it's a, well, I'm just going to read it, read the little bit here too.
Symbolic computation.
So like you're in math class or something.
We realized early on with programming that you can, if you punch things into the calculator
too fast, it just mucks things up because you have rounding and various things like that.
So symbolic computation deals with the computation of mathematical objects symbolically.
This means that mathematical objects are represented exactly, not approximately.
And math expressions are with unevaluated variables are left in symbolic form.
And SymPy allows you to do that with Python.
And it's sort of blasted cool.
I've got a little example of doing an integration of, of the sine of X squared over negative infinity
to positive infinity.
And it will tell you what the answer is.
And these sorts of symbolic math manipulations for a lot of people, boy, if I had to do this
by hand, I'd be in trouble.
I did not do that well in math.
And so being able to do this programmatically is cool.
And the introduction and the website is pretty awesome too.
It has a bunch of live, it's got engine in the back that runs it.
So you can try the examples out and pop up a little window and, and do it interactively.
So this is neat.
Yeah.
There's a ton of cool stuff that comes out of this.
So for example, you can say X comma Y equals symbols X and Y.
And then after that, you can do algebraic expressions, like truly algebraically.
So like expression equals X plus two Y, not in quotes or anything, just like as if it were
regular math.
And then you could like add one to that expression and it'll reform the equation and stuff like
that.
You can ask it to do integration.
Like the example you have in our show notes is to integrate sine of X squared from a minus
infinity to positive infinity.
Instead of giving you the answer of, oh my gosh, what is that?
Like 1.5 dot, dot, dot, dot.
You know, it just says that's square root of two pi over two.
Like the exact answer.
That is pretty awesome.
You know, we just wrecked the whole math experience for so many of our listeners who are students.
They're like, you know what?
That calculus class, I just solved that problem.
Well, I would have loved this while I was taking calculus.
Yeah, for sure.
Yeah.
You could totally check your work.
Like there's no answers in the book.
Oh yeah, really?
Hold on.
That's pretty awesome.
So if you can take your laptop to your tests, you're set.
Yeah.
Probably not likely.
All right.
So this next one that I found, Brian, it's pretty cool.
So something that I'm going to, I've been digging into lately behind the scenes, and I'm going
to be talking more and more about probably in the next few weeks is async programming in
Python.
Like I've really been doing a lot with that lately and we'll have some cool stuff to share
pretty soon.
But that means I'm running across all this cool async stuff.
So you've heard of WSGI, WSGI, which is the web service gateway interface.
That's like how Pyramid, Flask, Django, all those things work.
None of them do a great job of supporting async programming because fundamentally this WSGI
interface is synchronous.
It can't be made async.
Yeah.
So there's this other framework called ASGI for async gateway interface, I guess, that
allows these frameworks to be asynchronous.
So the thing I'm talking about this week is Starlet, which is an ASGI web framework.
And I like its little subtitle, the little ASGI framework that shines.
It's cute.
It is cute.
So it's basically intended to build high performance asyncio services.
So if you have anything that talks to a database, to caches, to file systems, things like that,
even calls other web services or microservices, super easy to build.
The API is basically Flask, like a Flask-ish API.
And you create like a web method.
You say async def regular view method.
And you go do a bunch of stuff.
And it has cool support for like response types.
So you can have a file response object that you just send back to the framework that's
based on async AIO files, which is an asyncio file based thing.
And there's a lot of nice integration like that.
Okay.
You're just interested in this or do you have an application that you're going to try to?
No, I'm building a course on it.
Oh, okay.
Trying to make a nice, well-rounded async concurrent programming Python course.
Oh, well, yeah.
So I've been building tons of little apps and stuff.
So here we go.
Here's one of them.
Cool.
If you want to build an app that is way more scalable, you know, 10 times more scalable
than regular web apps on the same hardware and whatnot, it's pretty easy to do if mostly
what that web app is doing is waiting, right?
You can just, you know, basically the asyncio web frameworks can just adapt to that more
easily because they're not blocking while they're waiting.
Also discovered a couple of cool things while looking into this.
One is they say you should install the ultra JSON package, which the pip install command
was ujson.
And that is a replace, basically a drop-in replacement for the JSON built-in that is like
between 50% and three times faster.
So if you're doing a lot of JSON, you can just use ultra JSON.
And that's pretty awesome.
Yeah.
Okay.
I have to check that out too.
Yeah.
So all you have to do is, you know, import a ujson as JSON.
And then that like makes your code faster.
Of course it has to be there, right?
But that's pretty sweet.
The other thing is you've maybe heard of G Unicorn for the traditional web frameworks.
There's a uveacorn, which is based on uvloop and gunicorn, which is also pretty awesome for
these async web frameworks.
Well, it's cool.
And I get the name, but eventually if we, if everybody starts using that, people forget
where that came from.
And it's just going to be a weird word, uveacorn.
I know.
Uveacorn.
No, it's uveacorn.
You got to understand where the name comes from.
Come on.
Get it together.
Well, that's it for our items.
I do have some extra stuff to share.
How about you?
Just one thing I wanted to point out if I remember it.
Okay, cool.
I'll go first.
You can think.
So really big news.
You and I, we had a good time at PyCon, right?
Oh, yeah.
I can tell you when we're going to have a good time again.
It's going to be, if we can go to a tutorial, it's May 1st and 2nd.
If you want to go to talks, it's May 3rd, 4th, and 5th.
And if you want to do the sprints, it's May 6th, 7th, 8th, and 9th.
So basically, the announcement is that the PyCon dates are out.
Yes, and it's not over.
I don't think it's over Mother's Day this year.
I hope it's not.
I also have a quick little follow-up.
You talked about the pre-commit package.
Also, another listener, Matthew Lehman, sent in some notes about how his team is using it
and basically talked about how they're using pre-commit, the Python package,
so that their Flake 8 and Black and other things that automatically run during continuous integration
also automatically run when people do Git commits.
So they have fewer failing builds, which is pretty awesome and has a couple of nice links.
So I threw that in there at the end.
And then, finally, you talked about the Gang of Four Patterns last week, right?
Yeah.
So John Tosher, I think he's right, sent us some messages pointing out another talk from PyCon AU
called You Don't Need That, which is pretty cool.
And it basically talks about how if you study the Gang of Four Patterns,
a lot of what they were doing was because they were using Smalltalk or Java or C++.
And in Python, here's a new way that you just basically don't need that pattern.
So pretty cool talk.
And that was a link to the video for that.
Yeah.
If you translated the Gang of Four book directly to Python, it would be like a pamphlet.
That's right.
Nice.
Do you remember your item?
I did not.
So...
Oh.
Save it for next week?
Yeah.
Save it for next week.
Yeah.
We'll do this again next week, right?
Yeah.
Maybe we should just do it every week.
Yeah.
All right.
Deal.
We'll do it every week.
Okay.
Cool.
All right.
Well, thanks for doing the show this week.
Thank you.
You bet.
Bye.
Thank you for listening to Python Bytes.
Follow the show on Twitter via at Python Bytes.
That's Python Bytes as in B-Y-T-E-S.
And get the full show notes at Pythonbytes.fm.
If you have a news item you want featured, just visit Pythonbytes.fm and send it our way.
We're always on the lookout for sharing something cool.
On behalf of myself and Brian Okken, this is Michael Kennedy.
Thank you for listening and sharing this podcast with your friends and colleagues.