Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.
This is episode 211, recorded December 2nd, 2020.
I'm Michael Kennedy.
And I'm Brian Okken.
And we have a special guest.
Yeah.
Matthew Feigert, welcome.
Yeah, thanks so much for having me on.
Yeah, it's great to have you here.
You've been over on Talk Python before, right?
Yeah.
Talking about some cool high energy physics and all that kind of stuff.
Yeah, I looked that up last night just to try and remind myself.
That was episode 144.
I was on with my colleagues, Michaela Paganini and Michael Kagan to talk with you about machine learning applications at the LHC.
Yeah.
And you do stuff over with CERN at the Large Hadron Collider and things like that?
Yeah.
So I'm a postdoctoral researcher at the University of Illinois at Urbana-Champaign.
And so there I split my time between working on the Atlas experiment and working as a software researcher at the Institute for Research and Innovation and Software for High Energy Physics.
And so like on Atlas, Atlas is this like huge five story tall particle detector that lives 100 meters underground at CERN's Large Hadron Collider.
That's just outside beautiful Geneva, Switzerland.
And so there I work with a few thousand of my closest colleagues and friends to try and look for evidence of new physics and make precision measurements of physics we do know about.
And then my IRISHEP work is kind of focused on working in an interdisciplinary and inter-experimental team to try and improve the necessary cyber infrastructure and software for us to be able to use in upcoming runs of the Large Hadron Collider.
And in what we call like a high luminosity run, which is going to be way more collisions than normal.
Have you guys ever turned it up to full power?
Have you turned it up to full power yet?
So, yeah, the design luminosity or the design energy of the LHC is at something called 14 tera electron volts, 14 TV.
And we've been running intentionally at a lower operating energy for the last couple of years, just a little bit below that.
But in the late 2020s, we're going to suck the entire Earth into it.
You know, no, no experimental evidence of black hole creation.
Yeah.
But kind of the cool thing is that if we even did make a black hole, the LHC due to something called Hawking radiation, it would evaporate well before it could actually ever do anything interesting gravitationally.
But yeah, it's really exciting.
Really.
I'm joking, but it's such a cool place.
Such cool technology.
I mean, that's right out of the edge of physics these days.
And the technology side is neat, too.
Yeah.
No, it's super fun.
Well, welcome over to Python Bytes.
Yeah, it's great to be here.
So I yeah, it's good to have you.
Thanks for coming.
And Brian, I think let's let's start with another one of my favorite topics.
Farms.
I love farming.
You know, you see the bumper sticker.
No farms, no food.
I like food a lot.
So I love farms.
No, no.
But the farm stack.
We've heard the lamp stack, other stacks.
Like lamp is not as useful as farm.
Right.
Farm sounds more useful.
So tell us about farm.
So Aaron Bassett, he's I'm not sure.
I think he's one of the spokespeople for Mongo or something like advocate or something like
that.
Anyway, he's he's doing he he wrote this article, but they've also done.
I think that there's been some talks given.
But this is this is a nice article.
It's a called introducing farm stack, which is FastAPI react and MongoDB.
So the I really actually appreciated the article and the code with it, because there's a GitHub,
a little GitHub to do credit app that they've put together.
And it the article describes basically all of the pieces of the application using it like
a little to do app.
But with FastAPI, you've got this is interactive, interactive documentation mode where you can
interact with the application just almost immediately.
You don't have to really do much to put it all together.
And and then for all your endpoints, you can actually interact with them, send send data,
do queries.
And there's a little animated gif to show how that's done.
But the article then, you know, goes through and says, you know, basically how how the endpoints
and routes get hooked up and then uses you, you, you, a corn to set up an async event loop and
get that going shows how easy it is to connect to a database and then defining models with and
how easy it is to set up a schema goes through.
And then it kind of hooks up, talks through the code discussion of you do have to write
code for the endpoints and and really how easy those are with all of these these pieces.
The React application is it's kind of a minimal app React application.
I'm not sure why they kind of included that, but it's kind of a neat addition.
There's a React application that's running that just sort of shows some of the interaction with
the the CRUD app and it gets updated while, you know, while you're changing things through
the interactive API.
And I just I like the demonstration of working through working with an API and working through
changing things and seeing it show up having a like a React app at the other end.
It's kind of a fun way to kind of experiment with an API.
This is a really neat thing.
And one of the other major stacks that's been used around Mongo is the mean stack.
And the farm stack is way nicer than the mean stack, not just because it uses Python and not
JavaScript.
But there's some interesting things here.
One of the examples is actually kind of blowing my mind in that it's an if statement using the
walrus operator awaiting an asynchronous call in an API method.
like the walrus operator and async, the await keyword.
I've never seen those together.
And it's kind of like it's inspiring.
It's nice.
It's good.
Yeah.
Such succinct code as well.
It's super nice.
I mean, it uses FastAPI, which is fantastic.
It's using motor, which is MongoDB's officially supported Python async library because you need an
async capable library in order to do things against MongoDB.
You know, this actually comes from the developer blog at MongoDB.
There also are some ORM like things, some ODMs, object document mapper stuff that also supports
async and await from MongoDB.
So if you're more in the ORM style, you might check that out.
But other than that, this looks pretty neat to me.
Yeah.
And I do know that a lot of people use the ORMs, but I like I appreciated the example without
an ORM for people because you throw an ORM example in there and then people that don't
use that particular one get lost.
Yeah.
Matthew, do you guys do anything with MongoDB?
Any of these kind of things?
FastAPI?
Yeah.
I have some friends that do.
I personally, myself, I'm not too versed in Mongo, but I've heard it on the show and
many, many times elsewhere.
So this is, I think, also just kind of paging through the article as Ryan was talking about.
It is pretty impressive.
So it's really concise.
Like, here's your four lines to completely implement the API.
Yeah.
Type of things, right?
Asynchronous, fast, like all the cool stuff.
Yeah.
There was an example, a case study of MongoDB being used at the Large Hadron Collider, but
that was many years ago and I don't know if it still is.
So I've completely forgotten where that is.
Yeah.
I don't know.
Yeah.
Cool.
So next thing I want to talk about another programming language.
Last time, Brian, I went on and on, maybe the time before, two times ago, about .NET and C-Sharp
because Anthony Shaw had done that work on Pigeon to get Python to run on .NET.
And we're like, well, why are we talking about C-Sharp on this project, right?
On this podcast.
Well, I want to talk about something even more advanced, AppleScript.
Wow.
Cutting edge.
Yes.
It's like the CMD shell script of Apple.
Have you ever programmed an AppleScript?
It's painful.
No, I have not.
It's like you say, like, tell this application that to like make a command.
Oh, it's bad news bears.
Let me tell you.
So what I've come across is this thing called PyAppleScript.
Now, this is not brand new, but it's brand new to me.
And there's a lot of talk about Macs and people may be getting new Macs.
So I thought I would say, hey, look, here's a cool way to automate your Mac or, you know,
Macs within your company or whatever.
With Python instead of this dreaded NS AppleScript.
Okay.
All right.
So basically it's a Python wrapper around NS AppleScript, allowing Python scripts or applications
to communicate with AppleScript and AppleScriptable applications.
So apps for which they basically implement AppleScript and let you do that.
So scripts get compiled either from source or they can be loaded from disk.
They have these, some of these ideas are from AppleScript as a standard run handler and user
defined handlers can be invoked with or without arguments.
They're automatically converted.
The responses to and from AppleScript are automatically converted either from AppleScript to Python
types like Python string versus AppleScript one or vice versa.
Right.
So you don't have to do the type coercion, which is cool.
And they're persistent.
So you can call your handle multiple times and it retains a state like AppleScript would.
And it also has no dependency on the legacy AppleScript library or the so-called flawed scripting
bridge framework, which is limited to OSA script executables.
So that's pretty cool.
If you want to automate things on your Mac, you obviously could use Bash.
But if you're talking to some kind of application that implements one of these scripts, like for
example, you want to tell this other application to grab something out of the clipboard and then
tell it to do something or something like that.
Right.
Like you couldn't reasonably do that with Bash.
Right.
Once it starts up, you kind of want to go back and forth with it.
So it sounds like AppleScript might be the thing to do.
Pretty cool.
Yeah.
I mean, not a lot to it.
Like if you've got a script, your Apple macOS stuff, do it with Python.
You don't have to do it with that AppleScript stuff.
No, it's neat.
Yeah.
So Matthew, you probably brought something to do with physics, data science, I'm guessing.
What's your first one here?
Yeah, a bit.
So we currently live in this like really nice age of having awesome CI services and all these
really nice metrics for all your GitHub projects and everything.
So, you know, if you're, I'm thinking of like coverage.
So if you're, you know, using pytest and, you know, making sure that you're reporting your
coverage, you have all these really great services to also track your coverage and report that
in a nice shiny badge.
But let's say you're developing some tool or some library and you have some sort of performance
metric that you care about.
Let's say like how fast some, the speed of evaluation for certain expensive functions.
And you actually want to try and like track that through the entire history of your code
base.
And that's not something that's like traditionally very super easy to do.
So recently I was really happy to find something.
So like if you're making changes.
So if you're going to be adding some feature or whatever, you are refactoring it.
So it's easier to write, but you're not sure if that makes it faster or slower.
And this would sort of give you that information from week to week or something like that.
Exactly.
Yeah.
So you might like, you might go ahead and say like, okay, well, you know, I have like
some, some tests that make sure that this function evaluates and under some period of time, if
it's an expensive function for your test.
But let's say you actually want to like track across like different, different parameter,
parameterizations, how that function actually is about is being performing and evaluating it
in your whole code base.
So I've recently found this super cool tool written in Python called airspeed velocity.
And so from the docs, ASV airspeed velocity is a tool for benchmarking Python packages over
their lifetime.
So it deals with runtime memory consumption and even custom compute values.
And the results are then displayed in a super nice web front end that's interactive and basically
just requires like static webpage hosting.
So it's, it's pretty impressive.
And just if you click on the docs, you can see that's developed by a community of people,
but led by Michael Dorritboom.
I'm probably getting your, your name wrong.
Very sorry.
And Pauly Burton.
But if you look at some...
He's the guy that who was behind a pie oxidizer at Mozilla.
Oh, really?
Oh, okay.
Yeah.
That's a super cool project.
Yeah, for sure.
Yeah.
Um, yeah.
And so, I mean, if you look at the other people that are on the contributor list, you can spot
a lot of names that are common in the SciPy and Jupyter ecosystem.
So it's, you already know that this is a nice community built tool.
And then also, as kind of some example cases, they give, current projects that
are using it like NumPy and SciPy and AstroPy.
So pretty well established projects.
And just as kind of like an example, if you click on like the SciPy project and go
to the interpolate function there, you can, you can just kind of look at a very nice
visualization of the actual evaluation, in time on the vertical axis across a whole
bunch of parameterizations, such as like CPython version and number of samples that are being
run.
And you can see this for the entire lifetime of the code base and you can zoom in on any
section just with the mouse.
And something I think that is super, super cool is if you, if you're looking at the visualization
of the plot and you see that, oh, there's like one commit where all of a sudden things go
funky and the evaluation time just jumps up.
You can just click on that node and it immediately opens up to that commit in GitHub, which is,
I think, super awesome that you don't have to go and like search through your commit history
to figure out what, like where that corresponds to.
It's just boom, right?
I'm looking at, it shows the, the Shaw from GitHub.
Yeah.
The, the, the, the unique identifier of the commit.
That's crazy.
Yeah.
So, wow.
Yeah.
So I've, I've, you know, a project that I'm working on, we've been interested in trying
to have the sort of like metric tracking for some of our, for some of our work.
So this is something that I'm actively kind of, looking at how we might be able to deploy
this for one of my projects with my coauthors.
Uh, but it's openly developed on GitHub.
It's up on pipe, pipe behind as well.
So just pip install ASV.
Uh, and then I think something that's kind of very cute and very kind of Pythonic is that,
uh, if you, when you go to the reporting dashboard for the different libraries that you're actually
benchmarking, it will, up at the top, say the airspeed velocity of an unladen X.
So the airspeed velocity of an unladen like numpy or an unladen scipy.
So, you know, keeping very true to the, you know, Python's roots there.
There's some Monty Python, the, the show Zen in there for sure.
Exactly.
Yeah.
This is impressive.
I mean, Brian, how do you see this fitting into like testing and stuff?
I actually love this.
I, I could use this right away.
There's, there's lots of, well, a lot of times it's, it's not, yeah.
Performance of performance is always something we care about.
And, and benchmarking systems, and, you know, testing, it's always, it's something
you forget about sometimes, like running, running stuff and it still works, but like
over time things slow down and it's good to, good to know that.
Yeah.
And if this could just be automatic and just part of your CI, you just go back and see the
updates.
That'd be very cool.
Definitely.
Yeah.
I don't think that this is something that at the moment, and I'm happy to be corrected about
this.
I don't think at the moment there is some way that this is currently being, given
as like a CI service, but I think that this is something that you could like set up and
run for yourself pretty easily.
Yeah.
You could probably plug it in.
Yeah.
Yeah, exactly.
But you could probably do some kind of web hook when, when there's a check-in automatically
kick it off and then save a result.
Right.
You could just hook into the GitHub actions and then have it just call you back and start
your, you know, let's take a, take a record of this or whatever.
Yeah.
Very cool.
This is a great idea.
Yeah.
Something else that I'm, I haven't really investigated yet, but that I'm looking into is if this can
also be used to do like GPU benchmarking.
So like, let's say you have a library that, you know, also that is going to be, you
can transparently, use the APIs to transparently move from CPU to GPU.
Like you have something like Jack's or TensorFlow or PyTorch, then this might be kind of a nice
way.
Uh, if it's, if it's based on those to be able to like benchmark your GPU performance
as well.
Yeah.
Well, and that's one of the things you might not test, right?
If it could run either way, you might just run it on your machine, whichever one of those
it is and forget to try the other style.
Right.
Exactly.
Yeah.
And I don't think there's too many CI services that are going to, you know, generously give
you some like really nice GPUs to be doing benchmarking on.
Yeah, that's for sure.
For sure.
All right.
Now for the next item, let me tell you about our sponsor.
This episode is brought to you by Tech Meme, the Tech Meme Ride Home Podcast.
They've been for two years recording episodes every single day.
And so they're Silicon Valley's favorite tech news podcast, and you can get them daily 15
to 20 minutes, exactly by 5 PM Eastern, all the tech news you want.
Right.
But it's not just headlines, much like by them bites.
Actually, it's a very similar show, but for the broader tech industry, you could have a robot
read the headlines or just flip through them, but it has the context and the analysis all
around it.
So it's like tech news as a service, if you will.
So the folks over at Tech Meme, they're online all day reading to catch you up and just search
your podcast app for the Ride Home and subscribe to the Tech Meme Ride Home Podcast or just visit
pythonbytes.fm/ride to subscribe.
I have a theory, a hypothesis about this.
I think that probably actually be a ton of work to put together a show daily on a time like
that.
But it's great that they're doing it.
Do you have any other hypotheses, Brian?
Yes.
My hypothesis is that there's not enough examples out in the world of how people are using a
hypothesis in the field in real world applications.
So I'm excited that Parsec put it together.
So Parsec...
Well, let's take a real quick step back just for people who don't know.
What is Hypothesis?
Oh, okay.
Right.
Hypothesis is a testing framework.
Well, it's not really...
It attaches to other testing frameworks.
So you can use it with unit test or py test.
You probably should use it with py test.
But it's a way, instead of writing a declarative single test or test case, you can...
It's a property-based testing.
So you describe kind of...
It's not like...
I expect one plus two equals three.
I expect if I add two integers and they're both positive that the result is going to be
greater than both of them.
You have these properties that you describe what the answer is.
There's a...
The examples that Hypothesis and other tutorials on how to use Hypothesis have given are more
of these like A plus B sort of things.
They're simplistic things.
And I do see a lot of value in Hypothesis and I know a lot of people are using it.
But there haven't been a lot of good descriptions for really how it's being used.
Like a real world example of how it's being used.
Because I'm probably not going to...
I don't have those little tiny algorithm things.
I've got big chunks of stuff.
And Hypothesis does have to run the test many times.
So how do you do this effectively on a large project?
So I love seeing this article.
So Parsec is a...
It's a client-side encrypted file sharing service.
I'd never heard of them before.
This blog.
But it sounds cool.
It's cool.
They describe themselves as the zero trust file sharing service like Dropbox where it's
end-to-end encryption for Dropbox.
You could share the files, but it only matters if you actually have the key, right?
Right.
Actually, I have no idea.
Sure.
I suspect so.
Yeah.
It sounds like a cool service, actually.
It sounds pretty neat.
So they describe kind of what they're doing and some of the problems.
It's a large four-year-old asynchronous Python project.
And then they describe this RAID redundancy algorithm that they need.
It's fairly complex with a bunch of servers and stuff, a bunch of data stores going on.
And what they need to test is they need to check things like if the blocks can be split into chunks and if the blocks can be rebuilt from the chunks that were split up before.
And then if you can rebuild them if you've got missing chunks.
And so this all sounds fairly, you know, yeah, I can understand how you could try to test that.
But there's a lot of variables in there.
How big is the chunk size?
How many chunks?
How much stuff should be missing?
And all that sort of stuff.
And then they're thinking, yeah, hypothesis would be good for that.
The normal tutorials talk about a stateless way to test with hypothesis.
But they're saying that for them, the stateful method that is supported is very useful because they're an asynchronous system.
And they describe how to do that.
It's actually a fairly complex description.
And it's kind of a lot to get through.
But it's neat that the power's there.
So it does, you know, walks through how they exactly how they set up a test like this.
And this is something I think the testing community of considering hypothesis has been missing.
So this is great.
They end with some recommendations, which it's great.
So the recommendation is for parts of your system that which parts should you throw hypothesis at?
That's a really good question because you don't want to throw it at everything.
Right.
Because there is some expense to set it up and also to run everything.
So they describe it as if the piece you're testing is kind of an encoder decoder thing, like theirs is you're splitting things into chunks and then rebuilding things.
It's a hypothesis is a no brainer for that because you can compare.
Is my input the same as the encoded then decoded output?
Yeah.
The other case is if you have a simple oracle, like it's simple to test the answer, but it's complex to come up with the answer.
I'm not sure what that is.
But in the case, you know, some of the cases are, you know, I've got a complex system.
And I just there's properties about the output that are easy to describe.
The other one is, yeah, I guess similar is if it's hard to compute, but easy to check.
Well, one example that just jumps out at me right away is anytime you have a file format, I'm going to save this thing, be able to save and load these files.
Right.
Because all you got to do is load up a whole bunch of random data, say save, load.
Is it the same?
If it's not, that's a problem.
Yeah.
And actually, I have talked with some people that that have thrown this at some of the standard library modules just on the side to test.
Because there's a lot of standard library stuff that's like kind of encoding, decoding sort of thing or two way conversions.
Yeah.
Cool.
Yeah.
This is super nice.
I'm going to have to really dig into this article in more detail.
I remember the first time I like learned about hypothesis was when one of the core devs gave a talk at PyCypys 2019.
And it just blew my mind then.
And so this is so cool to see this like very, very interesting application here.
Yeah.
It seems like there's a lot of uses in data science.
Data science seems tough to test like that scientific computation side because slight variations, you might not get perfect equality.
Exactly.
Close enough.
Right.
It's like, well, it's off, but it's like, you know, 10 to the negative 10th or something off.
Right.
That doesn't actually matter, but the equality fails.
Yeah.
You end up using NumPy's, you know, NumPy's approximation comparison schemes quite a bit in your.
Yeah.
And your pie test.
I can imagine.
Very cool.
All right.
Next one, Brian.
I told you about last time I talked about I'm still waiting on my Mac mini, right?
Yes.
I ordered the Apple, the M1 Mac mini maxed out.
And I'm a little bit jealous.
My daughter is getting a new Mac mini.
She doesn't, or Mac Air.
She doesn't know what about, but it's supposed to show up tomorrow and mine's still weeks away.
And I don't think that that's very fair.
But if you are an organization that depends on cloud computing and, you know, what organizations
don't these days, right?
They almost all do.
It was just announced at reInvent that AWS is going to be offering Mac instances as a type
of VM.
So until now you've been able to get Windows, Linux, that's it.
So for all those people out there who are offering some kind of iOS app, even if they're not like
a Mac shop, they still have to have Macs around because you can't compile and sign your IPA,
your Mac, whatever iPhone app format is.
You can't create those without a Mac.
So there's all these Macs that are around for like continuous, you know, CICD or checking those
things and whatnot.
So now you can go to AWS and say, I'll take a Mac mini, please.
That's pretty cool.
That's cool.
Yeah.
So you can do your tests up there and they don't have M1 yet.
Those are the Intel ones, but the M1 chips are coming later.
So you'll be able to do it.
What's interesting about this offering from AWS as is basically any cloud service, you would
imagine it's a VM, right?
But these, when you say I want one of these, you actually get a dedicated Mac mini.
That's you get pure hardware.
Well, that's why you can't get yours because Amazon bought them.
They did.
They had a huge truck full of them.
Well, they bought the Intel one.
So those were on sale, I bet anyway.
But no, they have some interesting, what do they call it?
Nitro.
I think they call it their Nitro service or something like that, which allows them to virtualize
actual real hardware.
So this is pretty neat.
You can sign up.
The billing is interesting.
You have to pay for at least one day's worth if you get it, which I think is like $24.
If you're going to run it continuously all the time, this is one pricey sucker.
Like the Mac mini you can get now is $700.
This is $770 a month.
Oh, okay.
So if what you need is like a couple of Mac minis, you're probably, and you need them
on all the time.
You're probably better off just buying a few and sticking them in a closet, especially the
M1s.
But if you just need one on demand every now and then, or you need to burst into them or
something like that, that could be interesting.
Yeah.
If you're back old school and you only release like once every three months.
Well, there was some conversations like, well, if your data is already stored in S3 and you
have like huge quantity of data and what you need to run is actually running like some video
processing on the Mac, you could do it by the data instead of transferring that kind of stuff.
Things like that might be interesting.
I don't know.
I would go ahead and throw out there also that this is all interesting.
I have links to this kind of stuff and whatnot, like the blog post announcing it and so on.
But there's also this thing called Mac Stadium.
And if you look at Mac Stadium, it's pretty interesting.
You go over there and say, give me a dedicated bare mini, a bare metal Mac mini in their data
center, $60 a month.
So you can actually get like a decent one for a decent price over there.
So if you just want one running all the time, it might be good.
But the thing is, if you're already like deeply integrated to AWS, maybe this is a good thing.
Yeah.
Is there anything you, yeah, go ahead.
I was just going to say, this seems pretty interesting.
I mean, I know one of the reasons that I love using GitHub Actions and Azure Pipelines
is the ability to be able to get access to Mac VMs for builds.
But if you, I could also see this being really interesting and useful if you have like some
very huge application or some like very large stack that you want to be able to be able
to do CI or tests on, that this could be really, really nice.
Especially if you don't just want to be like, you know, pounding and destroying like one Mac
over and over and over again.
Yeah.
This is nice.
Especially if you have a distributed team.
Yeah.
Which every team is basically a distributed team.
Yeah.
Welcome to 2020.
One thing that's interesting about this is you can literally press a button or even just
through the AWS, probably the Bodo API.
You can just make a new Mac instantly.
Like within seconds, you can have a clean, pre-configured Mac.
You can create AMIs, the Amazon machine image, which are like, install a bunch of stuff and
get it set up and then like save it so I can respawn new machines from it.
Those are pretty interesting options that just having a Mac mini in the closet.
You know, push a button, make a brand new one, try this, throw it away, make it a different
way, throw it away.
Like there are some use cases here that could be interesting.
That said, I won't be using it.
I'm just going to buy a Mac mini if I can ever get it.
All right, Matthew, what's this last one you got for us?
Yeah, I don't have any clever transition, but all right.
So maybe, I don't know about you, but I end up having to deal with a lot of JSON serializations
of different statistical models and different, and, you know, sometimes also getting, you know,
CSVs of different data sets that I want to be doing analysis on.
And, you know, your first instinct might just be to say, okay, I'm just going to open
this up in Pandas and start to get to work on it.
But if you kind of are used and comfortable to working in the Linux command line kind of
ecosystem of data tools, you might be itching a little bit and want to kind of just, you
know, peek inside at the command line level and kind of get to work there.
And so in that case, you might be really interested in this tool called VisiData.
So VisiData is written on this.
This is blowing my mind, actually.
Like, when I saw this, my jaw was kind of on the floor.
So we'll make sure that this is linked in the show notes because it has some really cool
videos.
But so from the docs, so it's VisiData is described as data science without the drudgery.
So it's an interactive multi-tool for tabular data.
It combines clarity of spreadsheets with efficiencies of being at the terminal and also, you know,
the power of Python 3 on a really lightweight utility that can handle millions of rows with
ease.
I can attest to that personally.
I've opened up to like four gigabyte CSV files before and it just, you know, drops
right in and starts asynchronously loading like a champ.
It's in addition to that, it supports kind of a really astounding number of file formats
that it supports.
Currently on the website, it says it supports 42 different file formats.
So it, you know, supports things that you would expect like CSV and JSON, but then it also
supports things like JIRA.
I guess like whatever JIRA uses for their sort of like tabular stuff.
It also can like read my, my SQL.
And I guess it can also even deal with PNG, the image file format, which I was, you know,
impressed by.
So this is all openly developed.
The output is a terminal, right?
Yeah.
Not a thing like text.
Yeah.
So this is all openly developed on GitHub by a guy named Saul Pawson, I think.
And if you go to the, if you go to the Visadata website, it also has plenty of links to live
demos of him doing kind of interactive examples of visualizations.
There's one lightning talk that he's given at, I think, PyCascades 2018 or something like
that, where he's able to just call up a CSV file of, of like, 311 components.
And then through the, through using Visadata, just kind of hone down onto certain boroughs
and then be able to do a filter on, on different complaint types to be able to basically find
complaints about rodents and then filter on rat complaints and then plot that inside of Visadata
still on the terminal to basically make a visualization of like rodent distribution in the New York City
boroughs.
So I thought that was, you know, quite amusing and really cool.
It's also, you know, this is a Python application.
So you might not want to, you know, continuously install this in every single virtual environment you make.
So, I mean, it is up on PyPI, so you can just do pip install Visadata.
But since it's an application, you probably might also want it just kind of as a generic tool on your machine.
So it's distributed through a lot of, you know, nice common package managers.
So if you're on Linux, they've got it on apt, as well as things like NX and GUX.
But I didn't see it on Yum.
So if you're on Fedora or CentOS, you might be a little bit out of luck.
You might have to do it manually.
It's, of course, on Homebrew and even Condo Forge.
And it's not listed there, but a very, very cool tool that's been featured on the show before,
which is PipX by Chad Smith.
Yeah, PipX is awesome.
It's so good.
I love it.
I tested this last night.
I just fired up a Python 3.8 Docker container and, you know, went ahead and installed PipX
and then used PipX to install Vizidata and was able to drop right into Vizidata as expected.
So it's very, very cool.
And just the power that you can have with it, I think, is worth checking out for anybody who
is doing data analysis with tabular data.
This is super cool.
I love when people build these tools that are kind of, you don't really expect them to be
so powerful.
And you talked about how you just dropped in and grabbed some random data and started answering
questions.
And that's super neat.
Yeah.
The number of inputs.
And because it's open source and because of all the other examples of data types, I
think even if you have a different data type, it shouldn't be too hard to modify this to
handle something different.
I do notice I'm excited about it.
It does have PCAP files for packet capture.
These are for communication packets.
Talking to all your devices and all your hardware at your company, right?
Well, this is like even Wi-Fi packets and cellular packets.
That's how we debug those.
So nice.
It's very cool.
Yeah.
Very cool.
And PipX is great.
I install a bunch of apps like Glances, which is a fantastic, like visualize the state, you
know, like top, but way, way better.
The ACPy, which is great for it's a better, but much, much better curl.
But the most important thing I install that way is a Py joke.
So now I can type Py joke on my command line and we're always right there.
So speaking of which, move on to our extras.
That's all of our main topics.
Brian, you got anything this week?
Oh, I did.
I haven't dropped them in.
Where'd my extras go?
Yeah, well, you got it.
I just wanted to bring up that the PyCon 2021 is going to be virtual and there's a website
up.
It's us.pycon.org slash 2021.
And there's not a lot there yet, but you can check out what's going to happen.
It's not surprising that they have to start planning it and they may as well plan it as a virtual
event.
I'm just kind of hoping that we would have live, but I understand.
Yeah.
I mean, Hycon is my geek holiday.
I love it.
It's both work, but it's also just such a nice getaway to connect with everybody.
You, everyone else we know from the community, listeners, I'm going to miss not having it.
Yeah.
I'm glad.
Do you attend?
Sorry, Brian.
No, it's good that they, I always check whenever they announce the date to make sure it doesn't
overlap Mother's Day.
And it does not.
That's not good.
Yeah.
So I have unfortunately not attended PyCon yet in person or, I mean, it was canceled this
year.
So maybe, maybe I'll attend this year remote, but I'm a regular attendee of the SciPy conference,
which this, so this past year, SciPy 2020 was moved online.
And I thought that the organizers did a fantastic job of actually writing it online while still,
you know, keeping kind of that SciPy community feel.
So that was helped a lot also by, you know, plenty of bad puns.
So I think that might, might be something that still comes through for PyCon 2021, maybe.
Yeah, absolutely.
One of the live listeners, Mohammed said, ask if it's going to cost money or if it's going to
be free this year to attend.
Did you notice anything, Brian?
I haven't looked.
I'm looking around and I don't know that it costs anything.
It's from what I can tell.
I don't see any pricing.
What I saw was sponsor information to get sponsors to sign up to be part of whatever they're doing
there.
but I can't tell.
Yeah, not sure.
Somebody else throw in the chat or put it into the, you know, visit pythonbytes.fm/211
and put it in the comments down there.
All right.
I got a couple here.
First of all, we're trying out live streaming here and I think it's going pretty well.
Seems like it's working out.
There's a bunch of people watching.
So if you want to get notified and we happen to keep doing this, just visit pythonbytes.fm
slash YouTube and it should have like the scheduled upcoming live stream.
You can like get notified there.
So we'll, maybe we'll keep doing this.
It's been fun.
Thanks for everyone out there who's watching right now.
And in addition to PyCon, which you just announced or mentioned the announcement of, that is the
main way that the PSF is funded, but they're also doing a dedicated offering sort of fundraiser
thing with six companies to help raise some money for the PSF and Talk Python training is
being part of that.
And 50% of the revenue of a certain set of our courses that are sold during the month of
December goes directly to the PSF.
And people who buy those courses through the PSF fundraiser also get like 20% of a discount.
So there's a link in the show notes for people to take some of our courses and donate to the
PSF.
If you'd rather just directly donate, that's fine.
But if you're looking to get some of our courses anyway, you can do it this way and support the PSF.
They're hoping to raise $60,000.
You know, hopefully we can do that for them and we'll see.
Brian, you announced Big PyCon.
Another thing that got announced is Small PyCon, PyCascades.
PyCascades being the mountain range that connects Portland, Seattle, and Vancouver.
And traditionally, this conference is cycled between those three cities.
I don't even remember anymore what it's supposed to be this year.
I think it's supposed to go back to Vancouver, but it's not going to Vancouver because nobody's
going anywhere.
So PyCascades is online.
And those do cost money.
It's $10 for students, $20 for individuals, and $50 for professionals to support that conference.
But I'll link to that one since that's one of our local conferences, if you will.
Yeah, they're trying to push.
They often push what's going on, try new things.
So it's a neat conference.
Yeah.
I enjoy my time there as well.
All right, Matthew, what are you got for us?
Anything else you want to get a shout out to?
Yeah, just a few items.
So Advent of Code 2020 is started now.
It's day two, but there's still plenty of time to get involved with that if you want to.
And for those of you who might not know, Advent of Code is just an annual coding challenge that
takes place every December.
And it's just basically 25 days of fun and interesting programming challenges.
So it's always a great opportunity to try and brush up on your Python and maybe learn about
some interesting collections that you might not have known about in the standard library.
So that's going on right now.
Worth checking out, I think.
And then I'm going to sneak in some very small physics-related follow-up to Python Bytes
episode 205, in which awkward arrays were talked about.
So the lead developer of awkward arrays is my friend and colleague, Jim Povarsky, who is one
of my scikit-hep co-collaborators, as well as also a member of iris-hep.
And as of today, which is recording December 2nd, awkward v1.0 is a release can that is up on PyPI.
So by the time that this goes live, if you just do pip install awkward, you should get awkward 1.0 releases
instead of having to do the...
No more awkward 1.
Exactly.
No more awkward 1.0.
No more awkward 0.0.
All that jazz.
It's so good to have the actual install statement be awkward itself.
Exactly.
So that's a nice little tidbit.
And I think there's some nice links in episode 205 if people want to learn more about awkward.
But that's kind of a backbone of kind of the Pythonic ecosystem for physics right now.
And then finally, I just want to give some kudos to Python Bytes as well, specifically for making
full transcripts of the shows available to view on pythonbytes.fm.
Not only is this, I think, like a cool idea in general, but I think this also makes the show
more inclusive to the deaf Python community, which is definitely out there.
And one of my good friends and co-authors is deaf, and I know that he definitely appreciates
this.
So good job, Ben, you guys for being more inclusive of the wider community.
Oh, that's so cool.
I didn't know anybody was utilizing it.
Yeah, that's awesome.
Thank you.
I think it's absolutely critical for that because the format is only audio.
But a lot of folks have reached out and said they also appreciate it if they're English as
a second language and they're not as good with English as well.
So that also helps, I think.
They're like, what was I saying again?
What a weird word.
Awkward array?
Why would they talk about that?
It doesn't make sense.
Yeah, transcripts and closed captioning is just more inclusive for everyone.
So that's awesome.
Yeah, thanks.
All right.
Well, let's wrap it up with a joke, Brian.
Yeah.
All right.
So you guys, I'm going to need your help here.
I'm going to let Matthew, I'm going to let you pick.
Do you want to be Windows or Apple?
I'll be Windows.
All right, Brian, you'd be Apple.
So the idea is like the title here is how to fix a computer, any computer.
So instructions for Windows.
Go ahead, Matthew.
So step one, reboot.
And then the flowchart goes to, did that fix it?
If no, proceed to step two.
Step two, format your hard drive and then reinstall Windows.
Lose all of your files and quietly leap.
Brian, Apple doesn't have that problem.
There's some totally different solution there.
Okay.
For Apple, it's step one.
Take it to an Apple store.
Did that fix it?
If no, proceed to step two.
Step two is buy a new Mac.
Overdraw your account and quietly weep.
That's me right now.
All right.
I got the Linux fix.
It's so easy.
It's totally like you don't need those things.
So you learn to code in C++.
You recompile the kernel.
You build your own microprocessor out of spare silicon you have laying around.
You recompile the kernel again.
You switch distros.
You recompile the kernel again, but this time using a CPU powered by the reflected light from
Saturn.
You grow a giant beard.
You blame Sun Microsystems.
You turn your bedroom into a server closet and spend 10 years falling asleep to the sound
of wearing fans.
You switch distros again.
You abandon all hygiene.
You write a regular expression that would make any other programmers cry blood.
You learn to code in Java.
You recompile again, but this time while wearing your lucky socks.
Did that fix it?
No.
Proceed to step two.
Revert back to using Windows and Mac.
Or Mac.
Quietly weep.
There's really no good outcome here.
They all in and quietly weep.
As a Linux user for the better part of a decade, I can neither confirm nor deny how accurate
that last part is.
Yeah, they all have their own special angle.
It just takes longer to get there with Linux to get to your destination, I guess.
Yeah.
All right.
Well, that's fun as always.
And everyone watching on YouTube, thanks for being here live and everyone listening.
Just thank you for listening.
Matthew, thanks for joining us.
Hey, thanks so much for having me.
This was really fun.
Yeah, yeah.
Great for the items you brought.
Enjoyed them.
And Brian, thanks as always, man.
Thank you.
It's been fun.
Yep.
See ya.
Bye.
Thank you for listening to Python Bytes.
Follow the show on Twitter via at Python Bytes.
That's Python Bytes as in B-Y-T-E-S.
And get the full show notes at pythonbytes.fm.
If you have a news item you want featured, just visit pythonbytes.fm and send it our way.
We're always on the lookout for sharing something cool.
On behalf of myself and Brian Okken, this is Michael Kennedy.
Thank you for listening and sharing this podcast with your friends and colleagues.