Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.
This is episode 261, recorded December 2nd, 2021.
And I am Brian Okken.
I'm Michael Kennedy.
I'm Shell Genteman.
Welcome, Shell. Could you let us know a little bit about yourself?
Yeah, I'm a research oceanographer.
So I study the sea from space.
And I've been doing oceanographic research for NASA for a little over 20 years.
I do almost everything using satellite data.
So I never have to leave the comfort of my, used to be office, but now office at home.
That sounds so fascinating.
Is it fun?
Super fun.
Cool.
It's like math and physics and computers all mushed together.
It's like all my favorite things.
And oceans.
Yeah, it's fantastic.
And oceans.
Yeah, that sounds like such a cool job.
Welcome to the show.
Well, Michael, what do you got for us to start?
Well, let's talk about our clone.
So this one was sent in to us by Mark Pender.
Now, our clone itself, I believe it's written in Rust or something.
It's not Python.
So the story here is not, oh, here's a cool thing created with Python.
But it is a cool library that I think will be useful for Python developers.
Okay.
So this our clone thing syncs your files to cloud storage.
Let me basically see my summary.
So imagine you wanted to put some files in AWS S3, or you wanted to store something in Azure blob storage, or there's actually 40 different places where this can go.
So like Backblaze, Backup, Box, Citrix, ShareFile, Dropbox, Google Drive.
Let's see.
Some stuff with OpenStack, KeyCloud.
All these different places and formats, even just web dev and whatnot.
So if you want to either read or write files to that location, what you can do with our clone here is it will basically mount those different locations as just something on your hard drive.
So if you want to write to S3, you can just write to a file with Open slash S3 slash wherever it goes, and then write to it with Python or set up some kind of cron job that moves stuff.
So if you're trying to move like large data for data analysis up to the cloud, so then you can connect it to a notebook, or you're trying to move files that are the back end of your website or your API through S3 or somewhere,
then you can just copy files over sync different locations, like I said, mount it as a drive.
And it has a lot of cool support for things like if the file transfer gets interrupted, it'll fall back to the last one that was working and then continue uploading.
So it can be kind of interrupted and unstable and whatnot.
I think this is so cool.
It was just like, you know, when I first moved to the cloud, it was so frustrating having to figure out whether I was using S3 or the, you know, the Google commands or the Amazon commands.
And all I wanted to do, get my data to where I could use it.
I am so, so with you.
And sometimes it's like, well, how do I copy files here?
Well, here's our API.
Like, I don't want an API.
I want to go to the Finder or to the Windows Explorer and draggy dropy the file.
Can I do that?
They're like, no, no, no, you can't do that.
No way.
You can run our app maybe.
Yeah.
So this is, you know, in theory.
This is so cool.
Yeah.
I'm glad you like it.
That's awesome.
I think it'll allow people to move data around from, especially it seems relevant to scientists who need to put a bunch of data in the cloud and run it.
But then they might be want that data locally and keep it in sync and stuff.
And it's really frustrating when your expertise is in something else.
It's not in computer science.
And like everything I pick up is because I'm only forced to learn it.
And I don't want to learn the Amazon API and I don't want to learn the Google API.
This like gives me maybe one tool that I can just be cloud agnostic and move my data around in a way that I'm already comfortable with.
Yeah, I agree.
Yeah.
Here's the thing I was looking at.
Yeah.
So the virtual backends wrap local and cloud file systems and apply encryption, compression, chunking, hashing, and joining.
And it looks after your data.
It preserves the timestamps, stamps, verifies checksums all the time, transferred over limited bandwidth, intermittent connections.
It can be restarted.
It checks the integrity of your files, all those kind of things.
So, you know, like if you're out, I know you don't leave the house anymore, but if you're out doing research and like on a boat and you needed to like, you had this rickety connection, you know, maybe you could get stuff uploaded well this way.
So I think it's neat.
How do you like configure it?
You have to put in all your cloud stuff.
I suspect you, when you set it up, you have to give it like, let's see, it's for your Amazon.
That's Amazon drive.
I forgot that that existed.
Okay.
Let's see.
Yeah.
You've got to give it like your, your AWS keys and stuff, of course.
But yeah, they have a whole configuration section on what you give it here to set it up.
It looks like you create a config file for it, I think.
But yeah, pretty neat.
Brian.
Yeah.
Brian, what do you think?
Well, so I'm trying to figure out like, even for something, for a mental model, is this like a Dropbox without version control or is it a completely different space?
Well, I mean, it does have some tie-ins to there, right?
It's got like Backblaze and things like that, which is just a pure backup system.
I think it's just trying to match like, how do I move files around to the cloud?
And you can also, you can move it between the cloud, right?
You can mount two places and copy from one to the other.
Like I can copy from Citrix share file over to box, neither of which I really know how to do.
Yeah.
Oh, it even has Dropbox as one of the configs.
Yeah.
But different.
This is actually pretty cool.
I like it.
Yeah.
Very cool.
Let's see.
Kim out in the live audience says, I like this.
Very few people really need to know or care that S3 doesn't have real files and directories, for example.
And Sam says, it's funny.
My group was just talking about how to transfer a huge amount of training data to our compute resources earlier today.
I'm guessing that's machine learning training.
Very cool.
When you still have to go to Amazon or Google and set up the bucket, right?
So you're not spared that particular pain.
Just like try to click public until it's public, but not too public.
That's my approach.
You still have to do that.
But this seems like a really nice solution.
Yeah, for sure it does.
I guess over to you, Ryan.
Yeah.
So this has been suggested several times by several listeners.
So thank you, everyone that sent this in.
Oh, I'm on the wrong thing, aren't I?
I wanted to talk about check wheels.
So check wheels are check wheel contents.
So the idea around it is that there's...
So I'm often using Flit, and it kind of does all this for me.
But there's other backends that you can use for building wheels.
And if you configure something wrong, it might get the wrong stuff in there.
So by wrong stuff, you might have like a PyCache in there,
or you might deliver your tests with your wheels,
and that's just extra space.
You don't necessarily need that.
Maybe your documentation should be there,
but maybe it shouldn't be, depending on that.
I don't think that...
Actually, I went on a tangent with the documentation.
I don't think this checks for that.
But so there's a...
It's just a pip installable tool,
and then you can run check wheel contents.
And you can give it a wheel, but wheels are often long.
So I just...
When I've been trying it out, I've been just giving it my dist directory,
and it just looks all the wheels in there and checks things.
So what does it check for, though?
So it's checking for things like making sure
that you don't have any PYC or PYO files in there
because you shouldn't have those in your wheels.
Checks for duplicate files,
because maybe you've got, I don't know,
copies your directories or something.
And there's actually, I don't know,
like 10, 12, 13, 14, 15 checks or something like that.
I'm counting really quickly.
But there's...
What I really love about...
One of the things I like about this is
there's a lot of things that you...
Like if you configured it totally wrong
and your wheel's empty,
it'll check for things like that.
And yeah, you probably could test this and try it,
but it would be nice to actually have something in your pipeline
to automatically check for these things.
And it's really fast.
The other thing I like is the readme for this project
lists of...
Has a very good description of all the checks
and why something like that could go wrong.
So if, for instance,
you happen to have your tests in there,
but you don't want them in there,
how do you fix that?
Or it also says,
if you actually do want your tests in there,
how to go about putting it in there
so the check passes.
So interesting project.
Yeah, this looks really neat.
I think if you're going to be creating a package,
you definitely don't want to be releasing things
that are not intended to be in there.
I was looking through it.
I wonder if it's possible to say,
you know, check for certain files,
make sure that they don't get in there.
Like I'm thinking like a settings file
that has some sort of key,
like an AWS key,
like we were talking about or something.
But nice.
Can you...
So I don't make lots of packages.
So what's the wheel?
When you're using that term,
what does that mean?
It's the thing that you pip install.
It's the...
They used to be just tarballs.
It used to be tar, the JZs and whatever.
But what we do now for the most part is,
or hopefully,
is wheels are not...
If it's just pure Python,
it'll be the same for everything
and hopefully it will be.
But it can also specify
that it runs on Python 2 or 3
and that some of those sorts of things
can be built into the name
and what operating system.
Because if you're building on,
like say, just simplifying the world,
a couple versions of Unix
and Linux
and maybe Windows and Mac
and then also the new Mac
with the different architecture,
those will all be different wheels.
But when you...
So when you pip install it,
PyPI and pip will download
the correct wheel
for your operating system.
And that makes it so that
when you're installing something,
none of...
You don't have to compile anything.
It just brings it all down.
So it's a cool format.
Yeah, it's especially important
for the scientific community
because there's so many weird libraries
that have to get compiled
with things like Fortran,
as we were joking about.
And so wheels will basically
contain the pre-compiled version
so you don't have to have
like a Fortran compiler
on your machine
to pip install it or whatever.
It just downloads and unzips
really quickly
without all that steps.
I was told a simple mental model
of the difference of old and new
is the old style
with setup tools and stuff
would often have
a whole bunch of stuff
that you download
and then you run setup
to like build some things
and redo things.
Whereas a wheel is closer
to mostly just a zip file
that just unpacks things
and throws it in your site packages.
Nice.
And Sam also adds,
you can also package
extension modules in wheels,
which is their greatest strength.
Very cool.
Cool.
All right.
Brian, is that it
for the check wheel contents?
Yeah, I'm done there.
Right on.
All right, Shell, take it away.
All right.
So I thought we would
talk a little bit about
weather and climate data
and Python
and we're really trying
to get more Python programmers
involved in weather
and climate research.
And the data, I think,
it used to be really hard
to get weather and climate data.
It was in these really weird,
obscure formats
that only scientists
knew how to read
and they only wrote
Fortran routines to read them.
But now with Python,
it's becoming really,
really easy to get these data.
So the first thing is like,
where do you get the data?
So I'm just going to show
the open data at Amazon,
at AWS.
But really, you know,
Google has the equivalent
in the Earth Engine
and Google has all sorts
of open data sets.
And that means that
they're free egress.
So most of these you can get,
you know, you can access
data for free.
And Microsoft has
the planetary computer
and they're building up
the same thing.
And like you can see
lots of people
are putting data on here.
Like NASA has
a Space Act agreement.
There's the NOAA,
which is our weather agency,
the big data program.
And so like you can
look for data.
And one of the biggest
data sets that I work with
is ERA-5.
And if you just sort of
type in here
and it brings up
the data set
and you can click on that
and see they have it
in these two different formats.
So one is SAR
and one is NetCDF.
And most people
in sort of data science
work with, you know,
SQL databases
or maybe they're doing
CVS files
or tabular data.
So weather and climate data
is a little different
because it's three-dimensional.
And so there's these
different data formats
and really almost all
of the weather and climate data
now is currently
in this NetCDF format.
The goal is
let's just write
a Python library
and make it so you don't
care about the format, right?
The data formats,
the people who produce
the data should care about it.
But as a user,
what we want is
we want anybody
to be able to use it
and do anything
they can think of.
And so that's
this sort of X-Array.
So X-Array
is a Python library
that is designed
for sort of
three-dimensional
structured data.
And all the data
has labels
and it has these things
called data sets
so that it organizes
your data for you
and to read it
you just sort of say
open data set.
Nice.
And it understands
these formats?
Yeah.
And like I'm going to bring up
a little example here
but this ERA5,
I mean this is like
I think it's
35 terabytes of data.
So I took this
off of the AWS.
why did it take it off?
I ran it on AWS
and I subsampled it.
Because where are you
going to put it, right?
Yeah.
I mean it used to be
that like to get
this data set
you had to write a script
and then you would
download it for like
three months.
And now it's just
on AWS
which is like
mind-blowing, right?
Like I log on
and a few minutes later
I actually have access
to all this data
which is so cool.
So like with X-Array
I'm going to run this cell
and basically I just
import X-Array
as X-Array
to read the data
I just say like
open data set.
That's it.
And it figures it out.
And it'll read almost
it'll read a lot
of different formats
and it just has your data.
And so this is like
a really big data set
and it tells you
all about it
and you can look
at the different data
that it has.
and you know
sort of the goal
with this
is to make it
really really easy
for anybody.
Like let's say
you want to look
at you know
sales patterns
in San Francisco
or you want to work
at ship traffic
or you want to look
at how weather
is evolving
at your location.
Like you don't need
to know about
the data anymore.
Yeah.
Fantastic.
Just know how to work
with NumPy
like X-Array
stuff in your notebook
and that's all
you got to know.
Yeah.
It's all built
around pandas
and NumPy
and like
if you want to
like
let me find
a really
easy example
like what if
I want to plot
the data set.
You know
I just type
dot plot
right?
Oh wow.
And then it like
labels everything
and you understand
what you're looking at
and what day it is
and you can use
cell and ice cell
and just sort of
like pandas.
It almost looks
like an ocean
just right there.
It's
yeah.
Longitude and then
I guess
temperature right?
Yeah.
And so this is
like you just
typed plot
and it actually
tells you exactly
what you're doing
and what it's plotting
and what the color bar
so what do these
different colors mean?
And you know
you could do
a spatial plot
like this
where you do it
in time
or let's just
pick a particular
latitude and longitude
and the nice thing
is that you can
actually just tell it
your latitude
and longitude
and you can use
Google Map
to look up your
latitude and longitude
and then plot it
and it says
oh I'll make
a time series.
That's pretty cool.
Wow.
Yeah I remember
just struggling so much
getting into programming
and having to work
with custom file formats
out of like research projects
you're like
what do you mean
I have to read
this binary file?
This is going to be so hard.
Okay here we go.
Yeah and then like
you wanted to read
a different binary file
like start from scratch
write all that code again
and like X-ray
sort of took
all of the
back end work
that all the people
at the data archives
did with like
getting everything
in the same format
and labeling
all the data nicely
it sort of
took all that work
and just said
well we'll write
one library
that builds
on all of that
and can read anything.
Yeah awesome
great recommendation.
A couple of pieces
of real-time follow-up
Sam Morley
out in the live stream
says X-ray is great
I did an example
of using it
to open a net CDF file
in my book
and I'm learning
about his book
Applying Math
with Python
Practical Recipes
for Solving Computational
Math Problems
Using Python Programming
and its Libraries
that's awesome.
That looks like fun
actually.
Yeah it does.
Yeah and X-ray
linked to like SciPy
and it has all
a lot of statistics
and math built into it
so you can actually
compute trends
in one line
and all of that.
Yeah nice.
Cool.
I also have one
other piece of follow-up
here Brian
I don't want to
panic you all
but right here
in Portland
we have Panic
the software company
and I just want
to give a quick shout out
to this thing
called Transmit here.
This is what I actually
use to get stuff
up into
and out of S3
and it also
will let you talk
to Backblaze,
Box,
Dropbox,
Azure,
Google Drive
all these places
as well
and it's basically
like an old school
FTP program
where like on one half
it has your computer
and the other half
it has whatever
cloud storage
is that you're
working with there
and maybe you could
even put the other half
not just your computer
but somewhere else
as well.
So if you want
just like a UI
not something like
R clone
but just a UI
I'd strongly recommend
this thing.
They don't sponsor
the show or anything
but I definitely love it.
I use it all the time.
Neat.
All right.
Am I up next actually?
I guess I am.
Yeah.
I think so.
I am.
Number four
would be
I want to talk about
this announcement
from JetBrains
being one of the bigger
tool companies
tool builders
for the Python world.
They came up with this thing
called JetBrains
remote development
and buried at the end of this
is actually what I think
is the lead
got quite buried here
but we'll see.
So they introduced something
that I was not aware of
called remote development.
So the whole idea of this
is basically
what if
instead of running
like PyCharm
this works for any
of the IntelliJ stuff
but let's say PyCharm
instead of running PyCharm
locally on your machine
you could just
give it an SSH
destination
let's say
and it will go over there
and run PyCharm
the server
or the sort of
logic bits over there
but just have a light
front end
to your computer
here.
so like a lightweight
if you're on
some really wimpy laptop
and you wanted to
access
like a better server
at work
or in the cloud
or
in like Shell's example
near some massive data set
instead of far away
from some massive data set
so you could just
directly talk to it
and so on.
So yeah
it's super cool
you just basically
give it some SSH thing.
They also say
it's good for things
like
if your laptop
gets stolen
what data goes with it
you know
if you just
keep the data
somewhere else
right
then like
just revoke
the SSH key
and nothing's
nothing's bad.
You can also set it up
so that it'll create
pre-configured environments
like
when you connect to it
it'll automatically
give you
something with like
let's say
Conda setup
and all the right
libraries pre-installed
and that one weird
C thing
you gotta
you know
apt install
to make sure
it works like
it starts with that
just all configured
from different things.
So anyway
that seems all
pretty cool to me.
I thought it was
pretty neat.
That does look neat.
I think it's free
if you set up
your own server
but then I think
it costs money
if they provide you
the server
right
so kind of just like
firing up a VM
for you on your behalf.
All right
you ready for the buried lead?
Scroll, scroll
so here you can see
this is an example
of just like
connect over SSH
or you can go to
JetBrains space
and they'll create
one for you
right
but here's the buried lead
they announced this thing
called JetBrains fleet
which is
as far as I can tell
unrelated
I think it'll connect
one of these things
but is
is another
thing
so if you click
down at the bottom
or is there something
about learn more
and if you go to this
it is a
complete
rewrite
of the whole
IDE story
over at JetBrains
and basically
think VS Code
but from JetBrains.
Yeah
I'm interested
in watching this
and I just heard
about this last week
and they're doing
it an invite only
sort of
not invite only
but you have to like
early access
get approved
sort of thing
yeah
get approved
sort of thing
they're trying to limit
basically limit the feedback
so that they can
deal with the feedback
so
yeah
so it's like
super fast to open
it doesn't have
a project structure
in the same sense
that like PyCharm
or IntelliJ would
it just opens files
and it doesn't even have
the IDE features
unless you click
this little like
make it smarter button
and then it'll like
fire up
all the high-end
stuff that takes
you know
five seconds to start
the other thing
that's cool
as you can see
on the screen
right here
is there's like
three people typing
all at the same time
actually no
there's five people typing
so it's like
Google Docs
where you can all
like collaborate on it
in parallel
like right within it
so
I think those are all
super neat developments
in the whole editor space
which you know
we all write a lot of code
and kind of deal
with these tools
editor as a service
is something
that is happening
and I'm
it
is a hard thing
for me to wrap
my head around
because my brain
thinks I want
all my editor stuff
locally
but there's a lot
of times where you don't
so
yeah
I just like the group
Cody
yeah I know
I think that's really
neat as well
I think that would be
really valuable
to some people
on teams
instead of
you know
we've all been
in those screen share
meetings
like no
could you go over there
could you type this
no no no no
not after that
but inside the parenthesis
it's like please
no
that's exactly
what's going to do
it
to the left
no
a little bit more
to the left
exactly
and so
wait not a friend
yeah
let's see
a bunch of people
out there
really like this
RJL
and Sam
and so on
but
Kim has an interesting
comment
we've come full circle
ish
back to
talking to the one
mighty mainframe
over a lightweight
terminal
circa
1985
or
you know
for me
90
like
95
and like
X11
Xwindows
like
is your Xwindows
set up so you can talk to the server
yeah
yep
I'm just thinking the same thing
yeah
definitely
but these are interesting ideas
you know
for me personally
I love to use PyCharm
for working on projects
but if I've got just a
JSON file
or even a Python file
and I just want to look at the file
I probably won't open it in PyCharm
because it's going to create all this project
goo
that's going to be stuck in that folder
and it's going to expect
it's going to complain
there's no interpreter
I just want to look at it
you know
and so tools like this
I think are going to be really neat
yeah
and
Brandon's support
suggesting something crazy out there
like mobs
might run in
and no
mob programming
where you like working as a group
I think it's fun
yeah
and I'll be
we should play with this though
yeah
I think
it'd be fun to see
what all the interactions
feel like
and stuff
I totally agree
yep
alright
over to you
you know
I'm trying to remember
how I came across
the XY problem
and
and I was
doing some research
last week
and
I think I was down
some rabbit hole
of link
follow link
sort of thing
and I ran across
this problem
and
the XY problem
and
probably everybody else
knows about this already
but I
it was
the concept
was new to me
and I
yeah
I don't know
the XY problem
okay
and I studied math
come on
well so
it
isn't
really that mathy
but
so the XY problem
is essentially
you
is
you're trying to solve
problem X
and you think of a solution Y
that would
help work
to solve that
and you
you get down to
trying to solve
all the details of Y
and you get stuck
so you ask about Y
but what you're really
trying to do is X
and it's
that's sort of nebulous
an example
kind of highlights it
so
and
we've got
this example
in the show notes
that I pulled out
of one of the links
is
how do I
if somebody asks
how do I
get the last three characters
of a file name
and somebody says
oh
you just like do
and this is a shell
command
you just
do like
if it's in the variable
foo
you just do
dollar
curly bracket foo
and then do a colon
and then negative three
just grabs the last three characters
but
also
why do you want
the last three characters
is it because you
are trying to do
trying to pull off
the extension
somebody goes
yeah that's what I'm trying to do
and they're like
oh well then you don't want
the last three characters
because it might be a two character
or a four character extension
so teach them how to do
the real problem
and
in one of the
I'm going to link to a couple
a couple
like forum
answers and stuff
in there
because
I think it's
interesting
there's a lot of
verbiage around
the XY problem
that sort of
blames the asker
for asking a stupid question
and I think it's important
to not do that
because
we do this all the time
we break problems
in software
we break problems down
if I want to do A
then I need to do B and C
but to do B
I got to do D and E
and then
and then also F and G
and then
way down
into the rabbit hole
I get to get into
the X and Y problem
but how far back
do you back up
to give enough context
to somebody else
so it's hard to avoid
you'll run into it
and then I
I really like
there was one forum
that had
some great advice
both on asking questions
and on answering questions
so when asking questions
state the problem
that you're trying to solve
but also
state the higher
level thing
that you're trying to achieve
if appropriate
and then also
how that fits
into the wider design
and then
it also brought up
if you've thought
of other solutions
that you've
eliminated
for some reason
or another
go ahead and list those
because
somebody might
give you one of those
as an answer
and you've already
eliminated that
so give the reason why
and then
I think what's most important
is giving answers
to what
XY problems
or giving answers
to problems
because
although
I think
everyone
that's
on this podcast
and also listening
is probably
an expert
in some fields
and a novice
in other fields
so we're going to be
on both sides
of the fence
so when answering
questions
and you think
oh somebody's just
trying to get the extension
I'll just tell them
how to do that
that's not
necessarily helpful
so a great
there's a great
three-part
thing to do
and our example
follows those
is go ahead
and answer
the question
directly
but also
ask some questions
about the problem
say
just curious
why are you
trying to do this
is it because
you're trying
to do this
other thing
if so
you're
the thing
I just told you
might not be
appropriate
and then
once you figure
out really
what the real
problem is
then
you can help
and give
the final answer
so
it isn't
helpful
to just
say
oh you're
probably
getting the
extension
go ahead
and just
do that
anyway
I thought
this was
an interesting
thought process
around answering
and asking
questions
yeah absolutely
it seems to
be very
relevant to
stack overflow
type places
because you're
going to
pose
you're looking
for help
you say
I'm trying
to do
this
but a lot
of times
people will
and it'll
give you
very specific
answers
and the answer
could be
well why don't
you do this
library that
already understands
that format
like Shell
mentioned
earlier
like why
don't you
use x-ray
instead of
trying to
understand
how to
parse this
thing
just use
that
oh well
that's
way better
thank you
I see
that a lot
on stack overflow
that exact
it reminds me
also of my
like when I
went to school
and you're
trying to ask
a question
to your
professor
or to get
help on
anything
right
you're like
this is my
problem
what really
is your
problem
please tell
me about
it
and like
that's
what you're
asking
right
like tell
me what
the actual
problem
is
and if you
can do
that
clearly
you're
going to
get a
much
better
answer
yeah
absolutely
and a lot
of people
just don't
I mean
it's also
just a
different
perspective
thing
they know
that
they know
they have
the toolbox
of things
they know
how to
solve
and ways
they've
solved
them
and if
a new
problem
and this
is a
related
thing
is
people
don't
sometimes
don't
even
think
that
there's
a really
simple
solution
out there
like
oh that
tool
you're
using
it
already
has a
flag
that
does
exactly
what
you
want
but
you
didn't
know
the flag
was there
so
it took
me
when I
started
learning
Python
and I
was so
used to
Fortran
77
where there
was never
any help
they just
don't even
try
that
when I
started
learning
Python
it
took
three
or
four
months
before
I
finally
just
said
anything
I
want
to do
someone
has
done
better
and they
are out
there
I
just
have
to
find
out
how
to
ask
the
question
correctly
to
find
them
because
it's
true
like
everyone
is
most
people
have
tried
to
solve
the
same
problem
there's
someone
out
there
who's
worked
on
the
same
problem
in
all
likelihood
yeah
there's
so
many
libraries
with
PIP
or
Conda
that
you
can
if
you
knew
it
existed
exactly
yeah
exactly
all
right
okay
so I
guess
I'm
am I
next
you
are
next
okay
so
what
I
wanted
to
show
this
library
that
is
called
Kerchunk
it's
a great
name
yeah
brand
new
so can
you see
my snail
screen
yeah
so
we had
this
problem
where
like
as
Noah
and
NASA
everyone
starting
to
throw
all
these
net
CDF
files
or
all
these
different
files
onto
the
cloud
and
then
it
turned
out
that
access
in
S3
was
really
slow
and
so
people
got
really
frustrated
because
like
the
cloud
is
supposed
to
be
fast
right
this
is
going
to
transform
science
we're
going
to
do
it
better
now
that's
the
promise
yeah
that's
the
promise
but the
grass
isn't
always
greener
so
this
is
this
library
that
I
think
has
really
maybe
some
broad
applications
it's
being
developed
right
now
and
the
idea
behind
it
is
like
we
have
all
these
data
formats
that
we're
sort
of
stuck
with
lots
of
data
but
sometimes
it's
slow
on
S3
so
is
there
a
way
that
we
can
fix
this
and
the
idea
is
that
you
create
a
reference
file
system
and
so
you
do
this
by
going
to
each
of
your
files
and
just
taking
the
data
that
you
need
for
that
file
like
just
the
metadata
so
like
what
size
is
it
what
its
dimensions
and
coordinates
are
what
variables
does
it
contain
so
you
just
take
those
little
bits
and
pull
them
out
into
a
JSON
file
and
so
then
you
have
this
reference
file
that
just
contains
the
important
information
but
it's
really
small
and
so
that
makes
it
faster
to
access
and
then
you
construct
this
JSON
file
and
I
have
some
benchmark
tests
in
here
but
then
you
construct
a
mega
JSON
file
and
you
basically
virtually
aggregate
all
of
your
data
so
that
in
one
call
again
you
can
just
get
access
to
everything
and
nice
because
you
might
not
need
actually
the
data
you
might
need
to
know
what
time
frame
is
this
so
do
I
need
to
read
in
that
file
or
not
right
yeah
and
in
some
ways
because
you're
doing
a lot
of
what
one
of
the
things
with
x-ray
back
to
that
other
library
is
that
does
the
lazy
loading
so
like
this
is
a
16
terabyte
data
set
that
I'm
loading
here
but
I'm
just
loading
the
data
about
the
file
I'm
not
actually
loading
any
data
until
I
need
to
touch
it
and
so
I
can
load
this
giant
data
set
in
a
little
bit
over
less
than
two
minutes
by
doing
this
virtual
aggregation
with
KrippChunk
and so
all it's
doing
is
it's
reading
these
aggregated
JSON
files
and
right
now
it works
for
three or
four
different
types
of
data
sets
so
if
you
have
big
collections
of
data
that are
going
on
to
S3
they have
lots
of
different
little
files
this
is
a
way
to
sort
of
virtually
aggregate
them
into
one
big
data
set
that
you
can
then
subset
oh
that's
really
cool
it
seems
like
this
is
one
of
those
that
comes
as
part
of
the
FS
spec
project
which
we
talked
about
pretty
recently
as
well
yeah
and so
this
is
part
of
FS
spec
and
it's
kerchunk
it was
just
released
and
it's
a
unified
way
to
represent
compressed
data
formats
and
it
creates
this
virtual
data
set
so
that's
where
it's
located
yeah
super
cool
see
Kim
has a
question
do you
keep
the
individual
JSON
files
with
the
data
you
can
so
the
nice
thing
about
this
the
data
can
be
anywhere
and
again
this
is
idea
to
make
data
invisible
and
easy
you can
either create
them yourself
and just
keep the
little
JSON
files
public
and
then
you
just
make
the
one
aggregated
JSON
file
public
and
anybody
could
use
that
JSON
file
to
access
the
data
this
way
fantastic
that
looks
really
helpful
for
working
with
large
data
yeah
I think
it's
cool
it looks
awesome
all right
Brian
does that
bring us
to
the
extras
yeah
I guess
it does
how many
extras
you got
today
I just
have one
entertaining
extra
I thought
as
some
people
have amusingly
noticed
I am
attempting
to grow
my hair
out
and
I went
to
Florida
last
week
and
it's
very
humid
in
Florida
and
I
looked
like
a
cotton
swab
it
just
like
poofed
anyway
that's
it was
amusing
to me
but
you should
have
sent us
some
pictures
or
something
yeah
maybe
those are
the pictures
you don't
really
went out
there
but
yeah
so I
wish I
could have
seen
like
yeah
because
I was
at
Disney
World
and we
were doing
like
rides
and stuff
and I
really
wish I could
have seen
like
the
flowing
hair
on the
roller coaster
or something
like that
perfect
I
nice
let's
see
what
shells
got
first
okay
so
what
are
extras
just
something
that
we
did
last
week
well
just
whatever
you
want
to
also
just
give
a
shout
out
to
while
we're
here
before
we
call
it
I
think
I'm
pretty
good
I'm
really
excited
like
NASA
starting
a big
transformation
to
open
science
which
is
exciting
they
started
a new
they
announced
just
last
month
a new
40
million
initiative
to
try
and
help
scientists
move
to
open
practices
and
Python
is
a big
part
of
that
because
a lot
of
this
was
the
open
community
that
Python
helped
develop
over
the
last
decade
and
all
of
the
tools
that
now
is
making
it's
not
just
science
easier
it's
making
it
easier
for
more
people
to
participate
in
science
I
think
there's
a lot
of
synergies
and
similarities
between
the
scientific
goal
and
open
source
yeah
because
it
used
to
be
like
scientists
like
you
would
share
your
knowledge
right
you'd
publish
paper
yeah
and
that
was
it
and
if
you
like
that's
what
graduate
like
I
remember
in
graduate
school
you
would
go
through
and
they'd
be
like
okay
derive
the
equations
in
this
paper
because
they
wouldn't
show
you
all
the
steps
and
you
would
do
that
and
if
you
wanted
to
code
it
up
you
would
just
open
up
a
new
window
and
start
coding
and
now
you
know
people
are
starting
to
publish
their
code
so
that
you
can
actually
reproduce
their
results
and
then
build
on
them
and
move
faster
and
the
whole
reproducible
science
thing
as well
fantastic
yeah
awesome
Sam
in the
audience
says
yes
more
open
reproducible
science
is
great
for
everyone
yeah
all
right
I got
some
extras
as well
as you
can
imagine
surprise
I don't
remember
when I
was going
on
maybe
this
was
actually
in
talk
python
but I
was going
on and
on that
visual
basics
six
just
the
I
want
to
drag
a few
things
on the
screen
and
write
a little
bit
of
code
made
it
so
easier
for
people
to
build
apps
you
know
what
Kojo
is
this
replacement
thing
so
if
you're
trying
to
build
some
desktop
apps
and
you
want
to
try
to
work
on
some
integration
between
those
things
but
there's
a little
demo
where in
six
minutes
they build
a web
browser
which is
kind of
neat
so
very
visual
basic
feeling
so
is it
Python
it's
not
Python
no
it's
not
Python
it's
more
VB6
feeling
I don't
know
if
it's
actually
VB6
which
is
even
worse
it's
sort
of
kind
of
but
not
exactly
I
just
did
a
webcast
10
reasons
you
love
high
charm
even
more
in
2021
with
jet
brains
and
Paul
ever
we
did
five
reasons
so
I'll
link
to
that
people
care
about
that
and
then
who
doesn't
love
a little
good
tech
shock
and awe
and
being
I don't
know
outrage
I guess
is the
word
I'm
looking
for
so
Microsoft
edge
is
this
browser
that's
sort
of
Chrome
based
and
they
just
announced
like
a
Linux
version
and
it
runs
on
macOS
which
all
these
things
surprised
me
and
there
was
getting
a lot
of
traction
and
there's
this
whole
thing
where
Microsoft
the team
at
edge
just
added
a
buy
now
pay
later
thing
built
into
the
browser
from
some
third
party
company
not
as
an
extension
but
integrated
into
the
browser
that
you
can't
not
get
when
you
go
shopping
it
says
would
you
like
to
use
this
for
payment
program
it's
almost
like
adding
payday
loans
baked
into
the
browser
it's
insane
I know
it's
such
a
bad
idea
so
there's
an
article
it
says
users
revolt
this
all
feels
extremely
unnecessary
for a
browsing
experience
and
the
comments
are
you
go to
the
comments
they
are
really
there's
256
comments
which is
an
awesome
number
of
comments
for
the
moment
but
there's
just
almost
nothing
but
why
is
it
this
is
unbelievable
to me
I
can't
believe
this
is
so
it
makes
it
feel
so
shady
and
trashy
right
like
the
next
thing
you're
going to
do
is
get
like
bail
bonds
offerings
inside
your
browser
if
you
get
your
browser
just
weird
stuff
so
anyway
I
thought
people
might
enjoy
just
reading
through
this
and
taking
a
little
bit
of
that
in
it
must
work
right
because
we all
have
this
experience
where
you
I
mean
there's
been
this
has
been
going
on
for
20
years
like
with
their
browser
remember
it
used
to
install
all
this
stuff
on
your
machine
and
have
to
delete
it
all
and
then
that
was
really
illegal
so
they
had to
take
it
they had
to
separate
them
out
and
they
just
keep
finding
ways
to
get
back
in
yeah
there's
some
really
interesting
stuff
you know
they're
now
sort of
putting
ads
in
the
start
menu
and
stuff
and
then
the
ads
are
forced
to
open
in
edge
not
your
default
browser
it's
just
like
there's
layers
of
like
why
are
you
doing
it
makes
me
happy
that
I'm
not
using
Windows
11
at
the
moment
whereas
I've
been
looking
forward
to
using
the
new
terminal
and
oh my
posh
shell
on
Windows
which
looks
amazing
so
I think
there's
a
different
groups
this
is
definitely
a
different
group
than
the
VS Code
group
of
people
this
is
going to
take
us
back
to
1995
and
we're
just
going to
be
using
a
terminal
window
to
access
anything
so
we
don't
get
annoyed
by
all
this
there's
no
ads
in
the
Linux
browser
now
if
they
could
just
get
the
ad
companies
to
be
able
to
just
collect
your
credit
card
information
and
then
instead
of
showing
you
the
ad
just
buy
it
for
you
and
set
you
up
on
a
payment
plan
we
already
know
who
you
are
just
click
here
if
you
want
it
or
just
send
it
to
you
anyway
and
just
charge
you
later
exactly
so
I
feel
like
this
almost
could
be
the
joke
but
I've
got
a
different
joke
for
you
all
right
so
the
joke
for
this
week
comes
from
a
solid
source
xkcd
as you
may
know
this
is
about
workflows
and
changing
software
so
here's
the
one
that
says
workflow
and
it's
just
in
the
change
log
or
some
sort
of
conversation
flow
maybe
a
github
release
or
something
it
says
changes
in
version
10.17
the
CPU
no longer
overheats
when you
hold down
the
spacebar
and
then
there's
a
frustrated
user
comment
it says
long time
user
four
writes
this
update
broke
my
workflow
my
control
key
is
hard
to
reach
so
I
hold
the
spacebar
instead
and
I've
configured
emacs
to interpret
a rapid
temperature
rise
as
pressing
control
the
admin
writes
that's
horrifying
the
user
writes
look
my
setup
works
for
me
just
add
an
option
to
reenable
spacebar
heating
oh I
remember
like
enabling
all the
weird
emacs
things
that
only
you
would
know
about
exactly
and
the
subtitle
is
every
change
breaks
someone's
workflow
I love
it
yeah
actually
it's
interesting
because
python's
even
like
more so
like that
because
of the
introspection
and
everything's
really open
unless you
really work
hard to
make it
I mean
you can't
really
hide
too much
stuff
with
python
so
even
if
you
have
a
comment
around
a
function
or
an
access
point
to
say
this
is
not
part
of
the
API
this
is
subject
to
change
you
can
change
it
and
it
will
break
somebody
because
somebody
has
reached
inside
and
used
the
thing
you
told
them
not
to
use
yep
yeah
those
double
underscores
and single
underscores
they're just
there to
slow you
down
but they
don't
that's
so you
notice
what
you're
not
supposed
to
do
those
are
where
the
interesting
parts
are
exactly
they
wouldn't
give me
the
feature
but I
can
just
do
it
right
here
all right
well I
think that's it
Brian
yeah
it was a good
episode so thanks
everybody for showing up
yeah
thanks everyone
yeah
thanks y'all for being
here great to have you on the show
thanks Michael
thanks Brian
take care
Bye, everyone.