Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.
This is episode 281, recorded April 27th, 2022.
I'm Brian Okken.
I'm Michael Kennedy.
And I'm Anna Story.
Welcome, Anna.
Thank you.
Before we jump in, tell us a little bit about yourself.
Yeah, definitely.
So I'm a data engineer, or at least at the moment.
I'm a byte trainee.
I'm a linguist.
So I'm debating both theoretical linguistics and computational linguistics.
So I'm really about how the information is encoded in our brains and how we share this information.
And that's why I work in the tech.
Nice.
Since I got my master's in computational linguistics, I worked at Amazon, at thelex.ai.org for a while.
The first worked as a language engineer, actually.
So I was more on the side of linguistic side of things and dealing with extracting the semantic and the meaning really out of the data for Electrophone.
And then gradually I switched over to just data processing and been in the role of data engineer for about three, four years now.
And I'm currently with Decafone, which is the worldwide sports retailer.
So I'm still working with lots and lots of data there.
Okay.
Wow.
Interesting.
That is fascinating.
Yeah, it's been right.
Yeah, it's really neat how we can speak to our devices these days and they kind of actually work, do amazing things, right?
Like I know when Alexa first came out and Siri especially, it was like, I don't really want to, that thing is so not getting it.
And now I talk to my devices all the time.
It's amazing.
Yeah.
There are some things that are really sophisticated that they haven't been doing now.
Sometimes I can't even believe where we're actually getting there.
So it's pretty exciting.
Yeah.
And sometimes I admit that, you know, in several things that they're like, really, you can't do it.
Yeah.
But I realized that having worked on that, actually, I realized that sometimes it's just kind of thing that, you know, like from a professional standpoint, it might seem like kind of trivial to me.
But I realized that, you know, there's so much work with things.
And then they say, I love the actual device.
But sometimes just like you, you, you don't get, you know, go over the little like corners.
Right.
So one of the things that I got to work at some point was actually helping Alexa kind of know when she needs to stop.
When she needs to stop talking about things and telling you about things like whatever she thought on Wikipedia or whatnot.
So, yeah, it's funny.
Fantastic.
So, well, for our first item, Michael, do you want to kick it off?
I will definitely kick it off.
Let's take it to the next level with this one.
So this is an article by Eluda called Take Your GitHub Repository to the Next Level.
And there's kind of 13 levels, but, you know, I guess it's a spectrum.
You decide which level you want to take it to.
So here are basically 13 ideas on how your GitHub repository can be better.
So there was a topic I was going to cover after I explored more.
I decided, eh, not so much.
But as part of it, there was a conversation about some WebAssembly stuff in Python.
And I checked it out.
It's really cool.
They're like, we're going to use this library.
This is the fundamental thing that makes it work.
And I go to the GitHub repo for that.
And it says, here's how you build it.
And that's it.
I'm like, wait.
OK, great.
But why do I want it?
What can I do with it?
How do I use it?
I don't care about how do I build it.
Like, that's the last.
I'll just download the WASM file.
But what do I do with it once I get it?
Right.
It was just none of that.
And so that's kind of, you know, this article helps you think through those ideas.
Nice.
So number one, and you know it's Python friendly because it starts with zero, step zero, rather
than one, make your project more discoverable.
Now, every one of these comes with a recommendation, a bit of a description, and then examples, which
is cool.
Nice.
So for example, this one says, what you can do is to help people find your project.
If the name of your project does not carefully describe what it is, you can put tags, basically.
So like refactoring or science or things like that might be something you put on there that's
not immediately obvious from it, right?
So you can tag subject areas and whatnot, and they have some examples.
So for example, there's this thing called Well app, which is like a mindfulness app for the
Mac.
Of course, it's for the Mac, isn't it?
So it has tags such as macOS, productivity, happiness, mental health, but also Flutter and
web app if people wanted to check out a Flutter web app, right?
Okay, so that's, you know, there's other examples as well.
That's step zero.
Step one is choose a name that sticks.
Something that's available on PyPI, something that people can Google, something that people
want to say.
It doesn't sound silly or unprofessional if they were to use it.
You wouldn't call your web app Fancy Pants Server, right?
You wouldn't say, well, our Fancy Pants Server is really scaling today.
Like, you wouldn't want to speak that way necessarily.
So don't name it that way, right?
Yeah.
So choose a name that sticks.
And that we can say on air.
Yes, exactly.
And is, you know, somewhat predictable in the pronunciation maybe because that's also a
challenge.
But so there's some examples of like...
Yeah.
Yeah, yeah.
Anna, what do you think?
Yeah, absolutely.
Just thinking about the name, something that I ran into today, particularly with Python,
some of the services or applications and libraries as well that help them.
And in PY, and sometimes you don't know if it's PY or P in that case.
It's like, you know, confusing.
And then you're talking to somebody else who's looking about the same thing.
They're like constantly confused.
So, yeah.
Yeah.
I agree.
It matters a lot.
Let's see.
So some of the things are conduct a thorough internet search for the name, avoid hard to
spell names, get the dev or .io domain if you really, really care about it.
You know, is it some random small little package or are you trying to create the next FastAPI?
Right.
A name that conveys some meaning.
I was thinking about Jupyter, for example.
Like, Jupyter is pretty interesting because it's kind of hard to spell.
But once you know it, you just know it.
And it very clearly works well in a search.
There's probably no domain name that's like a misspelled planet type of thing.
You know, I mean, it was probably a really good choice, even though it kind of breaks the
maybe hard to spell at first.
Yeah.
But it's easier to search.
Right.
So.
Yeah.
So the example they get for this one is size limit is the name.
And what does it do?
It calculates the real cost to run your JavaScript app or lib.
Keep good performance.
It'll show an error in a PR if the cost, basically file size, exceeds the limit.
That's cool.
The next one, I'm all about this.
Display a beautiful cover image.
So if you go to a repo and it's just the text, that's not amazing.
You want some color and you don't necessarily have to have like an amazing logo.
So they come back to this well app and it's just a W with like a little connection smile
or something under it.
One thing I did learn about this, though, that I thought was interesting.
Like, how do they center this image but not have it go all the way across the readme?
If you go to the readme and you actually look at it, apparently GitHub will let you put full
HTML inside of your readme for the segments that need lots of formatting.
I thought I thought they wouldn't.
I know some markdown does fall back that way, but I didn't think GitHub did.
Anyway, apparently, yes, you can.
Also, this one's quick badges like a CI passing.
What's the license and so on.
Is there a YouTube link to like a YouTube channel that shows people how to use it?
Some more of those as examples.
Write a convincing description in a paragraph or two.
Add things like, what is this repo or project?
How does it work?
Who will use it?
What is the goal?
And so on, right?
Real simple one.
And again, they come back to the size limit.
It's a performance tool that'll crash your CI if it's too big.
Here we go.
Getting to the ones that Brian and I love.
Port Visuals to attract users.
Yes.
So you might think there's no UI aspect, but here's a full-on CLI example.
That is create Go app CLI.
And all it does, imagine this, it creates Go apps on the CLI.
It's a good name that can base what it does.
But if you go to see, it's like, how do I create one?
It has the option, but then under it, it has an animated GIF doing the things that creates
the app and showing you the tree structure that results, the file structure that results and so on.
Then a full video and a documentation to that thing and so on.
So that's pretty awesome.
And how about you?
Brian and I are always trying to quickly jump into a project and figure out what is it about?
Is it polished and so on?
But that's because we run this podcast.
How do you see this sort of pictures and animations for repos?
Yeah, that's super helpful.
I really like the idea with the animation.
Just basically taking you through the kinds of things that this particular app, for instance,
can do.
That's super helpful.
More and more people are doing it.
I don't think it's super popular yet.
I don't know about how about you guys, but I haven't seen it for a lot of times.
Yeah.
But it, yeah, it definitely looks nice.
Yeah, I really like it as well.
All right, let's see.
Another one is create a practical usage guide, like how to use it with some examples, some
templates, answer common questions like an FAQ.
I use it on Windows or does it require admin support?
I don't know.
Something like that.
Build a community.
So maybe you have a, this is probably further down the line, but like, do you have a Discord
community for your project?
Or you can even just enable discussions on the GitHub repository.
I'll end up with people opening issues on my various repositories saying, I have a question.
Like, okay, a question's not an issue.
An issue is the thing that is wrong or a thing to be improved, but they don't have another
way to communicate traditionally.
But GitHub now has, in addition to issues, they also have a discussion section that's more
open-ended.
So I think that's off by default, if I remember correctly, at least on the older ones it is.
So I go and turn that on.
Code of conduct.
That's all good.
Contributor guidelines.
Choose a license, the right license.
Remember, if you don't choose a license at all, that means it's unlicensed and people
can't really use it.
So add a roadmap.
Create GitHub releases.
One thing that I didn't pull up that's pretty cool is release drafter.
I'm not sure if you all are familiar with this, but this is a pretty cool thing as well.
Release drafter.
Drafts your next release notes as PRs are merged into master or main, depending on how you set
up your repo.
That's pretty cool.
Customize your social media preview.
So if somebody shares your project, you can control what is shown in that little Twitter
card or other cards.
So apparently that that can be customized inside of your GitHub repository and launch a website.
Off it goes.
You can use GitHub pages or Netlify is really easy for easy and free for static sites and
so on.
So anyway, there's a bunch of things people can do to take their repo to the next level.
What do you think?
I think it's great.
Yeah.
I love this list.
It looks very nice.
I don't do any of these things and I probably should.
So I might have a picture.
I have a usage guide.
Oh, there's also one that talks about how to install it that I somehow skipped, but most
things don't need.
So one of the things that I see a lot is, I don't know if this covers it, but I see documentation
that's on read the docs, which is great, but I still think a quick start or a little like
this is how you install it and this is how you can do a little bit of something with it.
That should be in the read me, even if you have other documentation, because I don't
want to have to just go to the documentation to see if this is the right project for me.
So, yeah, this is great.
So we have a question of does, how does one create a CLI animated GIF?
And I don't know if the dot, if this article covers that, but I don't think so.
Okay.
We'll have to, we'll have to research that and get back to you.
Yeah.
Well, Alvaro, what I do is I'll use Camtasia and you can record a Camtasia video of just
the window.
And then there's different output options like just audio or just the video or an animated
GIF.
So that's one of them.
Jeremy Page points out there are a few tools to record that in a cinema.
I don't know.
Like ASCII cinema, basically.
I don't know how to say that.
It's often used pretty cool.
And Dean, hello Dean.
You know, the hook of names.
Exactly.
I'm at a loss on that one.
Claudia, who I just had on Talk Python, has a blog post about many of those things.
And he has a better for release drafter and badges.
Yeah, I covered that on Talk Python just recently about hypermodern Python.
Awesome.
Well, that's probably way more than people want to know about their GitHub repository.
But so often GitHub repositories these days serve as your CV or your resume when you go to apply
for developer jobs.
And if you end up at somewhere that looks like what they described here, rather than a bunch
of things with like weird commit messages and nothing like that's going to make a different
impression.
Or if you want people to adopt it and start using it.
Yeah.
And if you don't, then don't put this stuff in.
Yeah.
Exactly.
All right, Brian.
Let's go faster.
Well, let's go faster.
Speaking of CLI.
So this is a fun tool.
We're talking about faster row.
Faster?
I don't know.
Faster row.
I'm going to go with that.
So this is a time.
It's like time it on the command line.
So but it's pretty neat.
So this is by Arian Wasi.
And we've had we've covered something of his before.
So it was the type explainer thing.
Right.
I don't remember its exact name, but type explainer where you put a typed thing in there and
would humanize what those meant.
So I this is a simple little tool, but I'm loving it already.
So this one of the it does either it times stuff, but it also compares times.
So like in this, we're showing the website here, but and it I can't I can't tell what
they're timing.
So let's just pull over in the documentation.
It does have a bunch of examples.
So if you ran faster row with with two code snippets and in this example, we're showing
is just either just showing either a string or an F string timing those.
So that's pretty neat.
And those so those two code snippets.
If you run those, it'll run both of those a whole bunch of times and do some statistics.
Like in this example, it's running it 20,000 and 50,000 times.
No, 20 million and 50 million.
Wow.
And then it shows you a little progress bar and and then who wins.
But if you don't if you're not comparing two things, it'll just show one with the same
graphics, but you can do more than two.
I did like three or four just trying this out to time different things and compare them.
And this often that's why I'm timing something.
I'm comparing two things and I want to see which one's faster.
So this is a really cool feature.
You can either pass in code snippets or you can give it to Python file names and it'll
run both both those things.
One of the it's kind of a whole bunch of really cool features, actually.
And one of the things I like is you can if you've got some code snippet that you are
need some setup, but that the setup part isn't the part you're timing.
You can give it some setup code to do before it does the time part.
So that's pretty neat.
Anyway, just a really nice looking command line interface timing tool.
Yeah, that's very cool.
So you can sort of isolate the thing that you really want to time.
So you don't care about.
Yeah, I haven't tried the setup part, but it's cool that it has it in there.
There's there's a documentation is pretty thorough, actually, as well.
Quite a bit of cost customization available.
That's cool.
Yeah, I agree that that is nice that setup stuff, because so often if I want to profile
like some web app or something, it's the thing I want to profile is dwarfed by just loading
up the framework and scanning all the files.
And you're like, all right, now I got to hunt down that little fragment that actually represents
what I'm really after.
So cool.
Yeah.
Maybe I'll try one of those sometimes.
Yeah.
And you can pass in strings of Python or you can pass in files.
Yeah.
And when I saw the strings a bit, I'm like, all right, there's a good use case for semicolons
in Python.
Right?
You can use them.
Do it.
Yeah.
So.
Exactly.
It makes you feel better.
Awesome.
That's a good one.
All right.
Anna, on to you.
What's your first one here?
Yeah.
So I wanted to talk a little bit about, well, data, my line of business.
And I was just thinking that something that could be really interesting, especially for
that part of our audience that works with data science projects.
Well, in general, you're collecting data.
You definitely, in most cases, you get some kind of noisy data that you need to clean up
and filter out in some way.
And particularly, so I imagine we have a pretty large international audience as well.
And also, on the other hand, if you're working with data from social media, which is very popular
right now, one of the questions that you have to solve there is identify the human language
of the data that you're working with on content.
You want to filter out the pieces of data that are maybe, for example, are not an individual
if you're going through social media posts or something.
All right.
You get that little, translate this to your language, little button at the end, if for some
reason the popular post is in Spanish or something, right?
Exactly.
Yeah.
And some of the platforms, their APIs rather, do provide this kind of filtering on their backhand.
I know Twitter does that.
But also, as I know, sometimes it's not as reliable, really.
I guess maybe, again, like I could imagine that maybe it's not really sort of the ultimate
goal.
That's either maybe not putting as much love and caring to this question.
So that's something that I had to do a few times also.
And a couple of libraries that I've worked with are LANGID and LANGDETECT.
There are a few more out there.
And these ones have been out there for a while, actually.
And LANGID hasn't been actually sort of worked on actually for a few years now.
But it's still kind of one of those benchmark libraries for this kind of questions.
And both of those are super neat, actually.
So LANGID is really popular.
And one of the things that I really liked about it is that it actually covers a lot of languages.
So I've actually had different pieces of information depending on the documentation that I was using,
either at Hi-Fi or at the GitHub page.
So at some point, I saw it was covering 97.
And I think there did have pages saying 97.
97 is a lot of languages.
I couldn't name 97 languages.
I'm a linguist.
I would have trouble naming, you know, 97 languages off the top of my head.
I definitely don't speak 97 languages.
And some of the nice things about it is that you can use it as sort of like a standalone, you know, module,
like a command line tool, for instance.
But you can also use it as launched as a web service.
So that's really neat about it.
And some more like needy-free things that were really helpful when I was trying it out for some of my project was that
when you try to identify the human language using one ID, it actually outputs the weight and the calculations done,
which is very typical in like a log space.
So you have like these funky numbers in the end, you know, truly speaking.
But the good thing is that you actually can convert them to a bit more confidence force that especially a data scientist are used to.
And that actually comes in super handy because sometimes you, when you're trying to filter out the data
and you know that this kind of tools are like obviously not, you know, 100% reliable,
you can also use this confidence force to maybe use it as again as like, okay, I'm taking this answer and I'm relying on that.
Or, okay, maybe I'll just like drop this piece of data altogether because it looks like the language identified is not super actually sure.
Like what kind of language this is, you know, if you're targeting a specific language.
And another...
Yeah, this is wild.
Yeah.
So you get, you basically might say we're 80% sure it's English, but it might also be Spanish or something.
Exactly.
Yeah.
Like English can be easily confused with maybe German or sometimes French just to use it so much of the vocabulary circling around with those two languages.
So, yeah.
So the identifier is not going to be like 100% sure that, you know, this is language.
And the funny thing is that I'm not so sure about Lang ID.
Yeah, Lang ID is also statistical actually.
No remembering.
And so Lang Detect as well.
And sort of the flip side of that is that it actually works very well.
The bigger piece of data that you're fitting into it, the more confidence going to be.
Like, right?
That's how specific work.
Yeah.
That's how machine learning works, sort of generally speaking.
And if you're working specifically with this kind of short tweet, social media posts, if it's like really short phrase, sentence, interspersed with like emojis and stuff, it's probably not going to be super confident.
So the bigger piece of data, the more confident, the better the performance of the languages.
And there will be something to keep in mind when you're working on data and you're trying to filter it by language.
Yeah, that makes sense.
If you have one word or something, it's very hard to go off.
Yeah, exactly.
So this being one file, sorry, right.
This being one file is insane.
Like it acts as a web server and does all sorts of stuff.
Crazy.
This is crazy.
Yeah.
And it's something that I really like about it.
It's a pretty lightweight, sort of, well, isolated, low dependency kind of package, which is fascinating.
Based on a kind of not a super sophisticated, naive-based Helgern, if I'm remembering it actually correctly.
And so, yeah, that's really, really fun.
It's really nice.
It works so nicely.
And the other one that I wanted to sort of kind of juxtapose to it was Lang Detect, which is in my second tab.
I wanted to do a lot of things.
And it's also really neat and easy to use.
And the great thing about the basic usage is very straightforward.
It's like one of those packages you discover, like, you know, we need to do what's doing,
how it's doing it.
And, you know, like, you really can understand in five minutes if it's going to be something
that, you know, is going to suit well in my project when I put it.
Sure.
So the main methods are the text and the text language.
So you can either just call it in a piece of data and try and get the most probable language
that this package thinks it is.
Or you can have returned a list of possible languages.
So it's going to actually to order them.
English and then there's a fraction of probability that's going to be in the German or something
like that.
And then you can decide for yourself.
And yeah, so overall, from my experience, language is a little bit better than language, but that
sort of looks, you know, empirical.
Yeah, that's great.
It seems super useful for anyone that needs to parse text and can't be sure it's all in
one language.
Yeah.
So if anyone else is working on some kind of data science project or working with human
language data, I would highly recommend.
And probably one of the things while I think it's a little bit more confident and robust,
I know that it covers fewer languages.
So I think it's 55 languages total compared to 97 for language I do with it.
But yeah.
Yeah.
Interesting.
It's both great.
Nice.
Well, Michael, let me tell you about our sponsor for this episode.
Before we move on.
It's a podcast.
Amazing.
So this episode of Python Bytes is sponsored by the Python Bytes.
Compiler Podcast from Red Hat.
So everyone out there, just like you, Brian and I, we're both fans of podcasts, listening
to podcasts all the time and stuff.
That's why we started some.
We like them.
And so I'm happy to share a new one from a highly respected open source company, Compiler
Interactive.
And I'm a fan of the Python Bytes.
And so I'm happy to share a new one from a highly respected open source company, Compiler,
an original podcast from Red Hat.
With more and more of us working from home or being more disconnected, it's important to
keep our human connection with technology.
Compiler unravels industry topics, trends, and things you've always wanted to know about
tech through interviews with the people who know best.
So on Compiler, you'll hear a chorus of perspectives from diverse communities behind the code.
These conversations include questions like, "What is technical debt?"
"What are tech hiring managers actually looking for?"
"Hint, see item one to some degree."
"And do you know how to code to get started with open..."
"How do you know how to code to get started with open source?"
All right.
I was a guest on Red Hat's previous podcast called Command Line Heroes.
And that was a super produced and polished podcast.
It was a really cool experience.
And so Compiler follows along in that excellent tradition and that polished style.
So I checked out episode 12, how we should handle failure, which I found really interesting.
I really value their conversation about making space for developers to fail,
so they can learn without fear of making mistakes, you know, like taking down the production website and so on.
Right?
People grow through experimentation, but they also fail.
They try new things.
So you got to make sure that they get a chance to grow.
So learn about the Compiler podcast at pythonbytes.fm/compiler.
The link is at your podcast player show notes right at the top.
You can listen to it on all the places that you would think.
So thanks to Compiler podcast for keeping this podcast going strong.
And Brian, also just real quickly want to point out, I know people can just go to their podcast app,
wherever that's Pocket Cast or Overcast or whatever and type in Compiler and search.
But please visit pythonbytes.fm/compiler and there's a place to subscribe with all of your various podcasts destinations.
That way they know it came from us rather than just out of the ether.
So if you're going to subscribe or check them out, please do through that link just so people know.
Nice.
Yeah.
So how about we talk about watching some things like files?
Yeah.
We were listening.
Now we're watching.
We were listening.
Now we're going to watch.
But watch them for changes, not watch what they are.
So this one comes to us from Samuel Colvin of Pydantic fame.
So you know, it's a pretty cool experience behind developing this API.
And the idea is it's a simple, modern and high performance ways to watch files for changes.
So there's a lot of reasons you might want to do that.
You might want to say if somebody drops a file into this directory, I'm going to kick off a job to like load it up and process it in some kind of batch processing.
Or I want to have my web framework automatically restart if this, any of the files in here get changed, right?
Any of the Python files or whatever.
So you could use it for things like that.
But the modern part's pretty interesting.
So that's a good thing.
It hooks into the underlying file system, the underlying OS notification systems, and is done through, that's done through the notify rust library.
So basically it's a low latency, high performance, native, non polling way of watching the files.
And then just goes to the operating system and says, hey, I, in this directory tree, if anything changes, call the callback.
Nice.
That's pretty awesome.
Yeah.
So there's real simple uses here.
Like I can say from watch files import watch, and then just for changes in watch some path, then you can process those changes.
So here's an example of an app that just starts and its job is to, as things change here, take them up.
That might be an example of what I said about kicking off something over to like load it and parse it and decide what to do, and then maybe pass it to Celery for background work, right?
On the other hand, you might want to do other things while you're watching for changes as well in your app.
In which case there's also an A watch, an asynchronous watch.
So if you're doing other work and it's all asyncio based, here you can just say, kick off the watching bit and await for the changes to happen.
And then do other async processing like FastAPI or web or database calls, you know, web with HTTPX or database calls with Beanie or whatever other asyncio things.
And it's sort of less you run them in parallel, which is cool, right?
Yeah.
And if you want to go even further, you can kick off a separate process and say, start a process that will watch for changes here and then call back this function if those things change.
So that's pretty cool too.
There's all these different ways in which you can use it, but yeah, it's pretty neat.
It's based on this REST library and it seems pretty powerful.
There's also a CLI, which I did want to point out one other thing over here.
Like this, I thought this might impress you, Brian.
Definitely.
I can do a command line.
Watch files command that will say, watch this directory.
And if anything changes, rerun the failing tests.
That's very cool.
That's cool, right?
So you just do watch files and you run the string,
pytest - - lf, which is pytest, rerun the failing tests.
If anything changes.
I think that's neat.
The, the, the command line stuff is actually cool.
I, I, I check it out just for the command line usage, but the, the ability to use it
programmatically too with an API, that's impressive.
And I'm, I'm very happy they included that.
Yeah, absolutely.
This is, if you're going to use it through the CLI, this is the perfect pip X install type thing, right?
Yeah.
Pip X install watch files.
And it's not really tied to any of your projects.
It's just always there.
Anna, what do you think?
Yeah, but that, that looks super neat.
Um, just made me immediately think about, file triggers that are one of the things
that it's built in mostly, or at least widely used in, you know, and, cloud storage as well.
Yeah.
And it's like, yep.
I can imagine like all the possible ways that it can be used.
So yeah, that's really neat.
Yeah.
I wonder if they, in their documentation, they actually provide any popular use features or anything.
They might not do that, but I'm curious if they actually do.
Yeah.
I didn't see any, any in particular, just a couple of examples on how you might use it and all, but yeah.
Yeah.
There's an older project called watch God.
I don't know anything about that one, but I'm glad I didn't learn about it.
Cause now there's a new one called watch files.
But if you use an old one, this is the successor to that as well.
It's a funny name, but I could see why some people might not want to use it.
So yeah, well, I can see item one, right?
Pick a name that people are willing to talk about.
Exactly.
Yeah.
Well, I want to talk about a new tool as well.
Coverage, not.
So hopefully all of us are familiar with coverage.py.
So it's maintained by Ned Batchelder, a really cool tool, but there's a new guy on the scene
and the new person on the scene is slip cover.
So slip cover.
And actually I heard about slip cover through the coverage.py Twitter account, which was interesting.
And so not surprising though.
Ned's a pretty open-minded guy.
Okay.
Um, but, so slip cover is, is coverage, but it's, it's pretty new.
So some of these, commits that's just within the last week or so that things, this came in.
So there's a, it's still at like, I think the version is 0.1.1 or something like that.
Um, you even just got a new one out this morning.
So why would you want to use something different?
Well, the, the, the big selling point of this is it's really fast.
It uses a different, a different process for, for getting the coverage information.
And it supposedly is only a 3% overhead, which, depending on your code coverage that probably
can be, can sometimes slow down your code, significantly.
Um, and if you've got a really long running test suite, making it even 20% faster, but sometimes
coverage can make it like twice as slow.
Um, so if, if you've got a five minute test suite that makes it 10 minutes and that's a little
painful.
So, this might be worth checking out.
It's quite a bit faster.
I tried it against flask, as an example and the, the flask numbers.
Um, so flask is, they've got a pretty tight test suite anyway, but, so just straight
pie test on my machine.
It was like 2.7 seconds.
Um, with coverage was, about four, 4.3 seconds.
Um, and then with slip cover, it was just a little slower than just pie test.
So pie test 2.7 with slip covers 2.88.
So just a little tiny bit more and you get coverage information.
That's pretty cool.
Um, it is in the early stages though.
There's some, there's some kinks to work out still.
So, I would try it out and watch this space.
Um, I think they're doing some really cool things.
Definitely worth watching.
But, like for instance, I ran into issues on projects that use pie test plugins.
I don't know why, but the plugins don't get loaded.
So the, like for instance, I tried to run this, this flask example, but with xdist,
uh, so that I could run all the tests in parallel to see if it's sped up parallel runs.
Also, it didn't recognize the parallelism.
So I'm not sure what's going on there, but I am in communication with, Juan, one
of the maintainers of this or, let them know what, what I found out.
I'm not just griping, and not trying to make it better.
I'd love to have this be a really cool tool.
So it looks neat.
Yeah.
Go ahead, Anna.
Yeah.
And so the data near zero overhead is mostly due to how they, managed to, to provide that.
Uh, we talk about the involved recommendations.
It's really interesting.
Yeah.
With such a little overhead, I'm tempted to think of a more diabolical use of it.
Like I've got, I'm handed some crummy old app that doesn't really have tests.
And I got to figure out, well, what part of this is dead?
Cause I don't know if you've ever picked up some old app that's evolved and evolved.
And there's just stuff people don't take out cause they're afraid to just run this in
production for a while.
Oh yeah.
And just go, okay, these things don't look like they're doing it.
I mean, there might be some case I need to track down, but this gray area over here,
that's not touched.
Let me look for things to delete over here.
That'd be kind of fun.
That's my favorite use of coverage is looking for dead code.
Um, yeah, exactly.
Uh, before we move off this, Brian of our ass, does it have a pie test plugin?
I know you said it doesn't work to run plugins, but this is the reverse question.
Um, I don't, I don't think so.
So you're running, you're running a slip cover and pie test at the same time.
I don't think you really need a pie test plugin for it.
Um, it, I w it, it does run it work with pie tests.
So you can run a pie test, operations on with it, but nice.
Just not the bells and whistles yet.
Right.
Uh, but I'm sure they'll get there.
Yep.
I would love to sort of circle back to the, the, the data.
Great.
Sound like a perfect record, but that's my favorite topic.
No, it's great to have you on to talk about it.
Cause Brian and I don't live in the data science world.
Right.
So it's really cool.
Yeah.
Um, well, you're, you're welcome in our, in our world.
There's a lot of fun stuff happening here.
And, well, actually if you think about it from the actual, the very beginning, right.
Even before trying to wrangle the data and trying to, for any interesting information
of the data, you have to get it somehow.
And, sometimes if you're like particularly working on some, sort of, side projects
on your own, and you, you want to maybe try out a new tool or maybe new, you can, if you're
doing like a, machine learning project on modeling approach, you usually need,
um, some very specific data to, to work on.
Um, and how do you get through the data?
Well, you have to actually go and maybe find some examples of the data on your own.
And so something I wanted to talk a little bit today was, actually webclawing,
webclawing, webclawing and a couple of, tools for that.
Um, so one that is, quite popular and it's actually like, industrial grade kind of tool
is, well actually either scrapey or scrape by as for both, variants.
Yeah.
Um, and it is a pretty great tool.
Um, so one of the great things that you're doing, you're doing, you're doing, you're
a great tool.
Yeah.
One of the great things like from the get go about is that, it actually has a built
in shell.
Um, so you can just, go ahead and sort of try out things in the CLI, get response
from a URL for instance, and then try to, look around it and then test out it behavior,
which is really nice.
And then see what kind of things you might want from there.
Um, and if you're actually, sort of go ahead and, and, and use it, for your,
uh, for your module to, to acquire, to get some data rather, it provides all sorts
of, reunite functionality to begin with.
Um, for instance, the, the choice between using other, there's a selectors for the, the, the,
the content of the pages or an expat, which is, obviously a little bit more flexible.
It's more flexible.
Yeah.
It's more fragile though.
Cause if they make any change to the page.
That also, yeah.
Um, but so yeah.
Well, it's part of the game.
Yeah.
Um, yeah.
And then, some other really nice things about it is that actually, they do a lot
of like heavy lifting for you in terms of translating.
So you can, there's, built-in methods for start project and you can, you know, run
that and right away you have the whole structure and by cost of the boilerplate, kind of code.
You just fill in the, certain pieces for, I can processing, which is in there.
I plan modules, you think some of the setting, et cetera.
And, there you go.
You know, I have like a huge amount of work already.
So it's, pre set for you, pre done candidates for you.
Um, and then, some other nice things about it is that, also provide you with, like,
a, a, pre set for you, pre done candidates for you.
Um, and then, some other nice things about it is that, also provide you with, like,
um, numerous choices actually for, exporting the data and for storing the data as well.
I mean, in a few places and the format that you would love to, to use for it.
Um, but all the typical standard things like PSA, JSON, some more, some, some left frequent options.
Yeah.
Another thing that's pretty interesting about this whole project is that there's a web scraping as a service company.
Yeah.
Behind it. Right. It used to be called scraping hub. Now it's Zite, Z-Y-T-E.
Zite.
Yep.
And you can basically, you can basically go in there and just, you know, sign up and hand it one of these, these, spiders and it'll just run it on the different servers, try to avoid getting blocked, all that crazy stuff.
Exactly. Yeah. so, therefore it's so elaborate and they really put a lot of, just like I was talking before, a lot of, loving carrots, like all those sorts of, functionality, like covering all those corners of like what you might want from, it's a web crawling, tool. and some other examples that, I found particularly useful for instance, the link extractor class is like really getting to like reading.
You know, parts of the tool where you can, extract further links from the page. but don't need those ones that, you know, to hear to a particular pattern for instance. and the list that you get is already. so once again,
tool.
Some other examples that
I found particularly useful, for instance,
is the link extractor
class is really getting to
need reading parts of
the tool where you can
extract further links
from a page,
but don't use those ones that
adhere to a particular pattern, for instance.
And
the list that you get is already
decouped. So
once again, it's already
so much of the
dirty work on your part.
So that's really great. And they
do provide, actually,
ways to interact with
the pages as well. There's a
format class that you can use
as it does
provide some functionality where you can interact with the page,
but I haven't used
it as much myself, so
I can barely show how
fascinating it is.
But it's probably well done
as well. And another
library that I wanted
to touch on briefly
today as well was
Robux. That's actually something
new for MacBoss. That's
something I'm in the
process of exploring,
so I haven't
had a chance to work a whole lot
with it yet.
But it's been really, really interesting,
and I would love, I would be happy if, you know,
I got to hear from somebody else to write it out
or something.
Because it, well, in the first place,
it's still some type of
HTTPX and beautiful soup,
beautiful soup 4, rather.
They're super popular
in sort of the data
processing line of work
and particularly web
and web scraping.
But it adds
some, you know,
really useful
functionalities,
and it looks like it
allows even more
of this interruption
with the pages
in a, like,
very neat
and clean way.
And you can probably
find examples
where you're here
in the documentation.
It looks so, you know,
nice and clean
and straightforward.
looks lovely.
So, yeah,
I'm really excited
about this package.
I'm hoping to
have an opportunity
to test it out
soon.
Yeah, Roblox looks very interesting.
It looks very
Selenium-like
where you can actually
control the page
as well,
like, fill in this,
fill in the comments with this,
fill in the first name
with that,
and then submit.
The other thing
that's cool about it
is it has
async support
for doing all those things.
Exactly, yeah.
You can scale it.
Yeah, that's fantastic.
Awesome.
Thanks.
Nice.
Well, where are we at now?
We have
Extras.
Extra, extra, extra.
Here all about it.
I only got one.
How many you got?
I got zero.
Zero.
All right.
And anything else
you want to give a quick
shout out to
while we're here?
No.
No?
Okay, cool.
Well, I wanted to tell you
all about
my terminal adventures,
I suppose we'll call them.
So I've been using
Oh My Z Shell,
which is amazing.
I love Oh My Z Shell.
But I also started
playing with Oh My Posh
and Please
and some of these
other things.
And I thought,
oh, well,
how am I going to decide
between say Oh My Posh
and Oh My Z Shell?
Well, it turns out, Brian,
you don't have to decide.
You get both.
So here's a little
animated video
I'll throw up
for people who are watching
and I'll put it
in the links as well.
So here's,
you can see this cool
prompt,
which is all driven
by Oh My Posh,
but you can see
like auto-complete
into local Git branches
through Oh My Z Shell
for either branch
or checkout.
And then on top of that,
we can do like PLS,
which is amazing.
You can do,
oh, and McFly.
We talked about McFly before,
which gives you
auto-complete into your history
and sort of a Emacs style
editor type of AI complete.
Then PLS for LS replacement
that is developer friendly
with like little icons
for the file types
and it uses Git ignore
to hide stuff
that you don't want to see.
And it's like Python friendly,
like understands V and Vs
and de-emphasizes them
and all that kind of stuff.
So anyway,
people have been trying
to decide between these things.
It turns out
they all go well together.
You don't have to decide.
That's pretty cool.
Yeah.
I really go with VSH
and that looks even,
yeah.
Yeah.
All the stuff that works,
you don't have to give up any of it.
The only thing that isn't there
is the prompt
and the prompt is not
all that great,
honestly.
I mean,
I know you can customize it,
but I think it's better
in Oh My Posh,
which is pretty amazing.
So people who are listening,
they can check out
the little video.
I'll link to somehow
find a way to do that
in the show notes
so you all can check it out.
Okay.
That's my extra.
Yeah.
How about a joke?
And I guess,
how about a joke?
So we're all starting
to go back out to dinner,
restaurants,
COVID's over,
I hear,
not necessarily,
but here's one
from a slightly different perspective.
It says,
hello,
I'm your server today.
Brian,
can you just describe
for people listening
what's in this picture?
there's two robots
at a restaurant
sitting down
and there's a server rack
next to them.
Yeah.
Okay.
And the subtitle is
when you go out
for a bite,
B-Y-T,
he says,
the server
is by the table
where the robots
are drinking.
He says,
my name is
D-H-X-005972
and I will be
your server this evening.
I'll follow this one.
Thanks.
All right.
That's what I got for us
for our joke today.
Nice.
Well,
thanks,
Anna,
for joining us today.
Thank you.
Thank you for having me.
Yeah,
it was great.
Thank you,
Brian,
as always,
and everyone out there
listening.
Thanks so much.
Very good on our own.