Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.
This is episode 256, or as Anthony Shaw likes to put it, 2 to the 8th, recorded October 27th, 2021.
Again, unless you're Anthony, which is probably like a totally different day in the future because he's in Australia.
I'm Michael Kennedy.
And I'm Brian Okken.
And I'm Anthony Shaw.
Hello.
Hey, Anthony.
How is the 28th?
Is the next day going to be good or things are okay?
Yeah, it's pretty sunny today.
It's nice.
Yeah, right on.
Okay, so the world hangs together for one more day.
Fantastic.
You've been here before.
You've been on Talk Python a bunch of times.
It's a friend of the show, all sorts of stuff.
So I'm sure many people know you, but, you know, just tell people a bit about yourself.
You're doing more techie things these days.
You're a little closer to the code, maybe.
Yeah, so earlier this year, I started working at Microsoft and work with Nina Zakarenko on Python inside Microsoft.
And, yeah, a lot of what I'm doing at the moment is just running around, breaking things, sometimes on purpose.
Yeah, just seeing how we can improve our experience and working with VS Code and Azure and a whole bunch of other stuff.
So, yeah.
Yeah, it's been a while since the last episode was episode 100, I think.
Wow.
You're hitting the big numbers.
So, yeah.
This two to the eighth is a significant milestone, I think.
It is.
It's pretty cool.
Yeah, awesome.
Well, we're happy to have you here.
Thanks for being here.
Also, something to do with a puppy I've seen on Twitter.
Oh, yeah, I got a puppy as well.
Oh, how cute.
I like to go to Golden something.
He's a Border Collie, but he's kind of golden colored.
And he's not in the room at the moment.
He's not allowed in here while I'm recording.
I thought it would be a bit chaotic.
Yeah, my puppy sometimes is here, but it's very bizarre the way that puppies socialize around COVID.
Instead of us being gone and then we come home, she now knows and understands the expressions I make to end a Zoom call.
So, she'll sit quietly for an hour.
And as soon as I say goodbye in Zoom, she's like, we're ready to go.
Let's go.
It's super bizarre.
But, yeah, that's the world we live in.
So, enjoy the new puppy.
Brian, you want to kick us off with our first topic here?
Lucas Lenga, he's, what is he again?
The developer in residence?
Yes.
For Python?
Anyway, he wrote an article called, Where Does All the Effort Go?
Looking at Python Core Developer Activity.
And I kind of really like this article.
He not only talks about really what's going on with developers and who's doing what.
To start off with, he talked about how he got this data.
So, this is also sort of a data processing sort of information scraping sort of article.
He's looking at the GitHub repository data for CPython, of course.
And specifically, he's going to be able to do that.
We've covered that on the show.
And even lists the SQL queries that he's got.
Oh, also, since the data is from the time when CPython moved to GitHub.
So, that's February 10, 2017.
And he mentions that it's up through October 9th is when he pulled the data.
So, but all the information is there.
So, you could grab it yourself if you want.
Even the little scripts he's got for modifying some of the data.
So, some of the interesting things.
The top parts of CPython that are modified.
It's probably not that surprising that coval.c is involved in 259 merge requests.
It's the top merged file.
Coval.c.
Yeah, that's where the bytecode processor is.
So, yeah, that's where all, that's the center point or the tunnel.
Everything flows through.
So, that makes sense.
Yeah.
And then goes on and looks at which contributors merged, have merged.
And this is an interesting thing, or had been involved in PRs.
And it lists the top, he lists the top 50 people, but it includes some bots, which is interesting.
I was going to ask that.
I thought Bedivere is probably going to be up there or Mrs. LinkedIn.
Yeah.
Both bots, by the way.
So, this is a, I'd actually love to talk to, or either me or Michael or somebody talked to one of the, or Python people to talk about the different bots that are used and why they're used.
Because that's an interesting thing of large projects using bots to help out with some of the work.
And anyway, the non-bots, there's a couple of people that stand out.
Victor Stinner and Serhi Storkaka.
So, I apologize for messing up your name.
But that's, they're really up there.
So, that's pretty interesting that they're involved a lot.
And then there's a description here, a nice note that Lucas writes, clearly it pays to be a bot or a release manager since that's naturally causes you to make a lot of commits.
Victor and Serhi are neither of these things and still generate an amazing activity.
Kudos.
And also, it's not a competition, but it's still interesting to see who makes all these recent changes.
By the way, this, that top PR thing was only since the beginning of January 2020.
So, taking a look at the more recent stuff.
And then, one of the things that's interesting looking at who contributed where, I didn't know this.
There's an experts index.
So, that was linked.
Oh, it's asleep.
An experts index that is part of the Python Developers Guide.
I didn't know this was here.
It kind of lists some parts of the system, but there's blanks.
And so, there's, so Lucas also listed, did a script and pulled out the top five contributors to each file, which is kind of an amazing list of all of the different, of, you know, the top five people for every file within CPython.
So, if, this is kind of neat, because if you're going to do a PR, or you're working on a fix or something, and you're a little confused by some of the code, one of these people might be able to help you out.
So, this is kind of a neat list.
So, there's a, at the bottom of the article also, it talks about some of the, some of the takeaways from this.
I don't have this right off the top of my head.
Merging, how long it takes to merge a PR.
So, it's hard to draw information from this data, because it's all over the map.
The standard deviations are pretty large.
But, if a core developer merges their own PR, it takes, on average, about seven days to get through the process, give or take 42 days.
And, and then core developer authoring a PR, which is merged by somebody else, takes longer, about 20 days, give or take 78.
And then community authors, up to 20 days, give or take 80.
But, I mean, I work on commercial projects that are not really that much faster than this.
So, it's not too bad.
What do you think of this article?
Yeah, Anthony, what do you think of this?
You spend a lot of time inside the CPython code.
I mean, you did write a book, CPython Internals, which people can check out, right?
Yeah.
Yeah, I did write a book about CPython source code.
It's interesting.
I'm, first of all, I'm super excited about Lukash being the new developer in residence.
I think he's got the right approach.
And he's already made, you know, really promising progress, I think, in terms of how, trying to make the community contribution process a bit slicker.
Yeah, that's that at the bottom.
Like, just watching the GitHub repository, core developers working on the repository and making changes and stuff, from the outside, looks fairly seamless.
My own personal experience has been, sometimes it's quite, like, if your PR gets responded to within the first week, then it'll probably get merged pretty quickly.
And then, if it doesn't, then it just kind of ends up in the pile.
And I've had ones in there for, like, three years.
Well, right, the average was seven.
But it could go out at, like, another 40 days.
And it's probably, like, really quick or really far.
Well, that metric is how long they take to get merged, which I guess requires that they are merged.
Oh, yeah.
So that's not their fault.
I mean, there's basically just, like, loads of people contributing stuff.
And there aren't enough people with enough time to sift through it all.
And it just makes it really tricky.
And the project needs to continue marching forward.
And there's people who are dedicated to working on the core developers.
But, you know, some of the community contributions are really valuable.
I think that's what's promising to me is that Lukash is kind of looking at that and not just taking this role on as I'm going to be 100% core developer.
Yeah.
Because, yeah, there's already lots of other people on the team who are making some amazing contributions.
You know, Pablo has been working on the new parser.
And now he's working on this, like, stack list changes in 3.11.
Yeah, there's so many things going on at the moment in CPython.
So it's really encouraging to see.
Yeah, it's super encouraging.
I think Lukash is doing a good job sort of smoothing out the edges to just make it easier for everyone to go faster, which I think a lot of times in teams,
you know, not specifically here, but in general, there's these people who are kind of, oh, that's the person you can ask to make the CI work again when you break it.
This is the person you asked, like, to set up a new machine and remembers how to do that.
And like, you don't necessarily get direct credit for doing that work.
But without them, it's just way harder.
And I feel like he's doing that for CPython behind the scenes.
Yeah, the experts index is really helpful if you want to get involved in bug triaging.
So that's something that people are open to help with.
If you go on bugs.python.org and you want to help to triage bugs, often what you have to do is kind of look at it, make sure that the person who's reported it has filled in all enough information,
and then basically add people on the experts index to something called the nosy list, which is like a CC list, basically, on the bug.
And then, yeah, it's just kind of directing it to the right people.
Once you've done that for a while, then you kind of get given like a triage flag in your user.
And then if you've been doing that for even longer, then you could be promoted up to a core developer.
And there's a few people who've gone through that route over the last couple of years.
All right, Anthony, while you're talking, I got two things to share out of the audience.
Dimitri Figgel.
Hey, Dimitri.
Great to see you here.
Dimitri says, thanks for inviting Anthony.
He's someone I look up to.
Very nice.
Thanks, Dimitri.
Good to see you.
Yeah.
And Waylon, who was recently on Talk Python.
Hey, Waylon.
Says, what a great lineup here.
Also kind of for you.
And also, Henry Schreiner.
Hey, Henry.
Also recently on Talk Python.
Says, both PRs I've been involved with to CPython got in in about a day, I believe.
Which that's pretty amazing.
That's pretty good.
Yeah.
That's great.
Yeah.
So before we move off from this one, Brian, this is a good pick.
One thing I just want to point out as well is all these cool stats and these graphs and everything we're seeing here apply to CPython because it's on GitHub, right?
Yes.
But you can run the same code and run data set from Simon Willison against it.
But against a different repo, I would imagine.
Right?
Oh, yeah.
Yeah.
So if you run a project, you could probably do a similar analysis for your project.
That's a good idea.
Yeah.
All right.
Speaking of good ideas.
And it's interesting that Henry is out in the audience because I feel like we might have been responsible for this article.
Clearly, we did not write it.
We may have triggered is what I'm saying.
Mostly me.
In not the positive way.
Right?
So this is a cool article by Paul Gansel, who was also over on Talk Python talking about the mysteries of date, time and stuff.
There's all sorts of cool things.
He maintains the date, utility package and set up tools, projects and so on.
Over on episode 271.
So he wrote an article said why you shouldn't invoke setup.py directly.
And the reason I think I might have somehow had something to do with this is Henry was on talking about CI build wheel and all the proper ways to build packages.
I said, oh, you can run setup Python, setup py space, you know, wheel or bdisk or something.
And they're like, no, no, no.
You could.
But please don't do that.
And then here we have this article like two days later.
So I don't know if that was part of that conversation, but it's a really good article talking about the state of building Python packages.
And it says, you know, look for a long time, setup tools and distutils were the only game in town when it came to creating Python packages.
Right?
So you could do something like invoke Python setup, bdisk, sdisk, wheel and so on.
Wait, I see.
So Paul is actually in the audience.
Real time.
Fantastic.
Hey, Paul.
Paul says, I think I did it because Matthew Fikert asked for it on Twitter and I got sniped.
Yeah, perfect.
Okay, good.
So I'm just a coincidence.
Fantastic.
But yeah, so my, the reason this is extra interesting to me and thank you, Paul, for writing it is I was still doing this, you know, Python setup UI various commands.
And I was talking to Henry.
He's like, no, you shouldn't do that.
You should do it this other way.
I'm like, he said, well, okay, well, how should I do so?
You should use build the build package.
What is this build package you speak of?
You know?
So we've talked about pyproject.toml a bunch of times.
We've talked about things like flit and stuff that will use it, right?
This all comes from PEP 517.
And there is a package called build.
You can pip install build.
And then you do things like Python-m for module, run build.
And you can say, I want an sdisk.
I want a wheel and things like that.
And this acts as a front end to things like setup tools to the various backends that do building for Python.
Yeah.
All these different things that understand it, right?
So it says, all direct invocations.
Paul says, all direct invocations of setup.py are effectively deprecated in favor of purpose-built, standard-based CLI tools like pip, build, and tox.
So this is quite a long article.
There's a lot to go through.
It has some interesting history.
So in the early days, there wasn't even distutils.
And then in Python 2, distutils got added.
And then setup tools came along.
And then while they work, there's still problems.
Like, for example, you might have dependencies that you have to install to run the setup.
But the way you install stuff and figure out what you depend upon is by running the setup.
So what do you do?
So an example of that would be Cython, right?
So you might have to import Cython.
And in the invocation of calling setup, you tell it how to Cythonize the PYX files, right?
But that's obviously not going to work because you're going to have to have Cython installed.
But how do you express that?
You know, it's like this chicken and egg problem, right?
So let me pull up my notes here.
Yeah.
So basically, one of the big questions was, why am I not seeing deprecation warnings?
Let me go down a little further.
Yeah.
So if I'm not supposed to do this, why isn't screaming from the top of its terminal?
Stop, stop, stop.
Why are you doing this, right?
So there's a lot of commands that still have indirect uses of the distutils and stuff.
So it's a little tricky to deprecate it.
But basically, you should consider it deprecated.
You know, at the end of the day, it's better to replace your setup commands with tools like
build instead of setup PY sdist or bdist wheel or talks and knocks instead of setup PY test
and other commands backed by projects intended to support that.
Yeah.
That sound good to you guys?
Where are you on this?
Brian, you go?
Well, I don't use...
Have opinions.
I mean, I kind of indirectly use build, but I basically just use flit.
So I'm not writing things with C extensions.
So pure Python stuff, I just do a flit build or whatever.
It works fine.
Yeah.
So that's kind of...
I mean, that's using the pyproject.toml stuff, right?
Yeah.
Anthony?
I kind of...
If I'm starting a project now, then I use pyproject.toml and the project doesn't have a setup.py.
There were some reasons why I had to add one in the past, but that's mostly fixed now.
So I'm either using flit or something similar like poetry.
Yeah.
And I've worked on projects years and years ago where the setup.py was like, just ended up just
being a script to run ad hoc commands.
Like there was a test, setup.py test.
And then there's like, and lint.
And just run them like odd jobs.
Yeah.
And what does that have to do with installing software, right?
Why is that?
Yeah, nothing.
It was just like...
Yeah, it just ended up being an entry point to do things.
And one happens to be installed, but there's a bunch of other stuff you might randomly do.
Yeah.
And it's fine that it's being deprecated, but it just, you know, cpython still does that.
Like the setup.py and cpython is still used in that way and invoked directly in the source
code this is.
So yeah, it's good that it'll be deprecated, but I don't think the tooling is quite ready yet.
He's not really saying to get rid of setup.py.
Just don't use it.
Yeah.
Don't run it directly.
Yeah.
Find something better.
Pip should do that.
Pip should do the discovery for you, the PEP 5.17.
Yeah.
And run the correct steps for you.
So yeah, absolutely.
So a couple of comments out in the live stream is that while recommending build, it's nearly
impossible to Google to find it.
And racist.
I love and hate the name.
So authoritative.
So un-Google-able and a bit hard to use in conversation, but yeah, yeah, for sure.
So I think if you want to take away from this conversation, right at the top, there's a TLDR
section that Paul put in.
Click on the summary.
Picks you down to a summary and you can go to a table and it says, I was about to type
this.
What should I do instead?
I was about to type setup.py sdist.
What should you type?
Python-m build.
Having build installed.
Or if I was going to type setup.py bdist wheel, I should type python-m build --wheel
or something like that.
Setup.py test.
Oh, maybe py test or talks or knocks.
We covered knocks recently with Preycin, which was really fun, I believe.
So setup.py install.
No, that's pip install.
Python setup.py develop.
No, that's pip install dash E.
And then as well as upload, it goes back to twine.
So yeah.
Anyway, I think this is the most actionable bit here.
Yeah, it's good.
Yeah, indeed.
All right.
Well, Anthony, let's talk about keeping an eye on things.
Yeah.
So I wanted to highlight a project which has been in the works for a while, but they've
just recently finalized the specification.
So this is called OpenTelemetry.
It's a part of the Cloud Native Computing Foundation, the CNCF.
And it's a cross-language event tracing, performance tracing, logging, sampling framework for
applications, in particular for distributed applications.
So if you've got an application which is spread across multiple microservices and you want to
trace things or monitor performance or whatever across all of the stack.
It's a super hard problem, right?
Maybe you've got a Docker container running this thing.
That Docker container calls some other service on a different Docker container.
And maybe the logs are even transient.
What are you going to do to know if something went wrong?
Yeah, exactly.
And if you've got an application that's spread across, well, if it's built into multiple
microservices and one of those services has a fault, it's really hard to know where that
fault came from.
So if it just says error, blah, blah, blah, blah, you're like, okay, so what triggered that
error?
Which request from a user at the front end?
Or how did that error happen in the first place?
And how can I fix it?
And also like identifying, I guess, tracking performance across your application and looking
at that.
So there's been attempts at doing this in the past.
Open tracing and open sensors were the two kind of projects beforehand.
So this new project, Open Telemetry, is a merger of open tracing and open sensors.
There's engineers from some big companies working on this, including Microsoft, Amazon, Splunk,
Google, Elastic, New Relic, and a whole bunch of others as well.
Including actually full-time engineers from some of those companies working on this.
So yeah, I've been working with an engineer at Microsoft who works full-time on this project.
He works on, actually, there's a few people who work full-time on this, but the person who
works full-time just on the Python components to this.
So the Python SDK basically allows you to instrument lots of different frameworks.
So you can basically drop it into Flask or Django or Starlet.
So if you're using FastAPI and you can sort of instantly get capture of what requests are
going into the application, when there's been a crash, like where that exception's gone,
all the logging information.
You can look at performance records and stuff.
I've been sharing some examples of where I've wrapped it around a FastAPI app.
And then I can see like performance of what's the average request time for each of these parts of the application.
And where is that time spent?
Even down to like...
Can you say like this is the data layer section and this is the business logic and here's the civilization or whatever?
Exactly.
So I can kind of see like almost like a cool stack, but across the actual components of the app.
So here's where it came into FastAPI.
Here's where it went into database.
Like here's how long the query took.
Here's how long the ORM took to remodel it.
Here's how long Jinja2k to build the template.
Like so you can kind of see a breakdown of all the different components and how things are being pulled together.
So there's two parts of OpenTelemetry, but there's actually more than two parts.
I am actually really appreciative of even though there are lots of engineers from big companies, this hasn't been over-engineered yet.
And I'm really hoping it doesn't.
Is there a factory factory method in here?
Yeah, exactly.
Especially because it's like so generic.
There's a real danger of it being just over-engineered.
So if you go on the website and go to registry and then pick Python on the right-hand side, you'll see the kind of different extensions you can get.
So instrumentation is basically like this is the thing I want to monitor.
And it could be like ASCII or Async Postgres, for example, Database, Celery, Django, Elasticsearch, Flask.
Like there's a stack of app stacks that you can just drop it into and it will give you all the tracing information.
And then there's these things called exporters, which is basically like once it's got the information, it can send it to somewhere like Datadog or New Relic or Azure and AWS, obviously, and Google monitoring as well.
And yeah, actually, I just worked on recently, if you just want to hack around with it, there's an exporter for rich that just basically prints it on the console.
So you can see everything that's happening.
In Keller, probably, yeah.
Yeah, yeah, yeah.
So it's all kind of color-coded.
It's really nice, actually.
So yeah, I'm really excited about this.
I've been mostly trying it with FastAPI as there aren't really many frameworks for setting up like decent monitoring and tracing in FastAPI applications.
And yeah, I think it's really promising.
So I suggest you check it out.
And if you see a framework that needs support or something, then this is all open source and they're all accepting contributions as well.
And it's fairly straightforward to add support.
Yeah, it's got Postgres, MySQL, MongoDB, Pyramid, Redis, all sorts of good stuff in here.
Another thing maybe worth pointing out here is because this crosses languages, right?
There's a Python one, but there's also a .NET one.
There's a Swift one and so on, which means there might be scenarios where I've got like, say, a mobile app written in Swift.
And then I've got the backend written in Python and FastAPI or something.
And you want to put those together.
Because it goes across those languages, theoretically, that's a thing that could happen.
Absolutely, yeah.
And you can pull that all together.
And it would give a request a trace ID.
So when a request comes into the front end, a trace ID could carry across the different stacks as well, which is pretty cool.
Yeah, very cool.
This is neat.
Awesome.
Thanks for covering it.
Now, before we move on, Brian, we have a sponsor for this episode.
That's cool.
Yay.
Thanks to Shortcut.
Shortcut, formerly known as Clubhouse.
So they're a really cool project management tool.
And they ask the question, have you ever really been happy with project management?
You know, how's your Jira or whatever, right?
How much are you loving it?
So they basically say most are either way too simple for growing engineering teams to manage everything or too complex and just throw in the kitchen sink.
And you don't want to work with it.
You've got to constantly tweak it to make it work for you.
So Shortcut, who used to be known as Clubhouse, is different.
They try to be simple.
It's project management built specifically for software teams.
It's fast, intuitive, flexible, many other nice positive adjectives.
So some of the highlights are team-based workflows.
Individual teams can use shortcuts, default workflows, or customize them to match the way they work.
Also, organizational-wide goals and roadmaps.
So these workflows automatically get tied into larger goals and feed into like a bigger system outside the team.
Good source control integration, GitHub, GitLab, Bitbucket, all those types of things.
One thing that I really love is the web app has hotkeys.
So it's keyboard-friendly, just like HR and VS Code, whatever, right?
I don't know why more web apps don't have hotkeys.
It's not particularly hard, but they do, which is great.
Iteration planning, so you can set your priorities and let Shortcut run the schedule.
You get nice little burndown charts and so on.
So check them out at shortcut.com slash pythonbytes, shortcut.com slash pythonbytes, because you shouldn't have to project manage your project management.
That does not sound fun.
So let them do it.
It's their job.
Now, before we move off to the next topic, Robert Robinson on the audience.
Hey, Robert, this open telemetry sounds interesting.
He wants to try it out.
I do as well.
I feel like this is the kind of stuff that you just keep putting off, integrating into your system.
And then once you finally do, you're like, oh, look how awesome this is.
We can see what's going on.
And it's actually, did you know this part was crashing?
No, I didn't know that.
Nobody looked at the log and it was just eating the exception, right?
Yeah.
Tricky, tricky.
All right.
Brian, you got the next one.
So Python's got a few built-ins.
Not a ton, but quite a few.
So there's an article from Tushar Sadwani called Understanding All of Python Through Its Built-Ins.
And it's a pretty, like, he's got a pretty ambitious goal here to understand everything.
But I actually kind of really enjoyed even the first part of it.
So I started reading it.
I've been especially giving it a shot.
I got a shout out to him.
He's been fairly involved on Twitter, answering questions and being involved in conversations.
So that's a good way to get noticed.
But there's a, there's, there's a, starts off talking about scope.
So what is built-ins are not just things that Python has built in, but there's also, it has
a relevance to the scoping rules.
And he called it the LEGB scoping rules.
So it's when Python, if Python sees a symbol, first it looks in the local scope, then the
enclosing scope and global scope, and then the built-in and built-ins really are just anything
that's in the built-in package.
So, and that actually that discussion, it's a really pretty good discussion and it helps
it kind of, it's good for especially newbies to, to understand, but even advanced beginners
sometimes don't quite understand what's going on here.
Yeah.
Brian and Anthony, you both come from C style languages historically, right?
Or at least I've spent a lot of time there, right?
Brian, do a lot of C++.
Anthony, I know you've done some C# and stuff.
Did the scoping story of Python confuse you and kind of leave you a little uncertain in
the beginning?
Yes, definitely.
Especially coming from C++ where it's, it's very well-defined and different.
If it's in the curly braces, it's alive afterwards.
It's gone, right?
Like, wait a minute.
That's not the story at all.
Right.
And also you've got so many nested curly braces.
It could be anywhere and it's not really, there's, it seems like, actually we just
don't do that too much in Python, but Anthony probably know better than me.
If I've gotten like multiple nested, name or curly braces, we don't have curly braces,
but multiple nested indentations.
Uh, does the scope sort of look in outer and outer and outer ones?
Is that what non-local means?
Uh, I, non-local, there's a non-local keyword, which is like a whole other thing.
That's a completely different thing.
Okay.
I think I've every, I don't know.
Closure, variable capture, basically.
Yeah.
But the, the, the, the difference in global really freaked me out because really we were
pounded into our heads everywhere is to never use global variables, but global is different.
The global namespace is not a global variable.
It's more like a module level.
Yeah.
Or like, like a static variable in a class maybe would be what, what other people might
call it.
Yeah.
It's not a dangerous thing in Python.
So, yeah.
So I didn't mean to derail you that much, but I think it's interesting to think about the
built-in scope, the global scope, these different scopes, because it's such a different world
from the intuition you get coming from all the C languages.
Yeah.
Um, also just sort of just really enjoyed looking at the language through the scope of built-ins.
It's an interesting take on it.
Um, one of the, I will pull out a few things that, that he, he mentions.
And one is all the constants.
Um, I guess I'd never counted them before, but there's five, there's five constants in
Python, true, false, none, ellipsis and not implemented.
Um, I do like ellipsis.
Uh, we talked about that the other day, or I guess, one or two weeks ago using dot,
dot, dot, instead of pass.
Are you going to start doing that?
I've already started doing that.
Have you?
I'm all about, I think I'm, I'm up for it as well.
Um, I, I don't, I guess I, I don't think I've ever used not implemented or even looked for
it, but.
Interesting discussion.
Um, also just like, I like looking around.
So here's a section on compile exec and eval.
It's not an alphabetical listing of everything.
It's a more grouping them together.
Um, it's quite an, a big article, but I would suggest people just like skim through
the list.
Cause it's got a good table of contents at the top and you can just sort of, skim
through what, what he's talking about and pick a couple and go read about it.
You'll probably learn something.
So, anyway, good shout out to Tushar for writing this.
Yeah.
This looks super handy.
Yeah.
Some of the built-ins are super handy.
Um, I often have a Python repo open just to do, things that would otherwise be annoying
to do on a calculator, like converting hex, integers and vice versa.
There's a hex built in, which is really helpful actually, doing this stuff.
Yeah.
I use hex a lot because I'm often, looking at, looking at data elements in a, in a,
uh, a packet or something like that and trying to convert those.
So, yeah.
Very nice.
Nice one.
Before we move on, Anthony, how do you feel about dot, dot, dot?
They should have called it yada, yada, yada.
Um, yeah, I think that would be, it's way better than ellipses.
Come on.
Yeah.
I use it for type stubs, and that's it.
So, yeah.
There's times we use pass, right?
And I feel like, you know what?
Dot, dot, dot kind of says, I kind of know what I'm not ready for stuff here yet.
I think we should start calling instead of ellipses, we should call it dun, dun, dun.
Exactly.
All right.
How about we hand out some awards?
Okay.
Best open source software of 2021.
Now who gets to vote on this?
Who gets to say, well, info world in this example.
So this is according to info world, but there may be other rules, but I've, I've found this
pretty interesting.
Actually.
I heard about it, learned about it because Sebastian Ramirez from FastAPI said, yay, we've
been voted one of the best open source projects.
So this is called the info world bossy 2021 awards.
But what I thought was interesting is going through here, there was 30 different projects
that won awards.
I'm like, oh, that's interesting.
Oh, I didn't know about that.
Oh, check this out.
Yeah.
So I wanted to touch on a couple.
So there's some things that may or may not be interesting to you like Svelte, which is a
JavaScript front end like viewer react.
That's not interesting to me.
But Minikube, Minikube is pretty interesting.
Minikube is a way to run like a baby Kubernetes cluster right on your computer.
Just say Minikube start.
And guess what?
You've got a cool little cluster running.
So that might be really helpful for Python people.
Let's see.
Pixie.
Zoom back a little here.
Number five is FastAPI.
Now we're all fans of FastAPI.
I think it's really awesome that it won.
And we're maybe giving a quick shout out to how they described it as Django and Flask have
been leading the Python web frameworks for years.
FastAPI now deserves to be mentioned in the same breath.
I agree.
Calls out the main features, which are it's truly modern, truly modern Python web framework
written from the ground up using type hinting, async and high speed components by default.
That's true.
And I also really like that they pointed out that while its name indicates it's primarily
for APIs, it's also really good at writing more conventional websites with like Jinja
templates or even chameleon templates.
So way to go.
Anthony, you want to add it or Brian want to add anything?
Well, I just think that I think you're partly to thank for people considering FastAPI for
not just APIs, because you've been beating that drum a little bit as well.
Yeah, thanks a bunch.
I even created some decorators that make it really easy to like render templates as response
values and stuff.
Yeah, that's fun.
I tried out the chameleon thing.
The one you wrote, actually.
Yeah, because I'm working on this FastAPI course with you at the moment.
Yeah, that's going to be fun.
So yeah, I'm a big fan of FastAPI.
I think it's brilliant.
And testament to Sebastian, really, because he really kind of builds on something which is
quite complicated, but he makes it seem so effortless.
And just working with FastAPI, like the documentation is excellent.
The framework itself is just is really logical.
And, you know, it's really easy to use that in terms of like the I've been keeping an eye
on the popularity of the different frameworks and stuff over the last few years.
And Django and Flask are kind of neck and neck and have been for a while.
And FastAPI now is the third most popular, according to the metrics that I've seen.
Yeah.
Out of nowhere to third most popular.
Yeah.
And I know JetBrains are doing the latest PSF developer survey.
So yeah, we'll see kind of what happens in this year's number.
But I'd imagine FastAPI would still be the third most popular.
So yeah, it's brilliant.
So yeah, I think it's a good solid pick.
In terms of writing like full apps with it at the moment, like there's still a lot you have
to do for templating.
Like you pretty much have to like build in a whole bunch of other templating stuff and
picking an ORM at the moment isn't easy, but there are some brilliant ones to have a play
with.
Yeah.
There's a couple interesting ones.
I want to give a shout out to.
Yeah.
That give like some are even integrating with Pydantic, which is sort of the natural exchange
of FastAPI.
So you want to give a shout out to Tortoise, you say?
Yeah.
That's my favorite so far.
I've used, tried out six different ones so far.
Yeah.
And Tortoise, I think it's my favorite at the moment.
So right on.
Well, maybe next year we'll be talking about the award for SQL model, which is built on
top of Pydantic plus SQLAlchemy by Sebastian as well.
So who knows?
A lot of good ones out there.
A lot.
It's good to see a lot of the excitement and new ideas coming along there.
All right.
What else we got?
Crystal.
Don't care.
Windows terminal.
I think is actually pretty interesting.
Windows has traditionally been not on par with its terminal experience.
And I think, you know, the windows terminal, PowerShell seven, all my posh, all these things
come together, nerd fonts to make it like quite an amazing place to be.
Actually.
Windows terminal is an open source project.
It didn't start out that way, but now it is.
Yeah.
Okay.
Yeah.
So that's a good one.
OBS studio.
If you're doing video stuff, that's amazing.
There's a bunch of stuff in here that may apply to people that you can all check out that
are interesting, but I don't want to cover them.
BASC though.
BASC is a big data science one.
Scale computation, like pandas operations and whatnot.
Across cores, across clusters, across compute that's larger than the RAM you have by streaming
it off disk and all sorts of interesting stuff.
I have no idea why my browser is jumping up and down.
We'll have to ignore that.
I'm not in control of it.
I'm sorry.
It seems like.
I, you know, I'll tell you why this is happening.
I'm, I'm looking up and I see I'm not running my VPN, which would block ads.
And so there's some kind of ad off, off the screen that's just running.
And if I turn on my VPN, we'd be good.
All right.
Blazing SQL is another great one.
Rapids from NVIDIA.
And I feel like there's one more I want to give a shout out to.
Hugging face.
I don't know anything about that.
Now that was it.
So I just going through that list.
I thought it called out a lot of neat projects in addition to just FastAPI.
Yeah.
Cool.
Yeah.
Any, any of those jump out at you guys either that I've just screened by?
Lots of, lots of stacks that I don't use.
So same.
Yeah.
There was a bunch of ML stuff though, which I don't use, but I think would be relevant
to people who are listening.
Maybe.
Well, we're not to extras yet, Michael.
No, no, I know.
I just closed it because the jumping was driving me insane.
Okay.
All right.
Anthony, you've got the last main one, right?
All right.
Yeah.
So I think Lukash is taking up like half of this episode.
So we're going to get back to Lukash's blog and evolve the discussion that was started
last week on this discussion.
Yeah.
I'm, let's put it mildly.
I'm excited about this.
I think if, if this happens, it's probably going to be the biggest thing to happen in
CPyton in the last five years, in my opinion.
And this being the GIL removal.
This be the GIL removal.
But not the gillectomy.
Not the gillectomy.
Not exactly.
Yeah.
So no GIL or let's just go with no GIL.
Yeah.
No GIL.
So almost seemingly out of nowhere, Sam Gross, who works at Facebook, basically likes
submitted to the core developers, this research paper and a working branch of a gill-less
Python.
And just quickly recap, I guess, on what that means.
This article is pretty heavy in technical detail and the stuff that's, yeah, the stuff that's
being discussed in the article, again, is pretty complicated.
And I actually didn't understand a lot of it.
And I've written a book on the Python compiler.
So if you read this and it's confusing, don't worry.
So the GIL is basically the global interpreter lock.
And it exists as a way of making Python thread safe when it comes to keeping reference counts
of specific objects.
So if you create a Python object, for example, there's a counter of how many things are referencing it.
Because you don't want to just destroy an object.
And then you're working through a list of objects, for example.
But then one of the items in the list just disappears, has been deallocated.
Because everything is a pointer in Python.
That pointer just goes nowhere.
Or actually, there's a magic pointer that Python uses when it deallocates objects, which I know from a very painful experience.
So you don't want that to happen.
And if you've got multiple threads kind of working with the same objects all at once, you don't want them to...
It's incredibly hard to keep track of what's happening.
Threading is great because you can have multiple threads working on a computer.
And the operating system can do the scheduling of which threads run on which cores and which CPUs, etc.
So in theory, it's a way of making your Python applications a lot faster if you write them to be multi-threaded.
But Python is basically built in this lock, which says, okay, in the evaluation loop in C eval,
don't let anyone else run an instruction whilst this thread is running the instruction.
With the exception of IO-bound tasks.
Yeah.
And it seems like this is a thing to control threading.
And really, it's just a thing to protect memory management.
But it has this huge blocking effect for threading, right?
Yeah.
So it's a thing to basically make the reference counter thread safe.
Without locking.
So it's fast.
Without locking.
Yeah.
So you don't have to wait to add an income.
So to give you an idea, like if you run the GC by hand, you'll just see how many tens of thousands of objects are just created,
like all the time in Python applications.
So what Sam had put together, I say seemingly out of nowhere, but if you go through the article and what he proposed,
he's actually been working on this almost full time for two years, which is astonishing.
And it's a real feat of engineering, to be honest.
So kind of what he's proposed is a way of removing the GIL so that there's essentially like almost two ways of keeping references into objects.
And one of them is specific to the local thread.
And then there's also another reference count, which is for other threads.
So why is that important?
Well, let's say, for example, you've got a Python dictionary with values in it, and then you have multiple threads or working on the same dictionary.
Like that's a complicated problem to solve.
How do you make sure that the references to the keys or the values don't disappear?
And he does actually go into detail about how that's been handled.
And also objects like Python dictionaries are not thread safe at the moment either.
So if you have two threads working on a dictionary, adding values, for example, to a dictionary, do you have to lock the hash table?
Anyone who's worked with multi-threading in low-level languages knows that the complexities of doing this.
So what he's proposing is that in his prototype, he basically replaced the Python memory allocator with another one called Mimalloc,
which is a sort of thread-safe memory allocator.
It's actually a Microsoft project.
But I think it could have been any other thread-safe memory allocator.
Writing memory allocators is very involved for them to be performant and efficient.
And then basically objects get tied to the thread that created them.
And then there's a non-atomic local reference count with the owner thread.
And then there's basically a separate mechanism for what would be slower, basically, reference counting from other threads.
So single-threaded performance is equivalent with this proposal.
But there's still a performance impact of multiple threads working on the same object, which is to be expected.
There's always a little overhead for that.
Yeah.
But to give you an idea, in his note, he implemented a few common problems as a multi-threaded implementation.
And he said, if you give it 20 threads, it runs 19.84 times faster than it would in just regular CPython.
So for certain types of problems, this can have enormous impact in performance.
But it is really complicated.
And that's why I think it's an interesting discussion to see, okay, how do we get from this is a cool idea to this actually being released and being used by millions of people?
And I don't know, Python's like running on satellites in space and stuff.
Like, how do we go from a fork that someone's been hacking around with to something that's like production ready?
And this is kind of what the article goes into.
So like, you know, how would this work?
Would it be a feature flag?
Which version would we target?
And so at the moment, it's targeting 3.9 alpha 3, actually.
So it wasn't even the release of 3.9.
So he needs to do some work to update that to the latest version of 3.9, which is 3.9.7.
And then I think the target release, if the core developers agreed to kind of like explore this, if that was 3.11.
Or I don't think anyone wants to touch the Python 4 topic.
But 11 is like a year away.
Is that even possible?
Or would it most likely be a couple years out?
Yeah, it seems pretty soon to me.
And I like sub-interpreters, for example, is like an experimental feature.
I think the issue with this is that it's that the volume of changes is so broad that it's quite hard to kind of like have it in as a feature toggle.
So like sub-interpreters was in as like a hidden package that you can use and it's experimental.
Whereas this is like changing...
Everything.
Yeah.
Well, not everything.
But like it's a pretty wide sweeping change.
And changing the memory allocator is a massive change.
So the question is more, how can we introduce this softly, I think, and have it either as a feature flag?
And what would this break?
And the main thing is that C extensions haven't really had to worry about thread safety because the GIL kind of handles that for them.
So C extensions essentially would need to...
If they use the mechanisms that are here, that's fine.
But C extensions often have other objects which they haven't used the reference counter for.
So they've basically kind of like allocated their own objects and variables and stuff like that that would not be thread safe.
And they had just not ever had these kind of collision issues in the past.
So introducing this would potentially break some C extensions.
So, you know, how could that be introduced gently?
I think what was interesting in the article is there's a mention of NumPy.
And NumPy has actually done a lot of its own work already on basically kind of making it thread safe and more scalable.
But one of the tricky ones is PyBind 11 is called out in here as being anyone who's using PyBind 11 potentially might have to do some refactoring to support this if it was supported.
And then in closing, Lukash, who wrote this review of posts, sort of said, you know, the team have been really impressed with Sam's work and invited him to join CPython Project as a core developer.
And he's interested and Lukash is going to mentor him.
So I think that's brilliant.
Like, oh, yeah, that's brilliant.
Just to come up with this over like even two years is like a really short amount of time for a problem that people have been trying to solve for well over a decade.
So, yeah, very exciting.
Yeah, this is great.
I think we have a record number of core developers in the audience right now.
Yeah.
So some great comments from Steve Dower.
Hey, Steve.
The big thing needed here is a path forward for native extensions.
They could all need rewriting or else importing them could re-enable the GIL.
That discussion is happening now.
It's very early.
And Henry Schreiner also has similar comments that they're considering that.
But yeah.
And Henry also says we would be up for refactoring Pybind 11 if needed, I believe.
This is also interesting.
But this is exciting.
There's a lot of stuff coming here.
I think another thing in addition to the no GIL is I got the sense that Sam had added several other optimizations that were independently worth adding to Python.
Yeah.
One of the things I know that there's a lot of tension around whether or not to do 4.0.
But if it ends up being that all of the extensions need possibly tweaked, then that might be, then it's an API change.
And I think a shift to 4.0 might not be terrible.
It would be the right number.
Yeah.
Well, we should just go to Python 5.0 so no one's worried about 4.0 and we'll skip the whole conversation.
It'll be fine.
We'll do an AngularJS.
We'll just make a big fuss about going from 1 to 2 and then just all of a sudden they're on version 10 or something.
Yeah.
We'll just go crazy.
Yeah.
That's fantastic.
I'm actually having Guido Van Rossum and Mark Shannon, I believe, on Monday on Talk Python to talk about performance in the future and stuff.
And I'm sure we'll talk about this stuff a little bit.
Yeah.
So it should be a lot of fun.
This was Guido's suggestion when I asked internally if anyone wanted to share anything.
This is what he sent over.
Okay.
Fantastic.
Yeah.
So I'll try to take that up with him again.
All right.
Well, Brian, does that bring us to our extras?
We are at extras.
Do you have any extras?
Yeah.
No, you go first.
Tell us about PyCon.
It's a conference, right?
Call for Proposals is open for US PyCon.
I'm pretty excited about that.
I already wrote down like six ideas of things I might want to talk about.
So and of course, there's no guarantee.
No matter who you are, there's no guarantee that you're going to get in.
But it's fun.
It's fun to come up with proposals anyway.
And it's fun.
I'm definitely going.
So I'm pretty excited about that.
And anybody else going to propose?
Anthony, you going to try to talk there?
Yeah, I've been thinking about that, what I'm going to put forward.
I want to put together a talk on performance anti-patterns.
Oh, that'd be fun.
Propose that for next year.
Yeah.
Because of your name?
Like anti-patterns?
Also.
We're not on the direct section yet, Brian.
If anybody doesn't know, I wrote a book and then I rewrote it.
And I'm finished with it, actually.
So it's not out yet, but I'm pretty excited that I'm finished.
All the betas, there's beta seven out, has all chapters in it.
So if you're waiting for it to be done, it's done.
It's not in print form yet.
That's going to happen in January or February.
So I'm pretty excited to get that done.
I'm hoping for my copy at PyCon, Brian.
I'm pretty sure I paid for the last one as well.
I actually, I paid you in cash.
So I'm going to give you a copy of my book.
I'll bring, I'll bring at least.
Maybe we can do a swap.
Swap.
Yeah.
That'd be great.
Yeah.
Anthony, I got your book over there.
I'm not sure what I can trade it for though.
That's awesome.
Congratulations, Brian.
Thanks.
Anthony, you got any extras you want to share?
Yeah, I'll be shipping fairly soon.
The JIT compiler that I've been working called Pigeon.
I'll be going version one in two weeks.
So it's a Python 3.10 JIT compiler.
It's a, you basically just drop it into CPython and turn it on and then run your code and it
just JIT compiles it in the background.
And in some cases makes it a lot faster.
In other cases makes no difference.
But yeah, some of the benchmarks I've been doing, like floating point math and integer
math, like makes a massive difference.
So yeah, like 60%.
Like the scientific side of thing, right?
Yeah.
So stuff that you would otherwise think, oh, I'm going to redo this in Cython or something
like that.
You don't have to add all the extra stuff.
You just kind of turn it on.
Yeah.
And body benchmarks now 60% faster than standard CPython.
That's great.
And yeah, some of the other benchmarks I've got are 60% upwards.
That's super cool.
So this work with Sam and the NoGill, does that throw a spinner in the works or a spinner in
the works?
It would make my life quite hard for a few weeks if it gets merged.
Yeah.
So yeah, that could be interesting.
And I'm also working on another secret project, but I'll share that in a few weeks.
Yeah.
Pigeon does, there's a comment in the chat.
Pigeon does use scikit build, which I did want to call that out when we were talking about
setup.py earlier because, yeah, so Pigeon uses, is all C++ and it uses CMake, which generates
make files.
So yeah, and it uses scikit build, which is a CMake extension, I guess, around Python extension
modules.
So that's how it kind of compiles.
It's really cool.
Big fan of scikit build.
Yeah.
And I recommended using build earlier.
Henry, on our episode together, mentioned that if you have external non-Python code like
C code or Fortran or whatever, then instead of build, scikit build would be a good option
to build the binary bits for that.
This is the other question I wanted to ask, and Steve Dower beat me to it.
He states it as an assertion.
I was going to ask you the question.
I bet once Pigeon ships, you'll get people interested in helping add optimization.
Yeah, so it's one thing to JIT compile.
It's another to just then straight up run it versus go, oh, we can inline this method.
Oh, and I see we can do this.
And then we could actually reuse this field because it's not used below and early free,
all that kind of stuff.
Where's the optimization of that look?
Yeah, I've got it on the documentation page as an optimization section, and I've kind of
written up a lot of the optimizations and how they work, assertions that they make
and compromises and stuff like that.
So yeah, if you're interested, there's some info on there.
But yeah, I'd love more help on this.
The learning curve on the project is quite steep, but I'm trying to make it easier.
I mean, it is a compiler, so like, yeah.
And I just added ARM support as well.
So M1, Apple M1.
And I tested Linux ARM 64 and, in theory, Windows ARM.
But I don't have access to any machines to test the Windows one.
And I could only test the Apple one remotely.
If you need a periodic test, you can reach out.
I got Windows 11 running on ARM.
Oh, really?
Okay.
Yeah.
Yeah, maybe I'll take you up on that.
Right on.
Very cool.
All right.
I have a couple throughout there as well.
Python Software Foundation on Twitter, the PSF analysis.
We're happy to announce the Python Developer Survey 2021.
Take part in it.
This is the one that is then hosted, and then the data analysis is done by JetBrains, but
not influenced by JetBrains.
So I'll link to that in the show notes.
Be sure to get out there and take that.
Henry, out in the audience, I have something as well, the feature for what you said the other
day on Twitter.
I said, after Python's bytes mentioned on yesterday's show, I asked for a new feature, and it's
already in pip.
PipX.
PipX run PyPI command line wheels.
Basically, this is added to PyPI command line, and it'll tell you all sorts of cool stuff,
like the details of the wheel.
So you could run pipX, basically run pyPI-command-line wheels numpy, however you run that.
And it'll tell you, like, for numpy on macOS 10 universal, does it have a signature?
Is there a binary distribution?
What versions are updated?
Supported?
How old is it?
How big is it?
Same thing for Linux architecture, ARM on Windows, and so on, so on.
So you get, like, just this cool graph using rich of, like, tables of tables telling you
about the status of wheels on different platforms straight out of PyPI, which I thought was cool.
Nice.
Yeah.
So that's pretty good.
So, Henry, thanks for making that happen.
Also, on the last episode, out on the YouTube not live comments, we got a message from, I want
to make sure I give it the attribution, from Bahram, and said, we talked about, what is it?
T-Bump.
That was it.
T-Bump for bumping the versions.
He said, oh, that's cool.
I use bump to version, which is another option to do some similar types of things.
I could work with or without source control, all kinds of stuff.
So, fun one to check out.
And, Brian, you sound really good this time.
Like, last time I thought maybe a bee had gotten into your microphone.
What was the story of that?
It's a long story.
Basically, I had to throw a mic.
So, I had a bad mic and a bad cable.
But I have a new XLR cable.
It's tough when the two things that are connected together are both broken at the same time.
The buzzing, I think, was definitely my cable.
I think there was a feedback thing going on.
Were you getting an SMS?
And then I was examining everything in my audio chain and just got rid of the stuff that wasn't
working.
Yeah.
You sound great.
New mics, even better than before.
So, like a phoenix, you're back.
Nice image, too.
Yeah.
And then, have you got your Mac Pro yet?
No, I just bought a Mac a couple years ago.
I'm not going to buy another one right now.
Anthony, are you using one of these to test your own version?
No.
I don't have a spare $4,000 for another laptop.
And also, I was like, I don't really need a laptop because I never leave the house.
So, like.
Yeah, that is a big problem.
I mean, I am so loving my Mac Mini and my 4K monitor that I'm just like, I don't want
to leave.
I don't want to leave.
All right.
Well, that's it for the extras.
I think it's time for a joke.
Yeah.
Maybe Robert's got the first one out there.
Can't complain about Brian.
It's all about the hair.
You got to see the live stream for that one.
But yeah, I agree with that.
Next Halloween, I want to go as cousin it.
So, I got a ways to go.
Anthony, are you up for doing this joke?
Yeah.
I got it on the, it's got it on my screen.
Oh, you got it on yours?
Yeah.
I'll put yours.
All right.
Okay.
So, I got it.
It's a picture.
So, I'll have to describe it.
I couldn't stop laughing at this when I saw it.
So, this is Frodo explaining to Gollum.
And there's Gollum sitting at a computer looking quite confused, looking at a picture of the
ring.
And it says, buy now one ETH.
As in Ethereum, right?
Yeah.
As in Ethereum.
And Frodo is basically trying to convince Gollum to buy an NFT of the ring instead of actually
having the ring.
And underneath.
My digital precious.
So, underneath it says, so you can't own the precious physically, but you can pay to have
your name listed as its owner in an online distributed database.
It's only, what is that, like 400 US dollars, 500 Australian, something like that.
I don't know.
That's a lot for a listing.
I don't own any NFTs yet, nor have I sold any.
I don't plan to either.
Man, I feel like we're missing an opportunity to brand some of our former episodes.
Maybe like, I could just take screenshots of Brian laughing at different times out of the
live stream and then like turn it into a stream of NFTs that we'll retire upon.
Oh yeah.
Let's, let's do that.
Yeah.
Oh, fantastic.
All right.
Oh, that was a good one.
Thanks, Anthony.
And thanks for being here on this big episode 256.
Yeah.
I feel like we've maybe gone slightly over to this.
This is not really a Python bite this week.
It's more of a Python lunch.
Sandwich.
Yeah.
It's a proper meal, a Python dinner, but it was a good one.
We talked about a lot of stuff and a bunch of great people in the audience gave us like
really good inside information on where things are going.
So.
Yeah.
So thanks everyone.
It's everyone.
Yeah.
Thanks Brian.
Yeah.
All right.
Bye y'all.