Transcript #245: Fire up your Python time machine (and test some code)

00:00 Hey there, thanks for listening.

00:01 Before we jump into this episode, I just want to remind you that this episode is brought to you by us over at TalkBython Training and Brian through his pytest book.

00:10 So if you want to get hands on and learn something with Python, be sure to consider our courses over at TalkBython Training.

00:16 Visit them via pythonbytes.fm/courses.

00:20 And if you're looking to do testing and get better with pytest, check out Brian's book at pythonbytes.fm/pytest.

00:27 Enjoy the episode.

00:28 Welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.

00:34 This is episode 245, so it's not the first time. Recorded August 4th, 2021. I'm Brian Okken.

00:41 I'm Michael Kennedy.

00:42 I'm Juanpe.

00:42 So Juanpe, thanks so much for coming on the show. Can you introduce yourself a little bit before we get into it?

00:50 Thank you very much for having me. So my name is Juanpe. I'm from Spain, and my PhD in particle of Physics, I've been working at CERN for four years.

00:59 Then two years after my PhD finished, I decided to step away from academia and start working in industry.

01:06 And right now I'm working at FinancialForce, which is a company that develops products for Salesforce.

01:11 So I'm not developing products, I'm in the analytics team.

01:14 So my job is to analyze internal data, as well as usage, product usage from a customer to allow the board to take data-driven decisions on how the company should go forward.

01:25 - Nice. - Yeah, super interesting.

01:27 Give us your thoughts real quick on one hand, working for a place like CERN, and then the other working on a place that provides enhancements to Salesforce.

01:36 Like those sounds so different.

01:37 Are they really that different or are they similar or what's the story?

01:40 - Part, I mean, of course they're different, but there is a big part which is very much the same, at least in the team that I'm working on.

01:47 Because at CERN, basically what you do is you don't know the answer to anything and you need to first know what you need to ask yourself.

01:54 And this is very similar to what happens today in my current job, because for instance, marketing come and say, we have this whatever campaign and we want to know if we are targeting right.

02:07 And I need to know what do I need to do to answer that question, but neither marketing knows.

02:12 So it's like, let's figure things out.

02:15 But yeah, I mean, it's a pretty drastic change, but I don't know, I got the feeling that I needed to switch.

02:23 I like coding a lot and I felt at some point that I was enjoying more the coding part of being a physicist than the physics part.

02:31 So I said, I mean--

02:32 - I mean, you basically described why I'm not in math anymore.

02:35 (laughing)

02:37 - Yeah.

02:38 - I was working on projects and I was having so much fun and writing code on these Silicon graphics, like huge computers and stuff.

02:44 And then there'd be parts where I'd be like, "Ah, this part's not so fun.

02:47 This part's amazing." And I realized the programming parts were amazing.

02:50 And then when it had to get down to solving the math bits, I'm like, "Ah, darn, gotta put it away.

02:54 "Go work on the math again." - I mean, I remember the last year and a half I was working on a project that was literally designing a system that worked within GitLab CI to automate paper review and publishing.

03:08 So you don't need to have a lot of people reading the paper and say, "Oh, this rule is not matched "or you need to fix this image." So I built an entire pipeline in Python to check all of this that works based on pull requests and groups and so on in GitLab CI.

03:22 And I thought, I've been a year and a half not doing almost any physics.

03:26 So my physics work, it was related to review paper because I was a chair of an editorial board.

03:32 So I had an analysis, it was pretty cool, but I wasn't doing it.

03:36 I received version, reviewed them, made comment, fixed this, fixed that, then go back to write a pipeline to make the pipeline.

03:43 - Yeah, exactly.

03:44 Yeah, it sounds really cool.

03:45 - To Brown. - Yeah, go ahead.

03:47 - Will you kick us off today?

03:49 - Will, you wanna hear about the state of the developer world?

03:52 How's that sound?

03:53 - I like state.

03:54 - Yeah, yeah.

03:55 So here's an interesting survey results, I guess, put together by JetBrains, the state of the developer ecosystem 2021.

04:02 And I thought this would be fun to talk about because we do cover like the PSF state of Python survey and things like that.

04:11 But I thought it'd be fun to just have a quick look at the broader landscape, what people are doing and where Python fits into that.

04:17 JetBrains has done a really good job with a PSF survey.

04:20 So I thought, you know, this will be similar.

04:22 So let's check that out.

04:23 So let me give you some stats and some rundown on this.

04:27 So basically the idea is it presents the results of the fifth annual developer ecosystem survey conducted by JetBrains, and it went out to 31,000 or had input from 31,000, 32,000 people.

04:39 All right, so there's a couple of interesting things they're doing in the presentation here, but let's say in that world, JavaScript is still the most popular language.

04:47 Not a few as stack overflow, but of those 32,000 people or whatever, Python's more popular than Java overall.

04:54 However, Java is used more as the main language.

04:58 So there's more people using Python for extra things or for other things and so on, which I think that jives pretty well with my understanding of Python is that it's this really amazing way to like bring in a little interactivity, bring and a little bit of analysis, a little Jupyter notebook or something, even if it's not your main focus, right?

05:16 You might be an economist, you're not even a programmer, but you're still a Python person in a sense, whereas you probably wouldn't be a Java person as an economist most of the time.

05:25 - Yeah, I use Python for testing.

05:28 - Yeah, for sure.

05:29 So let's see, the top five languages that developers are planning to adopt are Go, Kotlin, TypeScript, Python, and Rust.

05:38 And the fastest growing languages are Python, TypeScript, Kotlin, SQL, and Go.

05:42 So for example, JavaScript, was the most popular language.

05:47 Java was the most popular main language, but it's, they're neither the fastest growing language.

05:51 So that's pretty much a pretty interesting of this group.

05:54 71% people work on some sort of web backend APIs, Flask apps, or, you know, Go apps or whatever.

06:02 So they have a lot of interesting stuff here in terms of analysis.

06:07 So they have these blocks that show how popular a language is, it's been used, or how likely people are to adopt it and pick it up.

06:16 So there's a bunch of grids if you go to the report and you can check them out.

06:19 And basically the orange is the current state of the world and there's a darker, almost black, that is like the derivative.

06:27 Like how quickly is that world changing for the upswing?

06:31 So for example, JavaScript has more orange squares, but it doesn't have as quick of a growth or a planned growth, I guess.

06:39 So Python has one of the larger ones of those.

06:41 So does TypeScript as well.

06:43 And those are interesting to look into.

06:44 You can compare those against different things.

06:46 You can also see the popularity of the language over time.

06:49 Python's been going up and up and up, although this year is kind of plateaued in this report.

06:53 So that's maybe something worth noting.

06:57 There's obvious things going down like Objective-C.

06:59 You'd be insane to work on Objective-C right now when Swift is like replaced it, although that's going down as well.

07:05 Let's see, there's a few more things.

07:07 They have these really interesting graphs that are both like grids, but also heat map.

07:11 So it'll let you answer questions like, okay, if I am currently a Swift developer, what is the likelihood that I'm going to adopt Python?

07:19 6%.

07:20 But if I'm a Kotlin developer, that's 8% likelihood that I'm going to adopt.

07:25 Oh no, I'm going the wrong way.

07:27 If I'm a Kotlin developer, I'm 10% likely to adopt, to move to Python and so on.

07:33 So there's a lot of sort of like flow from one language to another.

07:36 I haven't seen any analysis like this anywhere else.

07:39 Have you?

07:39 No, that's pretty interesting.

07:40 My hands looking at correlation or something.

07:42 What's the first row?

07:43 I'm curious.

07:44 I'm not planning on changing.

07:46 So they are the most likely to change.

07:48 Yeah.

07:50 There's just stock.

07:51 There's days.

07:51 Yeah.

07:52 All right.

07:52 Let's see.

07:53 also interesting operating systems people use for development.

07:56 61% windows, 47% Linux, 44% Mac iOS, which that's pretty high for macOS given its general popularity amongst like the computing world, I think.

08:06 Yeah, I think Linux is pretty high too.

08:09 Yeah.

08:09 This doesn't surprise me.

08:10 Yeah, exactly.

08:11 And then 1% other, who knows what that is?

08:14 Also questions about people using the windows subsystem for Linux and stuff like that.

08:19 there's also, if you're interested, a similar heat map for like what type of software do you develop?

08:25 So if you're trying to understand where, like if I develop this kind of software, what is the distribution for programming languages there, right?

08:33 Like it's interesting to say Python is popular or JavaScript is popular, but if I'm an embedded system developer, is JavaScript still popular?

08:41 I don't know. Probably not.

08:42 Maybe, but maybe not.

08:44 Maybe C is really popular.

08:46 So there's a really cool thing called what types of software do you develop?

08:49 There's like a grid plus heat map plus intersection of language and type.

08:53 So if I develop security software, there's a nine percent chance that I would be doing Python versus a 6% chance of Java.

09:02 On the other hand, if I do Blockchain, how does that break down and so on?

09:05 Let's see where is Python notable on utility little scripts.

09:09 It's quite popular there.

09:11 Database backends, pretty popular in that area.

09:16 See another one that maybe is standout, would be programming tools.

09:20 Actually, that's pretty interesting and so on.

09:23 What do you guys think of this?

09:24 >> I think it's weird that there's 39 percent of the C++ developers are developing websites.

09:29 What the heck?

09:30 - Yeah.

09:31 - Yeah, what are they doing back there?

09:32 - Maybe the backend.

09:33 She reported in the middle, I don't know.

09:37 But it's weird, yeah.

09:38 - Yeah, yeah, that is quite interesting.

09:40 - And then you get the standard business intelligence, it makes sense.

09:44 - Yeah, the business intelligence one, that one Python is definitely crushing it there, right?

09:48 It's like 30% versus 10, 15, 20% for the others.

09:51 Yeah, I guess one more, there's all these different things you all can dive into.

09:56 I guess one more area that might be worth addressing is they broke down like the type of coding and software activities you do based on gender.

10:04 So for example, if you're male, like how likely are you to do straight programming versus testing versus user experience type stuff or versus female?

10:13 And let's see, so there were some takeaways.

10:16 It says women are more likely than men to be involved in data analysis, machine learning, UI design and research, but less likely to be directly doing infrastructure development or DevOps.

10:27 But I mean, I kind of had that sense as well, but just from talking to people.

10:30 - I mean, my personal experience is completely the opposite.

10:34 So most of the DevOps people I work with are women.

10:38 - Oh yeah.

10:39 - But I think it kind of makes sense, I mean, in the industry for what I'm seeing.

10:43 But for my, it's completely the opposite.

10:45 - Interesting.

10:46 Yeah, so I'll leave this out here for people to go dive into and explore more.

10:50 I feel like I've gone probably over enough details there to give you all a sense, but there are some interesting things to be learned, I think.

10:56 - Yeah, definitely.

10:57 - Pretty cool.

10:58 - Yeah, and Matt out there in the live stream points out that that might be more than 100%.

11:02 I'm not sure which part you're talking about.

11:03 I do believe a lot of these had multiple, you could check multiple things, like which languages am I willing to adopt?

11:09 Well, I might be adopting both SQL and Python in the next year, something like that.

11:14 - Yeah, and I think a lot of people are perpetually going to start learning Rust or Go, but never starting.

11:22 >> That's true. It's a solid landing goat.

11:24 >> It's not much.

11:25 >> All right. Cornell.

11:28 This was suggested by Yale Mintz.

11:32 Michael, you know where Cornell comes from apparently?

11:35 >> I'm thinking Soundgarden.

11:37 Some good 90s grunge.

11:40 >> Okay. Maybe. Cornell is a record and replay mock server.

11:47 We're going to link to the tool, but also there's an introduction blog post about it.

11:54 It supposedly makes it really simple to record and replay features to perform end-to-end testing in an isolated test environment.

12:04 The gist around it, there's a cool tool called VCRPi, which saves cassette files.

12:12 You send a request and you get replies back, and then you can save those request reply sessions and stuff.

12:19 This is bundling VCRpy with Flask to make a little server.

12:24 It's actually really a cool idea.

12:27 The idea, one of the things around it is that you can do, you're not just mocking one service, you can just mock in any external service that you're dealing with.

12:36 It'll do replays on that.

12:39 One of the benefits over rolling your own mocks or rolling your own test server or test service is that you can, that you don't really have to think about designing the whole thing.

12:50 It just replays everything.

12:52 >> Yeah, that is cool.

12:53 >> Looks pretty fun. I haven't played with it yet, but definitely want to.

12:56 >> Hey, speaking of play with it, click on documentation, I think it is, on that page right there, on the bottom left.

13:01 >> Okay, documentation.

13:03 >> Then click on the documentation of that page.

13:06 There you go. So you have this series of animated GIFs of scene and action and I think that that's kind of cool, right?

13:13 Like you can, it'll go along and say, oh yeah, here you're recording a bunch of API calls and then the workflow of like how you create it.

13:20 I just want to give a shout out to the animated GIFs for like, is this interesting to me?

13:24 Let me just watch the GIFs instead of actually take the time to try to adopt it.

13:28 It's simple but it seems really effective.

13:31 Pompei, what do you think?

13:31 >> Good idea. No, it's a really good idea.

13:33 I mean, there are many projects that you think this might be cool to work with and then you start reading walls of texts and halfway through, I don't know if it's interesting, but I mean, half an hour, so how much I give is eye catching, yeah.

13:48 - Yeah, for sure, for sure.

13:50 Yeah, also I just want a quick shout out to the live stream.

13:53 German points out from his experience, the data analysis have more men than women, but it could be biased due to the tech sector having more men in general.

14:00 I do think that that's true.

14:01 I think what they said is, if you're a woman, what are you more likely to be doing?

14:04 If you're a man, what are you more likely to be doing?

14:07 And it's like, of that population, what areas do you work in?

14:11 Not that user experience has more men or women.

14:13 I don't think it addresses that question.

14:16 I think my thoughts here, there's a lot of women who end up in programming not down the traditional computer science path.

14:24 You know, they go into biology and then they're like, "Oh, I've actually learned a little Python "and now I really like it and I work here and it's amazing." But you know, they kind of get pulled in tangentially where there's a lot of guys that like sign up for computer science and they just go through that path.

14:37 And some of the areas that were called out are more likely to take the straight computer science path people, rather than the, I got interested and I came in through like psychology or something else where there would be more women.

14:49 So, I don't know, I would love to have more women in there, but I think that this is my, and broadly speaking, but I think this is my thoughts about why maybe those different areas seem to attract people not so directly down the computer science path.

15:02 Anyway, yeah.

15:03 All right, Juanpe, you're up.

15:06 Talk to us about the next thing you got here.

15:07 - Sure.

15:09 So I wanna talk about Factory Boy.

15:11 I think it's a very well-known library to basically mock objects when you're running tests.

15:19 And both this and the next tool I'm gonna talk about came because we were working on a system that replicates an entire Salesforce org.

15:28 So we have infrastructure we've built that takes everything you have, every object you have in Salesforce and copy it to a database.

15:35 This is a way we have to have daily snapshot of the data that we can do a time series and analysis and all the models that we have on it instead of being a few minutes, let's say, when you modify it, it's lost.

15:46 So for this, we obviously need to communicate a lot with the API in Salesforce.

15:51 And when you get API responses, you need to not only treat the JSON plainly, say, just the plain JSON object.

16:02 But you would like also to have some kind of object representation.

16:05 And for this, I think it's not news for anyone.

16:07 The Pydantic right now is taking the floor.

16:12 And the biggest issue came when we needed to start writing tests for it.

16:16 Because we get the JSON file.

16:20 We stick it in the Pydantic object.

16:22 It validates everything.

16:23 Everything's beautiful and works fine.

16:25 But then we have a bunch of objects, a bunch of fields on the object that cannot be nulled, for instance, or they are not optional.

16:31 So they need to come in the API and we need to validate for those because if the API does not return any of those, it should break and tell us, look, this is wrong.

16:38 It's not what you expected.

16:39 So when we wanted to write tests for those and we wanted to create objects for those in each test, we noticed that out of hundreds of fields, we might need to fill, I don't know, probably 80, 90 of them because they were mandatory.

16:54 And it started to be very tedious.

16:56 And I remember I opened an issue in the Pydantic.

16:59 I say, "Hey, have you thought about probably allowing creating an object with random fields that validate properly?

17:07 Like this field is an integer between 10 and 20.

17:10 So I just don't want to feel it.

17:11 I don't want to feel any of those because I don't care for this test.

17:14 Is there a way that I can say, "Okay, just fill whatever it validates." And they say, "No, it's out of the scope of Quedantic," which also makes sense.

17:20 I just wanted to ask in case.

17:22 They said that probably in Factory Boy, they might be interested in this.

17:25 So I went to Factory Boy and I read the documentation.

17:28 It was pretty cool because it allows you to create, you define an inside class.

17:33 So it's a meta class, not a meta class in the terms of Python meta classes, but it's a class called meta within the factory that you want.

17:43 It's weird because every time you say, yeah, this is the meta class, why do meta class?

17:47 No, it's a class.

17:48 So you inherit from factory, then you define a class called meta, meta where you define what is your model.

17:54 So this is the object I want to mock with this factory.

17:57 and then you can define many fields with their default values.

18:02 The cool thing about this is that it implements also Faker.

18:05 You can say, if I have a username, I don't want to fill it, just give me a username, and Faker will give you.

18:10 >> Yeah, Faker is really cool for generating stuff like that.

18:12 >> Really cool.

18:13 >> Yeah.

18:13 >> The amount of plugins you find for Faker is outstanding.

18:17 You can fake almost anything you think of.

18:20 The cool thing about this is that it's not only you plug in the class that you have and it will fill it, but you also can work with ORMs, like you can use SQL, KMORIM, or Django ORM.

18:31 And it will generate an object for this ORM based on whatever you set these are the default values.

18:38 So I thought it would be great if I could do this also for Pydantic.

18:42 So I could just say, OK, these are the mandatory fields.

18:45 It puts something fake.

18:46 You can think about it.

18:47 And then we're ready to go.

18:48 But reading the documentation, it didn't appear anywhere.

18:51 And I thought, hmm, maybe I cannot use it for this case.

18:53 So I went ahead and opened it easy and say, Are you thinking about putting this also to work with Pydantic?

18:58 I mean, it's now is booming and everyone is using it.

19:01 And if you are reading JSON from an API, it's very likely that you have hundreds of fields you don't care about.

19:06 You might want to fill it with whatever.

19:08 And I remember the author said, I didn't know this.

19:11 I didn't know Pydantics, cool you mentioning it, but internally what FactoryVault is doing is creating a dictionary with the parameters you want to fill in and just unpacking it in the construction of the class.

19:22 Have you tried it?

19:23 And I was like, no, I have not.

19:24 And when I tried it worked.

19:26 So it was perfectly.

19:27 I mean, maybe there are some quirks of Pydante that it kind of covers.

19:31 But if you're using Pydante to store your data from API calls and so on, from JSON validates and so on, Factory is pretty cool.

19:40 I mean, the amount of things you can do with this, you can create many factories for the same class.

19:46 You can create fixtures like, I don't know if you want to mock a user, you can have an admin or a buyer or whatever.

19:51 And then you can just define different factories and it will give you the usage you've defined.

19:56 And it's also pretty cool because the faker is randomized beneath it.

20:02 So if there are parts of your object that your code does not care about, it's also a good test to have those parts being random.

20:11 Because if it really doesn't care, you don't care what those fields are.

20:14 And then at some point your test fail, it happens once, it means that you actually can fix something.

20:19 Yeah, absolutely.

20:20 - Absolutely.

20:21 I did see that if you need repeatable tests, but you want Faker to generate random stuff, there's a way to seed Faker.

20:28 - Exactly.

20:29 - Generate the random values, but do it in a consistent way.

20:31 And one way you might want that is if you have an edge case or some value that breaks the test, and then you want to put a break point and press debug and go through it again, but like, you know, how are you going to get it to hit that case again in a predictable way, right?

20:45 So if you trigger, if you tell it to say, always do the same thing, but randomly, you know, >> You can go back and look at it a second time and figure out what's up.

20:53 >> Yeah, you can fix that. Sometimes it's also good to have them fixed, even if you don't care.

20:58 I mean, you need to have a date time, and for some reason you need to have the date time being whatever, and whatever, but you can validate for it.

21:04 You can just or either set it or ensure that it's fixed.

21:08 Yeah, there are many use cases that you can exploit that thing and it's actually really cool.

21:12 >> Yeah, I almost always seed Faker because I'm not using it because I want the randomness.

21:20 I'm using it because I don't want to come up with the data.

21:23 >> Yeah, exactly. So make it so that it does the same thing every time, just gives you the random data that you want.

21:28 That's right. Agreed. Very, very cool.

21:30 All right. The next one up actually is pretty related to that.

21:35 It's called PyInstrument.

21:36 Have either of you heard of PyInstrument?

21:38 >> Not until now.

21:39 >> Yeah.

21:39 >> I read the notes and it sounds pretty cool.

21:41 >> Yeah. Brian?

21:42 >> No, I haven't.

21:43 >> Yeah. So it's a call stack profiler for Python, which is pretty cool, right?

21:48 it's just going to tell you where your code is slow, but it looks really clean.

21:53 When you look at the output, it can actually give you the results in the terminal.

21:59 If you want to see, like you run this thing, instead of saying Python, my Python.py file, you would just say py instrument, that same file and it'll run it, but then at the end, it's going to generate a whole bunch of things about how long it took and whatnot.

22:16 And then you actually get like colored output in the terminal showing which lines of code are spending how much time in different places.

22:24 And it seems like it's a real good way to just sort of quickly dive in on where you're spending your time.

22:29 - Yeah, I'm definitely gonna try this.

22:30 It's cool.

22:31 - Yeah, one thing I like about it is the simplicity of like pip install pyinstrument, pyinstrument your file.

22:37 That'll give you the answer, right?

22:38 - That for me solved it.

22:40 I mean, every time you want to do some profiling, feel like they spent some time tweaking things.

22:45 So you get what you want.

22:47 The fact that this is just running with PI instrument, whatever script you want.

22:50 I mean, I'm going to try for sure.

22:52 - Yeah, yeah, for sure.

22:53 And when you do profiling, you end up in this sort of quantum mechanics world of if you observe a thing, you've changed it.

23:01 And so there might be code that is like, this half is 50/50 and this half is 50/50 at the time, but one is calling an external system once and the other is a tight loop.

23:11 And if you profile that with instrumentation, it's going to wreck it.

23:15 it's going to make the loop look way slower 'cause now you've added a bunch of overhead to each step where there's very little overhead to this external service sort of thing.

23:23 And this one uses sampling, and the sampling doesn't really have that effect.

23:28 It just every so often, every millisecond or something that says, what are you doing now?

23:33 What are you doing now?

23:34 Who called you?

23:35 What are you doing now, right?

23:35 And so it's more of a polling sort of thing rather than slowing down line by line code.

23:43 So that's probably worth doing as well.

23:45 >> Yeah, it's pretty cool.

23:46 >> Yeah.

23:46 >> Yeah.

23:46 >> It looks like you can specifically jump in and just do a section of your code that you care about also.

23:52 >> Exactly. If you want to say, so one of the things that I hate about profiling is, it'll say 87 percent of your time was in the startup code and the imports.

24:01 You're like, "Yeah, okay, that's not relevant to me.

24:04 What I want to know is the part that I'm actually trying to test here.

24:08 How long did I spend there?" Please don't pollute that with other junk about like starting up Python or loading modules or whatever, right?

24:16 And so you can, there's an API and you can say from PyInstrument import profiler, and then you can do a context block in there and run it.

24:27 And just that code will tell you like how long it takes.

24:30 Does anything else jump out there at you, Brian, in like with this example I got on the screen here?

24:34 That would be hard.

24:35 - It's an async example for one.

24:38 - Yeah, as an async and a wait.

24:39 And so they recently released PyInstrument 4, which will actually give you the information about the async code as well, right?

24:50 So it'll, let's see what it says.

24:52 Has async support, PyInstrument now detects when an async task hits an await and tracks the time spent outside of the async context under the await.

25:01 Whereas before it would basically just profile the asyncio event loop or something silly like that, right?

25:06 So if you're trying to profile async await, in asyncio, this might be your best option 'cause it specifically supports that.

25:14 - That's good.

25:15 So what happened before if you use a different profile?

25:19 - It would say, yeah, it says you only see the time spent in the run loop and it'll basically tell you like, here you see like run once, the select and then the queue control built in.

25:30 It's just like, there's this asyncio event loop that's cranking around waiting for the signal for the IO to be done.

25:36 and it just says, well, you're waiting on this, you're in the loop.

25:39 You know what I mean?

25:39 - Yeah, yeah, yeah.

25:40 - Yeah, so, yeah.

25:42 So now you get a little bit better.

25:44 Like it says, okay, you're awaiting on this line of your code for a second or whatever it is.

25:48 Yeah, there's also, I'll shout out a few more things here.

25:51 Is it in the stock?

25:52 Yeah, so there's also a bunch of simple authentication they did previously about network calls and stuff.

25:59 And there's an interactive Vue.js app that you can get with Flame Graphs.

26:04 So instead of looking at it in the terminal, you can look at it in your web browser and explore into those.

26:09 Yeah, there's a lot of neat little things here pulled out of the show notes, but it seemed like a really nice way to do some profiling and you just PyInstrument your code and you have a look.

26:18 - Yeah, I personally kind of like the default output.

26:21 I know that a lot of people like flame graphs, like they don't really do much for me.

26:25 They look like, I don't see the data, but it's cool that it has both.

26:29 - Yeah, a couple of things from the live chat.

26:31 Maddy says, "PyInstrument is a statistical "or sampling profiler, "which is better prepared for profiling." I think it depends.

26:38 I mean, the instrumentation ones do give you more precise information, but it's also skewed with the overhead of that information.

26:48 So it depends, but this is the least influential one for sure.

26:51 And then Avaro does, "How would you use PyInstrument with an entry point?" That's a good question.

26:58 Not knowing the answer off the top of my head, maybe make another Python file that just imports your library and calls the entry point and then profile that.

27:05 But there's a real quick cheat, just make it call it and then PyInstrument that file.

27:12 But there may be some way to say like -m and give it a module and a thing to do.

27:18 So yeah, that's it, Brian.

27:19 That's it for PyInstrument.

27:20 - Cool.

27:22 Well, I just wanted to remind everybody that Python 3.10 release candidate one came out yesterday.

27:27 So Pablo announced it just on the 3rd, I think.

27:31 I think it was yesterday.

27:33 The 4th today?

27:34 Yeah, anyway.

27:35 So 3.10 is out.

27:36 If you've got, well, 3.10, RC1 is out.

27:41 The timelines that we're looking at then, we're getting excited, it's coming up.

27:46 So the September 6th is the plan for RC2.

27:49 And then October 4th is the plan for the official release.

27:53 So we're just really right around the corner.

27:55 It's nice.

27:57 This is definitely a time, I know we've brought this up before, but if you maintain any third-party Python packages, you probably should have already been testing it against 3.10.

28:07 But if you haven't, definitely do it now to make sure that people that use your stuff, it doesn't break when they need to.

28:15 Then in the show notes, we put just a reminder of some of the new changes in 3.10.

28:21 We've definitely talked about some of these before, structural pattern matching is the switch statement thing.

28:29 Yeah, lots of these other things we've covered.

28:33 Actually, I like the union types because there's a lot of stuff that I write that the default is none but the normal type is something else.

28:43 You can really easily say the type is none or int or something like that.

28:48 That's a lot cleaner than before.

28:52 I've already started using 310 to test everything that I support.

28:56 I hope everybody else has as well.

28:58 >> Yeah, cool. I like the optional length checking in zip.

29:01 Zip taking two collections and you want to pair up the items, like if those things don't match, that should be a problem.

29:07 Also like the or for the types information.

29:10 I think DIC and some of those types don't require a from typing imports.

29:15 >> Oh, right. Yeah.

29:16 >> Yeah. I don't see it called out here, but one of the problem was, maybe that's explicit type aliases, I'm not entirely sure.

29:24 But if you want to say this type is a dictionary of strings and integers, you would have to say from type in import capital D dict, and then dict square bracket string comma int.

29:36 Whereas now you can just use the lowercase d-i-c-t, and you don't have to have that import, and you can do that sort of thing to it.

29:43 I'm looking forward to that.

29:44 >> Yeah. With this, a lot of the common type hints, you won't have to do the import anymore, and that's great.

29:53 I think that's really all I was using the import for, was things like dict and set, things like that.

29:58 >> Yeah, exactly.

29:59 >> Didn't that, I mean, I seem to remember that 3.10 was the one that was including these built-in types without having to import from typing.

30:07 Didn't that update might break some of the libraries that is typing?

30:13 - Like Pydantic and FastAPI.

30:15 The thing that that was, was to use it in, basically use it as a string and not actually evaluate the type.

30:23 I think that, like, so if you had your own type, your own Pydantic type that was a customer, I think you could put customer, but it wouldn't be actually evaluated until a type checker hit it or something like that.

30:34 - Like a forward typing.

30:35 - Yeah, yeah, exactly.

30:37 So this ability to specify the type on like lowercase ddict is related, but it's not the same.

30:43 - And I'm pretty sure that that fear around Pydantic is not in 3.10.

30:48 - Yeah, it either got postponed or rolled back or modified.

30:52 Yeah, yeah.

30:53 - I just want to talk about the one that says, what was the number?

30:59 Six to six, do you have it?

31:02 - Yeah, the precise line numbers for debugging in other tools.

31:06 - Yeah, I think it's very underrated.

31:09 (laughs)

31:11 It's gonna be one of those things that when people get used to it, it's like, I don't know how you live without this.

31:15 - Oh yeah, yeah.

31:17 There's not a good example shown right off the bat, but it's pretty cool.

31:22 - Yeah, yeah, absolutely.

31:23 Very cool, and then we also have better stack trace error messages, right?

31:26 - Yeah. - Yeah, those are coming.

31:27 A lot of good things to look forward to.

31:29 All right, Juanpe, you got the last item.

31:31 - Great. - I think it's time for it.

31:33 You wanna take us out, right?

31:34 - Sure. (laughs)

31:36 Yeah, so let's talk about Time Machine.

31:39 I said we were building this tool that copies an entire Salesforce org.

31:44 One of the things that we need to orchestrate everything is to timestamp almost every action we do.

31:49 This means that in many places all over the code, we have a daytimeUTCNow method call.

31:57 And when we are testing it, we need to be able to mock it.

32:00 And if you've tried to patch daytimeUTCNow with the usual patch method, it works, you can do it, But you need to do it with a patch and then you pass the string, but the module where this UTC now call is, and then you're good to go.

32:16 But when you have this in many files in the same test, you need to patch every one of those because otherwise it wouldn't work.

32:21 So I tried to use patch object and patch daytime and say, okay, I want to patch the UTC now method, this object.

32:27 And it will of course complain and say, you cannot patch a built-in type like daytime, daytime.

32:34 So I was looking for how we could patch this and I found FreeScan, which is a very well-known library thing to patch this kind of things.

32:44 But suddenly I noticed that once I started using FreeScan, all of my tests took much longer to complete.

32:51 It's not like deal breaker, so it went for probably five minutes to seven, or seven and a half, but it was very surprised because our pipeline or deployment pipeline ready to take a long time.

33:05 So every time I can reduce a minute, it's good for me.

33:08 And when I saw it going up two minutes, I was surprised, why is this happening?

33:11 And then I learned that what Freescan is doing is actually scanning all your dependencies and make a batch for every call you make to the methods of data.

33:20 And then in trying to see if there was something else, I found out Time Machine.

33:26 Time Machine is a very cool, not so well-known, I think, library that allows you to do basically the same that Freescan allows you to do.

33:37 So you can just patch almost any method call in daytime or time with a simple decorator in your test.

33:45 It also supports pytest fixtures that you can use.

33:48 The good thing about this is that it does not scan for imports of date and daytime.

33:53 And what it does is actually change the underlying C-level calls that you make to get the time.

34:00 So every time you say, I want to patch any call to be on January 1st of 2019, for instance, it will just call it normally, but the C, the underlined C calls that will be made will return this time instead of the other ones.

34:14 You don't need to scan everything to patch it.

34:16 Another thing that I thought it was pretty cool is this, you can let the time tick after you patched it.

34:22 So you say, this is for February 1st of 2018.

34:27 And once you enter the mock, either with a decorator or with a context manager, you can also use like standard patch call.

34:35 Then time start passing, starting on that time that you mocked it for.

34:41 So you can do perf counters and all this thing normally, but if you need to stay in a given day for a test, you can do it.

34:47 So I thought it was pretty cool.

34:49 It solved my two extra minutes running because we have many places and many files in the project where we used it C now.

34:55 And it was pretty well.

34:57 So this must have had incremental, I mean it has a little bit of time that it has to do its work, but it's fast enough that you're not noticing it then?

35:05 - No, I'm not noticing anything.

35:06 I mean it runs more or less the same.

35:08 - Okay, wow, that's pretty cool.

35:10 - I imagine there should be some delay, but it's not as noticeable as what happened with Freescan, 'cause it took some time.

35:16 - Yeah, I'm really glad you brought this up.

35:18 This is cool.

35:19 - Yeah, we have a bunch of tests.

35:21 - Yeah, exactly.

35:22 I was gonna say, Brian, you probably, this is kind of in your world, right?

35:25 like dealing with time as a dependency?

35:27 >> Definitely. Sometimes you want it fixed because you really want fixed answers because like your timestamps and stuff are in your data.

35:36 You're going to have to, I mean, it's good to compare against known oracles.

35:40 But there's also times where you, and this is where FreezeGAN isn't so bad, but maybe this would be really useful too, is if you want to test certain things.

35:52 There's weird quirky dates, you want to make sure that your software deals with certain times.

35:56 Fine. Does it work fine when it's running overnight on December 31st to January 1st, things like that when the year changes and things like that.

36:05 >> Exactly.

36:06 >> Yeah.

36:07 >> You always want to test your boundary conditions, and crossing over time or weird cases like March 29th, stuff like that.

36:15 You're like, let me just try that and see if this is going to survive.

36:18 >> Yeah. But then to be fair, I think most of the time, things like this are used are, like was brought up is that the time shows up in the data.

36:29 In order to compare the log or something, in order to compare those apples to apples, it's nice to have the same dates there.

36:36 I can't tell you how many times I've had to compare two log files and strip out the times because those are the, every line is different because the timestamp is different.

36:47 >> Yeah, very cool. Nice find.

36:50 So that's all for time machine.

36:52 >> Yeah, super.

36:53 >> Well, that's our six items, everybody.

36:57 Have you got anything extra, Michael?

36:59 >> Well, I have the old thing that is new again.

37:02 Let's see. I have some bandits.

37:04 So the drama around supply chain vulnerabilities and open source repositories goes on.

37:11 So this one, I think actually, the other article I'm going to talk about, comes to us from Joe Ridley.

37:18 Thank you, Joe, for sending that in.

37:19 But basically there's some more malicious things in PyPI again, and people just remind everyone to be careful and be white list stuff.

37:29 Yeah, this one, I don't know what this one was.

37:33 If it was typo squatting this time around or it was just something else that got put up there.

37:37 Yeah, there's one headline is credit card stealing malware found in official Python repository.

37:43 And the other one is the same one about our technical article says software downloaded 30,000 times from PyPI ransacks developer machines, developers machines, expect to see more of these Frankenstein type things 'cause it'd say basically a systemic threat, like how does it get dealt with?

37:59 Right, I'm not sure if they list out.

38:01 Yeah, so they used, they did interesting stuff as well.

38:04 Like they did simple obfuscation of the code that was being run.

38:08 So you couldn't look at it and, you know, say look for a credit card or look for a Bitcoin wallet and then go do your evil deeds in Python source code.

38:15 So they would do things like base 64 and code the Python code and then just in memory, decode it, then run it.

38:23 So they were trying to get around things like that.

38:25 So anyway, people can check that out.

38:27 And it's not ideal, but just a reminder to beware.

38:31 - Yuck.

38:33 - Yuck, yuck, yuck.

38:34 This is why we can't have nice things.

38:35 Come on, people.

38:36 - This is why we can't have nice things.

38:37 Well, I got a couple of things I wanted to bring up, just things I've been up to.

38:42 just released episode 162 of Test and Code.

38:46 I run through all the different flavors of test-driven development that I know of.

38:52 There are quite a few versions.

38:54 Check it out if you're interested in test-driven development.

38:57 Then I'm just working on wrapping up the talks and continuous integration chapter for the second edition of the pytest.

39:05 It'll be coming out hopefully within a week.

39:06 >> Very cool. Good to see you making progress there.

39:08 >> Do you have anything extra, Juanpe?

39:11 >> No, not from my side.

39:12 I'm very happy to be here.

39:14 >> Let's go to a joke.

39:15 >> Yeah, it's good to have you here. All right.

39:17 Let's go to a joke. So this one's a visual.

39:20 If you're listening, you're going to have to scroll down to your podcast show notes at the bottom.

39:26 Just click on the joke link.

39:27 One of the things you guys I like about Python is there's a lot of stability in the code that we write.

39:33 If I wrote something on Flask five years ago, chances are it'll still run.

39:37 If I write my Python code now, it's probably still going to run.

39:40 Yeah, there's new things.

39:41 There's new shiny visualization frameworks and stuff, but generally it's pretty stable.

39:44 You know, what is the opposite of that?

39:46 JavaScript.

39:47 So, so here's a little animation say, and it says JavaScript developer bouncing from framework to framework.

39:54 And it's this incredible, like almost people are awesome type of thing where somebody set up, you know, 50 workout balls on a running track, the whole straight of a quarter mile running track.

40:06 and somebody jumps on it and just like glides from one to the next.

40:10 What do you all think?

40:11 >> The fact that he's able to do this is surprising.

40:14 >> It's really impressive that he pulls it off.

40:17 It's on one of these like sandy, gritty running tracks.

40:21 It's going to hurt like crazy if he misses it.

40:23 So maybe there's the motivation.

40:26 >> Yeah. I remember the GenBrain report you said before, I was thinking, I didn't say anything, but I didn't want to mean what you're saying.

40:34 How likely are you to change languages?

40:36 And it was like, we're JavaScript.

40:37 They're going to change a lot.

40:39 Then I thought, oh, they're languages, not frameworks.

40:42 - Exactly.

40:43 How likely are you to change your framework?

40:45 Well, that's like, nearly a hundred percent.

40:48 - Yeah, that's true.

40:50 I mean, people stick around like, you got Django developers have been doing it for years.

40:55 - Yeah, 10 years and they're more excited today than ever about it, right?

40:58 They're not like, we're ditching this.

41:00 - Yeah.

41:01 - All right, well, that's, does that count?

41:03 Does that count as a joke?

41:04 >> Yeah.

41:04 >> I laughed.

41:06 >> All right, perfect. Well, that's what I brought for you all.

41:09 >> Well, thanks everyone for showing up and had a fun day today.

41:14 Hope everybody else did.

41:15 >> Thanks a lot for having me here.

41:16 >> Thanks, Brian, and thanks for being with us, Pompeii.

41:18 >> Bye.

41:18 >> Bye-bye.

41:19 >> Thanks for listening to Python Bytes.

41:21 Follow the show on Twitter via @pythonbytes.

41:24 That's Python Bytes as in B-Y-T-E-S.

41:27 Get the full show notes over at pythonbytes.fm.

41:29 If you have a news item we should cover, just visit pythonbytes.fm and click submit in the nav bar.

41:34 We're always on the lookout for sharing something cool.

41:37 If you want to join us for the live recording, just visit the website and click "Live Stream" to get notified of when our next episode goes live.

41:44 That's usually happening at noon Pacific on Wednesdays over at YouTube.

41:48 On behalf of myself and Brian Okken, this is Michael Kennedy.

41:52 Thank you for listening and sharing this podcast with your friends and colleagues.