Transcript #43: Python string theory, v2

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.

00:06 This is Python Bytes, episode 43, recorded on September 13, 2017.

00:12 And I'm Michael Kennedy.

00:15 And I'm Brian Okken.

00:16 Hey, it's great to be with you guys. Great to be with you, Brian.

00:18 We have a bunch of really cool stuff lined up.

00:21 There's a few things that really surprised me and maybe even inspired me to go create this project

00:26 that we were just chatting about before we hit record.

00:27 So very, very cool stuff.

00:30 But before we get to it, let's just say thank you to Rollbar.

00:32 Those guys are sponsoring the show.

00:34 You check out their offers at pythonbytes.fm/Rollbar, and we'll get to them later.

00:38 I want to talk about taking the future and sending it into the past, like Marty McFly style.

00:44 What do you think?

00:45 Yeah.

00:45 Yeah, this popped up recently, just this last week.

00:51 And I'm not sure what the impetus was, but it is Future F-Strings.

00:56 It's a project.

00:57 You could just do pip install future dash F-Strings.

01:01 And then you've got a little bit of a little...

01:04 It isn't a library that you include.

01:06 It changes the encoding.

01:09 So I'll just jump to the chase.

01:11 It's a back port of F-Strings, even down to Python 2.7.

01:15 And I've tried it in 2.7.

01:18 I haven't tried it in anything else.

01:20 But it is kind of fun to be able to play with F-Strings everywhere.

01:23 So I had very mixed thoughts about this when you told me about this project.

01:28 On one hand, I look at it like, you know, you're taking one of the really amazing and smooth features of Python 3.6 and making it available in legacy Python.

01:39 Which is kind of like, do we really want to like encourage that sort of thing?

01:43 But at the same time, I feel like this also lets you, as you're migrating your code, move to the latest, greatest syntax in Python 2.

01:53 But then when you actually make the switch, you're not going to have to go like, well, we used format because that was the best option in Python 2.

01:59 But now we've got F-Strings.

02:01 And so I feel like that's really cool.

02:04 The other area where I think it's great is Python 3.

02:06 There's plenty of people running Python 3.5.

02:09 That's the default on Ubuntu right now and things like that that don't have F-Strings.

02:13 I've decided after further contemplation that this is a really cool project and I want to check it out.

02:19 So tell people what F-Strings are.

02:22 I mean, most people maybe know, but what's this thing, this F-String thing?

02:26 Well, it's, so if you're familiar with the format part that you could be, I don't know how long ago you start using that.

02:36 I think that's available in 2.7 as well.

02:37 Yeah, I think so too.

02:38 In the string that you're printing, you can like leave little brackets to show where you're going to insert something.

02:44 And then you say like .format and then all the things that you're going to put into the string.

02:49 In F-Strings, you just put an F before the string and then within your brackets, you just put the variable name for whatever you're going to put in there.

02:56 Or any expression can go in there.

02:59 Right.

02:59 So you could say like F, quote, hello curly thing dot upper and like uppercase the thing just in that expression or something like that, right?

03:07 Yeah.

03:07 Yeah.

03:07 That's really cool.

03:08 Yeah.

03:09 At first I was like, I don't know if I like it, but once you start using F-Strings, you can't go back.

03:14 That's really one of the reasons why I like this project is because I do, I usually use Python 3.6, but there are times where I have to write some stuff in 3.5 or 2.7 or something.

03:26 And I don't want to go back.

03:27 Yeah, I know.

03:28 I hear you.

03:28 That's great.

03:29 I'm going to put a lot of tests around it, so I'm not going to rely on this yet because I don't know if it's really ready for primetime for everywhere.

03:36 So I would say use caution, dip into this slowly if you're going to use it.

03:41 But it's a fun thing.

03:43 I'm not sure I like how it's doing it though.

03:46 It's doing it in an interesting and odd way.

03:49 Yeah.

03:50 I don't know how else it would do it, but it's interrupting the encoding.

03:54 Yeah, it installs a separate encoder.

03:56 So it extends the UTF-8, I think, the UTF-8 encoding to also do this.

04:02 So if you're not using UTF-8, I don't know, maybe you could hack it to do something like you're using.

04:08 Yeah, yeah, it's pretty cool.

04:09 So you put like a little encoding flag at the top and you install a special encoder and off it goes.

04:14 Yeah, pretty interesting.

04:16 All right, so I want to stick with this Python 3.6 feature angle, even though that's bringing that stuff to other versions of Python.

04:23 I want to talk about the fun of reinvention.

04:26 So this was a really amazing presentation done by David Beasley.

04:31 And it's done in the style of some of his other presentations.

04:35 And in my mind, like the way technical presentations should be done.

04:38 You should be up there writing code, showing how things work, not just slides and pictures and stuff,

04:44 but like showing things, right?

04:46 And so what he shows, though, is kind of insane and incredible.

04:50 And your mind will probably explode if you see this.

04:52 You'll definitely come away with a greater appreciation for Python 3.6.

04:55 So what he does is he goes and builds upon some of the new 3.6 features that are below the surface

05:04 that a lot of people wouldn't even know about, like hooking into when a class gets defined.

05:09 So you can basically redefine what that means and things like that.

05:13 So, for example, he says, well, let's talk about type-ins.

05:15 These type-ins are cool, but they don't do anything.

05:17 We can add these constraints and assertions inside of our code, and that does something,

05:22 but we can't really check that in the tooling.

05:26 So what if we could rewrite, create this little framework using inheritance and all sorts of other stuff

05:32 that lets us add these constraints as type-ins, and then they're enforced at runtime automatically.

05:38 It's pretty amazing.

05:39 That's incredible, yeah.

05:40 Yeah.

05:40 So whether you think it's a good idea or a bad idea to do that, the fact that you can do that is really, really cool.

05:46 So you can basically set up a function, and as like a type hint constraint,

05:50 you can say, and this is a positive integer between 0 and 200.

05:54 And it will actually check that.

05:58 It's really something.

05:59 But mostly it's an amazing look inside.

06:01 Yeah, it's an amazing look inside of Python 3.6, down at the low-level stuff.

06:07 So, Brian, knock, knock, who's there?

06:10 Yeah, I am.

06:13 Sound recognition is there.

06:14 Yeah, sound recognition.

06:16 So there's an interesting article called Sound Pattern Recognition with Python,

06:21 and I really liked just the – I didn't know you could do this with SciPy.

06:26 So there's apparently part of SciPy.io.wave file, and I'm not sure if there's other types you can do – use.

06:36 But this article uses SciPy to read a WAV file and then just has it as data, like an array of numbers,

06:45 and then does some math on it.

06:47 And it does – this particular one is assuming that you've got some WAV files

06:51 that have NOCs in them and trying to detect where the NOCs are and how far away they are.

06:56 And just some basic little logic to try to figure out, yeah, like I said, where the peaks are.

07:02 I think the author noticed that you need to have some minimum values for how you tell where the peaks are and then some distance between them

07:10 so you don't have, like, ringing within one peak.

07:12 I like it because it's kind of a – I think it would extend easily to do –

07:17 I'd like to take it and run with it and try to do some basic oscilloscope measurements

07:21 with this kind of a thing.

07:23 It'd be fun.

07:24 Yeah, it'd be super fun.

07:25 It's really accessible.

07:26 Like, the actual code to figure out the NOCs and to process the WAV file,

07:30 it's really small and approachable.

07:32 And to me, what I thought of when I heard this was, oh, you could build something that is actually kind of smart.

07:38 So it shows you how to identify NOCs, right?

07:42 And you should be able to identify, like, a door jiggling, a key noise.

07:46 You should be able to identify, like, a few of these really distinct patterns.

07:50 And if you could say, like, identify, like, what does a door jiggle sound like?

07:54 What does a knock?

07:56 What does a doorbell?

07:57 What does a key sound like?

07:59 You could put a little Raspberry Pi just by your front door that says, someone's home, someone unlock the door, someone's knocking.

08:04 You know, just cool stuff.

08:06 Wouldn't that be – I mean, it seems like that's a weekend project with this.

08:10 It doesn't seem major.

08:11 Yeah, that would be fun.

08:12 You could hook it up with Twilio and get texts when somebody's knocking at your door.

08:16 Right, and hook it up with a camera to take a picture of whoever's out there,

08:19 see what they're doing, right?

08:20 Yeah.

08:21 Yeah.

08:21 Send yourself a note, hey, kids came home, right?

08:23 They unlock the door at 3 o'clock when school's out or whatever.

08:26 That would be fun.

08:26 It sure would.

08:27 All right.

08:28 So I think people should take this idea and run with it.

08:31 And if they do, they should send us a message.

08:33 Or even better, go to pythonbytes.fm/43 and leave a comment at the bottom about what they created.

08:39 Yeah, definitely.

08:40 Definitely.

08:41 Or even better, they could write a blog post and then email that to us, and we'll highlight it on the show.

08:45 Yeah, maybe we'll even cover it.

08:46 That'd be awesome.

08:47 Yeah, yeah.

08:47 Cool.

08:49 So before we get to the next item, which is sort of mind-blowing, I want to talk about Rollbar.

08:53 You guys hear me talk about Rollbar a lot probably, but that's because they're great.

08:57 Basically, if you run any sort of web app, you should have real-time error monitoring and reporting.

09:02 So if something goes wrong with the website, for example, pythonbytes.fm, we'll get a notification.

09:08 It could be in Slack.

09:09 It could be on our phone.

09:10 They don't have all the details.

09:11 You know, this person went to this URL in this situation.

09:14 Here's the variables they passed to the function that caused a crash, things like that.

09:19 So if you want to get this for your site or application, just go to pythonbytes.fm/rollbar and check it out.

09:25 So we've talked a lot about async stuff, right?

09:29 Like async both on the server and on the client.

09:32 It's pretty mind-blowing stuff.

09:35 But it turns out that, you know, like all changes to software, there's like cascading consequences of adopting a new model or trying something new, right?

09:44 And one of the cascading changes is in many of the high-performance async processing loops.

09:52 I'm thinking async.io, but also UV loop, which is a really super fast, powerful one.

09:57 Things like Sanic are, I think Sanic is based upon it, the web framework and so on.

10:01 Those things are ultra fast because they rely on async.io, but async.io doesn't use threads.

10:08 And so that means thread local storage doesn't have any meaning anymore when all of these like concurrent operations are running on the same thread.

10:16 Okay.

10:17 Right?

10:17 So like if I was running a website, I might store into thread local storage like the cookies or the authentication.

10:23 And then later I might ask for it back.

10:26 But that could be changed because you have a blocking.io that released the thread that somebody else then wrote their cookie to.

10:32 Right?

10:32 So it's kind of like crazy.

10:33 And there's also in other really interesting places like decimals.

10:36 Working with decimals does this.

10:38 NumPy error state.

10:39 Warnings.catch warnings.

10:40 Profiling.

10:42 Tracing.

10:43 All these things use this kind of stuff.

10:45 Right?

10:45 But it doesn't work in this async await world.

10:48 So we have a new PEP.

10:50 PEP 550 that defines a new execution context.

10:54 And this is from the guys at magic.io who created UV loop.

10:58 So like their motivation is really obvious.

11:00 Like they want UV loop to work.

11:02 And this is kind of something in the way.

11:05 So the PEP adds a new generic mechanism for ensuring consistent access to non-local state in the context of out-of-order execution such as generators and coroutines.

11:16 Wow.

11:17 It's pretty fascinating, right?

11:18 Yeah.

11:18 It's very fascinating.

11:19 I hadn't even realized that.

11:21 I mean, you think it through, obviously, it's a problem.

11:23 But I hadn't even realized, like, oh, my gosh.

11:24 Like, some of these, like, low-level things are just going to be broken.

11:27 Like how, for example, state is processed in a web request.

11:30 I'll be very interested to watch this and see where it goes.

11:33 Yeah.

11:34 So maybe it makes it into Python 3.7.

11:36 Maybe not.

11:37 But it's kind of cool that this is not just like, oh, we made UV work properly.

11:40 But we're going to make all the Python work properly.

11:43 So they've got some really nice examples on the article we're linking to.

11:47 That's really cool that they did it instead of just trying to hack their own way.

11:51 Yeah.

11:51 For sure.

11:52 Speaking of threads and processing and all that.

11:55 I often work in single-threaded or single-process sort of tasks.

11:59 I really like this article called Intro to Threads and Processes in Python.

12:04 It's a very accessible beginner's guide to parallel programming.

12:08 And really kind of what's the difference?

12:11 Where would you use threads?

12:12 Where would you use processes?

12:13 And it's even got pictures with some algorithms to see to show how, like, if you're a certain

12:20 kind of a job.

12:21 I can't remember what the job is that he's doing.

12:23 He's doing some data science project based on Amazon and trying to analyze a bunch of different

12:29 stuff from space.

12:30 And so he's got to run tons of algorithms and tweak them with different parameters and stuff

12:35 like that.

12:35 It shows, like, basically, if you use one thread, two threads, or four threads, and then the same

12:41 with processes, how it's affected with running times.

12:45 And you can watch to see that things are running in parallel.

12:49 Good use of little simple diagrams, too.

12:51 That's cool.

12:51 Yeah, the pictures were great.

12:52 The rundown is, conclusion at the end is, if you're waiting on I.O. for something, threads

12:58 are just fine.

12:59 And if you are CPU heavy, then you want to go with multiprocessing.

13:04 Yep.

13:04 Until the galectomy happens, which who knows if that will, but until then, this is definitely

13:08 a good introduction to it.

13:09 Yeah.

13:09 And I'm actually, I'm one of the, I don't know, I think there's probably a lot of people,

13:13 but having the GIL doesn't really bother me.

13:16 It does simplify how you program.

13:19 I don't think it's that terrible.

13:20 So.

13:21 Yeah.

13:21 Yeah.

13:21 It's, you know, it's in the extreme cases.

13:23 I think the main group of people who suffer under the GIL are those doing computational,

13:28 CPU computational work.

13:30 Outside of that, you're mostly okay.

13:32 Okay.

13:33 Yeah.

13:34 So in that, like async and await and stuff like that, I'm going to expose my, unknowledge

13:39 of this stuff is, so async and await, is that dealt with, with, multiple threads

13:44 then?

13:44 No, it's all one thread, but it has a mechanism to say, anytime you hit blocking I.O., go put

13:53 this part of the code to sleep and allow it to run another, another task, another thing until

13:59 it hits blocking I.O.

14:01 And the idea is like most of the time you're waiting on databases or networks or something

14:06 like that.

14:06 And then you wake up for a second, you do a little processing and then you go back to wait

14:09 on another network or database or web service.

14:12 And as long as that's the kind of stuff you're waiting on, the GIL is not a problem because

14:16 it gets released.

14:17 But if you're like trying to compute pi with a, you know, power series or something, something

14:22 like that, then it would just, it would stop.

14:24 And the async wouldn't help you with that.

14:25 Yeah.

14:26 I just wanted to, so the async and await is going to use, it's going to be in the

14:31 same sort of a family as when threads would be good for you.

14:34 Yes.

14:34 Yeah.

14:35 It's in exactly this similar spot.

14:37 Yeah.

14:37 But just more efficiently.

14:38 So let's talk about another low level thing.

14:41 And I found this, I think it maybe even was recommended by a listener, some low level

14:46 part of it.

14:47 And then I found the whole, whole thing.

14:49 One of the problems that we can have when we're doing unit testing is working with files,

14:53 right?

14:53 That can be a serious pain.

14:55 Yeah.

14:55 Yeah.

14:55 And if you want to save your files to like say user storage, or you want to process different

15:02 types of files, like zip files, regular, regular files, these are all sorts of pains that you

15:07 have to deal with.

15:08 But there's this really cool project called alternative file systems for Python that makes

15:13 this all seamless.

15:14 So the idea is you can work with files and directories in like zip archives, in memory,

15:21 on the cloud, or just as easy as if they're on your hard drive.

15:24 So they give some examples in this project saying you could write your code now just with open,

15:30 you know, such and such as fin and start working with it.

15:34 And then you decide what that open means.

15:36 Does that open mean normal files?

15:38 Does that open mean something on S3?

15:40 Things like that.

15:41 You can unit test your code.

15:43 You can just have in-memory files, but open reads and writes to those in-memory files.

15:48 You can upload your files to like S3 or OneDrive or something like that by just writing to that

15:55 folder, you know, in that file system and all sorts of stuff like that.

15:59 Yeah.

15:59 So the, like the memory file system or the temporary file system that they have would be great for

16:04 like parallelizing tests and things like that.

16:06 Right.

16:07 It keeps it all separate and isolated.

16:08 So some of the back ends they have is they have application data.

16:12 This is like the special user data locations in various operating systems.

16:16 They have Amazon S3 file systems.

16:19 They have FTP memory.

16:21 Let's see what else.

16:22 That's pretty cool.

16:23 They've got zip for reading and writing zip files.

16:26 And they also have like SSH file systems for writing to remote servers, all kinds of stuff.

16:32 And even tar files.

16:34 Good old tar files.

16:35 Good old tar files.

16:36 Yeah.

16:36 You can just whiff open on those babies.

16:38 Yeah.

16:38 So this is really pretty interesting and I haven't tried it yet, but it looks, it looks quite promising

16:43 and it's extensible.

16:44 So you can add more of these back ends if that makes sense for you.

16:47 That's awesome.

16:47 I actually think this is pretty fun.

16:49 I got to play with this.

16:50 Yeah.

16:50 Yeah.

16:51 Yeah.

16:51 Me too.

16:51 So definitely check that out.

16:53 And Brian, that's it for our six items for the week.

16:56 That's incredible that we've already done.

16:58 So.

16:59 I know these are, they're all fun and all interesting.

17:02 I really enjoyed researching them this week.

17:05 So what else are you up to?

17:06 I got an interesting email this morning saying that they've taken off the beta off of the

17:12 Python testing with pytest on the Pragmatic website.

17:16 And you can order the book now and it's supposed to ship next Monday.

17:21 That is awesome.

17:22 Congratulations.

17:23 Yeah.

17:23 Thanks.

17:24 So that is what I've been thinking about.

17:26 I bet.

17:27 Nice.

17:28 I've been thinking about switch.

17:29 Well, I have been too as of this afternoon.

17:32 So.

17:32 I'm working on a project that would add the switch statement to the Python language without

17:38 extending it.

17:39 Just like a class that you can basically use.

17:41 But it's a pretty clever use of like defining blocks and stuff.

17:45 And I think it might be interesting.

17:46 I'll put a link to a gist or a GitHub repo or something for you at the end of the show notes.

17:51 And you guys let us know.

17:52 I really do want feedback on that because I want to try to nail this.

17:55 And I think if we could get it into something, we could stick on PyPI or have you do the

18:00 work actually.

18:01 pip install switch.

18:02 There you go.

18:03 That might be a thing.

18:04 Don't do that.

18:04 Yeah.

18:05 Don't do that.

18:05 Switch laying.

18:07 Who knows?

18:07 But yeah, I think it would be a really cool feature if we could make it work for the language

18:11 without actually changing the language because it's already flexible enough.

18:15 Yeah.

18:15 Cool.

18:16 Cool.

18:16 All right.

18:17 Well, thanks for everything, Brian.

18:19 And good to talk with everyone.

18:21 Thank you for listening to Python Bytes.

18:24 Follow the show on Twitter via at Python Bytes.

18:27 That's Python Bytes as in B-Y-T-E-S.

18:29 And get the full show notes at pythonbytes.fm.

18:33 If you have a news item you want featured, just visit pythonbytes.fm and send it our way.

18:37 We're always on the lookout for sharing something cool.

18:40 On behalf of myself and Brian Okken, this is Michael Kennedy.

18:43 Thank you for listening and sharing this podcast with your friends and colleagues.