Transcript #43: Python string theory, v2

00:00 Hello and welcome to Python Bytes where we deliver Python news and headlines directly to your earbuds This is Python Bytes episode 43 recorded on September 13th 2017 and I'm Michael Kennedy and I'm Brian Okken. Hey, it's great to be with you guys good to be with you Brian We have a bunch of really cool stuff lined up Like there's a few things that really surprised me and maybe even inspired me to go create this project that we were just chatting about before we hit record.

00:28 So very, very cool stuff.

00:30 But before we get to it, let's just say thank you to Rollbar.

00:33 Those guys are sponsoring the show.

00:34 You check out their offers at pythonbytes.fm/rollbar, and we'll get to them later.

00:39 I want to talk about taking the future and sending it into the past, like Marty McFly style.

00:44 What do you think?

00:45 - Yeah.

00:47 Yeah, this popped up recently, just this last week.

00:51 And I'm not sure what the impetus was, but it is future f-strings.

00:56 It's a project.

00:58 You could just do pip install future-fstrings.

01:01 And then you've got a little bit of a--

01:05 it isn't a library that you include.

01:08 It changes the encoding.

01:09 So I'll just jump to the chase.

01:11 It's a backport of f-strings to even down to Python 2.7.

01:16 And I've tried it in 2.7.

01:18 I haven't tried it in anything else.

01:20 but it is kind of fun to be able to play with f-strings everywhere.

01:24 So I had very mixed thoughts about this when you told me about this project.

01:28 On one hand, I look at it like, you know, you're taking one of the really amazing and smooth features of Python 3.6 and making it available in legacy Python, which is kind of like, do we really want to like encourage that sort of thing?

01:43 But at the same time, I feel like this also lets you, as you're migrating your code, move to the latest greatest syntax in Python 2.

01:53 But then when you actually make the switch, you're not going to have to go like, well, we used format because that was the only that was the best option in Python 2.

02:00 But now we've got f-strings.

02:02 And so I feel like that that's really cool.

02:04 The other area where I think it's great is Python 3.

02:07 There's plenty of people running Python 3.5.

02:10 That's the default on Ubuntu right now and things like that, that don't have f-strings.

02:14 I've decided after further contemplation that this is a really cool project and I want to check it out. So tell people what f-strings are, right? I mean most people maybe know but what is what's this thing this f-string thing? Well it's so if you're familiar with the format like the format part that you could be I don't know how long ago you start using that I think that's available in 2.7. Yeah I think so too. In the string that you're printing you can like leave little brackets to to show where you're gonna insert something and then you say like dot format and then all the things that you're going to put into the string.

02:48 In f-strings, you just put an F before the string and then within your brackets, you just put the variable name for whatever you're going to put in there or any expression can go in there.

02:59 Right.

02:59 So you could say like F quote, hello, curly thing dot upper and like uppercase the thing just in that expression or something like that, right?

03:07 Yeah.

03:07 Yeah.

03:07 That's really cool.

03:08 Yeah.

03:09 At first I was like, I don't know if I like it, but once you start using f-strings, you can't go back.

03:14 That's really one of the reasons why I like this project is because I usually use Python 3.6, but there are times where I have to write some stuff in 3.5 or 2.7 or something, and I don't want to go back.

03:27 Yeah, I know.

03:28 I hear you.

03:28 That's great.

03:29 I'm going to put a lot of tests around it, so I'm not going to rely on this yet, because I don't know if it's really ready for primetime for everywhere.

03:37 So I would say use caution.

03:38 Dip into this slowly if you're going to use it.

03:41 But it's a fun thing.

03:43 I'm not sure I like how it's doing it, though.

03:46 It's doing it in an interesting and odd way, yeah.

03:50 I don't know how else it would do it.

03:51 But it's interrupting the encoding.

03:54 Yeah, it installs a separate encoder.

03:56 So it extends the UTF-8, I think, the UTF-8 encoding to also do this.

04:02 So if you're not using UTF-8, I don't know.

04:05 Maybe you could hack it to do something like you're using.

04:08 Yeah, yeah, it's pretty cool.

04:09 You put a little encoding flag at the top and you install a special encoder and off it goes.

04:15 Pretty interesting.

04:16 All right.

04:17 I want to stick with this Python 3.6 feature angle, even though that's bringing that stuff to other versions of Python.

04:23 I want to talk about the fun of reinvention.

04:27 This was a really amazing presentation done by David Beasley.

04:32 It's done in the style of some of his other presentations, and in my mind, the way technical presentations should be done.

04:38 You should be up there writing code, showing how things work, not just slides and pictures and stuff, but showing things.

04:47 What he shows though is insane and incredible.

04:50 Your mind will probably explode if you see this.

04:52 You'll definitely come away with a greater appreciation for Python 3.6.

04:57 What he does is he goes and builds upon some of the new 3.6 features that are below the surface that a lot of people wouldn't even know about, like hooking into when a class gets defined. So you can basically redefine what that means and things like that. So for example, he says, Well, let's talk about type ends. These type ends are cool, but they don't do anything, we can add these constraints and assertions inside of our code. And that does something, but we can't really check that in the tooling. So what if we could rewrite, create this little framework using inheritance and all sorts of other stuff that lets us add these constraints as type hints and then they're enforced at runtime automatically. It's pretty amazing.

05:39 That's incredible. Yeah. Yeah. So whether you think it's a good idea or a bad idea to do that, the fact that you can do that is really really cool. So you can basically set up a function and as like a type hint constraint you can say and this is a positive integer between 0 and 200 and it will actually check that. It's really something. But mostly it's an amazing look inside. Yeah. It's an amazing look inside of Python 3.6 and down to the low level stuff.

06:07 So Brian, knock knock, who's there?

06:12 Yeah, I am.

06:13 Sound recognition is there.

06:15 Yeah, sound recognition.

06:16 So there's an interesting article called Sound Pattern Recognition with Python.

06:21 And I really liked just the--

06:24 I didn't know you could do this with SciPy.

06:26 So there's apparently part of SciPy.

06:29 There's scipy.io.wav file.

06:32 and I'm not sure if there's other types you can do, use.

06:36 But this article uses SciPy to read a wave file and then just has it as data, like an array of numbers, and then does some math on it.

06:47 And it does, this particular one is assuming that you've got some wave files that have NOX in them and trying to detect where the NOX are and how far away they are.

06:57 And just some basic little logic to try to figure out, Yeah, like I said, where the peaks are.

07:02 I think the author noticed that you need to have some minimum values for how you tell where the peaks are, and then some distance between them so you don't have like ringing within one peak.

07:13 I like it because it's kind of a--

07:14 I think it would extend easily to do--

07:17 I'd like to take it and run with it and try to do some basic oscilloscope measurements with this kind of a thing.

07:23 That'd be fun.

07:24 Yeah, it'd be super fun.

07:25 It's really accessible.

07:26 Like the actual code to figure out the NOX to process the WAV file.

07:31 It's really small and approachable.

07:33 And to me, what I thought of when I heard this was, "Oh, you could build something that is actually kind of smart." So it shows you how to identify knocks, right?

07:42 And you should be able to identify a door jiggling, a key noise.

07:46 You should be able to identify a few of these really distinct patterns.

07:51 And if you could say, identify what does a door jiggle sound like, what does a knock, What is a doorbell?

07:57 What does a key sound like?

07:59 You could put a little raspberry pie just by your front door that says, "Someone's home.

08:03 Someone unlocked the door.

08:04 Someone's knocking." You know, just cool stuff.

08:07 Wouldn't that be...

08:08 I mean, it seems like that's a weekend project with this.

08:10 It doesn't seem major.

08:11 Yeah, that would be fun.

08:12 You could hook it up with Twilio and get texts when somebody's knocking at your door.

08:16 Right, hook it up with a camera to take a picture of whoever's out there, see what they're doing, right?

08:21 Yeah.

08:22 Yeah.

08:23 Send yourself a note, "Hey, kids came home, right?

08:24 They unlocked the door at three o'clock when school's out," or whatever.

08:26 That'd be fun.

08:27 It sure would.

08:28 All right.

08:29 So I think people should take this idea and run with it.

08:31 And if they do, they should send us a message or even better, go to pythonbytes.fm/43 and leave a comment at the bottom about what they created.

08:39 Yeah, definitely.

08:40 Definitely.

08:41 Or even better, they could write a blog post and then email that to us and we'll highlight it on the show.

08:46 Yeah, maybe we'll even cover it.

08:47 That'd be awesome.

08:48 Yeah.

08:49 Yeah.

08:50 Cool.

08:51 So before we get to the next item, which is sort of mind blowing, I want to talk about roll bar.

08:53 We talk about roll bar a lot probably, but that's because they're great.

08:57 Basically, if you run any sort of web app, you should have real-time error monitoring and reporting.

09:03 If something goes wrong with the website, for example, Python bytes.fm, we'll get a notification.

09:08 It could be in Slack, it could be on our phone.

09:10 It'll have all the details.

09:12 This person went to this URL in this situation.

09:15 Here's the variables they passed to the function that caused a crash, things like that.

09:19 If you want to get this for your site or application, just go to pythonbytes.fm/rollbar and check it out.

09:27 We've talked a lot about async stuff, async both on the server and on the client.

09:33 It's pretty mind-blowing stuff, but it turns out that all changes to software, there's cascading consequences of adopting a new model or trying something new.

09:45 One of the cascading changes is in many of the high-performance async processing loops, I'm thinking async I/O, but also UV loop, which is a really super fast, powerful one.

09:57 Things like Sanic, I think Sanic is based upon it, the web framework and so on.

10:02 Those things are ultra fast because they rely on async I/O, but async I/O doesn't use threads.

10:10 That means thread local storage doesn't have any meaning anymore when all of these concurrent operations are running on the same thread.

10:16 Okay.

10:17 All right.

10:18 So like if I was running a website, I might store into thread local storage like the cookies or the authentication.

10:24 And then later I might ask for it back.

10:26 But that could be changed because you have a blocking IO that released the thread that somebody else then wrote their cookie to, right?

10:32 So it's kind of like crazy.

10:33 And there's also another really interesting places like Decimals.

10:37 Working with Decimals does this NumPy error state, warnings.catchWarnings, profiling, tracing, all these things use this kind of stuff, right?

10:46 But it doesn't work in this async/await world.

10:49 So we have a new PEP, PEP 550, that defines a new execution context.

10:54 And this is from the guys at magic.io who created UV loops.

10:58 So like their motivation is really obvious, like they want UV loop to work.

11:03 And this is kind of something in a way.

11:04 So the PEP adds a new generic mechanism for ensuring consistent access to non-local state in the context of out-of-order execution such as generators and coroutines.

11:16 Wow.

11:17 It's pretty fascinating, right?

11:18 Yeah, it's very fascinating.

11:19 I hadn't even realized that.

11:21 I mean, you think it through, obviously, it's a problem, but I hadn't realized like, oh my gosh, like some of these like low-level things are just going to be broken.

11:27 Like how, for example, state is processed in a web request.

11:30 I'll be very interested to watch this and see where it goes.

11:33 - Yeah, so maybe it makes it into Python 3.7, maybe not, but it's kind of cool that this is not just like, oh, we made UV work properly, but we're going to make all of Python work properly.

11:43 So they've got some really nice examples on the article we're linking to.

11:47 - That's really cool that they did it instead of just trying to hack their own way.

11:51 - Yeah, for sure.

11:52 Speaking of threads and processing and all that.

11:55 - I often work in single threaded or single process sort of tasks.

12:00 I really liked this article called intro to threads and processes in Python. It's a very accessible beginner's guide to parallel programming and really kind of what's the difference? Where would you use threads? Where would you use processes? And it's even got pictures with a little some algorithms to see to show how like if you're certain kind of a job. I can't remember what the job is that he's doing.

12:23 He's doing some data science project for based on Amazon and trying to analyze a bunch of different stuff from space and so he's got to run tons of algorithms and tweak them with different parameters and stuff like that. It shows like basically if you use a one thread, two threads, or four threads and then the same with processes how it's affected with running times and you can watch to see that things are running in parallel. A good use of a little simple diagrams too that's cool. Yeah the pictures are great. The rundown is, conclusion at the end is if you're waiting on I/O for something, threads are just fine. And if you are CPU heavy, then you want to go with multi-processing.

13:04 Yep, until the Gillectomy happens, which who knows if that will, but until then, this is definitely a good introduction to it.

13:09 Yeah, and I'm actually, I'm one of the, I don't know, I think there's probably a lot of people, but having the GIL doesn't really bother me. It does simplify how you program.

13:19 I don't think it's that terrible.

13:20 Yeah, it's, you know, it's in the extreme cases. I think the main group of people who suffer under the GIL are those doing computational CPU computational work.

13:29 Outside of that you're mostly okay.

13:33 Okay. Yeah. So in that like async and await and stuff like that I'm going to expose my knowledge of this stuff. So async and await is that dealt with with multiple threads then? No it's all one thread but it has a mechanism to say "Anytime you hit blocking I/O, go put this part of the code to sleep and allow it to run another task, another thing, until it hits blocking I/O." The idea is most of the time you're waiting on databases or networks or something like that.

14:06 Then you wake up for a second, you do a little processing, and then you go back to wait on another network or database or web service.

14:12 As long as that's the kind of stuff you're waiting on, the GIL is not a problem because it gets released.

14:17 but if you're like trying to compute Pi with a, you know, power series or something, something like that, then it would just, it would stop and the async wouldn't help you with that.

14:25 Yeah, I just wanted to, so the async and awaiter is going to use, it's going to be in the same sort of family as when threads would be good for you.

14:34 Yes, yeah, it's in exactly this similar spot, yeah, but just more efficiently.

14:38 So let's talk about another low-level thing and I found this, I think it maybe even was recommended by a listener, some low-level part of it, and then I found the whole thing.

14:49 One of the problems that we can have when we're doing unit testing is working with files, right? That can be a serious pain.

14:55 Yeah.

14:55 Yeah, and if you want to save your files to, like, say, user storage, or you want to process different types of files, like zip files, regular files, these are all sorts of pains that you have to deal with.

15:08 But there's this really cool project called Alternative File Systems for Python that makes this all seamless.

15:16 The idea is you can work with files and directories in zip archives, in memory, on the cloud, or just as easy as if they're on your hard drive.

15:25 They give some examples in this project saying you could write your code now just with open, you know, such and such as fin and start working with it.

15:34 Then you decide what that open means.

15:36 Does that open mean normal files?

15:38 Does that open mean something on S3?

15:41 Things like that.

15:42 You can unit test your code, you can have in-memory files, but open reads and writes to those in-memory files.

15:48 You can upload your files to like S3 or OneDrive or something like that by just writing to that folder, you know, in that file system and all sorts of stuff like that.

15:59 Yeah, so the memory file system or the temporary file system that they have would be great for like parallelizing tests and things like that.

16:06 Right, keeps it all separate and isolated.

16:08 So some of the back ends they have is they have application data.

16:12 This is like the special user data locations in various operating systems.

16:16 They have Amazon S3 file systems.

16:19 They have FTP memory.

16:21 Let's see what else that's pretty cool.

16:23 They've got zip for reading and writing zip files.

16:27 And they also have like SSH file systems for writing to remote servers, all kinds of stuff.

16:32 And even tar files, good old tar files.

16:35 Good old tar files.

16:36 Yeah, you can just width open on those babies.

16:38 Yeah, so this is really pretty interesting and I haven't tried it yet, but it looks quite promising and it's extensible so you can add more of these backends if that makes sense for you.

16:47 That's awesome.

16:48 I actually think this is pretty fun.

16:49 I got to play with this.

16:50 Yeah, yeah, me too.

16:52 So definitely check that out.

16:53 And Brian, that's it for our six items for the week.

16:56 That's incredible that we've already done.

16:59 I know.

17:01 They're all fun and all interesting.

17:02 I really enjoyed researching them this week.

17:05 So what else are you up to?

17:06 I got an interesting email this morning saying that they've taken off the beta off of the Python testing with pytest on the Pragmatic website and you can order the book now and it's supposed to ship next Monday.

17:21 That is awesome.

17:22 Congratulations.

17:23 Yeah, thanks.

17:24 So that is what I've been thinking about.

17:27 I bet.

17:28 Nice.

17:29 I've been thinking about Switch.

17:30 Well, I have been too as of this afternoon.

17:33 I'm working on a project that would add the switch statement to the Python language without extending it, just like a class that you can basically use, but it's a pretty clever use of defining blocks and stuff.

17:45 I think it might be interesting.

17:47 I'll put a link to a gist or a GitHub repo or something for you at the end of the show notes and you guys let us know.

17:52 I really do want feedback on that because I want to try to nail this and I think if we could get it into something we could stick on PyPI or have you do the work actually.

18:01 pip install switch.

18:02 There you go.

18:03 That might be a thing, don't do that.

18:05 >> Yeah, don't do that.

18:06 >> Switching, who knows.

18:07 But yeah, I think it would be a really cool feature if we could make it work for the language without actually changing the language because it's already flexible enough.

18:15 >> Yeah.

18:16 Cool.

18:17 >> Cool.

18:18 All right.

18:19 Well, thanks for everything, Brian, and good to chat with everyone.

18:23 Thank you for listening to Python Bytes.

18:24 Follow the show on Twitter via @pythonbytes.

18:27 That's Python Bytes as in B-Y-T-E-S.

18:30 get the full show notes at pythonbytes.fm.

18:33 If you have a news item you want featured, just visit pythonbytes.fm and send it our way.

18:38 We're always on the lookout for sharing something cool.

18:41 On behalf of myself and Brian Okken, this is Michael Kennedy.

18:44 Thank you for listening and sharing this podcast with your friends and colleagues.