Transcript #40: Packet Manipulation with Scapy

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.

00:05 This is episode 40, recorded August 21st, 2017.

00:09 I'm Brian Okken. We have a guest host again, filling in for Michael while he's still on vacation.

00:15 This week we have Eric Choo. Did I pronounce that right? Is it Eric Choo?

00:20 Yeah, so I would usually pronounce Eric Cho, but it's all good. Eric Cho. Yeah, okay.

00:24 Well, Eric has agreed to do it with us.

00:27 He's the author of the book, "Mastering Python Networking," and a self-proclaimed network automation nerd.

00:34 That sounds great.

00:35 Let's jump in with Eric's first topic.

00:37 Yeah, sure.

00:38 Hey, Brian.

00:39 So it's good to be here.

00:40 As you mentioned, I first recently finished my book.

00:43 And in the book, I introduced about a dozen or so Python tools and modules.

00:48 And I thought this would be a great chance to kind of introduce three of the top ones, my go-to modules that I mentioned in the book so that I could share with everybody if they proved to be helpful for other people.

01:00 These topics also show up, we have links in the show notes, but you have information about these in your book as well?

01:05 Yeah, totally.

01:06 So these are actually, so for example, the first tool that I will mention a bit, Ansible, you know, I actually spent two chapters in the book on Ansible.

01:15 First just kind of introducing and then the second would be like more advanced topics and so on and so forth.

01:20 the ones that I kind of live by the book as a reflection of the experience I have.

01:24 So okay, great. So let's, let's talk about Ansible.

01:28 Yeah, sure. So if you're, you know, I know we chat a little bit about ops and so on. But, you know, if you're involved in operations of your company, you probably heard of Ansible. So it's a general purpose automation framework. And so not just for networking equipment, but you could automate both, you know, servers, cloud operations, like Azure and AWS through what they call the modules like, you know, in their quote unquote playbooks.

01:55 It's a good framework because I think especially for network engineering, because it has the most vendor support. So in the network engineering world that it's always the tool kind of lives and dies by vendor support. And Ansible has the most network support because they're agentless, and they could execute locally calls the API's on the remote network devices, and therefore allows a lot of flexibility, allows the vendors to shine, and so on.

02:22 That's cool.

02:23 And Ansible is written in Python?

02:26 That's neat.

02:26 Yeah, it is written in Python.

02:28 So that's another thing that's great.

02:29 It's that you obviously start out as a user of the tool.

02:33 But if you know a little bit about Python, you can actually start contributing to the modules rather quickly.

02:39 So I'm a network engineer by nature, and I don't write a lot of code.

02:44 But I was able to write modules for it rather quickly.

02:47 But you can also drive it then, obviously, from if you've got Python scripts or Python modules to control your Ansible tool chain?

02:56 Yeah.

02:57 Or is it something you use outside of that?

03:00 The heart of the tool is actually revolve around playbooks.

03:04 So they have this playbook, and that is in a YAML format.

03:07 And in the YAML format, you can actually specify the modules that you want to use.

03:11 For example, you want to do Cisco iOS for NTP configuration, for example.

03:16 So you use the iOS command module and within your playbook, you only have to specify the parameters for your like NTP server IP address, for example.

03:26 That allows you to quickly spin up and be a user of Ansible, but the iOS command module that is running in Python.

03:33 So, you know, if there's a module that you're, you want to use, but does not exist, you could just go ahead and write that module.

03:41 So for example, for me, I work for ATEN networks and ATEN networks didn't really have a lot of modules.

03:46 So I went ahead and ported one of the modules into our latest API and so on.

03:52 - Okay, cool.

03:53 - Yeah.

03:54 - Well, thanks.

03:55 - Yeah, thanks.

03:55 - Well, I'm gonna jump into a article that I found and I like the topics.

04:01 The article is Python Practices for Efficient Code, Performance, Memory and Usability.

04:07 And I gotta like preface this that I've got a lot of opinions about some of this stuff, but I don't wanna, it's not meant as a disrespect to the author and I'm gonna probably get his name wrong but it's I think it's Satwik Kensal. I wanted to say it because it's a cool name.

04:22 But it's got six different practices he talks about.

04:26 I'll just list them off. Try not to blow off memory. Whether you use Python 2 or 3.

04:31 Write beautiful code. Speed up performance.

04:34 Analyze your code and testing and continuous integration.

04:37 I like that these are, it's a pretty short article but these are it's a really good overview of a lot of things that you should care about when you're getting serious about code.

04:47 So, first off, the memory bit. I think it is important to talk to people about using generators when they're dealing with large data sets because that's something that by default my brain doesn't think generators right off the bat.

05:00 So making sure that generators are used when you can for big things.

05:04 Big number crunching, use NumPy. And a little bit on using classes and strings actually don't see a lot of people using the mentions don't use plus for formatting large strings and I just don't see that that much and then his recommendation for classes if you're going to use classes use slots and I say just use adders if you're going to use classes because adders is awesome Python 2 vs Python 3 kind of goes on this a lot but I guess I just say use Python 3 unless you really have to writing beautiful code he has a lot of good reasons I think for why you should, because you're writing for other people as well as not just yourself, but I'm more solid for why I like...

05:49 I used to be like everybody should just learn how to read other people's code but now I think it's a maintenance cost thing and maybe it's my foray into management but just picking a style, sticking with it, using static analysis it's a cost thing it's gonna save you money. The next couple topics I can't really talk about too much. Speed up your performance, multi-processing versus multi-threading. I guess I'm I gotta admit that I don't really use either in most of my Python code so far. And then analyzing your code with cprofile and memory profiler and stuff. I haven't done enough of that to really comment. And then the last bit I have to testing and continuous integration. I really appreciate that he links to python testing.net for some of these tutorials.

06:33 That's awesome.

06:35 But I think for testing, the shorter answer is use pytest.

06:39 I didn't really tell you all about what his thoughts are, but it's a well-written article, so go read it.

06:44 Yeah, I agree.

06:45 I read it, and I totally agree with your point about Python 2 and Python 3.

06:49 I think it's almost--

06:51 I've read studies on how much users were on Python 3 and Python 2.

06:56 It seems like last year we reached the inflection point where most users are starting to write their new code in Python 3 as opposed to Python 2.

07:03 I don't know if you agree or not.

07:04 Yeah, I think so.

07:05 And also, even a couple of years ago, there was a problem with some really crucial projects and modules and packages not being available for Python 3.

07:16 But I think that's totally changed.

07:18 I think it's just some of the little corner case things that may not even be being used on new projects anymore that are Python 2 only.

07:27 It seems like once Django moved over, it's like the world is wide open now.

07:32 Yeah.

07:32 And I think that has a lot to do with just people don't--

07:35 a lot of new people didn't--

07:36 they're just following the tutorial and doing the default.

07:39 I think the next hurdle will be trying to get all of the default distributions and even Mac to ship with Python 3 instead of Python 2.

07:49 That'd be good.

07:50 Yeah, good luck with that.

07:52 Because I think-- yeah, I think--

07:54 what is it, Ubuntu 16.04 ships with 3.4.

07:57 But default, you know, if you--

07:59 I forgot, when you type in Python, Does it come out with 3.4 or 2.7 on my Mac? It was 2.7, but I've had this Mac for a while So I'm not sure so that's all I got on that one I'm actually pretty interested in this next tool that you're gonna talk about So let's jump into that. Yeah, so the second tool I want to mention is the the scapy program So it's a Python based interactive packet manipulation and library So you could use it both in an interactive mode where you could just kind of fuzzing and trying stuff out And once you're happy with it, you can actually write a program and just reuse it.

08:33 And so the biggest difference between Skapi and other tools like, and I'm just going to be mainly security focused. So like I say, HPING 3, for example. So biggest difference between Skapi and HPING 3 is that for Skapi, you're crafting a packet from the ground up. So you're not really limited to the features and functions that was provided by the program itself. So for example, you know, using Skapi, I was able to write a mail form packet, IP version 3, for example, in a packet and see if it passes through my network, or if it crashes my infrastructure, my servers or anything like that. So mainly, very much security focused tool for me. But I could imagine other people have other use for it. So on the tutorial on the for Skapi, and it's in the the show notes that it shows how to use ping of death, where you have a large payload of the packet and that's a ping, see if it crashes your host.

09:31 It's pretty much an outdated vulnerability, but who knows?

09:34 Some new service comes out, we still have this vulnerability, you have your land attacks, and so on and so forth.

09:41 There's a bunch of useful things that you could test with, or even just fuzzing, or TCP scanning, and stuff like that.

09:47 >> This is actually awesome.

09:49 I can't wait to play with this, And in testing and stuff, like being able to muck with 8 or 11 frames.

09:55 Exactly.

09:56 And yeah, this is great.

09:59 Yes, and it's been around for a while.

10:01 A lot of people, I think if you're running just say something quick and something that's already included in say HP3, go ahead and just use HP3, right?

10:10 Because it's quick, you don't have to spend all this time crafting your packet.

10:14 But for the 20% of stuff that you really need to clamp down on what you want inside of that packet either through the header or the payload, Skappy is kind of your go-to tool for that and it's based in Python. So and use the Python shell for interactive to play with it so that's even better. Oh that's great, cool. You definitely have to pass that along with some of the people I know. Yeah let me know how it goes, you know. I wanted to talk about headless Chrome which I haven't actually played with but I just heard about it because Miguel Grinberg wrote a post on using headless Chrome with Selenium and With testing web applications. It's a common practice to use Selenium and Python but usually a lot of people are using a Firefox as their default browser to interact with a website and This is a little he starts off with basically an example that does use Think he's using Firefox but if not, it's using a normal. Yeah starts off the Firefox example and then shows how to change it He's using unit tests I got to get on him about using by test more but Using a headless Chrome and then the nice thing about that is you don't get the with a headless browser You don't have the the dialog window or the the window pop up all the time when you're running tests. So The magic trick I guess is that there's an option to to selenium with the web driver you pass it a headless option but like I said all this we got a link to Miguel's post in the show notes and he just shows you how to do it. Yeah, I'm a big fan of Miguel's work, you know obviously he did the mega tutorial on flask and I learned a lot from that and also back to his Kickstarter campaign that's gonna come out next year when he's revising his Mega tutorial so I've definitely recommend anything that anything you publish I would just recommend go read it Yeah, he's a super nice guy, too. He was the only person in Portland that I knew that was going to EuroPython He took a bunch of rocket stickers for the pytest book He just like carried him over there with him so I could have him handed out there. So that was cool Yeah, I saw him on the last year's PyCon in Portland, but I was just too shy to go up and say hi, right?

12:32 So, you know, maybe next time I'll just say hey Miguel like it work. You should he's a he's a super nice guy He's pretty underspoken.

12:40 He won't go out and tell you how awesome he is right off the bat.

12:45 We all know that's wrong.

12:46 He is.

12:46 That's right.

12:47 Again, I'm excited looking at this list.

12:49 It's kind of fun getting new people in to talk about things.

12:52 I love graph visualization tools.

12:55 So tell me about GraphViz.

12:56 Yeah, so as a network engineer, it seems like we spent a large chunk of our time, and rightfully so.

13:03 It's part of our job to document the network.

13:06 and you spend a lot of time drawing the graph and using, you know, whether your choice of tool is a Visio or it's a OmniGraffle, you spend a lot of time diagramming your network.

13:16 But by the time you're done, it's almost the case that your graph is outdated, right?

13:22 Like the network actually just grows.

13:23 You either add a new VLAN, you add a new box, or just whatever it is that it's almost, because you take so long and it's not automated, you're actually manually graphing these network diagrams.

13:36 So this is especially a big problem with the data center being bigger and bigger.

13:40 So now you have what they call the factory network, where the middle is huge, where you have server to server communication as opposed to server to client communication.

13:50 So this huge network presents a big problem because you're having hundreds of links between each of the network layers.

13:58 And doing it manually is just not an option anymore.

14:01 So I think a couple of years ago, I went out and started thinking about trying come up with a solution and GraphViz is what I saw where you could actually basically describe your network in terms of a text file. So I would say in terms of nodes and edges and say these two nodes should be placed together at the very top tier and these four nodes would be in the middle tier and they should line up and so on so forth. So GraphViz is a great tool for doing that and there's a Python module that's also named GraphViz that allows you to come up with this what they call like a dot file where you know you you come you use the Python module you generate this dot file you run it through graph viz and voila you have this beautiful graph that's up to your liking so the only thing that I guess it's me not being didn't learn it enough that you know like coloring the graph like say if I wanted to color all the spine boxes or if I want to color all the leafs but leave devices in a certain color I still haven't learned how to do that.

15:01 So sometimes when I do need that colorization, I kind of launch OmniGraffle Pro.

15:07 And you could import that graph and then color it to my liking.

15:11 But it's an awesome tool.

15:12 It allows you to automatically generate your graph.

15:16 So it allows you to have less of a drift between your actual network versus your diagram.

15:22 OK.

15:23 Nice.

15:24 Actually, I didn't know about the Python module for it, or the Python package to use it.

15:32 Because GraphViz has been around for a long time.

15:35 That's right.

15:35 But working with the dot files, even though the format's pretty easy, it still is something else to learn.

15:42 And yeah, I'll have to check out the Python package, because that's pretty cool.

15:47 I like that.

15:48 Yeah, so I think just the biggest--

15:50 I mean, if you want a quick graph, you could use PyGal or Matplotlib even, and you just quickly draw that graph.

15:57 But GraphViz to me is the most powerful.

16:00 And I know a lot of companies use that for drawing their eye candy kind of graph.

16:06 Like you've seen the Facebook graph where it just have lines all over the globe and so on.

16:11 I mean, that's the kind of graph that GraphViz is capable of doing really, really quickly.

16:14 Well, yeah.

16:15 I don't know if those are helpful, but yeah.

16:18 Those are helpful for marketing materials.

16:20 Yeah.

16:21 Some of the automated code processors use it for drawing class hierarchies and stuff.

16:26 Right, right.

16:27 I mean, and in a shameless plug for my book, I also use the, the graphics tool.

16:31 I use an example of where you're inside of a, so use your Python script to go query your network devices and say, basically say, tell me who your neighbors are and you just kind of walk layer by layer by layer and you construct a graph is file that represents your network in real time.

16:49 So every time you could just execute that script and you will give you your most up to date and network graph automatically.

16:56 So I think that's kind of a good use case for the tool, where, like I said, the biggest problem that I face with network graphs is there's always this drift between the actual network topology versus the graph that you present.

17:08 And if the graph is wrong, nobody's going to use it.

17:10 Yeah.

17:11 Oh, hey, I just remembered a cool example of the dot files.

17:15 One of the projects I work on, one of the things that we log is basically a running state machine example, an internal state machine that is basically a state machine as it's running and where we are currently.

17:32 That's being output as a log file in the dot format so that whenever we get a crash or something, we can take that log file and just throw it through GraphViz and see exactly where we're at visually.

17:48 So even a manager could understand it.

17:52 Yeah, like me.

17:53 Yeah.

17:54 So cool.

17:54 Yeah.

17:55 Last topic I'd like to bring up is PyCascades.

17:59 So this is a new thing this year.

18:01 I think it's-- I'm going to get it wrong, but I think it's a whole bunch of different Python groups in the area coming together to try to do a regional Northwest Python conference.

18:12 Oh, interesting.

18:12 It is in January.

18:14 It's January 22nd, 23rd.

18:16 And then they're going to do sprints the 24th.

18:18 But I like the format.

18:20 I like what they're doing.

18:21 The tickets are not very expensive. I think they're like a hundred bucks or something. Okay, they're not bad and it's in Vancouver, BC Which I even though I live in Portland, I've never been there So I'm looking forward to going and seeing that and the call for proposals for talks is the 28th So that's like Monday or something like that So when this comes out people only have a couple days, but I think there's still slots open So, please check it out. And if you want to do a talk local talk, I think this is good The talk slots are 25 minutes too, which is a nice length.

18:51 I think that anybody that has a, if you've got expertise on really anything in your job, 25 minutes shouldn't be too hard to fill.

18:59 So I'm going to submit a couple talks and encourage anybody else to as well.

19:04 Yeah.

19:04 So Vancouver is about three hours away from me as well.

19:07 So I have to check it out.

19:08 I mean, I've never been to the PAI Cascades, but it seems like a good format.

19:13 And like you said, tickets are cheap.

19:15 Yeah.

19:15 Well, yeah, it's the first one this year.

19:17 Oh, okay. Got it. Yeah. Yeah, that's our six topics that went by really fast. It did it did so tell me a little bit About what's up with you? How long is mastering Python networking been out?

19:29 So it's been out since a June 28th, but the only the digital version was out So I think you could start ordering the physical book like mid-july So it's been out for about a month and a half I have the link in the show notes go check it out. If you if you could I really appreciate any kind of feedbacks or Just any criticism good bad ugly, whatever it is, you know, I'm open to it. So I appreciate that nice Yeah, so what's happening with you Brian?

19:54 Well, the this last week was the fixing up the cop I got got the copy editing changes for the Pyth test book back Oh, that's right. And I just got those in yesterday That's gonna be out and I'm so excited to get the the physical copy out. So the physical form. Yeah Yeah, the physical form will be like in October, but I'm like as of like there's a like one minor tweak I got a fix tonight. Other than that, my hands are completely off. Hopefully I keep saying this every week But once it goes in past copy editing and through the rest of it It's really a author doesn't get to interfere process until it comes out as a hard copy So from start of this project to actually at this point, how long did it take for you?

20:37 That's a long answer, but my interaction with Pragmatic started, I think, June of last year, June of 2016. So a little over a year, but I think the first four or five chapters, I think I rewrote those like four times to try to get it right. Yeah. Initially, I was kind of coming along the same lines as the tutorials I have on the Python testing.net, but getting a lot of trying to understand how to write in a book format instead of a blog format is a it was a learning experience for me and but still even after doing all that I think that I don't think it looking back a year isn't that terrible but there's a one of the other authors Python authors is Luciano Romalo I wish I knew which book he wrote it's like I'm gonna get it wrong he's one of the reviewers and his feedback was really influential but added like two or three weeks of my work.

21:34 For me, I think the biggest issue was always not on myself, but on the people around me.

21:40 It's almost like I have to be concentrated and focus on the book, and then so the family has to go away.

21:47 My wife has to take on the burden of taking the kids away from me for the duration I was writing the book.

21:52 So, yeah, it's definitely both a draining experience, but I'm very excited for you.

21:57 I think when you see that paper copy, you're just going to be like, "Yay!" You know, makes it real.

22:03 Yeah, it definitely is going to be cool.

22:06 Like you said, trying to find time within your life and trying to get distractions out.

22:10 Because like you, I've got a full time job, and kids, and family.

22:14 So it was quite a few late nights at the coffee shop trying to get away from everything.

22:19 But it'll be out.

22:20 So that's all we got for today.

22:22 And thanks a ton for helping out, Eric.

22:26 And we'll keep in touch.

22:27 Yeah, absolutely.

22:28 Thanks, Brian.

22:28 being fine.

22:29 Thank you for listening to Python Bytes.

22:32 Follow the show on Twitter via @pythonbytes.

22:36 That's Python Bytes as in B-Y-T-E-S.

22:40 Get the full show notes, including links at pythonbytes.fm.

22:43 If you have a news story you'd like featured, visit pythonbytes.fm and send it our way.

22:49 We're always on the lookout for sharing something cool.

22:52 This is Brian Okken.

22:53 On behalf of myself and Michael Kennedy, thank you for listening and sharing this podcast with your friends and colleagues.