Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to your earbuds.
This is episode 40, recorded August 21st, 2017.
I'm Brian Okken. We have a guest host again, filling in for Michael while he's still on vacation.
This week we have Eric Chu. Did I pronounce that right? Is it Eric Chu?
Yeah, so I would usually pronounce Eric Cho, but it's all good.
Eric Cho. Yeah, okay. Well, Eric has agreed to do it with us.
He's the author of the book Mastering Python Networking and a self-proclaimed network automation nerd.
That sounds great. Yeah. Let's jump in with Eric's first topic.
Yeah, sure. Hey, Brian. So it's good to be here.
As you mentioned, I first recently finished my book.
And in the book, I introduced about a dozen or so Python tools and modules.
And I thought this would be a great chance to kind of introduce three of the top ones, my go-to modules that I mentioned in the book,
so that I could share with everybody if they're proved to be helpful for other people.
These topics also show up. We have links on the show notes, but you have information about these in your book as well?
Yeah, totally. So these are actually...
So for example, the first tool that I will mention a bit, Ansible, you know, I actually spent two chapters in the book on Ansible.
So first, just kind of introducing, and then the second would be like more advanced topics, and so on and so forth.
So these are the ones that I kind of live by. The book is a reflection of the experience I have.
Okay, great. So let's talk about Ansible.
Yeah, sure. So if you're, you know, I know we chatted a little bit about ops and so on, but, you know, if you're involved in operations of your company, you probably heard of Ansible.
So it's a general purpose automation framework.
And so not just for networking equipment, but it could automate both, you know, servers, cloud operations like Azure and AWS through what they call the modules, like, you know, in their quote unquote playbooks.
It's a good framework because I think especially for network engineering, because it has the most vendor support.
So in the network engineering world that it's always the tool kind of lives and dies by vendor support.
Ansible has the most network support because they're agentless and they could execute locally, calls the APIs on the remote network devices, and therefore allows a lot of flexibility, allows the vendors to shine and so on.
That's cool. And Ansible is a, it's written in Python? That's neat.
Yeah, it is written in Python. So that's another thing that's great. It's that you obviously start out as a user of the tool.
But if you know a little bit about Python, you could actually start contributing to the modules rather quickly.
So I'm a network engineer by nature, and I don't write a lot of code. But you know, I was able to write modules for it rather quickly.
But you can drive it, you can also drive it then obviously from if you've got Python scripts or Python modules to control your Ansible tool chain?
Yeah.
Or is it something you use outside of that?
The heart of the tool is actually revolve around playbooks. So they have this playbook, and that is in a YAML format. And in the YAML format, you can actually specify the modules that you want to use. For example, you want to do Cisco iOS for NTP configuration, for example.
So you use the iOS command module. And within your playbook, you only have to specify the parameters for your like NTP server IP address, for example, that allows you to quickly spin up and be a user of Ansible. But the iOS command module that is written in Python.
So, you know, if there's a module that you're, you want to use, but does not exist, you could just go ahead and write that module. So for example, for me, I work for A10 networks, and A10 networks didn't really have a lot of modules. So I went ahead and ported one of the modules into our latest API and so on.
Okay, cool.
Yeah.
Well, thanks.
Yeah, thanks.
Well, I'm going to jump into a article that I found. And I like the topics. The article is Python practices for efficient code, performance, memory and usability. And I got to like preface this that I've got a lot of opinions about some of this stuff. But I don't want to it's not meant as a disrespect to the author. And I'm going to probably get his name wrong. But it's I think it's Satwick Kensal. I wanted to say it because it's a cool name.
But it's got six different practices that he talks about. I'll just list them off. Try not to blow off memory, whether you use Python 2 or 3, write beautiful code, speed up performance, analyze your code and testing and continuous integration. I like that these are it's a pretty short article. But these are it's a really good overview of a lot of things that you should care about when you're getting serious about code. So the first off the memory bit, I think it is important to talk to people about using generators.
When they're dealing with large data sets, because that's something that by default, my brain doesn't think generators right off the bat. So making sure that generators are used when you can for big things, big number crunching, use NumPy. And a little bit on using classes and strings.
Actually, I don't see a lot of things. Actually, I don't see a lot of things. Actually, I don't see a lot of people using the mentions don't use plus for formatting large strings. And I just don't see that that much.
And then his recommendation for classes, if you're going to use classes, use slots. And I say just use adders if you're going to use classes, because adders is awesome. Python 2 versus Python 3 kind of goes on this a lot. But I guess I just say use Python 3 unless you really have to.
Writing beautiful code. He has a lot of good reasons, I think, for why you should, because you're writing for other people as well as not just, not just yourself.
But I'm more solid for why I like, I used to be like, everybody should just learn how to read other people's code. But now, I think it's a maintenance cost thing. And maybe it's my foray into management.
But just picking a style, sticking with it, using static analysis, it's a cost thing. It's going to save you money.
The next couple topics I can't really talk about too much. Speed up your performance, multiprocessing versus multithreading.
I guess I got to admit that I don't really use either in most of my Python code so far. And then analyzing your code with cprofile and memory profiler and stuff. I haven't done enough of that to really comment.
And then the last bit, I have to testing and continuous integration. I really appreciate that he links to pythontesting.net for some of these tutorials. That's awesome.
But I think for testing, the short answer is use pytest. I didn't really tell you all about what his thoughts are, but it's a well written article. So go read it.
Yeah, I agree. I read it. And I totally agree with your point about Python 2 and Python 3. I think it's almost I've read studies on how much, you know, users were on Python 3 and Python 2.
It seems like last year we reached the inflection point where most users are starting to write their new code in Python 3 as opposed to Python 2. I don't know if you agree or not.
Yeah, I think so. And also, even a couple years ago, there was this, there was a problem with some really crucial projects and modules not and packages not being available for Python 3. But I think that's totally changed. I don't, I think it's just some of the little corner case things that may not even be being used on new projects anymore that are Python 2 only.
It seems like once Django moved over, it's like, the world's wide open now.
Yeah, and I think that has a lot to do with just people don't, a lot of new people didn't, they're just following the tutorial and doing the default.
I think the next hurdle will be making, trying to get all of the default distributions and even Mac to ship with Python 3 instead of Python 2. That'd be good.
Yeah, good luck with that.
Because I think, yeah, I think, what is it, Ubuntu 16.04, it shifts with 3.4. But default, you know, if you, I forgot, when you type in Python, does it come out with 3.4 or 2.7?
On my Mac, it was 2.7. But I've had this Mac for a while. So I'm not sure. So that's all I got on that one. I'm actually pretty interested in this next tool that you're going to talk about. So let's jump into that.
Yeah, so the second tool I want to mention is the SCAPI program. So it's a Python-based interactive packet manipulation and library. So you could use it both in an interactive mode where you could just kind of fuzzing and trying stuff out.
And once you're happy with it, you can actually write a program and just reuse it. And so the biggest difference between SCAPI and other tools, like, and I'm just going to be mainly security focused. So like, I say HPing 3, for example.
So the biggest difference between SCAPI and HPing 3 is that for SCAPI, you're crafting a packet from the ground up. So you're not really limited to the features and functions that was provided by the program itself.
So for example, you know, using SCAPI, I was able to write a mail form packet, IP version 3, for example, in the packet and see if it passes through my network, or if it crashes my infrastructure, my servers or anything like that.
So mainly very much security focused tool for me, but I could imagine other people have other use for it.
So on the tutorial for SCAPI, and it's in the show notes that it shows how to, you know, use ping of death where you have a large payload of the packet and that's a ping.
See if it crashes your host.
It's pretty much an outdated vulnerability, but who knows, right?
Like some new service comes out, we still have this vulnerability, you have your land attacks, and so on and so forth.
So there's a bunch of useful things that you could test with, or even just fuzzing or, you know, TCP scanning and stuff like that.
This is actually awesome.
I can't wait to play with this.
And in testing and stuff, like being able to muck with 8 or 211 frames.
Exactly.
And yeah, this is great.
Yes.
And it's been around for a while, you know, a lot of people, I think if you're running just say something quick and something that's already included in say HP 3, go ahead and just use HP 3, right?
Because it's quick.
You don't have to spend all these time crafting your packet.
But for the 20% of stuff that you really need to clamp down on what you want inside of that packet, either through the header or the payload, SCAPI is kind of your go-to tool for that.
And it's based in Python.
And use the Python shell for interactive to play with it.
So that's even better.
Oh, that's great.
Cool.
You definitely have to pass that along with some of the people I know.
Yeah, let me know how it goes, you know.
I wanted to talk about Headless Chrome, which I haven't actually played with, but I just heard about it because Miguel Grinberg wrote a post on using Headless Chrome with Selenium.
And with testing web applications, it's a common practice to use Selenium and Python.
But usually a lot of people are using Firefox as their default browser to interact with a website.
And this is a little, he starts off with basically an example that does use, I think he's using Firefox, but if not, it's using a normal.
Yeah, starts off the Firefox example and then shows how to change it.
He's using unit test.
I got to get on him about using pytest more.
But using a Headless Chrome.
And the nice thing about that is you don't get the, with a Headless browser, you don't have the dialogue window or the window pop up all the time when you're running tests.
So the magic trick, I guess, is that there's an option to Selenium with the web driver.
You pass it a Headless option.
But like I said, all this, we got a link to Miguel's post in the show notes and he just shows you how to do it.
Yeah.
I'm a big fan of Miguel's work.
You know, obviously he did the mega tutorial on Flask and I learned a lot from that and also backed his Kickstarter campaign that's going to come out next year when he's revising his mega tutorial.
So I would definitely recommend anything that, anything he published, I would just recommend go read it.
Yeah.
He's a super nice guy too.
He was the only person in Portland that I knew that was going to Europython.
He took a bunch of rocket stickers for the Pi Test book.
He just like carried them over there with him so I could have him handed out there.
So that was cool.
Yeah.
I saw him on the last year's Pi Con in Portland, but I was just too shy to go up and say hi.
Right.
So, you know, maybe next time I would just say, hey, Miguel, like your work.
You should.
He's a super nice guy.
He's pretty underspoken.
He won't go out and tell you how awesome he is right off the bat.
We all know that's funny.
That's right.
So again, I'm excited looking at this list.
It's kind of fun getting new people in to talk about things.
I love graph visualization tools.
So tell me about GraphViz.
Yeah.
So, you know, as a network engineer, it seems like we spent a large chunk of our time and rightfully so.
It's part of our job to document the network.
And you spend a lot of time drawing the graph and using, you know, whether your choice of tool is a Visio or it's Omnigraffle.
You spend a lot of time diagramming your network.
But by the time you're done, it's almost the case that your graph is outdated.
Right.
Like the network actually just grows.
You either add a new VLAN, you add a new box or just whatever it is that it's almost because you take so long and it's not automated.
You're actually manually graphing these network diagrams.
So this is especially a big problem with the data center being bigger and bigger.
So now you have what they call the factory network where the middle is huge, where you have server-to-server communication as opposed to server-to-client communication.
So this huge network presents a big problem because you're having hundreds of links between each of the network layers and doing it manually is just not an option anymore.
So I think a couple of years ago, I went out and started thinking about trying to come up with a solution.
And GraphVis is what I saw where you could actually basically describe your network in terms of a text file.
So I would say in terms of nodes and edges and say these two nodes should be placed together at the very top tier.
And these four nodes would be in the middle tier and they should line up and so on and so forth.
So GraphVis is a great tool for doing that.
And there's a Python module that's also named GraphVis that allows you to come up with this what they call like a .file where you use the Python module, you generate this .file, you run it through GraphVis and voila, you have this beautiful graph that's up to your liking.
So the only thing that I guess it's me not being didn't learn it enough that, you know, like coloring the graph, like say if I wanted to color all the spine boxes or if I want to color all the leaf devices in a certain color, I still haven't learned how to do that.
So sometimes when I do need that colorization, I kind of launch OmniGraphle Pro that you could import that graph and then color it to my liking.
But it's an awesome tool.
It allows you to automatically generate your graph.
So it allows you to have this less of a drift between your actual network versus your diagram.
Okay.
Nice.
Actually, I didn't know about this, the Python module for it or the Python package to use it because GraphVis has been around for a long time.
That's right.
But working with the .files, even though the format's pretty easy, it still is something else to learn.
And yeah, I'll have to check out the Python package because that's pretty cool.
I like that.
Yeah.
So I think just the biggest, I mean, if you want a quick graph, you could use PyGal or Matplotlib, even.
And you just, you know, quickly draw that graph.
But GraphVis, to me, is the most powerful.
And I know, you know, a lot of companies use that for drawing their kind of eye candy kind of graph.
Like, you know, you've seen the Facebook graph where you just have lines all over the globe and so on.
I mean, that's the kind of graph that GraphVis is capable of doing really, really quickly.
Well, yeah.
I don't know if those are helpful, but yeah.
Those are helpful for marketing materials.
Yeah.
Some of the automated code processors use it for drawing, like, class hierarchies and stuff.
Right, right.
I mean, and in a shameless plug for my book, I also use the GraphVis tool.
I use an example of where you're inside of a...
So use your Python script to go query your network devices and say, basically to say, tell me who your neighbors are.
And you just kind of walk layer by layer by layer.
And you construct a GraphVis file that represents your network in real time.
So every time you could just execute that script and it will give you your most up-to-date network graph automatically.
So I think that's kind of a good use case for the tool where, like I said, like the biggest problem that I face with network graphs is there's always this drift between the actual network topology versus the graph that you present.
And if the graph is wrong, nobody's going to use it.
Yeah.
Oh, hey, I just remembered a cool example of the .files.
One of the projects I work on, one of the things that we log is basically a running state machine example.
An internal state machine that is basically a state machine as it's running and where we are currently.
That's being output as a log file in the .format so that if whenever we get like a crash or something, we can take that log file and just throw it through GraphVis and see exactly where we're at visually.
So even a manager could understand it.
Yeah, like me.
Yeah.
So cool.
Yeah.
Last topic I'd like to bring up is PyCascades.
So this is a new thing this year.
I think it's, I'm going to get it wrong, but I think it's a whole bunch of different Python groups in the area coming together to try to do a regional Northwest Python conference.
It is in January.
It's January 22nd, 23rd.
And then they're going to do sprints the 24th.
But I like the format.
I like what they're doing.
And the tickets are now very expensive.
I think they're like 100 bucks or something.
Okay.
They're not bad.
And it's in Vancouver, BC, which I, even though I live in Portland, I've never been there.
So I'm looking forward to go and seeing that.
And the call for proposals for talks is the 28th.
So that's like Monday or something like that.
So when this comes out, people only have a couple days, but I think there's still slots open.
So please check it out.
And if you want to do a talk, local talk, I think this is good.
The talk slots are 25 minutes too, which is a nice length.
I think that anybody that has a, if you've got expertise on really anything in your job, 25 minutes shouldn't be too hard to fill.
So I'm going to submit a couple talks and encourage anybody else to as well.
Yeah.
So Vancouver is about three hours away from me as well.
So I have to check it out.
I mean, I've never been to the Py Cascades, but it seems like a good format.
And like you said, tickets are cheap.
Yeah.
Well, yeah, it's the first one this year.
So.
Oh, okay.
Got it.
Yeah.
That's our six topics that went by really fast.
It did.
So tell me a little bit about what's up with you.
How long has Mastering Python Networking been out?
So it's been out since June 28th, but only the digital version was out.
So I think you could start ordering the physical book like mid-July.
So it's been out for about a month and a half.
I have the link in the show notes.
Go check it out if you could.
I really appreciate any kind of feedbacks or just any criticism, good, bad, ugly, whatever
it is.
You know, I'm open to it.
So I appreciate that.
Nice.
Yeah.
So what's happening with you, Brian?
Well, this last week was the fixing up the copy.
I got the copy editing changes for the Pyth test book back.
Oh, that's right.
And I just got those in yesterday.
That's going to be out.
I'm so excited to get the physical copy out.
So the physical form.
Yay!
Yeah.
The physical form will be like in October.
But I'm like, as of like, there's like one minor tweak I got to fix tonight.
Other than that, my hands are completely off.
Hopefully.
I keep saying this every week.
But once it goes in past copy editing and through the rest of it, it's really an author
doesn't get to interfere process until it comes out as a hard copy.
So from start of this project to actually this point, how long did it take for you?
That's a long answer.
But my interaction with Pragmatic started, I think, June of last year.
Right.
June of 2016.
Okay.
So a little over a year.
But I think the first four or five chapters, I think I rewrote those like four times.
Yeah.
To try to get it right.
Yeah.
Initially, I was kind of coming along the same lines as the tutorials I have on the
pythontesting.net.
But getting a lot of trying to understand how to write in a book format instead of a blog
format, it was a learning experience for me.
But still, even after doing all that, I think that, I don't think, looking back, a year isn't
that terrible.
But there's one of the other authors, Python authors, is Luciano Romalo.
I wish I knew which book he wrote.
It's like, I'm going to get it wrong.
He was one of the reviewers and his feedback was really influential, but added like two
or three weeks of my work.
Yeah.
For me, I think the biggest issue was always the, not on myself, but on the people around
me.
It's almost like, you know, I have to be concentrated and focus on the book.
And then, so the family has to, you know, kind of go away.
My wife has taken the burden of taking the kids away from me for the time duration I was
writing the book.
So, yeah.
So, it's definitely both a journey experience, but I'm very excited for you.
You know, I think when you see that paper copy, you're just going to be like, yay, you know,
makes it real.
Yeah.
It definitely is going to be cool.
The, like you said, trying to find time within your life and trying to get distractions out
because like you, I've got a full-time job and kids and family.
And so, it was quite a few late nights at the coffee shop trying to get away from everything,
but it'll be out.
So, that's all we got for today.
And thanks a ton for helping out, Eric.
And we'll keep in touch.
Yeah, absolutely.
Thanks, Brian.
This has been fun.
Thank you for listening to Python Bytes.
Follow the show on Twitter via at Python Bytes.
That's Python Bytes as in B-Y-T-E-S.
Get the full show notes, including links at pythonbytes.fm.
If you have a news story you'd like featured, visit pythonbytes.fm and send it our way.
We're always on the lookout for sharing something cool.
This is Brian Okken.
On behalf of myself and Michael Kennedy, thank you for listening and sharing this podcast with
your friends and colleagues.