Automating nmap and ndiff with PowerShell

I like to perform reconnaissance against my network border on a regular basis to identify new services and new hosts.  With nmap, some bash, and cron, this is pretty easy to do.  Unfortunately, the system I have outside our network to do this with is a Windows system with Kali running in a VM.  I say unfortunately, because for some scans, running from Kali within a VM is significantly slower than performing a scan from nmap in Windows.  I know you're thinking "But why don't you just get a *nix box in the cloud?"  You deal with what you have available to you.  Larger organizations sometimes come with less flexibility.

I wanted to be able to scan a given set of hosts and compare the results of each scan against the previous scan.  The ndiff utility included with nmap was designed to do just that.  I also wanted to get those results emailed to me.  I also wanted to encrypt these results before emailing them, because reasons.  I hadn't spent a lot of time with PowerShell, so this seemed like an opportunity to both get something done and learn PowerShell.  The result is scandiff, a partial wrapper for nmap, ndiff, and 7Zip written in PowerShell.

Scandiff does a discovery scan against the targets supplied looking for open ports to identify live hosts.  These hosts are then re-scanned using a larger set of ports and service version probing (-sV).  I chose this two step method to decrease overall scan times when scanning large IP spaces.  If the IP space being scanned is behind a firewall blocking pings and not returning RST, nmap will determine all ports are "open|filtered" and marks the host as up.  The downside to this approach is that  it is possible to miss hosts which have ports opened but are not included in the probed ports list.  It is advisable to review your infrastructure and include all common ports for your network's services in your discovery probe list.

Scandiff retains the previous output in an XML file, $basename-prev.xml.  Once the scan has completed, ndiff is run to compare the current scan results against the previous results to generate a -diff.txt file.

Once the ndiff operation has completed, the XML files, nmap log, and diff file are added to an encrypted 7Zip archive.  This archive is added as an attachment to a System.Net.Mail.MailMessage email object.  The results are then emailed to a designated recipient using the PowerShell Net.Mail.SMTPClient.

I used Gmail to relay the output to my inbox.  This section can be modified to relay through a different email server, and authentication can be disabled if not required by your mail server.  Email output can be disabled completely by specifying "-email 0" or "-email $False".

I'm still playing around with the nmap options to find the right balance of accuracy and thoroughness vs. performance.  The script also has very little error handling.  I will be working on adding this in the coming weeks.  I am also looking at being able to specify the majority of configuration options from a configuration file to make the command line operation less unwieldy.

Scandiff can be downloaded from github here: https://github.com/hardwaterhacker/scandiff.  Let me know if you find this useful.  I'd welcome any input on how to improve this script.

Automating Simple Website Reconnaissance Measures a.k.a An Ounce of Prevention

As a pen tester as part of an internal security team, I'm responsible for periodically sweeping our networks to identify web servers and determine if there are risks presented by those websites such as information disclosures, default credentials, or insufficient access and authorization measures.  (aside: yes, change control would make sure this never happened again.  In a world filled with unicorns farting rainbows.) On anything other than a small network, this can quickly become a time-consuming task. It didn't take long to decide to automate as much of this process as possible.

Since our vulnerability scanners are regularly touching all parts of our network, they are a good choice as a source for a list of hostnames, IPs, and ports for any service speaking HTTP or HTTPS. After massaging the data in Excel I have a list of URLs to test using either the FQDN or IP and the port number.

Once I have this list, typically several thousand different URLs to test, I need to quickly eliminate the systems I don't need or want to inspect.  To do this, I wrote a simple python utility which uses urllib2 to pull in the page associated with each URL and analyze it through a simple string.find() loop.  I built a dictionary of common sites that I know I won't need to inspect, such as

• Sites with the corporate authentication mechanisms presented
• Default Apache / IIS web pages
• Default Tomcat or JBoss install
• KVMs and SAN switch interfaces
• etc.

When the utility finds a URL matches something in the dictionary, it records this in the output file.This resulting report contains far fewer sites needing inspection than the original list.

The biggest return isn't in time saved, however. The real value comes when the utility isn't able to classify the site. These sites often contain information that should have been secured, or authentication mechanisms using weak/default credentials.  I can easily filter the output into additional tasks, such as testing for default Tomcat or JBoss credentials, etc.

In the past, I would take these unclassified results and dump them into a spreadsheet and then review them individually. Any site that would attempt to perform a Javascript redirect or refresh to a different landing page when '/' was requested would fool my utility as urllib2 is unable to follow the redirect. This lead to manually reviewing a lot of sites that would otherwise be easily identified if my utility could see the landing page.

A while back I experimented with being able to take a screenshot of each site to quickly eliminate these sites visually. Unfortunately, at the time, every utility I investigated was also stumped by the redirect. AJAX-heavy sites also fooled my utility as well as the other utilities I tested.

This summer Netflix released a tool they wrote - Sketchy - which they use to assist in their IR processes. Sketchy addresses the same issues I was experiencing with Javascript and AJAX sites. After reading about Sketchy, I knew that I wanted to try applying this to my processes to see if I could get better results and be more efficient.

Feeling inspired by all the incredible talks presented at DerbyCon,I decided it was time to start putting Sketchy to work. I blogged earlier about my experience setting up Sketchy, you can read about it here.

While Sketchy does have an API, a quick and dirty shell script worked for my needs.  The script supports grabbing a screenshot (sketch), grabbing the DOM as text (scrape), or grabbing the rendered HTML (html). For sites sketchy is unable to connect to, my script makes a log entry and does not produce an artifact.I can quickly view these resulting images and determine if the site is something that warrants further inspection.

Examples

Linksys router login page

Twitter login page

Conclusion

Reviewing websites is essential to identifying information disclosures, weak authentication mechanisms, and new web apps or devices that may have been deployed without your knowledge. Regularly reviewing these websites for this information prevents audit findings and helps keep your network and data safe from unauthorized access.

Sketchy was easy to install, and it didn't take long to whip up a functioning system.  With a few hours of setup, scripting, and testing, I'm able to automate what used to be several hours of work. In the end, I'm free to get more done, and much more of the proverbial low-hanging fruit is picked.

If you're using different tools to achieve the same end, I've love to hear about it. Leave me a comment or reach out to me on Twitter.

-Mike