Since our vulnerability scanners are regularly touching all parts of our network, they are a good choice as a source for a list of hostnames, IPs, and ports for any service speaking HTTP or HTTPS. After massaging the data in Excel I have a list of URLs to test using either the FQDN or IP and the port number.
Once I have this list, typically several thousand different URLs to test, I need to quickly eliminate the systems I don't need or want to inspect. To do this, I wrote a simple python utility which uses urllib2 to pull in the page associated with each URL and analyze it through a simple string.find() loop. I built a dictionary of common sites that I know I won't need to inspect, such as
- Sites with the corporate authentication mechanisms presented
- Default Apache / IIS web pages
- Default Tomcat or JBoss install
- KVMs and SAN switch interfaces
The biggest return isn't in time saved, however. The real value comes when the utility isn't able to classify the site. These sites often contain information that should have been secured, or authentication mechanisms using weak/default credentials. I can easily filter the output into additional tasks, such as testing for default Tomcat or JBoss credentials, etc.
A while back I experimented with being able to take a screenshot of each site to quickly eliminate these sites visually. Unfortunately, at the time, every utility I investigated was also stumped by the redirect. AJAX-heavy sites also fooled my utility as well as the other utilities I tested.
Feeling inspired by all the incredible talks presented at DerbyCon,I decided it was time to start putting Sketchy to work. I blogged earlier about my experience setting up Sketchy, you can read about it here.
While Sketchy does have an API, a quick and dirty shell script worked for my needs. The script supports grabbing a screenshot (sketch), grabbing the DOM as text (scrape), or grabbing the rendered HTML (html). For sites sketchy is unable to connect to, my script makes a log entry and does not produce an artifact.I can quickly view these resulting images and determine if the site is something that warrants further inspection.
|Linksys router login page|
|Twitter login page|
ConclusionReviewing websites is essential to identifying information disclosures, weak authentication mechanisms, and new web apps or devices that may have been deployed without your knowledge. Regularly reviewing these websites for this information prevents audit findings and helps keep your network and data safe from unauthorized access.
Sketchy was easy to install, and it didn't take long to whip up a functioning system. With a few hours of setup, scripting, and testing, I'm able to automate what used to be several hours of work. In the end, I'm free to get more done, and much more of the proverbial low-hanging fruit is picked.
If you're using different tools to achieve the same end, I've love to hear about it. Leave me a comment or reach out to me on Twitter.