Questions? Comments? Need a CTF? Contact Us

Contact

Please let us know your name.
Please let us know your email address.
Please let us know your message.
Invalid Input

OSINT Guide

Techniques by Bellebytes

Introduction

Buckle up. This is a long one. Alyse has compiled her methodology for gathering open-source intelligence on an organization and its personnel.

Part 1: Overall Company Reconnaissance

Setting Up

Organized Spreadsheet (optional):

  • Create an excel spreadsheet with the following sheets tabs across the bottom:
    • Total External Inventory (save this sheet tab to fill out last)
    • Competitive Intel
    • People & Contacts
    • DNS Harvesting
    • IP Blocks
    • Open Job Requisitions Information
    • Metadata Analysis
    • Google Dorks
    • Social Media
    • GitHub Presence online
    • Amazon AWS or Cloud Presence
    • Extra sheets can include Dark Web, Passive Observations, or any extra notes that you think deserves its own category
  • Note: These categories will be described further in-depth later

Set-up VMs for anonymity and OSINT tools:

  • Kali Linux OS has many OSINT tools installed by default and would only require API setup/ configuration beforehand.
  • Certain tools such as Cain and Search Diggity are only available for Windows OS, so know which tools you would like to use and prepare the operating system VMs accordingly.
  • Buscador is an OSINT Linux VM that is pre-configured for online investigations and was developed by David Westcott and Michael Bazzell. It can be found here:  https://inteltechniques.com/menu.html
  • Consider using a VPN. Private Internet Access (PIA) is a wonderful example.
  • Use throttling, obfuscation, and overall detection evasion techniques when performing standard port scanning, and passive observation gathering of externally open ports and services.

Competitive Intel

Get to know your target:

  • Industry type
  • Major competitors
  • Major businesses
  • Major customers
  • Major products and services, trade information, etc.
  • Corporate Officers and other VIPs
  • Map company relationships for potential whitelists to 3rd party vendors and shareholders for phishing campaigns
  • Misc. notes to estimate company net worth (market statistics, financial reports, etc.)
  • Recent press releases for company changes
  • Physical locations
  • Try to develop a sense for what may be the company’s biggest security concerns if possible (sensitive information disclosure, interruption of production processing, media release of website defacements, etc.)

People & Contacts

Spreadsheet columns to organize people and contacts (optional):

  • Name
  • Email address
  • Active/non-active (the Mailtester module in Recon-ng is one of the many helpful tools available to test if email addresses are still active)
  • Phone numbers
  • Job Title
  • How discovered (scraping tools, web searches, etc.)
  • Misc. Notes (background, interesting information, physical addresses, etc.)

Scraping Tools for obtaining entire list of employees and contact information:

  • LinkedInt python script (https://github.com/mdsecactivebreach/LinkedInt)
  • The Harvester python script (https://github.com/laramies/theHarvester)
  • Recon-ng modules for people (all APIs keys need to be in key list)
    • BuiltWith contact module
    • Have I Been Pwned hibp_breach and hibp_paste credential modules
    • FullContact module
    • Linkedin_auth module (Bing LinkedIn cache)
    • Whois_pocs module
    • Hashes_org module
    • Pgp_search module
    • Dev_diver module
    • Namechk module
    • Jigsaw and pwnlist modules (APIs need to be purchased beforehand)
    • Many other recon-ng modules such as the GitHub and twitter modules can help add to this list as well
  • Whois ARIN registrations
  • Geektools.com
  • Hunter.io
  • DataSploit python script (https://github.com/DataSploit/datasploit)
  • Metadata analysis (see Metadata Analysis for more information)
  • Google Dorking (see Google Dorking for more information)
  • Maltego
  • Many other tools and scrapers can be found here: https://github.com/jivoi/awesome-osint
  • Other search engines (DuckDuckGo, Bing, AOL, Yahoo, Internet Archive/ Wayback Machine, Baidu, etc.)
  • Employee information revealed on company websites (most of these get scraped using the tools listed above but worth trying if you want to)

DNS Harvesting

Spreadsheet columns to organize domains found (optional):

  • Domain
  • IP Address
  • Externally Accessible? (port information, status code, etc.)
  • How Discovered (DNS Zone Transfer, DNS reverse lookups, SSL certificates, etc.)
  • Can it be found in Shodan search?
  • Registrant Organization information (if hosted by external party)
  • BuiltWith recon-ng module or BuiltWith browser extension information
  • Passive observations of externally accessible domains

DNS Harvesting tools and techniques:

  • DNS Query commands (dig any, etc.)
  • Nslookup
  • Nslookup Recurse versus Norecurse for DNS cache snooping (http://blog.commandlinekungfu.com/2009/03/episode-17-dns-cache-snooping-in-single.html)
    • If a DNS server does not have the information we request, it can forward that request to other DNS servers to retrieve the information in a process known as a recursive lookup. By default, nslookup will ask for recursion from name servers it queries (RD set to 1) but can be configured to create queries that do not request recursion using the set norecurse syntax (RD bit set to zero instead). Example:
  • Whois and ARIN
  • DNS Zone Transfer (AXFR and IXFR)
    • Full zone transfer:
      dig site.com -t axfr
    • Incremental zone transfer:
      dig site.com -t soa
      • The serial number will be something like 2016011701 (using a date format in YYYYMMDDNN where “NN” is incremented during each day. Incrementing the serial number tells the secondary DNS servers to mirror the updated zone so decrement the number by 1
        dig site.com -t ixfr=2016011700
    • Nmap brute force DNS script:
      nmap --script=dns-brute site.com
    • DNSRecon python script (https://github.com/darkoperator/dnsrecon)
      dnsrecon -t brt -d site.com -D /usr/share/dnsrecon/namelist.txt
    • Nmap brute force using DNSRecon wordlist as well
      nmap --script=dns-brute site.com --script-args=dns-brute.hostlist=/usr/share/dnsrecon/namelist.txt
    • Nmap IP ranges for domains
      nmap -sL 198.133.169.0/24| grep \
    • Sublist3r python script (https://github.com/aboul3la/Sublist3r)
      • This tool is great for finding internal domains by scraping SSL certs as well
    • Recon-ng
      • BuiltWith module
      • Shodan_hostname module (finds subdomains for all domains in Recon-ng)
      • Shodan_net module (finds domains for IP ranges)
      • Census_2012 module
      • Censysio module
      • Interesting_files module
    • Shodan.io
      • net:198.133.169.0/24
      • geo: filter is good as well for sparsely populated areas
    • Netcraft
    • DNSdumpster (https://dnsdumpster.com)
    • The Harvester python script (https://github.com/laramies/theHarvester)
    • DataSploit python script (https://github.com/DataSploit/datasploit)
    • Belati python script (https://github.com/aancw/Belati)
    • Metadata analysis (see Metadata Analysis for more information)
    • Google Dorking (see Google Dorking for more information)
    • Maltego transformers for domains
    • Metasploit
      • auxiliary/gather/dns_bruteforce
      • auxiliary/gather/dns_cache_scraper
      • auxiliary/gather/dns_info
      • auxiliary/gather/dns_reverse_lookup
      • auxiliary/gather/dns_srv_enum
    • Burp Suite (web crawling, passive observations, etc.)
    • Many other tools and scrapers can be found here: https://github.com/jivoi/awesome-osint

IP Blocks

Provide company name, company acronym, or domain name to search fields

WHOIS References:

  • American Registry for Internet Numbers (ARIN) covers North America, including the United States, Canada, and certain Caribbean islands
  • The Réseaux IP Européens Network Coordination Centre (RIPE NCC) is the RIR for Europe, the Middle East, and parts of Central Asia
  • The Asia Pacific Network Information Centre (APNIC) covers the Asia-Pacific region
  • The Latin American and Caribbean Internet Address Registry (LACNIC) covers Latin America and most of the Caribbean
  • AfriNIC covers the continent of Africa

Open Job Requisitions Information

Spreadsheet columns to organize Job Requisitions (optional):

  • Job Position Opening
  • How discovered (job posting URL, etc.)
  • Information Revealed

Job openings can help us obtain information of current technology products used in the target organization (Web server types, firewall type, router information, etc.):

  • Google Dorking
    site:[companydomain] careers
    site:[companydomain] jobs
    site:[companydomain] openings
  • Search through the “Careers” section at target company’s domain
  • Search other job sites such as monster.com, hotjobs.com, indeed.com, etc.
  • Some technical employee resumes contain detailed information as well

Metadata Analysis

Spreadsheet columns to organize Metadata Analysis (optional):

  • File name
  • File system path /URL file was obtained
  • Metadata Date
  • Client-side software in use
  • Names, Accounts, and other interesting information disclosure
  • How discovered (Exiftool, strings, etc.)

Metadata Analysis tools and techniques:

  • Recursive wget
    • Example:
      wget -nd -r -R htm,html,php,asp,jsp,aspx,jspx,cgi -P /root/Documents/OSINT/metadata_ex targetdomain.com
    • Example:
      wget -nd -r -A pdf,doc,docx,xls,xlsx,xlt,ppt,pptx -P /root/Documents/OSINT/metadata_ex targetdomain.com
  • Exiftool (https://www.sno.phy.queensu.ca/~phil/exiftool/)
    • Example:
      ./exiftool /root/Documents/OSINT/metadata_ex/osi_implementation_guide_v2.2.pdf
    • For loop example to exiftool all files in a given directory:
      mkdir /root/Desktop/exifoutput; for i in `ls /root/Documents/OSINT/metadata_ex/`; do ./exiftool /root/Documents/OSINT/metadata_ex/$i >> /root/Desktop/exifoutput/$i.txt; done
  • Strings command
    • Example that shows strings only eight characters long
      strings -n 8 [filename]
    • Example that can search for metadata associated with target’s firewall rules
      strings [filename] | grep -I firewall
  • FOCA (https://github.com/ElevenPaths/FOCA)
  • Belati python script (https://github.com/aancw/Belati)

Google Dorking

Useful search directives:

  • “site:” directive searches only within given domain
    • Example:
      site:example.com “web app”
  • “intitle:” directive shows pages whose title matches the search
    • Example:
      intitle:index of passwd
  • “inurl:” directive shows pages whose URL matches the search criteria
    • Example:
      inurl:viewtopic.php
  • “filetype:” directive searched for specific kinds of files such as .pdf, .doc, .xls, etc.
    • Example:
      site:example.com filetype:ppt
  • “intext:” directive searches for certain keywords in the text of the webpage
    • Example:
      intext:"Index Of"
  • “related:” directive shows similar pages (not always useful)
    • Example:
      related:example.com

Google Hacking Database (GHDB):

  • https://www.exploit-db.com/google-hacking-database/
    • PGP and GnuPG private key rings example:
      intitle:index.of intext:“secring.skr” | “secring.pgp” | “secring.bak”
    • Nessus scan results example:
      intitle:“Nessus Scan Report” “This file was generated by Nessus”
  • Recon-ng GHDB module (limited results)

SANS Google Dorking Cheatsheet:

Bing Hacking Database (BHDB):

Shodan Hacking Database (SHDB):

SearchDiggity Suite (Windows OS only)

Social Media

Major Social Media websites (list is not all encompassing):

  • Facebook, Flickr, Google+, Hi5, Instagram, LinkedIn, Pinterest, Reddit, Snapchat, Swarm, Tumblr, Twitter, YouTube, etc.

Scraping to obtain social media profiles associated with company:

Keep communication tools and ticketing tools in mind as well (optional):

  • Adobe Connect, AnyDesk, AnyMeeting, GoToMeeting, Join.me, MatterMost, Skype, Slack, TeamViewer, WebEx, Zoom, etc.
  • ServiceNow, Remedy, BMC Software, etc.

Github

Scraping to obtain GitHub pages associated with company:

  • Recon-ng modules for finding Github presence
    • Github_repos module
    • Github_commits module
    • Gists_search modules
    • Github_dorks modules
  • Google Dorking

Cloud Presence

AWS:

Other Cloud Vendors:

  • Microsoft, Google, IBM, VMware, Rackspace, Oracle, Alibaba Cloud, DigitalOcean, etc.

Cloud Storage and File Sharing:

Pastebins and Dark Web

Pastebin Scrapers:

Dark Web Tools/Techniques:

  • Ichidan (ichidanv34wrx7m7.onion)
  • DeepDotWeb
  • Reddit Deep Web
  • Reddit DarkNetMarkets
  • Dark Net Stats
  • OnionScan
  • Tor Scan
  • Onioff
  • Hunchly Hidden Services Report
  • Docker-onion-nmap
  • Onion Investigator
  • Onion Cab
  • Hidden Wiki
  • Core.onion
  • Various Torrent search engines contain a lot of breached data for download as well

Other Dark Web sites and tools can be found here: https://osintframework.com/ and https://tomokodiscovery.com/free-tools-osint-socmint-dark-web-darknet-tor-bitcoin/

Total External Inventory

Spreadsheet columns to organize externally accessible inventory (optional):

  • Target IP address
  • Target name (hostname, domain name, etc.)
  • Target OS
  • How Discovered (revealed by target personnel, Google/web searches, DNS Zone Transfers, DNS reverse lookups, network sweep, Shodan.io, etc.)
  • Listening Ports (nmap or other port scanning results, recon-ng censysio, etc.)
  • Known vulns (passive observations based on what you find; obtain approval before actively testing exploits)
  • Accounts/ Passwords found
  • Misc. Notes

Other options for inventory storage

  • Collaboration tools such as Dradis, Faraday, EtherPad, Lair, etc.
  • Metasploit Databases
  • MediaWiki, and many more

Overall Company OSINT Checklist

Optional Sanity Checklist (Overall Company)












Copyright © 2018 Storm CTF | Design by Stitch