In April 2018, The Wall Street Journal reported missile strikes on Syrian government bases that killed dozens not long after Israel had been blamed for attacking an Iranian air station. The article stated, “An Israeli open-source intelligence site posted purported satellite imagery on Twitter, saying that the target of the attack was an Iranian base recently erected north of Hama airport.”

Indeed, open-source intelligence (OSINT) frameworks and tools are on the rise. The term refers to data collected - via various scripts, scraping and multiple tools often working in framework-  from available sources to be used by those seeking information in an intelligence context. Be it the blogs we browse, broadcasts we watch, or publications we read, there is an endless supply of information available that is hidden within the content or linkable to or from the content.

OSINT draws from the internet, traditional mass media (e.g., television, radio, newspapers, magazines), specialized journals, conference proceedings, think tank studies, photos and geospatial information (e.g., maps and commercial imagery products), as well as other information repositories.

Using an OSINT Framework

Justin Nordine is an IT security professional who created an OSINT framework that allows anyone to use the associated tools of the framework to gather informational artifacts from a multitude of sources for a variety of purposes. It gathers information about and from sources related to a cadre of information repositories and sources which include training, documentation, OpSec, threat intelligence, classifieds, public records, IP addresses, business records and many more. Developers can use such a framework, as can journalists, forensic investigators, and anyone else seeking to deep dive into the archaeology of something or someone.

Often the framework can be used not just for the information it provides, but also to find holes or vulnerabilities within an enterprise or network that can be patched or filled for security purposes. “Blue Team” security professionals can use it to find competitive information or seek intelligence. Companies may use it to gather corporate intelligence.

Open-Source Spirit, Collaborative Results

Nordine, in a podcast on Timothy Deblock’s Technology and Media site, posits that since the creation of his OSINT framework, response from others, in IT and other disciplines, has been overwhelming.

He welcomes visitors, outside of infosec, to use it and aid in the collaboration of building it further for the good of all. He’s hoping it can be, in the spirit of open source, a fully collaborative effort whereby others contribute and add to the development of the framework. The code is posted on www.github.com, where anyone can access and clone OSINT applications.

As Nordine describes, his OSINT framework began after he listened to a presentation by Johnny Long about “no-tech” hacking; he described the many ways he was getting information without attacking another computer system. This happened in the 2006-2007 timeframe. Long went on to write a book in 2008, along with some co-authors, entitled No Tech Hacking: A Guide to Social Engineering, Dumpster Diving, and Shoulder Surfing.

Nordine said his OSINT framework had its genesis from his own exploration into how to proceed in finding useful information. More specifically, if he could find one piece of information such as an email address, IP address, or domain name, how could it be used to “tile back in,” as he calls it, into one big picture?

Mind-Mapping Open Intelligence

From that genesis, he developed disparate tools that can work within the framework to perform other functions. Many of these tools were overlapping. Then he narrowed the tools down by contemplating other tools that he had successfully used. This evolved into a “mind map” that subsequently morphed into a framework that served his intention by allowing him to enter information and then use it to branch off to more information.

This mind map has tiers of information. Any observer who wants to know more information about anything then starts digging. "This is the 'no-tech hacking' form of thinking," he says on the podcast. For example, finding user names may be cross-referenced into different sites to gain more information. Nordine notes that anyone can use this tact to gather information. The framework originated from his information security background where he gathered information to know what to defend against.

An Arsenal for Information Seekers

Since the framework was offered, others have begun using it.  This includes fact checkers, journalists, background checkers, and many different industry interests that Nordine says he wasn’t exactly expecting. He believes it can be used in a more formal way to gather information and can be used as a tool to find information.

Nordine says future plans for the framework include building it further by adding new tools continuously. Then he expects to spend more time on some of the site’s code and functionality upgrades. “Blue Team” security professionals can use it to find competitive information or seek intelligence. Companies may use it to gather corporate intelligence.

So, what are some real-world examples of how OSINT is being used in today's world?

  • The BBC used OSINT tools to uncover facts about a viral video whereby a group of soldiers in the nation of sub-Saharan Cameroon were shown two women and their children being killed by the soldiers. According to Caroline Scott who wrote about the incident, open-source intelligence tools were unveiled by the BBC’s investigative unit when they published a detailed breakdown of the tragic video.
  • When the media probed deeply into the recent nomination of Justice Brett Kavanaugh to the Supreme Court, one source, ProPublica, used an OSINT network of crowdsourced information to investigate Kavanaugh’s $200,000 baseball ticket debt and tried to identify who sat with him at games.
  • During the World Cup in the summer of 2018, hackers from the militant Palestinian group, Hamas, tried to install spyware on the phones of Israeli soldiers by using fictitious social media accounts that prompted the soldiers to download spyware applications disguised as dating applications. Some prognosticated that the set up was made possible by open source intelligence tools.
  • A 39-year-old systems analyst at Indiana University used OSINT tools to hunt for Russian trolls, mostly just for fun. He became curious about Russian trolls after hearing about purported Russian intervention in U.S. elections.

OSINT Tools Abound

As stated, many tools have been developed that use OSINT. Here’s a sampling of other tools that use OSINT for specific functions of search and information extraction:

  • Maltego is an OSINT tool used by security professionals and forensic investigators to collect and study open-source intelligence by drawing from various sources and using such information to generate graphical results.

  • Shodan is an OSINT tool designed to serve as a search engine for hackers. Shodan presents information in a different way that allows hackers to see the assets attached to digital ones. It can spot devices such as laptops as well as webcams and other IoT devices—often a vulnerable security target—so analysts may target, test, and mitigate info security risks.

  • The Harvester is a tool used to gather email and domain-related information.

  • Metagoofil is a tool used to gather metadata of public documents.

  • The Operative Framework is a Python-based open-source intelligence (OSINT) tool used to find domains registered by the same email address.

  • Paliscopeis software developed to enable structured and secure OSINT investigations. It was created principally to conduct forensic and criminal investigations, but can be used for many different types of investigative missions.

Says Christian Berg, CEO at Paliscope: “It’s a way of solving crimes, but also a means of knowing who you are dealing with and separating facts from false claims. However, it is not a matter of just collecting and storing large amounts of data; online investigations have to be carried out with structure and purpose, especially in light of the new European General Data Protection Regulation (GDPR).”

While many tools built from OSINT are commercially available, the framework and its capabilities, in the spirit of open source, is free. Justin Nordine says his framework requires constant updating, but hopes such a tool will continue the collaboration of open-source developers to make OSINT a valuable asset for the many who stand to benefit from it and its capabilities, just as the Israeli site did when it posted satellite imagery of the Syrian missile attack on Twitter.

Want more? Visit MISTI's upcoming InfoSec World Conference & Expo in Orlando, Florida where some of the top leaders in the security industry will be sharing their knowledge.

Pankaj Patel