Open Source Intelligence (OSINT) Search Techniques

Open Source Intelligence (OSINT) is a valuable process for gathering, analysing, and reporting data obtained from publicly available sources for intelligence purposes. OSINT analysts employ specialised methods to explore a wide range of open sources and identify relevant data that meets their objectives. In many cases, OSINT analysts uncover information that is not widely known or accessible to the public.

OSINT encompasses both offline and online information that is publicly available, regardless of whether it is free, purchasable, or obtainable by request. Let’s explore some examples of offline and online information used in open-source intelligence.

Offline Sources

Offline sources of OSINT include:

1. Diplomatic: Government agency databases, law enforcement databases, court records, NGOs and international organisation records.
2. Academic: research papers, journals, dissertations, and academic publications.
3. Corporate: Annual reports, conference proceedings, press releases, employee profiles, and resumes.
4. Mass Media: Television, radio, newspapers and magazines.
5. Anything sent to you directly from another source.

Online Sources

Online sources of OSINT include:

1. Internet Search/Database: Search engines like Google, Bing, Yahoo, the Wayback Machine, and WHOIS.
2. Social Media Platforms: Facebook, Twitter, LinkedIn, Instagram, and other popular social media platforms.
3. Sharing & Publishing: YouTube, Flickr, Pinterest, Dailymotion, and other platforms for sharing and publishing content.
4. Blogging, Forums, and Online Communities: WordPress, Medium, Reddit, 4Chan, and other online platforms for discussions and sharing information.
5. Deep Web: Non-indexed web pages that are not reachable by internet search engines.
6. Dark Web: The dark web, accessible through darknets like Tor and I2Ps, where illegal content is often hosted.

It is important to note that the deep web and dark web require specialised tools and access methods to gather information. These sources may contain valuable intelligence, but also pose significant risks and legal concerns.

History of Open Source Intelligence

The origins of OSINT can be traced back to before the introduction of digital technologies and the internet. However, OSINT became a prominent intelligence discipline during the Cold War, particularly for gathering information on the Soviet Union and China. After the Cold War, advancements in technology, commerce, and politics further expanded the scope and capabilities of OSINT. The widespread distribution of media publications, the invention of television, and the advent of the internet all contributed to enhancing the intelligence community’s access to open sources.

Source: Mercado, S., 2004. Sailing the Sea of OSINT in the Information Age. Studies in Intelligence, [online] 48(3), pp.44-55.

Uses of Open Source Intelligence

OSINT is widely used by information security teams for two main purposes:

1. Discovering Public-Facing Internal Assets: OSINT analysts employ penetration testing techniques to identify an organisation’s publicly available assets. This ethical hacking approach involves testing the cybersecurity of computer systems, networks, and web applications to uncover exploitable security vulnerabilities. The intelligence gathered through penetration testing can help security teams identify weaknesses and implement appropriate remediation measures.
2. Identifying External Threats: Organisations must also consider external cyber threats when assessing their attack surfaces. OSINT plays a crucial role in third-party risk management programs, as third parties often become common attack vectors. By analysing publicly available information on social media platforms and other sources, OSINT analysts can identify potential vulnerabilities and threats that could be exploited by threat actors. This information helps organisations proactively address security gaps and protect their assets.

OSINT is also vital for optimising Operations Security (OPSEC), which involves identifying and mitigating potential vulnerabilities that could be exploited by attackers if combined with other data. By leveraging OSINT, organisations can proactively identify and address potential risks to their operations.

OSINT Techniques

OSINT reconnaissance (recon) techniques can be categorised into two main types: passive and active.

1. Passive Recon: This technique involves gathering information about a target network or device without directly engaging with the system. OSINT analysts rely on third-party information and utilise tools like Wireshark to analyze network traffic in real-time. By piecing together different OSINT data points, analysts can identify patterns and extract meaningful insights.
2. Active Recon: In contrast to passive recon, active recon directly engages with the target system to obtain more accurate and timely information. OSINT analysts use tools like Nmap, a network discovery tool, to gain a granular view of a network’s security. However, it’s important to note that active scanning can potentially be detected by intrusion detection systems (IDS) and intrusion prevention systems (IPS).

While information security teams may adopt unique OSINT techniques tailored to their specific needs, following a general process helps establish a solid foundation for effective intelligence gathering. The Open Web Application Security Project (OWASP) outlines a 5-step OSINT process:

1. Source Identification: Determine where to find the information relevant to the specific intelligence requirement.
2. Harvesting: Gather the necessary information from the identified sources.
3. Data Processing: Process the collected data and extract meaningful insights.
4. Analysis: Combine the processed data from multiple sources to gain a comprehensive understanding.
5. Reporting: Create a final report on the findings and intelligence gathered.

OSINT Tools

There are numerous free and paid OSINT tools available for various purposes, including:

1. Babel X: A multilingual internet search tool that finds publicly available information from sources like social media, forums, news sites, and blogs across 200 different languages.
2. BuiltWith: A website profiling tool that provides current and historical information about a website’s technology usage, versions, and hosting.
3. Creepy: An OSINT gathering tool that collects geolocation information through social networking platforms.
4. DarkSearch: A dark web search engine that allows organisations to research and access sites directly through Tor2Web.
5. GHunt: An OSINT tool used to find data associated with Google accounts, including account owner name, Google ID, YouTube, and other services like Photos and Maps.
6. Google Dorking: involves using advanced search queries to find security and configuration information about websites.
7. Greg.app: A search engine that searches code from public repositories on GitHub.
8. Intel Owl: An OSINT tool that gathers threat intelligence data about specific files, IP addresses, and domains through a single API request.
9. Intelligence X: A search engine and data archive that searches Tor, I2P, data leaks, and the public web by email, domain, IP, CIDR, Bitcoin address, and more.
10. Maltego: An OSINT and graphical link analysis tool used for gathering and connecting information for investigative tasks.
11. O365 Squatting: A Python tool used to identify typo-squatted domains within O365 infrastructure.
12. The OSINT Framework: An online directory that lists open source tools for OSINT gathering, sorted by source type.
13. reNgine: An automated reconnaissance framework used for OSINT gathering that streamlines the recon process.
14. Recon-ng: An open-source intelligence gathering tool used for web-based reconnaissance.
15. Searchcode: A source code search engine that indexes API documentation, code snippets, and open source repositories.
16. Shodan: A search engine used for gathering intelligence information from various IoT devices like webcams, routers, and servers.
17. Social Mapper: An OSINT tool that uses facial recognition to correlate social media profiles across different sites on a large scale.
18. Spiderfoot: A reconnaissance tool that automatically queries over 100 public data sources (OSINT) to gather intelligence on IP addresses, domain names, email addresses, names, and more.
19. Sublist3r: A Python tool designed to enumerate subdomains of websites using search engines like Google, Yahoo, Bing, Baidu, and Ask.
20. theHarvester: A penetration testing tool used to gather information about emails, subdomains, hosts, employee names, open ports, and banners from different public sources like search engines, PGP key servers, and SHODAN computer database.
21. TinEye: A reverse image search engine and image recognition tool.
22. Zmap: A network tool used for Internet-wide network surveys.

These tools can assist OSINT analysts in efficiently gathering and analysing data from various sources, enhancing their ability to uncover valuable intelligence.

Legality of OSINT

The legal use of open source intelligence is defined by the US Code as intelligence produced from publicly available information that is collected, exploited, and disseminated in a timely manner to an appropriate audience for addressing specific intelligence requirements. The specialised recon tools and techniques used by OSINT analysts are legal as they aid in the collection, analysis, and processing of publicly available information.

However, it is important to note that while OSINT deals with information that anyone on the internet can find, it often uncovers information that most people are not aware is public. This lack of awareness creates a “grey area” in the legality and ethics of OSINT. The legality and ethics of OSINT primarily revolve around how vulnerabilities are managed.

For example, if an organisation accidentally leaks employee credentials on a public storage bucket like Amazon S3, an OSINT analyst using a code search engine might discover this leak. While a threat actor could also discover this leak and exploit it for malicious purposes, an ethical OSINT analyst would alert the organisation to ensure prompt remediation.

Given the prevalence of scenarios like the one mentioned above, organisations must develop clear frameworks and guidelines for OSINT to ensure analysts follow correct procedures. Strict regulatory and compliance requirements, such as GDPR, further emphasise the need for ethical guidelines and responsible OSINT practices.

The Dangers of OSINT

The accessibility of OSINT appeals to both resourceful security teams looking to improve their cybersecurity and cyberattackers with malicious intent. While OSINT can be a powerful tool for gathering intelligence, it also presents certain risks and challenges.

For instance, OSINT analysts often use OSINT tools for network scanning during security assessments. These same tools can be used by threat actors to identify network vulnerabilities and exploit them. Additionally, threat actors can leverage OSINT to gather intelligence for various cyberattacks, including social engineering.

To mitigate the dangers of OSINT, organisations must implement effective information risk management practices. This includes regularly assessing and addressing vulnerabilities, ensuring proper data protection measures, and staying informed about emerging threats in the OSINT landscape.

In conclusion, OSINT search techniques are valuable for gathering intelligence from publicly available sources. By understanding and utilising the appropriate tools and techniques, organisations can enhance their security posture and proactively address potential vulnerabilities. However, it is crucial to operate within legal and ethical boundaries and to prioritise responsible OSINT practices to mitigate risks and protect sensitive information.