A penetration tester conducts OSINT for a client and discovers the robots.txt file explicitly blocks a major search engine. Which of the following would most likely help the penetration tester achieve the objective?
In OSINT, the tester’s objective is to gather information using publicly available, non-intrusive sources without altering the target environment. A robots.txt file is a crawler directive that can discourage or block specific search engine bots from indexing certain paths. If one “major” engine is explicitly blocked, the most practical OSINT adjustment is to use other search engines and data providers that may not be restricted in the same way or may already have historical indexing and cached results. This aligns with PenTest+ reconnaissance techniques that emphasize using multiple sources and pivoting between providers to maximize coverage and reduce blind spots.
Modifying the WAF or changing robots.txt would be active alteration of the client’s systems and is not an OSINT method; it also typically falls outside the intent of passive recon and may violate rules of engagement. A CSRF attack is an exploitation technique unrelated to discovering publicly indexed information. Therefore, leveraging a competing provider is the best way to continue OSINT collection when one crawler is blocked.
Contribute your Thoughts:
Chosen Answer:
This is a voting comment (?). You can switch to a simple comment. It is better to Upvote an existing comment if you don't have anything to add.
Submit