Service overview
This internal service manages automated indexing bots and crawler policies for protected environments. Access to restricted resources is regulated and documented.
Security policy
Some internal paths are not intended for public access.
Automated agents are instructed accordingly.
Manual inspection of crawler policies is discouraged.
System notes
Configuration files are updated automatically.
Legacy paths may still exist for backward compatibility.