Robots.txt is a text file at the root of a website that tells search engine crawlers which URLs they are allowed or disallowed to crawl.
Located at yourdomain.com/robots.txt, this file gives instructions to crawlers — which directories or pages to skip, and where the XML sitemap is. It is the first thing most crawlers check.
Robots.txt controls crawling, not indexing. A page blocked in robots.txt can still appear in search if it is linked elsewhere — to keep a page out of the index, use a noindex meta tag instead (and do not block it in robots.txt, or Google cannot see the noindex).
It tells search engine crawlers which parts of a site they may or may not crawl, and points them to the sitemap. It is used to keep crawlers away from admin, staging or low-value URLs.
No. It controls crawling, not indexing. A disallowed page can still be indexed if linked elsewhere. To prevent indexing, use a noindex meta tag and leave the page crawlable so Google can see it.
At the root of the domain — for example, https://example.com/robots.txt. Crawlers check this location automatically before crawling the site.
Tell us a little about your business and a senior strategist will send a tailored growth plan — no obligation.
We combine data, AI, and market insight to optimize every touchpoint — for maximum ROI and sustained growth.
We continuously track, analyse, and adjust campaigns in real time — turning data into instant, actionable improvements.
From market entry to scaling internationally, we help brands expand and localize with precision and efficiency.
Launching or accelerating, our agile approach lets you scale campaigns, teams, and budgets without friction.
We act as an extension of your business, collaborating closely to solve complex challenges. Whether it’s long-term innovation or short-term impact, our dedicated teams bring agility, expertise, and a results-driven approach to your most important goals.
Based on the strategy, we start working on a delivery plan and schedule, designing short- and long-term goals.
To power the relentless pursuit of growth — and redefine what’s possible for India’s enterprises.
Case studies ↗Through a team of revenue-obsessed operators who demand results over vanity — driven by transparency, ownership, and craft.
Meet the team ↗We engineer end-to-end growth through strategy, marketing, and data — measured in pipeline, not vanity metrics.
Get a proposal ↗Get a free, no-obligation growth plan from a senior strategist within 24 hours.
