robots-builder

The Problem It Solves

Every website needs a robots.txt file at its root. It's the standard protocol that tells search engine crawlers — Googlebot, Bingbot, GPTBot, and hundreds of others — which parts of your site they're allowed to visit.

High Risk

Block too much and Google stops indexing; block too little and AI scrapers take everything.

No Feedback

Writing it by hand offers no linting, no validation, and no immediate error feedback.

Complex Rules

Non-obvious rules like required empty Disallow: lines or specific directive ordering.

Installation

npm install robots-builder

Quick Start

Get up and running in seconds with full TypeScript support.

example.ts

import { build } from 'robots-builder';

const txt = build({
  rules: [
    {
      userAgent: '*',
      allow: ['/'],
      disallow: ['/admin', '/api', '/private'],
      crawlDelay: 5,
    },
    {
      userAgent: 'GPTBot',
      disallow: ['/'],
    },
  ],
  sitemaps: ['https://example.com/sitemap.xml'],
  host: 'https://example.com',
});

console.log(txt);

API Reference

build(config)

Generates a robots.txt string. Throws a RobotsBuilderError if the config is invalid.

build.ts

import { build, RobotsBuilderError } from 'robots-builder';

try {
  const txt = build(config);
  fs.writeFileSync('public/robots.txt', txt);
} catch (err) {
  if (err instanceof RobotsBuilderError) {
    console.error(err.errors); // ValidationError[]
  }
}

validate(config)

Returns { valid: boolean, errors: ValidationError[] }. Use this to check before building.

parse(content)

Parses an existing robots.txt string back into a typed config object.

Ready-made Presets

allowAll

Allows all crawlers with no restrictions.

blockAll

Blocks all crawlers from the entire site.

standard

Allows Google/Bing, blocks AI and internal paths.

blockAiCrawlers

Shields your content from GPTBot, CCBot, etc.

nextjs

Optimized for Next.js internal routes.

Framework Integration

Next.js (App Router)

app/robots.ts

import { build, presets } from 'robots-builder';

export default function robots() {
  return build({
    ...presets.standard,
    sitemaps: [`${process.env.URL}/sitemap.xml`],
  });
}

Astro

src/pages/robots.txt.ts

import { build, presets } from 'robots-builder';

export const GET = () =>
  new Response(build(presets.standard), {
    headers: { 'Content-Type': 'text/plain' },
  });

Powerful CLI

CLI Usage

# Generate robots.txt with a preset
robots-builder --preset standard > public/robots.txt

# Validate an existing file
robots-builder --validate public/robots.txt

# Parse an existing file into JSON
robots-builder --parse public/robots.txt > config.json

# Use a custom config file
robots-builder --config robots.config.js --comment