Next.js Sitemap and Robots

Module 26 — Sitemap and Robots in Next.js

A production website needs more than good pages and metadata.

Search engines also need to discover your URLs efficiently and understand which parts of your website should be crawled.

Two important files in Next.js are:

textCopy
1app/
2├── sitemap.ts
3└── robots.ts

They generate:

textCopy
1sitemap.xml
2robots.txt

Think of them as two different systems:

textCopy
1Sitemap
2   ↓
3"These are the important URLs on my website."
4
5Robots
6   ↓
7"These are the crawling rules for my website."

They work together:

textCopy
1                    Search Engine
2                         │
3              ┌──────────┴──────────┐
4              ↓                     ↓
5         sitemap.xml            robots.txt
6              ↓                     ↓
7       URL discovery         Crawling rules
8              │                     │
9              └──────────┬──────────┘
10                         ↓
11                     Your Site

Why Does a Sitemap Matter?

A sitemap is an XML document containing URLs that you want search engines to discover.

For example:

textCopy
1https://example.com/
2https://example.com/courses
3https://example.com/courses/nextjs
4https://example.com/courses/react
5https://example.com/topics/server-components

A sitemap can help search engines discover important pages, especially on large websites.

The architecture is:

textCopy
1Website
2   ↓
3Many URLs
4   ↓
5sitemap.xml
6   ↓
7Search Engine
8   ↓
9URL Discovery

A sitemap is particularly useful when your website contains:

• Many courses
• Blog articles
• Documentation
• Topics
• Tutorials
• Dynamic pages
• Frequently changing content

Sitemap Does Not Guarantee Indexing

This distinction is extremely important.

A sitemap tells a search engine:

"Here are URLs that I consider important."

It does not tell a search engine:

"You must index these pages."

The process is more like:

textCopy
1Sitemap
2   ↓
3URL discovered
4   ↓
5Crawler evaluates page
6   ↓
7Content / quality / technical signals
8   ↓
9Possible indexing

Therefore:

textCopy
1Sitemap
2≠
3Guaranteed indexing

sitemap.ts

Next.js supports generating a sitemap through:

textCopy
1app/sitemap.ts

A basic implementation looks like:

tsxCopy
1import type { MetadataRoute } from "next";
2
3export default function sitemap(): MetadataRoute.Sitemap {
4  return [
5    {
6      url: "https://example.com",
7      lastModified: new Date(),
8    },
9    {
10      url: "https://example.com/courses",
11      lastModified: new Date(),
12    },
13  ];
14}

Next.js can use this file to generate:

textCopy
1/sitemap.xml

The architecture is:

textCopy
1app/sitemap.ts
2       ↓
3Next.js
4       ↓
5sitemap.xml

Static Sitemap

For a small website, you might have:

tsxCopy
1import type { MetadataRoute } from "next";
2
3export default function sitemap(): MetadataRoute.Sitemap {
4  return [
5    {
6      url: "https://example.com",
7      lastModified: new Date(),
8    },
9
10    {
11      url: "https://example.com/courses",
12      lastModified: new Date(),
13    },
14
15    {
16      url: "https://example.com/about",
17      lastModified: new Date(),
18    },
19
20    {
21      url: "https://example.com/contact",
22      lastModified: new Date(),
23    },
24  ];
25}

This works well when your important URLs are mostly static.

But a course platform may contain hundreds or thousands of dynamic URLs.

For that, you need a dynamic sitemap.

Dynamic Sitemap Generation

Suppose your database contains:

textCopy
1Courses
2│
3├── Next.js Masterclass
4├── React Masterclass
5├── TypeScript Masterclass
6└── Node.js Masterclass

Your application can fetch these records:

textCopy
1Database
2   ↓
3Courses
4   ↓
5Generate URLs
6   ↓
7sitemap.xml

For example:

tsxCopy
1import type { MetadataRoute } from "next";
2
3export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
4  const courses = await getCourses();
5
6  return courses.map((course) => ({
7    url: `https://example.com/courses/${course.slug}`,
8    lastModified: course.updatedAt,
9  }));
10}

Now the sitemap automatically changes when your database content changes.

Real Course Sitemap

Suppose your database contains:

jsonCopy
1[
2  {
3    "slug": "nextjs-masterclass",
4    "updatedAt": "2026-08-20"
5  },
6  {
7    "slug": "react-masterclass",
8    "updatedAt": "2026-08-21"
9  },
10  {
11    "slug": "typescript-masterclass",
12    "updatedAt": "2026-08-22"
13  }
14]

The generated URLs become:

textCopy
1/courses/nextjs-masterclass
2/courses/react-masterclass
3/courses/typescript-masterclass

The architecture is:

textCopy
1Course Database
2      ↓
3course.slug
4      ↓
5URL Generation
6      ↓
7sitemap.xml

Course URLs

For a course platform:

textCopy
1/courses
2/courses/nextjs
3/courses/react
4/courses/typescript

You can generate them dynamically:

tsxCopy
1import type { MetadataRoute } from "next";
2
3export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
4  const courses = await getCourses();
5
6  return [
7    {
8      url: "https://example.com",
9      lastModified: new Date(),
10    },
11
12    {
13      url: "https://example.com/courses",
14      lastModified: new Date(),
15    },
16
17    ...courses.map((course) => ({
18      url: `https://example.com/courses/${course.slug}`,
19      lastModified: course.updatedAt,
20    })),
21  ];
22}

Notice that static and dynamic URLs can exist in the same sitemap.

Topic URLs

Large learning platforms often have topic pages too.

For example:

textCopy
1/topics/server-components
2/topics/data-fetching
3/topics/typescript
4/topics/authentication

Suppose your database contains:

textCopy
1Topics
2│
3├── Server Components
4├── Data Fetching
5├── TypeScript
6└── Authentication

You can generate them:

tsxCopy
1const topics = await getTopics();
2
3const topicUrls = topics.map((topic) => ({
4  url: `https://example.com/topics/${topic.slug}`,
5  lastModified: topic.updatedAt,
6}));

Then combine them with your course URLs.

Courses + Topics Sitemap

A more complete sitemap:

tsxCopy
1import type { MetadataRoute } from "next";
2
3export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
4  const [courses, topics] = await Promise.all([
5    getCourses(),
6    getTopics(),
7  ]);
8
9  return [
10    {
11      url: "https://example.com",
12      lastModified: new Date(),
13    },
14
15    {
16      url: "https://example.com/courses",
17      lastModified: new Date(),
18    },
19
20    ...courses.map((course) => ({
21      url: `https://example.com/courses/${course.slug}`,
22      lastModified: course.updatedAt,
23    })),
24
25    ...topics.map((topic) => ({
26      url: `https://example.com/topics/${topic.slug}`,
27      lastModified: topic.updatedAt,
28    })),
29  ];
30}

The architecture becomes:

textCopy
1                  Database
2                     │
3          ┌──────────┴──────────┐
4          ↓                     ↓
5       Courses                Topics
6          ↓                     ↓
7     Course URLs           Topic URLs
8          │                     │
9          └──────────┬──────────┘
10                     ↓
11                sitemap.ts
12                     ↓
13                 sitemap.xml

Fetching From an API

Your sitemap does not necessarily need direct database access.

For example:

textCopy
1Next.js sitemap.ts
2       ↓
3Backend API
4       ↓
5Django / Node.js
6       ↓
7Database

Then:

tsxCopy
1async function getCourses() {
2  const response = await fetch(
3    "https://api.example.com/courses"
4  );
5
6  if (!response.ok) {
7    throw new Error("Failed to fetch courses");
8  }
9
10  return response.json();
11}

Then:

tsxCopy
1export default async function sitemap() {
2  const courses = await getCourses();
3
4  return courses.map((course: {
5    slug: string;
6    updatedAt: string;
7  }) => ({
8    url: `https://example.com/courses/${course.slug}`,
9    lastModified: course.updatedAt,
10  }));
11}

This fits the architecture from the earlier API modules:

textCopy
1Next.js
2   ↓
3Service / API
4   ↓
5Backend
6   ↓
7Database

lastModified

The lastModified value communicates when the content was last updated.

Example:

tsxCopy
1{
2  url: "https://example.com/courses/nextjs",
3  lastModified: course.updatedAt,
4}

If your database contains:

textCopy
1updatedAt:
22026-08-20T10:30:00Z

you can pass that value to the sitemap.

The important principle is:

textCopy
1Database updatedAt
2       ↓
3Sitemap lastModified

Do not artificially update every URL's lastModified value on every deployment if the underlying content has not changed.

changeFrequency

You may also encounter sitemap fields such as:

tsxCopy
1{
2  url: "https://example.com/courses/nextjs",
3  lastModified: course.updatedAt,
4  changeFrequency: "weekly",
5  priority: 0.8,
6}

These fields should not be treated as a ranking guarantee.

In modern search engine crawling, simply setting:

textCopy
1priority: 1.0

does not force a page to rank higher.

The most useful sitemap information is generally:

textCopy
1Correct URL
2+
3Accurate lastModified

Use sitemap fields based on your actual content architecture rather than trying to manipulate rankings.

Canonical URLs and Sitemaps

Your sitemap URLs and canonical URLs should agree.

For example:

textCopy
1Canonical:
2https://example.com/courses/nextjs
3
4Sitemap:
5https://example.com/courses/nextjs

Good architecture:

textCopy
1             Course
2                ↓
3        Preferred URL
4           ↙       ↘
5          ↓         ↓
6     Canonical    Sitemap
7          ↓         ↓
8          └────┬────┘
9               ↓
10         Search Engine

Avoid having:

textCopy
1Canonical:
2https://example.com/courses/nextjs
3
4Sitemap:
5https://example.com/courses/nextjs-course

unless there is a deliberate reason and the URLs are actually equivalent in your architecture.

Sitemap and Redirects

Suppose an old URL is:

textCopy
1/courses/nextjs-course

and the new URL is:

textCopy
1/courses/nextjs

If the old URL redirects to the new URL, your sitemap should generally contain the final canonical URL:

textCopy
1/courses/nextjs

not the old redirected URL.

Think:

textCopy
1Old URL
2   ↓
3301 Redirect
4   ↓
5New URL
6   ↓
7Canonical
8   ↓
9Sitemap

This keeps your URL signals consistent.

robots.ts

The second important file is:

textCopy
1app/robots.ts

It generates:

textCopy
1/robots.txt

A simple example:

tsxCopy
1import type { MetadataRoute } from "next";
2
3export default function robots(): MetadataRoute.Robots {
4  return {
5    rules: {
6      userAgent: "*",
7      allow: "/",
8    },
9
10    sitemap:
11      "https://example.com/sitemap.xml",
12  };
13}

This communicates crawling rules to compliant search engine crawlers.

What Is robots.txt?

robots.txt is a file that tells crawlers which paths they are allowed or disallowed to crawl.

Example:

textCopy
1User-agent: *
2Allow: /
3
4Sitemap: https://example.com/sitemap.xml

The architecture is:

textCopy
1Crawler
2   ↓
3robots.txt
4   ↓
5Crawling rules
6   ↓
7Website

Blocking a Path

Suppose you have an internal path:

textCopy
1/admin/

You could specify:

tsxCopy
1import type { MetadataRoute } from "next";
2
3export default function robots(): MetadataRoute.Robots {
4  return {
5    rules: {
6      userAgent: "*",
7      disallow: "/admin/",
8    },
9
10    sitemap:
11      "https://example.com/sitemap.xml",
12  };
13}

The resulting concept is:

textCopy
1User-agent: *
2Disallow: /admin/
3
4Sitemap: https://example.com/sitemap.xml

Important: Robots Is Not Security

This is one of the most important concepts in this module.

robots.txt does not protect private data.

Do not think:

textCopy
1Disallow: /admin/
2       ↓
3Admin is secure

That is incorrect.

Instead:

textCopy
1Authentication
2      ↓
3Authorization
4      ↓
5Protected Route
6      ↓
7Private Data

robots.txt only communicates crawling preferences to compliant crawlers.

It should never replace:

• Authentication
• Authorization
• Access control
• Server-side validation
• API security

Robots vs Authentication

Compare these architectures.

Incorrect

textCopy
1/admin
2   ↓
3robots.txt disallow
4   ↓
5"Protected"

Correct

textCopy
1/admin
2   ↓
3Authentication
4   ↓
5Authorization
6   ↓
7Server validates user
8   ↓
9Admin content

You learned this distinction in the protected application architecture module.

Robots and Sitemap Work Together

A common production setup:

tsxCopy
1import type { MetadataRoute } from "next";
2
3export default function robots(): MetadataRoute.Robots {
4  return {
5    rules: {
6      userAgent: "*",
7      allow: "/",
8      disallow: [
9        "/admin/",
10        "/dashboard/",
11        "/api/",
12      ],
13    },
14
15    sitemap:
16      "https://example.com/sitemap.xml",
17  };
18}

Conceptually:

textCopy
1Public
2├── /courses/
3├── /topics/
4├── /blog/
5└── /docs/
6
7Private/Internal
8├── /admin/
9├── /dashboard/
10└── /api/

However, remember that disallowing a path in robots does not guarantee that a URL will never appear in search results. It mainly controls crawling, and it should not be used as your primary indexing or privacy mechanism.

Public Content Architecture

For Tech3Space-style content:

textCopy
1Public Content
2│
3├── /
4├── /courses
5├── /courses/[slug]
6├── /topics
7├── /topics/[slug]
8├── /blog
9├── /blog/[slug]
10└── /docs/[slug]

These are potential sitemap candidates.

Private application areas:

textCopy
1Private
2│
3├── /dashboard
4├── /profile
5├── /settings
6└── /admin

These generally should not be included in your public sitemap.

Complete robots.ts

A practical structure:

tsxCopy
1import type { MetadataRoute } from "next";
2
3export default function robots(): MetadataRoute.Robots {
4  const baseUrl = "https://example.com";
5
6  return {
7    rules: {
8      userAgent: "*",
9      allow: "/",
10      disallow: [
11        "/admin/",
12        "/dashboard/",
13        "/settings/",
14        "/api/",
15      ],
16    },
17
18    sitemap: `${baseUrl}/sitemap.xml`,
19  };
20}

The important idea is:

textCopy
1Public pages
2    ↓
3Allow crawling
4
5Internal routes
6    ↓
7Disallow crawling
8
9Sitemap
10    ↓
11Tell crawler where public URLs are

Complete Dynamic Sitemap

For a course platform:

tsxCopy
1import type { MetadataRoute } from "next";
2
3const BASE_URL = "https://example.com";
4
5export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
6  const [courses, topics] = await Promise.all([
7    getCourses(),
8    getTopics(),
9  ]);
10
11  const staticPages: MetadataRoute.Sitemap = [
12    {
13      url: BASE_URL,
14      lastModified: new Date(),
15    },
16
17    {
18      url: `${BASE_URL}/courses`,
19      lastModified: new Date(),
20    },
21
22    {
23      url: `${BASE_URL}/topics`,
24      lastModified: new Date(),
25    },
26  ];
27
28  const coursePages: MetadataRoute.Sitemap =
29    courses.map((course) => ({
30      url: `${BASE_URL}/courses/${course.slug}`,
31      lastModified: course.updatedAt,
32    }));
33
34  const topicPages: MetadataRoute.Sitemap =
35    topics.map((topic) => ({
36      url: `${BASE_URL}/topics/${topic.slug}`,
37      lastModified: topic.updatedAt,
38    }));
39
40  return [
41    ...staticPages,
42    ...coursePages,
43    ...topicPages,
44  ];
45}

This is the basic architecture you can expand as your platform grows.

Large Sitemap Architecture

A very large website should not blindly place every possible URL into one sitemap.

Sitemaps have protocol limits. A single sitemap can contain up to 50,000 URLs or 50 MB uncompressed, whichever limit is reached first.

For a large platform, use a sitemap index or Next.js sitemap partitioning.

Architecture:

textCopy
1                  Sitemap Index
2                       │
3          ┌────────────┼────────────┐
4          ↓            ↓            ↓
5     courses-1.xml courses-2.xml topics-1.xml
6          ↓            ↓            ↓
7       50,000       50,000       50,000
8       URLs max     URLs max     URLs max

For example:

textCopy
1sitemap.xml
2    ↓
3Sitemap Index
4    ├── sitemap-courses-1.xml
5    ├── sitemap-courses-2.xml
6    ├── sitemap-topics-1.xml
7    └── sitemap-blog-1.xml

This becomes important when your platform grows from hundreds to tens or hundreds of thousands of URLs.

Dynamic Sitemap With generateSitemaps

For very large datasets, Next.js also supports sitemap partitioning with generateSitemaps().

Conceptually:

tsxCopy
1import type { MetadataRoute } from "next";
2
3export async function generateSitemaps() {
4  const count = await getCourseSitemapCount();
5
6  return Array.from(
7    { length: count },
8    (_, id) => ({ id })
9  );
10}
11
12export default async function sitemap({
13  id,
14}: {
15  id: number;
16}): Promise<MetadataRoute.Sitemap> {
17  const courses = await getCoursesForSitemap(id);
18
19  return courses.map((course) => ({
20    url: `https://example.com/courses/${course.slug}`,
21    lastModified: course.updatedAt,
22  }));
23}

The exact implementation depends on your database and URL volume.

The architecture is:

textCopy
1Database
2   ↓
3Count URLs
4   ↓
5Split into batches
6   ↓
7Sitemap 0
8Sitemap 1
9Sitemap 2
10...

Database Pagination

Do not load hundreds of thousands of database records into memory unnecessarily.

Instead of:

textCopy
1Database
2   ↓
3Fetch everything
4   ↓
5Application memory
6   ↓
7Generate sitemap

prefer:

textCopy
1Database
2   ↓
3Pagination / batches
4   ↓
5Sitemap segments

For example:

textCopy
150,000 URLs
2     ↓
35,000 per batch
4     ↓
510 sitemap files

This makes large sitemap generation more manageable.

Course + Topic + Blog Architecture

A complete learning platform might have:

textCopy
1Database
2│
3├── Courses
4├── Topics
5├── Lessons
6├── Blog Posts
7└── Documentation

Only include URLs that represent actual indexable pages.

For example:

textCopy
1Courses
2    ↓
3/courses/[slug]
4
5Topics
6    ↓
7/topics/[slug]
8
9Blog
10    ↓
11/blog/[slug]
12
13Documentation
14    ↓
15/docs/[slug]

Avoid creating sitemap entries for every internal database object if there is no public page for it.

Do Not Sitemap Every URL

Suppose your application has:

textCopy
1/api/courses
2/dashboard
3/admin
4/search?q=nextjs
5/login

These are not automatically good sitemap candidates.

A sitemap should focus on canonical, indexable, public URLs.

Think:

textCopy
1Sitemap
2   ↓
3Important public content
4   ↓
5Canonical URLs

not:

textCopy
1Sitemap
2   ↓
3Every URL the application can generate

Sitemap Quality

A high-quality sitemap should contain:

textCopy
1✓ Public URLs
2✓ Canonical URLs
3✓ Working URLs
4✓ Indexable pages
5✓ Useful content
6✓ Accurate lastModified values

Avoid:

textCopy
1✗ Redirect URLs
2✗ 404 URLs
3✗ Private dashboard URLs
4✗ Duplicate URLs
5✗ Temporary URLs
6✗ URLs blocked by authentication
7✗ Parameter combinations

Search Engine Crawling

The relationship between your technical SEO files can be understood as:

textCopy
1                 Website
2                    │
3        ┌───────────┴───────────┐
4        ↓                       ↓
5   robots.txt               sitemap.xml
6        ↓                       ↓
7 Crawling rules           URL discovery
8        │                       │
9        └───────────┬───────────┘
10                    ↓
11               Search Engine
12                    ↓
13               Crawl Page
14                    ↓
15            Evaluate Content
16                    ↓
17                Indexing

This is why sitemap and robots should be designed together.

Crawling vs Indexing

These terms are different.

Crawling

A search engine requests your page.

textCopy
1Crawler
2   ↓
3GET /courses/nextjs

Indexing

The search engine decides whether to store and potentially serve that page in search.

textCopy
1Page
2 ↓
3Processed
4 ↓
5Index

Therefore:

textCopy
1Crawling
2≠
3Indexing

And:

textCopy
1Sitemap
2≠
3Guaranteed indexing

Canonical + Sitemap + Robots

A strong SEO architecture connects all three:

textCopy
1                    Page
2                     │
3          ┌──────────┼──────────┐
4          ↓          ↓          ↓
5      Canonical   Sitemap     Robots
6          ↓          ↓          ↓
7      Preferred    Discover   Crawl
8        URL          URL      Rules
9          │          │          │
10          └──────────┼──────────┘
11                     ↓
12               Search Engine

For a public course:

textCopy
1Canonical:
2https://example.com/courses/nextjs
3
4Sitemap:
5https://example.com/courses/nextjs
6
7Robots:
8Allow crawling

This is a clean architecture.

Common Mistakes

Mistake 1 — Sitemap Contains Old URLs

If your application has moved:

textCopy
1/courses/nextjs-course

to:

textCopy
1/courses/nextjs

do not continue generating the old URL in your sitemap.

Use the current canonical URL.

Mistake 2 — Sitemap Contains 404 Pages

Bad:

textCopy
1/courses/deleted-course

If the page returns 404, remove it from the sitemap.

Architecture:

textCopy
1Database
2   ↓
3Only published content
4   ↓
5Sitemap

Mistake 3 — Sitemap Contains Private Pages

Do not generate:

textCopy
1/dashboard
2/admin
3/profile
4/settings

as public sitemap URLs.

Mistake 4 — Robots Used as Security

Never rely on:

textCopy
1Disallow: /admin/

to secure administrative content.

Security belongs in:

textCopy
1Authentication
2+
3Authorization
4+
5Server-side access control

Mistake 5 — Incorrect lastModified

Do not use:

tsxCopy
1lastModified: new Date()

for every dynamic record if the content did not actually change.

Prefer:

tsxCopy
1lastModified: course.updatedAt

when that timestamp represents the actual content update.

Mistake 6 — Generating Every Database Record

Not every database record is a public page.

Use:

textCopy
1Published
2+
3Public
4+
5Canonical
6+
7Indexable

as your sitemap criteria.

Production Sitemap Strategy

For a growing learning platform:

textCopy
1                    Database
2                       │
3       ┌───────────────┼────────────────┐
4       ↓               ↓                ↓
5    Courses          Topics           Blog
6       ↓               ↓                ↓
7 Published          Public           Published
8       ↓               ↓                ↓
9 Course URLs        Topic URLs       Blog URLs
10       └───────────────┼────────────────┘
11                       ↓
12                 sitemap.ts
13                       ↓
14                  sitemap.xml
15                       ↓
16                Search Engines

And:

textCopy
1robots.ts
2    ↓
3robots.txt
4    ↓
5Crawl Rules
6    +
7Sitemap Location

Real-World Example

Imagine your site contains:

textCopy
1100 courses
2500 topics
3300 blog posts

Your sitemap could contain:

textCopy
11 homepage
21 courses index
3100 course pages
41 topics index
5500 topic pages
61 blog index
7300 blog pages

Approximately:

textCopy
1904 public URLs

The sitemap architecture becomes:

textCopy
1Database
2   │
3   ├── 100 Courses
4   ├── 500 Topics
5   └── 300 Blog Posts
6            ↓
7      URL Generation
8            ↓
9        sitemap.xml

As the website grows:

textCopy
1904 URLs
2   ↓
310,000 URLs
4   ↓
550,000 URLs
6   ↓
7100,000+ URLs

you can move toward sitemap partitioning.

Module 26 Learning Checklist

After completing this module, you should understand:

• What a sitemap is
• Why sitemaps matter
• sitemap.ts
• Static sitemap generation
• Dynamic sitemap generation
• Course URLs
• Topic URLs
• Blog URLs
• lastModified
• Sitemap URL quality
• Sitemap limits
• Sitemap partitioning
• generateSitemaps
• Database pagination
• robots.ts
• robots.txt
• Search engine crawling
• Crawling vs indexing
• Canonical URLs
• Sitemap and canonical consistency
• Redirect URLs
• Private routes
• Why robots is not security
• Public vs private content
• Large-site sitemap architecture

Final Mental Model

The most important relationship is:

textCopy
1                 Your Database
2                       │
3              ┌────────┼────────┐
4              ↓        ↓        ↓
5           Courses   Topics    Blog
6              │        │        │
7              └────────┼────────┘
8                       ↓
9                  sitemap.ts
10                       ↓
11                  sitemap.xml
12                       │
13                       ↓
14                 URL Discovery
15                       │
16                       ↓
17                 Search Engine
18                       ↑
19                       │
20                  robots.txt
21                       │
22                  Crawl Rules

And for each public page:

textCopy
1Public Page
2    ↓
3Canonical URL
4    ↓
5Sitemap
6    ↓
7Robots allows crawling
8    ↓
9Crawler discovers page
10    ↓
11Search engine evaluates content
12    ↓
13Possible indexing

The key lesson is:

sitemap.ts helps search engines discover your important public URLs, while robots.ts communicates crawling rules. A production SEO architecture should generate canonical, working, public URLs dynamically from your content database and keep sitemap, canonical, and robots configuration consistent.