How to build dynamic sitemap with Next.js
Try sitemap generator to generate your sitemap in this link
Now let's understand what is a sitemap and how to generate it for Next.js project for both server side rendering and static generation
What is a sitemap?
To succeed building it, you should first understand your project architecture, and how pages are served, usually in Next.js projects, served pages are under pages folder, so this is the main folder which will contain your pages. Next.js also allows to name js files with "[]" ,which mean what's inside the brackets will be served dynamically, in order to demonstrate, we will suppose that you have created your "[id].js" under pages folder and you would like to build your sitemap dynamically. Of course you should adapt this to what you have in your own project.
In the next lines, we will detail the steps to build the sitemap dynamically with Next.js:
Step 1: Serve sitemap.xml with Next.js server-side Rendering
The file have a specific format which is XML format, so the first thing to think of is how can you serve xml instead of HTML, which Next.js serve most of the time, to do so we need to use the Next.js server side rendering getServerSideProps function since this is the only method where we can read http requests and responses and specify which content to send to the user. In this method, we define the http response header for content type as xml, and we are building our URLs tags based on the array that we will discuss shortly.
Next.js Sitemap javascript file structure
Most search engines prefer to have a file called sitemap.xml under your root directory,or specify it's name in your robots.txt file to have an idea about the links that you have, to do so we will create a file called "sitemap.xml.js" under pages folder, that way it will serve our document with the name sitemap.xml under the root directory when sending a get http request to /websiteRoot/sitemap.xml.
Below what the sitemap.xml.js file will look like, it should be named sitemap.xml.js and created under pages directory:
import { baseUrl } from "../lib/constants";
// the main function to be exported as default
const Sitemap = () => {};
export const getServerSideProps = ({ res }) => {
//we get our array from the environment
const posts = JSON.parse(process.env.posts);
const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
${posts
.map((url) => {
return `
<url>
<loc>${baseUrl}${url.path}</loc>
<lastmod>${url.date}</lastmod>
<changefreq>daily</changefreq>
<priority>0.9</priority>
</url>
`;
})
.join("")}
</urlset >
`;
res.setHeader("Content-Type", "text/xml");
res.write(sitemap);
res.end();
return {
props: {},
};
};
export default Sitemap;
You can always try the next.js sitemap generator if you have trouble getting your sitemap
Step 2: Build sitemap from filesystem with Next.js
The less costly way is to build our content based on files on our system, like markdown files for example, so what we have to do is read our content from the filesystem and build our sitemap. The idea is pretty simple but with Next.js, once you are using getServerSideProps, your directory is no longer accessible even if you put it under public folder when the project is deployed using next build.
The reason why is that Next.js constructs and make a minimal package when build is finished, and the project structure deployed is different than the one in development. The use of getStaticProps function will give us access to filesystem but we can’t serve xml with it. Keep in mind that you cannot mix static and dynamic in the same page, since it is not allowed, at least till Next.js version 11.
Step 3: Separate development environment from production
To distinguish sitemap generation production from local environment, we can use the process.env.NODE_ENV like the way above, so create a file called lib/constants.js under the root directory and copy this :
//replace yourwebsite by your own website, the localhost port 3000 to the configured port
export const baseUrl = {
development: "http://localhost:3000",
production: "https://yourWebsite.com",
undefined: "https://yourWebsite.com",
}[process.env.NODE_ENV];
Step 4: Include a prebuild script
NPM allows to have a prebuild script to run before the npm build, so we are going to use this trick to read the files from our directory before project is built.
We are constructing an array which has our files and some meta info about each file, this array will be written to the .env.production file before build, that way our node.js process will have access to it using process.env.posts where posts is the name of the array we are writing to our env file, that way no need for any extra library for us to build our sitemap.
let’s take a look at the code, you can create a file called lib/sitemapcjs.js and copy the below content to it:
const fs = require("fs");
const path = require("path");
const mydir = path.join(process.cwd(), "your_directory_under_project");
function getArticles() {
return fs.readdirSync(mydir);
}
function getArticleMeta(article, fields = []) {
//your logic to read articles
return articleMeta;
}
const articles = getArticles();
const basicPosts = articles.map((slugFile) => {
//this depends on the way you construct you meta articles
// and how it is possible to deconstruct them
//here we got the filename, and the date when it was written
const { slug, date } = getArticleMeta(slugFile, ["slug", "date"]);
return { slug, date };
});
//build our articles array
const posts = [
//map static pages such as home, about and privacy
{ path: `/`, date: "2021-09-01T23:35:07.322Z" },
{ path: `/about`, date: "2021-09-05T23:35:07.322Z" },
{ path: `/privacy`, date: "2021-09-23T23:35:07.322Z" },
//map the others
...basicPosts.map((post) => {
return {
path: `/${post.slug}`,
date: post.date,
};
}),
];
//Save the array in the env file, luckily this one will be available on runtime, so we will have
//access to is as long as the app is running
fs.writeFileSync(
".env.production",
"posts=" + JSON.stringify(posts, null),
"utf-8"
);
Sample implementation of the getArticleMeta with Markdown files
Below a sample implementation to use if your files are Markdown files:
const fs = require("fs");
const path = require("path");
const matter = require("gray-matter");
//the directory where you keep markdown files
//you can copy and paste this code and the above code in the same file
//you need just to remove the mydir declaration as it is duplicated
const mydir = path.join(process.cwd(), "your_directory_under_project");
function getArticleMeta(slug, fields = []) {
const realSlug = slug.replace(/\.md$/, "");
const fullPath = path.join(mydir, `${realSlug}.md`);
const fileContents = fs.readFileSync(fullPath, "utf8");
const { data, content } = matter(fileContents);
const items = {};
fields.forEach((field) => {
if (field === "slug") {
//store the markdown file name in items[slug]
items[field] = realSlug;
}
/
if (field === "content") {
// store file content in items[content] with matter library
items[field] = content;
}
if (data[field]) {
//store any field you declare in the header markdown file to the item[givenName]
items[field] = data[field];
}
});
return items;
}
The reason the script above is written in common js is that when you create a Next.js app using create-next-app, the package.json is by default set to commonjs for all the js files contained in the project unless you specify {type:"module"} in it, you can check more in depth in the javascript history tutorial, but once this option is added, your project will not work since a Next.js project uses babel to transform "import syntax" scripts to the common js, and since you specify the module js for all files in the project, it does not understand the "import syntax", so it is better to keep the package.json as it is and write the prebuild script using common js.
Step 5: include the prebuild step to the package.json
Finally, in our package.json we need to add the prebuild script like so:
"scripts": {
"prebuild": "node lib/sitemapcjs.js",
"build": "next build",
"start": "next start",
"lint": "next lint"
},
Run the npm run build now, and open the file .env.production, you should see something like the following:
posts=[{"path":"/","date":"2021-09-01T23:35:07.322Z"},{"path":"/about","date":"2021-09-05T23:35:07.322Z"},{"path":"/privacy","date":"2021-09-23T23:35:07.322Z"}]
Now go to your browser and type yourAddress/sitemap.xml, in my case it is localhost:8003/sitemap.xml
Congratulations, you are done and your sitemap will be generated dynamically and set the correct URL whether you are in production or test environment everytime you run a new build.
Build sitemap from CMS in Next.js
You can also use the easy way to build your sitemap from a CMS, but it comes with the burden that you have already one, basically all the content needed is fetched through the API, so you need to make sure that the url constructed under tag loc is exactly the url of the page, you can define a field in the CMS where the name is the xml "loc" tag of your article, and place the call to your api right under where you define your getServerSideProps function.
Conclusion
In summary, the sitemap is generated dynamically, so whenever a build is done, we are generating a new array in the env file using the NPM prebuild capability, this array is available at runtime, and in our getServerSideProps method, we are reading this array and serving our sitemap file accordingly.
Do not forget to try out the sitemap generator, it will improve your website seo easily
