Technical SEO Basics for Developers: Sitemaps, Robots, Canonicals and Schema

Content and links get most of the attention in SEO, but none of it helps if search engines cannot crawl, understand and index the site properly. Technical SEO covers those foundations, and many of them sit squarely in the developer's hands.

This guide covers the basics every junior developer building Malaysian business sites should implement by default, and that business owners can ask about when reviewing a new site.

Crawlable, Meaningful HTML

Search engines read HTML. Use semantic elements such as headings in logical order, real links with href attributes, descriptive page titles and meta descriptions. Avoid hiding important content behind interactions that require clicks or scripts to load.

Modern search engines can render JavaScript, but rendering may be delayed and errors can hide content. For content-heavy pages, server-side rendering or static generation is generally safer than relying entirely on client-side rendering.

• One clear H1 per page describing its topic
• Unique title and meta description for each page
• Descriptive URLs using words, not random IDs
• Alt text on meaningful images

Internal linking is part of technical SEO too. Make sure important pages are linked from navigation or related content, rather than only reachable through search or filters. Pages with no internal links are hard for crawlers to find.

XML Sitemaps and Robots.txt

An XML sitemap lists the pages you want indexed, helping search engines discover content, especially on new or large sites. Generate it automatically from your CMS or framework so it stays current, include only indexable canonical URLs and submit it in Google Search Console.

The robots.txt file tells crawlers which areas not to crawl, such as admin pages or internal search results. Be careful: blocking a page in robots.txt does not guarantee it stays out of search results, and accidentally blocking the whole site is a classic launch mistake when a staging setting is copied to production.

Launch check: confirm that robots.txt and meta robots tags on production do not block indexing, and that staging sites are protected from being indexed.

Canonical Tags and Duplicate Content

The same content can appear at several URLs: with and without www, with tracking parameters, sorted product listings or printer-friendly pages. Canonical tags tell search engines which URL is the preferred version.

1. Add a self-referencing canonical tag to each indexable page.
2. Point filtered or parameter variants to the main version where appropriate.
3. Use one consistent domain version, with redirects from the others.
4. Avoid generating thousands of thin filter pages that dilute crawl budget.

For ecommerce sites, faceted navigation is a common source of index bloat; see performance budgets for related speed considerations, since fast pages are also crawled more efficiently.

Structured Data

Structured data, usually written in JSON-LD using the schema.org vocabulary, helps search engines understand page content such as organisation details, local business information, products, articles, FAQs and breadcrumbs. It can make pages eligible for rich results in search.

Only mark up content that is visible on the page and accurate. Test markup with Google's Rich Results Test and monitor Search Console for errors. For local businesses, accurate name, address, phone and opening hours in structured data complement a Google Business Profile.

Breadcrumb markup and organisation markup are good starting points for most business sites, since they are simple to implement and describe the site structure clearly.

Redirects, Migrations and Monitoring

When URLs change, use permanent 301 redirects from old URLs to the most relevant new ones. Website redesigns and migrations without redirect plans are one of the most common causes of sudden traffic drops. Keep a mapping spreadsheet of old to new URLs and test it before launch.

After launch, monitor Google Search Console for crawl errors, indexing issues and Core Web Vitals. Fix broken internal links and remove redirect chains where possible. Our pre-launch QA checklist is a good place to add these checks.

Technical SEO is not a one-time task. Every new feature, template or plugin can affect crawlability, so include SEO checks in routine development reviews.

For multilingual sites, use hreflang annotations so search engines show the right language version to users. Our guide on multi-language websites for Malaysia covers the details.

FAQ

Do I need a sitemap for a small site?

It is helpful even for small sites and easy to generate. It speeds up discovery of new pages.

Does structured data improve rankings?

It helps search engines understand content and can enable rich results, but it is not a direct ranking boost on its own.

Is a JavaScript framework bad for SEO?

Not necessarily. Use server-side rendering or static generation for important content to reduce risk.

Technical SEO gives good content a fair chance. Build crawlable semantic HTML, generate accurate sitemaps, use robots.txt carefully, set canonical URLs, add honest structured data and plan redirects for every URL change.