When and how to use HTML sitemaps for SEO and UX

Google search advocate John Mueller has always downplayed the value of HTML sitemaps. Recently, he reiterated his dissatisfaction with HTML sitemaps, saying: 

“I changed my mind on HTML sitemaps over the years, they should never be needed. Sites small [and] large should always have a clear navigational structure.”

This is a slightly stronger stance than what he’s expressed in the past:

“When it comes to SEO … for small sites, your site should be crawlable anyway (and if you’re using a common CMS, it’ll almost always be fine) & for large sites, they’re not going to be useful anyway (use sitemaps, use normal cross-linking, check with a crawler of your choice).”

While Mueller makes a good point, in my opinion, we can’t just completely drop HTML sitemaps. Links and website usability still matter. 

I also like what former Googler and now popular SEO expert Pedro Dias said in response to Mueller’s remarks, which reflects the reality of the web:

“Unfortunately it’s not easy to change the architecture on complex and large websites. Often, something like this, [sic!] can take years to change, as it depends on multiple teams. So, HTML sitemaps are often a welcome and temporary-ish solution. I used to dislike them… Nowadays, I see their use-cases.”

There is a compromise between both HTML and XML sitemaps, so as always in SEO, “it depends.” 

Read below why, when and how to use sitemaps to benefit your website and its users. 

What are sitemaps and why did they get invented?

In the early days of the web, many websites grew organically. 

Websites weren’t appropriately planned by information architects according to UX best practices. 

They started from scratch, and webmasters added content whenever needed in often far-from-usable ways. 

Those were the days before blogs, social media or content management systems. When you built a site, you did it yourself using HTML.

In most cases, you were manually coding the HTML in a text editor, using a messy WYSIWYG editor like Dreamweaver or one of the first often custom-built and pretty cumbersome CMS tools. Either way, website structures quickly became confusing.

So out of necessity, people started adding sitemaps to their websites to make some sense of the increasing disorder. Those sitemaps were created in plain HTML and looked like any other webpage, part of the fact that they displayed a map instead of written content.

Later search engines introduced so-called XML sitemaps meant to replace manual “search engine submission.”

More than 25 years later, HTML sitemaps are less preferred. But can we really stop using HTML sitemaps, just like we dropped guest books or webrings? 

HTML vs. XML sitemaps – the visible vs. the invisible

To make it short and simple, the main difference between HTML and XML sitemaps is that an HTML sitemap is usually visible to website visitors while an XML sitemap is not. (The latter primarily made for search engines.)

The XML sitemaps standard was introduced at the end of 2007 by search engines like Google, Yahoo and Bing (then MSN). 

By default, they are formatted in a way that does not display well or is readable for humans. Instead, such sitemaps are used mainly to ensure proper and faster indexing by search engines.

On the other hand, just because a resource is on an XML sitemap does not mean Google and Bing will index it, or it will be findable even if it’s in the index.

Another SEO colleague Rob Watts added a tongue-in-cheek, first-hand account to the debate by saying:

“I had sites do well for years that were absent of html or xml sitemaps. I removed my tin foil hat and began using xml sitemaps and enjoyed seeing how long it took for urls to appear in serps. I don’t think I saw any noticeable improvement performance wise for their use, but it made some analyses/reports slightly easier.”

So with XML sitemaps only theoretically ensuring crawling and indexing, do we have to focus on them and neglect visible or visual sitemaps? 

Get the daily newsletter search marketers rely on.

<input type="hidden" name="utmMedium" value="“>
<input type="hidden" name="utmCampaign" value="“>
<input type="hidden" name="utmSource" value="“>
<input type="hidden" name="utmContent" value="“>
<input type="hidden" name="pageLink" value="“>
<input type="hidden" name="ipAddress" value="“>

Processing…Please wait.

See terms.

function getCookie(cname) {
let name = cname + “=”;
let decodedCookie = decodeURIComponent(document.cookie);
let ca = decodedCookie.split(‘;’);
for(let i = 0; i <ca.length; i++) {
let c = ca[i];
while (c.charAt(0) == ' ') {
c = c.substring(1);
}
if (c.indexOf(name) == 0) {
return c.substring(name.length, c.length);
}
}
return "";
}
document.getElementById('munchkinCookieInline').value = getCookie('_mkto_trk');

Why should we create sitemaps solely for Google?

Over the years, Google spokespeople told us to create websites for users, not search engines. Why make an exception when it comes to sitemaps and only create them for Google and other search engines? 

It makes me wonder whether other SEO tactics that display one thing for search bots and something different for website visitors may also be a good idea.

Anyway, Google said so, so we must listen, right? Not really.

There must be a reason why Google dislikes HTML sitemaps and prefers XML.

XML sitemaps are usually automated and include all the content published as soon as it’s online. 

On WordPress, numerous tools create sitemaps for you. Yoast SEO also covers you, for example, even the basic free version.

In 2020, WordPress finally added XML sitemap support to its core, so you won’t even need external plugins.

The XML code format is also machine-readable, while HTML is messy. HTML has to work despite errors, while XML breaks when you forget a character.

So at the end of the day, the XML sitemap creation works mostly behind the scenes without error-prone human hands interfering.

Also, most people will never see your XML sitemap, they wouldn’t even know where to find it as it’s not linked anywhere visibly on the site.

Making XML sitemaps work for people

Luckily, there is always some middle ground. You don’t have to make XML sitemaps invisible. 

You can format XML sitemaps with stylesheets just like you do with any HTML pages.

This technique has been working as long as XML has been around. There are tutorials out there that help you with that.

Indeed for a while, it seemed that XML would replace HTML as the web standard, or it was a compromise called XHTML.

For compatibility reasons, that didn’t happen. Even though the XHTML format was cleaner and less error-prone. When an error happened, the website did break.

With HTML5 introduced instead, browsers embraced sloppily coded websites to make publishing easier but at the expense of automatic crawling.

Now you have ideally created an XML sitemap that can be read by people without coding skills. Remember that it’s still a bland list of all the content you have.

You may still consider creating a readable HTML sitemap for your website visitors.

When to use visual or HTML sitemaps

It’s not either/or. You can:

  • Decide to separate both visible and invisible sitemaps instead of making your XML sitemap visible to people.
  • Create both visible (HTML) and invisible (XML) sitemaps on your site for different reasons. 
  • Have more than one HTML or XML sitemap. 

It depends on what your site looks like and what your objectives are.

Use these questions as a guide to making a decision:

  • Do you want to create a quick overview page of your site? Use an HTML sitemap!
  • Do you want to get indexed as soon and as deeply as possible? Create an XML sitemap.

That said, there are edge cases, and many websites do not conform to typical standards. 

Here are example use cases for visual or HTML sitemaps.

Legacy and/or hard-coded sites

Believe it or not, there are still legacy sites from decades ago on the web, and even hard-coded sites made manually in HTML, CSS and/or JavaScript.

In fact, you’ve probably used one quite often. 

Amazon.com is almost the same site it was many years ago. It has been continuously refreshed over the years and improved to this day. 

An all-out redesign would have been too risky and disruptive for the bottom line.

Some basic legacy sites may have difficulty creating an XML sitemap on the fly. So it’s not a problem when they have a hard-coded HTML sitemap instead. 

Do you have to remove it because Google says so? Of course not.

Large sites

When Mueller said that you needed proper site navigation, he was generally right. But it does not always apply to larger sites. 

You can’t possibly fit a highly complex site with thousands of pages into a menu without it quickly spiraling out of control. 

For better findability, it’s useful for both humans and search bots to have sitemaps that reflect the overall site structure. 

You don’t have to list every page, as that would make it too large.

Alternative to mega menus

Many sites attempt to fit as much as possible into their site navigation and end up having so-called mega menus. 

You hover your mouse over the navigation, and the menu suddenly covers the whole site. 

Users get quickly overwhelmed visually with dozens of options to select from.

Also, sometimes, if a user is not fast enough to point at the right menu item, it will disappear when they lose the level one menu. 

Mega menus are tempting, especially with large companies where every stakeholder wants to be on top. 

But they are often a user experience nightmare, as Smashing Magazine founder Vitaly Friedman clearly demonstrates.

When doing web outreach, you might notice that many sites make it quite hard to contact them. 

Such is often the case with sites having no internal search feature. You will often struggle to locate their contact page. 

A search feature would be a logical solution, yet surprisingly some sites do not have one, even those run on WordPress, which has built-in functionality. 

Some themes or website owners simply remove the search input.

When I can’t find a standard page in your menu and there is no search feature available,a visual sitemap would be nice.

Orphan pages

When you manage to create a menu that is usable without becoming so huge that it does not fit some screens anymore, you will often end up with orphan pages. 

Orphan pages are webpages that are not linked internally from the menu structure or inside the content. 

This happens when content gets removed and updated, and some links disappear.

Oftentimes, orphan pages should be removed altogether. 

Others still make sense but might not be as significant to place into the main menu or the footer links. 

Such orphan pages would be fine when linked in a visible, readable sitemap so that people looking for them can find and access them.

Faster indexing 

Let’s be honest! Just because a page or other resource (think image or video) is listed on an XML sitemap does not mean Google will index it. 

Your site may not have enough authority to get all its content into the Google index. 

This was quite a common issue for new sites in the past. Especially with newly created sites and Google having more limited computing power, some content did not make it into the index on purpose.

It’s less of an issue now, but you still may want to link content internally to boost your chances of getting indexing. 

Link to the more important content on top, prominently, and/or more often.

When we talk about authority, we refer to the link equity that some SEOs still call “link juice.” 

An HTML sitemap that shows some links on top and others at the bottom regulates link equity flow to some extent. 

Pages linked on top are expected to have and receive more authority than pages at the bottom. 

With XML sitemaps, there is no such hierarchy, as far as I know. All resources listed on it are just that, items in a list. 

The item on top is merely the latest one, not necessarily the most important one or the one needing the highest authority. 

There is no inherent hierarchy on XML sitemaps unless you code one into it by declaring that some resources are more important and, thus, should be crawled more often than others.

A visible HTML sitemap allows you to link those resources on top that matter most and thus send more users and authority their way, just like in your site navigation. So it’s not “latest” on top but most important on top, ideally.

Usability

As we are talking SEO here, the most important aspect of sitemaps, improving usability, comes at the end. 

This is Search Engine Land and not UX Collective, after all. They advocate creating an (internal) UX sitemap for every website project, at least while establishing the information architecture.

HTML sitemaps improve the overall user experience, especially:

  • Accessibility.
  • Discoverability.
  • Findability.
  • Usability. 

These are still the strongest use cases for HTML sitemaps, just like in the ‘90s, way before Google started using links for indexing and ranking and long before XML sitemaps were introduced. 

Older users (I admit I’m among them) may even prefer clear and page-style sitemaps instead of shaky menus that vanish when you don’t touch them or search features showing irrelevant items. 

Some elderly people cannot use mouse-over and mouse-click actions efficiently. 

People with disabilities, particularly the visually impaired, need as few mouse actions as possible. Thus, a one-page overview might be best for them.

Useful sitemap examples in the wild

The New York Times 

I often use the New York Times as an example because almost everybody knows the publication, and it’s such an authority. 

This time I want to show the NYT as an example simply because they managed to do something right and stand out. 

The NYT uses visible HTML sitemaps in a way that is really helpful for readers, search users and search engine bots alike. 

The NYT has several sitemaps highlighted by fellow SEO Chris Long of Go Fish Digital on LinkedIn. It is a prime example of using sitemaps to benefit a website.

“60 Second SEO: The New York Times has one of the best HTML sitemaps I’ve seen. Here’s an example of how an article from 1912 is only 5 steps away from [their] home page:”

I don’t want to go too much into detail here, as Long has done an excellent job already at it.

Apple.com

Another great example of sitemap usage for a large site is Apple.com. Their plain and simple text sitemap is useful for people and search engines.

However, note how much the sitemap differs from the actual top navigation the Apple site has above the content.

The “Apple sitemap” starts with “About Apple” (not linked) and the following links:

Apple Site Map

The site navigation starts with the shop and specific products in contrast:

APPLE

Apparently, the SEO team decided to make the shop and products more visible to website visitors and search bots. At the same time, the true structure or importance of the pages seems to be completely different.

In any case, the huge Apple sitemap gives users a good overview of the site contents without overloading the page despite more than a hundred links.

Moreover, they have another sitemap in the footer reflecting the top menu structure instead of the standalone “sitemap” on the dedicated page.

It is smaller both by the font size and by the number of links listed but still very useful.

Apple footer site map

So no matter where you end up, you always have a site navigation or sitemap to help you. 

An “everything is easy to find” user experience is central to the Apple website design.

The post When and how to use HTML sitemaps for SEO and UX appeared first on Search Engine Land.

Original source: https://searchengineland.com/html-sitemaps-seo-ux-when-how-394763

78 thoughts on “When and how to use HTML sitemaps for SEO and UX”

  8. Having read this I thought it was rather enlightening.
    I appreciate you taking the time and effort to put this article together.
    I once again find myself spending way too much time both
    reading and commenting. But so what, it was still worthwhile!

    Also visit my web page :: buy herpafend

  12. Thanks for a marvelous posting! I certainly enjoyed reading it, you will be
    a great author.I will make certain to bookmark your blog and may come
    back in the foreseeable future. I want to encourage continue your great writing,
    have a nice evening!

    Here is my page nerve fresh

  22. Have you ever considered about including a little bit more than just your articles?
    I mean, what you say is fundamental and everything.
    Nevertheless imagine if you added some great pictures or videos to give
    your posts more, “pop”! Your content is excellent but with pics and
    video clips, this blog could undeniably be one of the best in its field.
    Great blog!

    Feel free to visit my site: lottery defeated software

  25. Hello There. I found your blog using msn. This is an extremely
    well written article. I will make sure to bookmark it and return to read more of your useful information. Thanks for the post.
    I will certainly return.

    Stop by my web blog fitspresso

  31. I wanted to send you one tiny word to be able to give many thanks as before considering the spectacular ideas you’ve featured in this article. This is simply particularly generous with people like you in giving without restraint precisely what a few individuals would have marketed for an e book to help with making some cash on their own, primarily seeing that you might have tried it if you ever decided. The smart ideas additionally worked to become good way to be sure that some people have a similar desire much like my personal own to see whole lot more in regard to this problem. I think there are lots of more pleasurable times ahead for individuals that scan through your blog. prodentim reviews

  32. I needed to send you one tiny word just to give thanks yet again on the pleasant knowledge you have discussed here. It has been really surprisingly open-handed with you to provide extensively just what a number of people might have advertised for an e book to get some bucks for their own end, precisely considering the fact that you might well have done it in case you wanted. Those tricks also served to become a easy way to fully grasp that someone else have a similar desire much like mine to learn more and more around this problem. I am certain there are a lot more fun sessions in the future for many who see your blog. provadent reviews

  33. I intended to put you a little bit of note to help thank you over again regarding the remarkable guidelines you’ve provided here. It is certainly open-handed with you in giving extensively what exactly a few people would have marketed as an e book to make some bucks for their own end, primarily seeing that you could have done it in case you wanted. The tricks additionally served like a good way to understand that many people have similar interest just as my own to learn very much more in terms of this issue. I am sure there are several more enjoyable situations up front for individuals that see your blog post. tonic greens reviews

  34. I needed to send you a very small observation to finally give many thanks as before regarding the pleasing strategies you’ve shown here. It’s seriously open-handed of people like you to grant freely what exactly some people might have offered as an electronic book to help with making some dough on their own, precisely seeing that you might have done it if you ever wanted. These ideas additionally worked to be the easy way to understand that other people have the same dreams similar to mine to realize a good deal more regarding this matter. I think there are lots of more enjoyable opportunities up front for people who read carefully your site. tonic greens reviews

  35. I needed to compose you one little word to help say thank you once again considering the pretty basics you have provided on this website. It is extremely open-handed of you to make unhampered what most people might have marketed as an electronic book to earn some profit for their own end, most importantly given that you could have tried it in the event you decided. Those solutions likewise worked to be a fantastic way to be sure that other people have a similar interest like my personal own to find out somewhat more around this issue. I am certain there are some more pleasant instances up front for individuals that start reading your website. lottery defeated reviews

  36. I needed to put you a tiny remark just to say thanks a lot as before on the wonderful solutions you’ve documented on this website. This is generous of you giving easily precisely what many people might have advertised for an ebook to make some dough on their own, especially since you might have tried it if you ever wanted. Those tactics as well served as the fantastic way to understand that most people have similar interest just as my personal own to learn a whole lot more in respect of this matter. Certainly there are thousands of more fun opportunities up front for many who check out your blog post. boostaro reviews

  37. I intended to compose you this little bit of remark in order to thank you the moment again for your amazing ideas you’ve documented at this time. It’s certainly strangely open-handed of people like you to allow extensively exactly what a lot of people could possibly have offered for sale as an ebook to generate some cash on their own, certainly considering the fact that you might have tried it if you ever wanted. The basics likewise worked as the easy way to realize that someone else have a similar zeal just like my personal own to grasp good deal more with reference to this condition. I’m sure there are several more pleasant opportunities ahead for people who start reading your blog post. java burn

  38. I intended to create you one very small word so as to thank you once again for the remarkable solutions you have provided on this site. This has been really pretty open-handed of you in giving extensively exactly what many of us would have distributed as an e-book to generate some bucks for themselves, specifically since you might have done it in the event you desired. These pointers in addition served as a good way to comprehend most people have the same dream like my personal own to learn more on the subject of this problem. I am sure there are some more fun occasions ahead for individuals that go through your site. prostadine reviews

  58. Hi, just required you to know I he added your site to my Google bookmarks due to your layout. But seriously, I believe your internet site has 1 in the freshest theme I??ve came across.Seo Paketi Skype: By_uMuT@KRaLBenim.Com -_- live:by_umut

Comments are closed.