Thierry Muller 11:50 am on June 12, 2019
Tags: feature plugins ( 121 ), feature projects ( 51 ), proposal ( 38 ), SEO ( 6 ), sitemaps ( 15 ), xml-sitemaps ( 15 )

XML Sitemaps Feature Project Proposal

Note: a follow post was published with more recent information about this project.

While web crawlers usually discover pages from links within the site and from other sites, sitemaps supplement this approach by allowing crawlers to pick up all URLs included in the sitemap and learn about those URLs using the associated metadata.

Today, WordPress coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. does not generate XML Sitemaps by default, affecting a high number of WordPress websites search engine discoverability. 4 out of the top 15 plugins on WordPress pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party repository currently ship with their own implementation of XML sitemaps, pointing to a universal need for this feature and a great potential to join forces.

This post proposes integration of XML Sitemaps to WordPress Core as a feature project. The proposal was created as a collaboration between Yoast*, Google** and various contributors.

Proposed Solution

In a nutshell, the goal of the proposal is to integrate basic XML Sitemaps in WordPress Core and introduce an XML Sitemaps APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. to make it fully extendable. Below is a diagram of the proposed XML Sitemaps structure:

XML Sitemaps will be enabled by default making the following content types indexable

Homepage
Posts page
Core Post Types (Pages and Posts)
Custom Post Types
Core Taxonomies (Tags and Categories)
Custom Taxonomies
Users (Authors)

Additionally, the robots.txt file exposed by WordPress will reference the sitemap index.

Developers

An XML Sitemaps API will be introduced as part of the integration allowing extensibility. At a high level, below is a list of the ways the XML Sitemaps may be manipulated via the API:

Add extra sitemaps and sitemap entries
Add extra attributes to sitemap entries
Provide a custom XML Stylesheet
Exclude a specific post type from the sitemap
Exclude a specific post from the sitemap
Exclude a specific taxonomyTaxonomy A taxonomy is a way to group things together. In WordPress, some common taxonomies are category, link, tag, or post format. https://codex.wordpress.org/Taxonomies#Default_Taxonomies. from the sitemap
Exclude a specific term from the sitemap
Exclude a specific author from the sitemap
Exclude a specific authors with a specific role from the sitemap

Non Goals

While the initial XML Sitemaps integration will fulfill search engines minimum requirements and cover most WordPress content types, below is a list of features which will not be included in the initial integration:

Image sitemaps
Video sitemaps
News sitemaps
User-facing changes like UIUI User interface controls to exclude individual posts or pages from the sitemap
XML Sitemaps caching mechanisms

i18ni18n Internationalization, or the act of writing and preparing code to be fully translatable into other languages. Also see localization. Often written with a lowercase i so it is not confused with a lowercase L or the numeral 1. Often an acquired skill.

The XML Sitemaps will leverage standard internationalization functionality provided by WordPress core.

Since there are plans by WordPress leadership to officially support multilingual websites in WordPress, the XML Sitemaps will be flexible enough to list localized content in the future as per web development best practices.

What’s next?

Your thoughts on this proposal would be greatly valued. Please share your feedback, questions or interest in collaboration by commenting on this post. After that we can decide on how to best proceed with this proposed project and set up a meeting on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. to kick things off.

* @joostdevalk, @omarreiss, @jonoalderson, @herregroen

** @swissspidy @albertomedina @westonruter @flixos90 @tweetythierry

#feature-plugins, #feature-projects, #proposal, #seo, #sitemaps, #xml-sitemaps

Dhanendran 12:12 pm on June 12, 2019

Thank you @tweetythierry for this proposal. This is one of the most needed feature in WordPress. As many people who uses WordPress doesn’t know much about sitemaps and struggling to get their content indexed in search engines like Google. This will be very useful in WordPress. I’m more interested in collaborating on this project.
- leodandesign 12:22 pm on June 12, 2019
  
  This is a good idea – I think it needs to be done in collaboration with all the major SEO plugins. You need to also build in the mechanism to “no-index” certain posts, or even post / taxonomyTaxonomy A taxonomy is a way to group things together. In WordPress, some common taxonomies are category, link, tag, or post format. https://codex.wordpress.org/Taxonomies#Default_Taxonomies. types. I think that there also needs to be the option to disable the sitemap functionality if you have an alternative solution. The no-indexing feature is important, as you are potentially exposing pages that users have kept hidden from the search engine results page. I am very much for this feature becoming standard, however, sitemaps are a really big deal for sites that rely on Google for Traffic – so needs to be properly implemented. You may also find that some of the SEO pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party providers may not be willing to collaborate as you are potentially taking away their sole purpose of the plugin (or at least a major feature) – which could impact the number of installs through the repo that they have and therefore the number of potential leads for their pro products or services. It would be great if this could happen. It would be a great and market disrupting feature.
  - Pascal Birchler 12:32 pm on June 12, 2019
    
    @leodandesign you raise some valid points!
    
    So far we’ve received good feedback from Yoast and Jetpack in private discussions. I’m sure other pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party vendors will express interest in this as well.
    
    As for the noindex feature you mention, the post already says that “UIUI User interface controls to exclude individual posts or pages from the sitemap” is not part of this proposal.
    
    At its coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., this proposed sitemap solution would contain the minimum functionality needed to have a good, functioning XML sitemap in WordPress. SEO plugins can then extend this functionality to add UI controls, disable certain things, etc.
    
    If anything, this proposal makes it easier for all these plugins to offer SEO solutions for WordPress, as there would be a unified, standardized solution in WordPress core that serves as a foundation for whatever features they might build on top of it.
    - leodandesign 12:40 pm on June 12, 2019
      
      I saw the part about excluding after re-reading the proposal.
      
      I really hope that this happens.
      
      The multilingual aspect of this sounds very interesting indeed.
Amaury Balmer 12:16 pm on June 12, 2019
It’s a great initiative. @tweetythierry

Have you integrated the performance aspect into your project ?
On sites with a high volume, current solutions are not perfect.

Some articles on the subject:
- Thierry Muller 12:50 pm on June 12, 2019
  
  Yes we have 🙂
  
  Take a look at the architectural diagram in the post, the root index sitemap will link to all the “sub-sitemaps” which will be paginated. This should address performance and scalability concerns.
  - Matthew Boynes 8:15 pm on June 12, 2019
    
    Performance should not be taken lightly. As simple and straightforward as XML sitemaps are, they present some relatively significant performance challenges at scale.
    
    As one for-instance, how many urls are going to be in each paginated (sub-) sitemap? A sitemap index file is limited to 50,000 sitemaps, so even though each sitemap is limited to a maximum of 50,000 urls, generating 50,000 urls in one page request would be extremely difficult and non-performant to do on-the-fly. If each paginated sitemap were limited to 10 posts (the default `posts_per_page` option value), that would mean a total capacity of only 500,000 pieces of content across all terms and posts. Even 100 per paginated sitemap would yield a relatively low upper limit of 5,000,000 pieces of content.
    
    I would like to get involved to help work through and overcome some of these challenges. What is the best avenue to do so, @tweetythierry?
    - Pascal Birchler 9:22 pm on June 12, 2019
      
      Raising very valid points like this and expressing your interest here is a great first step! 🙂
      
      I expect there to be some more comments flowing in on this post throughout the week. After that we can collect all the feedback so far and decide on the next steps, which likely means setting up a meeting on SlackSlack Slack is a Collaborative Group Chat Platform https://slack.com/. The WordPress community has its own Slack Channel at https://make.wordpress.org/chat/. to kick things off.
- Matthew Boynes 1:16 pm on June 13, 2019
  
  On sites with a high volume, current solutions are not perfect.
  
  @momo360modena if you haven’t already, check out https://github.com/Automattic/msm-sitemap. It is the best solution I’ve seen for sites with large volume; I’ve used it on sites with millions of posts without issue.
  - Amaury Balmer 1:46 pm on June 13, 2019
    
    Very interesting, I did not know! thank you 🙂
Ion Burdianov 12:33 pm on June 12, 2019

Oh, this is a super idea guys, I have never disregarded side plugins to create XML sitemap. I have been waiting for this since 2008. Like it.
rickcurran 12:37 pm on June 12, 2019

Hi,
This sounds like a great feature to have in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. to avoid having to install a potentially bulkier pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party just for that feature.

One issue though, in the ‘Non-Goals’ section above it says:

“User-facing changes like UIUI User interface controls to exclude individual posts or pages from the sitemap”

Does this mean that there won’t be any method to exclude Published posts (post / page / CPTs) from appearing in the sitemap file? This would be an essential requirement from my perspective as it’s pretty common to have content that is Published but is not intended to be listed on the site. E.g. it doesn’t appear in Navigation menus or any other list of pages such as regular user-facing Site map pages – I commonly use ‘wp_list_pages’ on 404 page templates to show a user-facing Site map using the ‘exclude’ parameter to exclude any post ID’s that I don’t want to be listed.

I don’t see there having to be a provide UI to exclude posts / pages from this, but a developer function so that they can be excluded from the sitemap XML file similar to ‘wp_list_pages’ would be enough. I would expect third-party XML sitemap plugins would also want this to be able to make use of the core feature but bring the added-value functions that their plugins provide.
- Thierry Muller 12:44 pm on June 12, 2019
  
  Thanks for you feedback.
  
  The goal of the proposal is to integrate basic XML Sitemaps in WordPress CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. and introduce an XML Sitemaps APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. to make it fully extendable. Take a look at the Developer section of the post, the XML Sitemaps AP will allow devs to granularly exclude urls.
- rickcurran 12:52 pm on June 12, 2019
  
  Thanks Thierry,
  Ack, sorry, I did read the whole article, honest! 🙂 I’m not sure how I missed that section though as it pretty specifically answers my question!
woodsandhillsplc 12:38 pm on June 12, 2019

Great news, though the risk here for me is that you have way more sitemaps than necessary for SEO – and turning them all on by default could lead Google up the proverbial garden path – wasting a site’s crawl budget.
- Pascal Birchler 12:43 pm on June 12, 2019
  
  Good point. Having a sitemap and actually submitting it to a search engine’s webmaster tools / search console are two different things. But I am sure any coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. solution and also any pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party adding sitemaps to WordPress can find a way to prevent any duplication in this regard 🙂
  - woodsandhillsplc 12:45 pm on June 12, 2019
    
    > Having a sitemap and actually submitting it to a search engine’s webmaster tools / search console are two different things.
    Google/other search engines crawl the web – if they come across a sitemap index (especially if it is linked from robots.txt) they will index it, crawl it, and crawl all the sitemaps listed – unless they are physically blocked from doing so.
Mukesh Panchal 12:44 pm on June 12, 2019

@tweetythierry thanks for sharing great proposal.

Such a great news that WordPress plan to added sitemap in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. like Site Health module. Also it was changes based on user need using actions/filters with other options so that user don’t need any extra pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party to generate there website sitemap.

Add an option to remove default sitemap so if needed user can use other plugin sitemap.

If any dev needed then i’m interested in collaborating on this project.
- Thierry Muller 12:58 pm on June 12, 2019
  
  Add an option to remove default sitemap so if needed user can use other pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party sitemap.
  
  Thanks for raising this. The goal is to standardize XML Sitemaps in CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. and provide an APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. for plugins to extend it rather than replacing it.
  - Mukesh Panchal 1:01 pm on June 12, 2019
    
    In that case all pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party developers need to changes there current functionality to generate sitemap and also needs to check backward comparability for old WP website
Tran Ngoc Tuan Anh 2:01 pm on June 12, 2019

This is a great idea and I love that. I’m developing a SEO pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party which has a very similar approach for the sitemap. I’m willing to help if possible.
Daniel Roch 2:56 pm on June 12, 2019
The idea looks good on paper, but I think it has many flaws:
- Users do not know what a “sitemap” is
- If it is not configurable without an extension, it will add many unwanted entries (especially in SEO). It would therefore be a coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. functionality that cannot be used without a dedicated pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party
- If the feature does not include automatic submission to Google’s Search Console, it has absolutely no use
- Pascal Birchler 6:36 pm on June 12, 2019
  
  Users do not know what a “sitemap” is
  
  Ideally, they don’t even have to know! As the post says, it should just work. No user-facing changes are planned.
  
  If it is not configurable without an extension, it will add many unwanted entries
  
  The proposal so far is to only add things to the sitemap that are publicly viewable anyway.
  
  If the feature does not include automatic submission to Google’s Search Console, it has absolutely no use
  
  As far as I know, automatic submission is not something smaller sites need to be concerned about. And as long as the sitemap is referenced in in the `robots.txt`, crawlers will pick it up.
  - Daniel Roch 7:36 am on June 13, 2019
    
    Users do not know what a “sitemap” is
    
    Ideally, they don’t even have to know! As the post says, it should just work. No user-facing changes are planned.
    
    You’re right.
    
    If it is not configurable without an extension, it will add many unwanted entries
    
    The proposal so far is to only add things to the sitemap that are publicly viewable anyway.
    
    I’m an SEO consultant. Unfortunately, many WordPress developers make mistakes when creating a Custom Post TypeCustom Post Type WordPress can hold and display many different types of content. A single item of such a content is generally called a post, although post is also a specific post type. Custom Post Types gives your site the ability to have templated posts, to simplify the concept. or Custom TaxonomyTaxonomy A taxonomy is a way to group things together. In WordPress, some common taxonomies are category, link, tag, or post format. https://codex.wordpress.org/Taxonomies#Default_Taxonomies.: they leave it “public”, which lead to SEO issues.
    
    The same is true for some content that is supposed to be private, such as a thank you page or WooCommerce’s private pages (“My account” for example).
    
    A sitemap without configuration will therefore only cause SEO issues. It will do more harm than good.
    
    If the feature does not include automatic submission to Google’s Search Console, it has absolutely no use
    
    As far as I know, automatic submission is not something smaller sites need to be concerned about. And as long as the sitemap is referenced in in the `robots.txt`, crawlers will pick it up.
    
    Do not trust Google on that. Sometimes, he does check your sitemap when you add it to the robots.txt file, but most of the time this is not the case (It can be easily seen by looking at the log files). The only effective method is manual submission.
    - Aurélien Denis 9:11 am on June 28, 2019
      
      Totally agree with Daniel!
rmcc4444 4:08 pm on June 12, 2019

This has been needed for quite some time. Thanks
keellye 4:19 pm on June 12, 2019

1. Yandex does not support images inside a sitemap.
It is necessary to make an option to exclude images from the sitemap.

2. Why sitemap will be enabled by default?
Millions of sites already have sitemaps. Are you sure that this will not lead to conflicts and disruption of sites and indexing errors by search engines?
- Pascal Birchler 6:39 pm on June 12, 2019
  
  Implementing image sitemaps is currently a non-goal, meaning there’s no intent to implement these. Plugins would be responsible for doing that.
  
  While millions of WordPress sites might already have a pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party installed that adds XML sitemaps, it’s definitely not the majority of sites. Remember, WordPress powers an estimated third of the web.
  - keellye 8:02 pm on June 12, 2019
    
    1. Оkay, but please make a note and if you decide in the future to add images to the site map, keep this in mind. Thank you in advance.
    
    2. Perhaps it is necessary to detect already installed plugins (All in one Seo Pack, Yoast Seo etc.) and/or the presence of sitemaps before “enable sitemap by default” and enable or disable this feature to avoid conflicts?
    It seems to me that the absence of sitemaps on a small site/blogblog (versus network, site) does not prevent its indexing by search engines – sites of less than several thousand pages are successfully indexed (all large sites/blogs are aware of sitemaps).
Arne Brachhold 4:23 pm on June 12, 2019

Hi,

Great idea! XML Sitemaps have really evolved to a standard every website should have, therefore it makes sense to include a basic functionality in the coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress..

The proposal looks very good to me. All published content should be included by default, because that is probably what the majority of users would expect. All functionality regarding filters based on posts, categories and so on should be pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party territory, since users probably would also like to have “noindex” tags on these pages. While the proposal already lists a list of possible exclusion criteria, I would also add a generic filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. to let plugins filter based on custom rules or post metaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress. data.

Priorities for individual entries can also be provide by a plugin using the additional attributes (=child elements?) mentioned in the proposal.

The structure looks fine for me too. One thing to keep in mind is that if you have a website with a million posts and delete or unpublish one, not all the paginated sitemaps should change. This would happen if you just paginate them by ID and a LIMIT. The root sitemap should also contain the correct modification date of the sub-sitemaps so they do not need to be re-indexed if nothing has changed.

Caching can be done by the popular caching plugins, so no need to include this in the core.

If the root sitemap is included in the robots.txt, Google will find it automatically. I am not sure if the “pingPing The act of sending a very small amount of data to an end point. Ping is used in computer science to illicit a response from a target server to test it’s connection. Ping is also a term used by Slack users to @ someone or send them a direct message (DM). Users might say something along the lines of “Ping me when the meeting starts.”” functionality to notify Google about changes, which is provided by most Sitemap plugins, can also be implemented via the existing “Update Services” feature in WP. (If this feature will still survive the next WP releases, since the days of Technorati and Feedburner are gone for a long time).

Regards,
Arne
cihaneken 5:19 pm on June 12, 2019

This is a good idea
Fernando Tellado 5:23 pm on June 12, 2019

Great idea @tweetythierry!!

All other major CMS have XML Sitemaps features in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. (i.e. Magento, Prestashop, etc) and it’s a long time demanded functionality for WordPress in every poll made in the past to users asking for future features.

I think that cons are widely overcomed by pros.

Let’s go for it!
robwent 5:53 pm on June 12, 2019

I don’t think this should be added without a way for users to configure what is in the sitemap.
Custom post types are used by plugins for loads of different things that shouldn’t be added to a sitemap.
For example, contact form 7 stores submissions as a custom post typeCustom Post Type WordPress can hold and display many different types of content. A single item of such a content is generally called a post, although post is also a specific post type. Custom Post Types gives your site the ability to have templated posts, to simplify the concept.. Slideshow plugins store configurations as custom post types.
Even when building a site with your own Costs and taxonomies you might not want them all in the sitemap.

Seems like it should be off by default rather than enabling something which could potentially cause problems for a lot of non technical users who then have to add code to disable it.
- Pascal Birchler 6:45 pm on June 12, 2019
  
  How useful is a feature that is off by default? 😉
  
  You have some good points about custom post types though. Ideally such a feature would only add posts and terms to the sitemap that are publicly viewable already. So internal post types or taxonomies would not be considered.
  
  As Matt says, let’s build a v1 of this and then iterate from there!
  - robwent 7:12 pm on June 12, 2019
    
    Yeah, my first thought was just enable posts, pages and their related taxonomies by default but then there needs to be a way to enable others.
    The public setting for the post type is a good indicator but a dedicated setting might also be useful. I seem to remember a case where posts had to be public even if you don’t want the single pages to be accessible. Possibly when you just want to show a collection of items on an archive page or page template.
    
    I would disagree that new features have to be enabled automatically to be useful. Enabling on a new installation maybe, but turning on a new feature which the site adminadmin (and super admin) may be unaware of and could conflictconflict A conflict occurs when a patch changes code that was modified after the patch was created. These patches are considered stale, and will require a refresh of the changes before it can be applied, or the conflicts will need to be resolved. with existing plugins and add unwanted content to a sitemap may be more unhelpful than helpful.
    That said, most people with a sitemap are probably using something like Yoast anyway so it should just be a case of updating to a version that uses the APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways..
    Yoast also turns everything on by default and most sites I look at have things in the sitemap that shouldn’t be there.
    
    Maybe it’s now time to also look at having coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. metadata 😀
  - woodsandhillsplc 4:55 pm on June 13, 2019
    
    Off by default is the best option, then allowing users to turn on individual sitemaps. Otherwise you are going to send Google towards trillions of URLs that Google doesn’t need to crawl and waste site crawl resources.
    - igrigorik 4:46 am on June 19, 2019
      
      Our crawlers are smart (enough) to prioritize crawling effectively. What’s most important and useful is explicit signals about URLs we should consider, which is exactly what sitemaps enable.
Matt Mullenweg 6:10 pm on June 12, 2019

This makes a lot of sense, looking forward to seeing the v1 of this in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. and for it to evolve in future releases and cement WP’s well-deserved reputation of being the best CMS for SEO.
- shawfactor 3:03 pm on July 1, 2019
  
  Why is this a good idea. Developers like me, look at WordPress as a framework. A framework that already supports a huge number of features (including sitemaps). via plugins.
  
  Making this part of coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. seems unnecessary, anti choice, creates bloat, and most importantly diverts attention from many of the APIs the WordPress framework is missing.
Jon Brown 6:47 pm on June 12, 2019

Love seeing features like this added to coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress..

I think yoast seo handles this brilliantly and is a great model to adapt.

The one (and only) problem we have with yoast seo’s sitemaps is on some of our very large sites (100,000+ pages) it can literally take down our server so we throttle it, cache it and run on a background cron.

I think not including a UIUI User interface to disallow on specific pages/categories/tags is a poor choice. This is a very important feature for us. It creates a lot of potential for pages appearing in sitemaps which shouldn’t be index or are set to noindex by any of the many other methods to do so.

I would also point to Google XML sitemaps brilliant feature to add items to the virtual robots.txt. Perhaps outside the minimal scope of this, but another feature that should be in core to properly manage these virtual files core creates.
Josepha 6:50 pm on June 12, 2019

Let’s do it!
patricedefago 10:03 pm on June 12, 2019
Hello,

For the sitemap to be effective for SEO and valid, it must contain:
- Only urls with HTTPHTTP HTTP is an acronym for Hyper Text Transfer Protocol. HTTP is the underlying protocol used by the World Wide Web and this protocol defines how messages are formatted and transmitted, and what actions Web servers and browsers should take in response to various commands. response 200
- Only canonized urls
Does not contain
- Urls 3xx, 4xx and 5xx
- Urls containing canonicals pointing to another url
- Urls in noindex
https://support.google.com/webmasters/answer/183668?hl=fr&ref_topic=4581190
- patricedefago 6:17 am on June 13, 2019
  
  Hello,
  
  It is also important to be able to choose whether or not to leave content types or urls.
  
  The possibility of integrating the hreflang is also to be taken into consideration.
Mitch Menghi 1:17 am on June 13, 2019

Great to see this coming, fantastic news…. this should be in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. for sure, but I do agree that by default maybe it should be off, and in general settings its a simple switch on that then enabled this.

If it is to be on by default, then I would like to see it on by default only for the most important post types leg PAGES and POSTS as default, It should not be on for CPT, like some said some plugins create their own mess that doesnt even need to be crawled like some sliders for example.

My only request here is that we should be able to easily turn on and off features of what is to be in the index file and what is not to be in the index file, if not I can see most people just turning it off and going back to a third party pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party that does what they want with XML sitemaps.

On a sidenote, why only extend this to XML sitemaps, how about HTMLHTML HyperText Markup Language. The semantic scripting language primarily used for outputting content in web browsers. sitemaps also, they both work in conjunction with each other and pretty much and SEO who does one ends up doing the other…. 😀
Breno Alves 3:11 am on June 13, 2019

I’m very interested to help in this project. I’ve been working with an APIAPI An API or Application Programming Interface is a software intermediary that allows programs to interact with each other and share data in limited, clearly defined ways. integration that generates thousands of pages in my WordPress apps that also need to be present in sitemaps.

I have implemented custom sitemaps using Yoast since, and had faced the same performance issues that were told above.

If this feature were supported natively we could probably find a better solution for those issues.
Michael Torbert 1:27 pm on June 13, 2019

This is awesome. Standardizing basic XML sitemaps in WP coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. is a great move for the future.
- Michael Torbert 1:30 pm on June 13, 2019
  
  As far as on or off by default, I think it wouldn’t make sense for something like this to be included in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. and then off by default.
  
  Having said that, there could be an exclude_from_sitemap or include_in_sitemap for CPT registrations, and maybe even in for custom taxonomyTaxonomy A taxonomy is a way to group things together. In WordPress, some common taxonomies are category, link, tag, or post format. https://codex.wordpress.org/Taxonomies#Default_Taxonomies. registration as well for the taxonomy archives.
EBrockway 2:07 pm on June 13, 2019

Anyone is seing like me how this could easily lead to lot of WordPress installations being hackedhacked ?

If you publish by default the usernames in a sitemap, you have the username of the admins, you just have to brute force the password!
Also on e-commerce websites it could lead up to hack of clients accounts…

Please atleast provide a way to active/deactivate certains sitemaps! And for security purposes leave user sitemap deactivated by default.
- Soren Wrede 2:48 pm on June 13, 2019
  
  This is not a new problem, the REST APIREST API The REST API is an acronym for the RESTful Application Program Interface (API) that uses HTTP requests to GET, PUT, POST and DELETE data. It is how the front end of an application (think “phone app” or “website”) can communicate with the data store (think “database” or “file system”) https://developer.wordpress.org/rest-api/. does the same. If you want to know more about this topic in WordPress, you can read this article: https://wptavern.com/why-showing-the-wordpress-username-is-not-a-security-risk
- RwkY 10:49 am on June 14, 2019
  
  as @soean said, this is not new, usually hosting providers have protections designed to throttle (hardware/software) for this kind of attacks.
RwkY 10:47 am on June 14, 2019

This is a long-waited feature for me, but without the control of what gets generated and added in the sitemap (eg. cpt / users), it could be nice but too much for some projects + maybe a security risk (considering the users get pushed in) as @ebrockway sayd.
modus 6:21 pm on June 17, 2019
While a sitemap is a must have, I have the following thoughts to add:
- Different use cases make use of WP in completely different ways. It would therefor be essential to many projects to be able to configure what the index includes. Users need control over what shows up in search engines.
- As the original proposal explicitly excludes the idea of such an interface, I just cannot keep from wondering whether this proposal is a “clever” marketing stunt for premium plug-ins, that serve the sole purpose of selling the user a level of control he already had before it got taken away from him.
Jon Brown 3:12 pm on June 18, 2019

At this point roughly half the comments above mention needing the ability to control what is included in the site map.

Several commenters have outlined clearly why including “all public post types” is insufficient. Many plugins set CPTs public for various reasons which still shouldn’t be included in a site maps and indexed. There must be a way to manually enable/disable inclusion in coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., without an additional pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party.

Will this concern be addressed directly in this proposal?

I really want to see this in core, but without addressing this the proposal is fatally flawed.
- Aurélien Denis 9:23 am on June 28, 2019
  
  You’re right!
Aaron Jorbin 3:53 pm on June 18, 2019

I love this idea and completely agree that there is no need for a UIUI User interface for most users. This should just be a hidden benefit of using WordPress.

I do think that when a post type or taxonomyTaxonomy A taxonomy is a way to group things together. In WordPress, some common taxonomies are category, link, tag, or post format. https://codex.wordpress.org/Taxonomies#Default_Taxonomies. is registered it should specify if it should be sitemapped or not.
- Jb Audras 8:11 pm on June 18, 2019
  
  Totally agree.
  By default I think any public post type/taxonomyTaxonomy A taxonomy is a way to group things together. In WordPress, some common taxonomies are category, link, tag, or post format. https://codex.wordpress.org/Taxonomies#Default_Taxonomies. should be listed in the sitemap. Then, we could add a parameter in the register functions to remove it from the sitemap 🙂
  - Aurélien Denis 9:24 am on June 28, 2019
    
    Public doesn’t mean you want it in sitemap, I’ve seen a lot of misconfiguration in themes or plugins. :/
Aurélien Denis 9:22 am on June 28, 2019

So now Google is determining how WordPress should be and they’re doing it only to reduce theirs costs. I was thinking that WordPress is for user, not bots.

As well explained by Daniel here : https://make.wordpress.org/core/2019/06/12/xml-sitemaps-feature-project-proposal/#comment-35998 it will cause more problem than it solves…
Benjamin Denis 10:09 am on June 28, 2019

This is the territory of the plugins, not the CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. to take care of that…
This will generate a lot of support for existing SEO plugins (like mine), but also errors in Google Search Console (because many sites / themes are poorly designed).
In the end, frustration and misunderstanding for everyone.
34% of the web risk overnight to see its SEO impacted.
Google shouldn’t be bothering with that.
The formatting of the web is definitively launched.
- DaveZ 10:25 am on July 1, 2019
  
  Index posts and pages only as the unique content (Goog hates dupes). Leave authors, tags, images etc to plugins and those making an educated choice.
shawfactor 2:57 pm on July 1, 2019

This is worse than a waste of time. It will bloat coreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. further and divert resources from needed features and APIs.
Please please stop with this dumb idea
Grzegorz.Janoszka 5:05 pm on July 1, 2019

Great proposal. I think users shouldn’t have too many options in the GUI, just maybe to turn it on/off. However filters and actions should allow to do anything a developer may wish. Please do it.
ghostqix 6:13 pm on July 1, 2019

I believe I read in one of the proposal/comments that image sitemaps wouldn’t be included in the initial implementation. If you do, please add logic that automatically excludes image pages that are only used on pages or posts that are excluded. I ran into this anomaly with one of the current SEO plugins and it was the only crawl exception on the Google Search Console. This is hardly fatal, but irritating for OCD types like me.

Food for thought.
Alain Schlesser 3:12 am on July 9, 2019

In the above proposal you state that caching mechanisms will not be part of this initial integration.

I think this will probably make it a non-viable solution right from the start for inclusion into CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress., as the main issue with sitemaps is not their basic generation (which is very straight-forward), but their scalability.

Doing only uncached requests will just not be an option on sites that are large, complex, high traffic or a combination of these. And as soon as you try to add caching, you’ll face the actual real issue with sitemaps: how to know what to update when (in paginated requests) without going through the entire dataset again.

There are solutions to this, of course, but scalability absolutely should be one of the design requirements.
- Aurélien Denis 6:13 am on July 9, 2019
  
  Good point, that’s why it should remains pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party territory.