Meta robots tags and X-Robots-Tags give website owners a way to tell search engine crawlers how they should handle specific pages and files. When used correctly, these directives can help control which content gets indexed, which links crawlers should follow, and what information may appear in search results.

Understanding these instructions is an important part of technical SEO because not every page on a website needs to appear in search engines. Some pages may be useful to visitors but unnecessary in search results, while others may contain resources that should be handled differently from ordinary HTML pages.

What Is a Meta Robots Tag?

A meta robots tag is an HTML element that provides search engine crawlers with instructions about how they should crawl, index, and display a particular page.

The tag is placed inside the <head> section of an HTML document. A simple example is:

<meta name=”robots” content=”noindex”>

This directive tells compliant search engine crawlers not to include that page in their search index.

Meta robots tags can do much more than prevent indexing. Depending on the directives you use, they can control whether search engines follow links, display snippets, show cached versions, index images, or generate certain search-result features.

Because these instructions can directly influence a page’s visibility in search results, they should be implemented carefully.

Meta Robots vs. Robots.txt

Meta robots tags and robots.txt files both provide instructions to search engine crawlers, but they operate at different levels.

• A robots.txt file is placed at the website level and provides crawling instructions for URLs or sections of a site.
• A meta robots tag is placed within an individual HTML page and provides instructions specifically for that page.

The distinction is important.

A robots.txt rule can prevent a crawler from accessing a URL, while a meta robots directive tells a crawler what to do with a page after it has accessed the page.

For example, if you want a page to remain accessible to visitors but do not want it included in search results, a noindex meta robots directive may be appropriate.

If you block the same URL through robots.txt, however, a crawler may not be able to access the page and therefore may never see the noindex instruction.

What Are Robots Meta Tags Used For?

Robots meta tags allow you to influence how search engines handle individual pages and their content.

Depending on the directive, you can tell search engines whether to:

• Include a page in search results
• Follow links found on a page
• Index images associated with a page
• Display cached versions of a page
• Show text snippets in search results
• Show video previews
• Control the size of image previews
• Offer translated versions of a page
• Display certain search-result features

These controls are particularly useful on websites containing large numbers of pages, temporary content, internal functionality, or resources that are not intended to receive organic search traffic.

How Do Robots Meta Tags Affect SEO?

Robots meta tags can help search engines understand which pages should and should not be included in their indexes.

This becomes especially useful on large websites where thousands of URLs may exist but only a portion of them are intended to appear in organic search results.

Examples of pages that may not need to be indexed include:

• Development or staging pages
• Thank-you and confirmation pages
• Login and account pages
• Internal search-result pages
• Shopping cart or checkout pages
• Certain duplicate or near-duplicate pages
• Temporary campaign pages
• Internal utility pages

Using indexing directives appropriately can help keep unwanted URLs out of search results and make a site’s search presence more focused.

Robots directives should also be considered alongside other technical SEO elements, including XML sitemaps, canonical tags, and robots.txt. Each serves a different purpose, and using them together correctly can help avoid crawling and indexing problems.

What Are the Name and Content Attributes in Meta Robots Tags?

A standard meta robots tag contains two important attributes:

• name
• content

Each serves a different purpose.

Name Attribute

The name attribute identifies the crawler or group of crawlers to which the instruction applies.

For example:

<meta name=”robots” content=”noindex”>

Using:

name=”robots”

means the directive is intended for all crawlers that support the instruction.

The value is not case-sensitive, so these are effectively equivalent:

name=”robots”

name=”ROBOTS”

name=”Robots”

You can also target particular crawlers by using their recognized crawler names. For example, a directive can be written specifically for Googlebot or another search engine’s crawler.

Keep in mind that crawler directives are not security mechanisms. A bot that does not follow these instructions can ignore them, so sensitive information should never be protected merely by adding a robots meta tag.

Content Attribute

The content attribute contains the actual instructions you want the crawler to follow.

For example:

<meta name=”robots” content=”noindex”>

Here, noindex is the instruction.

Multiple directives can be combined by separating them with commas:

<meta name=”robots” content=”noindex, nofollow”>

The content values are generally not case-sensitive.

Below are some of the most useful robots directives.

Default Behavior

If no robots meta tag is present, search engines generally assume that they are allowed to index the page and follow its links.

This default behavior is sometimes represented by:

<meta name=”robots” content=”all”>

You generally do not need to add all because indexing and link following are normally permitted unless another directive says otherwise.

Noindex

Example:

<meta name=”robots” content=”noindex”>

noindex tells compliant search engines not to include the page in their search index.

This can be useful for pages that should remain accessible through direct navigation but should not appear as organic search results.

Examples include certain account pages, checkout pages, internal search pages, and temporary pages.

It is important to remember that noindex is an indexing instruction, not a security feature. The URL can still potentially be accessed by people who know the address.

Nofollow

Example:

<meta name=”robots” content=”nofollow”>

nofollow tells crawlers not to follow the links found on that page.

This can be useful in situations where you do not want to provide crawling instructions through links on a particular page or where the page contains links you do not control.

The directive applies to links on the page rather than preventing users from clicking them.

Noarchive

Example:

<meta name=”robots” content=”noarchive”>

noarchive prevents a search engine from displaying a cached version of the page in search results where that feature is supported.

This may be useful for information that changes frequently or for pages where showing an older stored version would be undesirable.

Noimageindex

Example:

<meta name=”robots” content=”noimageindex”>

This directive tells Google not to index images associated with the page.

The images can still appear normally to visitors on the page itself. Therefore, you should use this directive carefully if image search traffic is important to your website.

Notranslate

Example:

<meta name=”robots” content=”notranslate”>

notranslate tells Google not to offer a translated version of the page in search results.

This may be useful when translations could create confusion around product names, technical terminology, branding, or other important language-specific information.

Nositelinkssearchbox

Example:

<meta name=”robots” content=”nositelinkssearchbox”>

This directive prevents Google from displaying a search box for the site in certain search-result contexts.

It affects the search-result presentation rather than preventing the page from being indexed.

Nosnippet

Example:

<meta name=”robots” content=”nosnippet”>

nosnippet tells search engines not to display a text or video snippet for the page.

It can also prevent the content from being used as a direct input for certain AI-generated search features.

One consequence is that the page’s meta description and certain rich-result information may no longer be displayed in the usual way.

If you only want to prevent a particular section of a page from appearing in snippets, you can use the data-nosnippet HTML attribute instead.

For example:

<p>This part of the content can appear in search results.</p>

<span data-nosnippet>

This section should not be displayed in snippets.

</span>

This approach gives you more precise control than blocking snippets for the entire page.

Max-snippet

Example:

<meta name=”robots” content=”max-snippet:100″>

max-snippet specifies the maximum number of characters that a search engine should use when creating a text snippet for the page.

For example:

<meta name=”robots” content=”max-snippet:100″>

requests a maximum text snippet length of 100 characters.

Two useful values are:

• 0 — no text snippet
• -1 — no specified limit

The actual search-result presentation remains under the search engine’s control.

Max-image-preview

Example:

<meta name=”robots” content=”max-image-preview:large”>

This directive controls the maximum size of an image preview that a search engine may display.

The supported values are:

• none
• standard
• large

Using large permits a larger image preview where supported.

This can matter for websites where visual search-result presentation is important, particularly publishers, travel sites, recipe websites, ecommerce sites, and other image-heavy websites.

Max-video-preview

Example:

<meta name=”robots” content=”max-video-preview:10″>

max-video-preview controls the maximum duration, in seconds, of a video preview that a search engine may display.

For example:

<meta name=”robots” content=”max-video-preview:10″>

requests a maximum preview duration of 10 seconds.

The values include:

• 0 — no video preview
• -1 — no specified limit
• A positive number — maximum preview duration in seconds

Indexifembedded

Example:

<meta name=”robots” content=”noindex, indexifembedded”>

indexifembedded allows Google to index content when it is embedded inside another page, even though the original page itself has a noindex directive.

This can be useful for media publishers and websites that host standalone resources that should not appear as individual search results but may still need to be discoverable when embedded elsewhere.

This directive is not supported by every search engine, so its behavior can vary.

Unavailable_after

Example:

<meta name=”robots” content=”unavailable_after: 2026-10-21″>

unavailable_after tells Google that a page should no longer be shown in search results after a specified date or time.

The date can be provided using supported formats such as RFC 822, RFC 850, or ISO 8601.

It effectively provides a scheduled search-visibility restriction.

This can be useful for temporary announcements, time-sensitive offers, event pages, or other content that should stop appearing in search after a defined deadline.

Combining Robots Meta Tag Rules

You can combine multiple robots instructions when a page requires more than one restriction.

There are two common approaches.

Multiple Directives in One Content Attribute

For example:

<meta name=”robots” content=”noindex, nofollow”>

This tells crawlers not to index the page and not to follow the links on it.

You may also encounter:

<meta name=”robots” content=”none”>

where supported, which is intended to represent a combination of noindex and nofollow.

However, support for individual directives can differ between search engines.

If multiple instructions conflict, search engines generally follow their own rules for resolving the conflict. Google applies the more restrictive instruction when conflicting robots directives are encountered.

Multiple Robots Meta Elements

You can also use separate meta elements for different crawlers.

For example:

<meta name=”robots” content=”nofollow”>

<meta name=”YandexBot” content=”noindex”>

In this example, the first instruction applies broadly and tells crawlers not to follow links, while the second adds a noindex instruction specifically for YandexBot.

This approach can be useful when you need different behavior from different search engine crawlers.

Search Engine Support for Meta Robots Directives

Not every search engine supports every robots directive.

A simplified overview looks like this:

Directive	Google	Bing	Yandex
noindex	Yes	Yes	Yes
noimageindex	Yes	No	No
nofollow	Yes	No	Yes
noarchive	Yes	Yes	Yes
nocache	No	Yes	No
nosnippet	Yes	Yes	No
nositelinkssearchbox	Yes	No	No
notranslate	Yes	No	No
max-snippet	Yes	Yes	No
max-video-preview	Yes	Yes	No
max-image-preview	Yes	Yes	No
indexifembedded	Yes	No	No
unavailable_after	Yes	No	No

Because crawler support can change over time, check the documentation of the search engine you care about before relying on a less common directive.

How to Implement Robots Meta Tags

The implementation method depends on your website platform and how much access you have to the site’s code.

Adding Robots Meta Tags to HTML

For a standard HTML website, place the robots meta tag inside the <head> section of the page.

For example:

<!DOCTYPE html>

<html>

<head>

    <meta name=”robots” content=”noindex, nofollow”>

    <title>Example Page</title>

</head>

<body>

    …

</body>

</html>

The directive should be included in the document’s <head> so that crawlers can discover it when processing the page.

Implementing Robots Meta Tags in WordPress

WordPress users can usually manage robots directives through an SEO plugin rather than editing theme files manually.

For example, many SEO plugins provide an advanced settings area for individual posts and pages.

A typical process is:

1. Open the page or post you want to modify.
2. Open the SEO plugin’s advanced settings.
3. Locate the robots or meta robots controls.
4. Set the page to noindex if it should not appear in search results.
5. Enable nofollow if you specifically want to prevent crawlers from following links on that page.
6. Use the advanced robots fields for less common directives when supported.
7. Save or update the page.

The exact names and locations of these options vary depending on the SEO plugin and its current version.

Always verify the generated HTML after making changes.

Adding Robots Meta Tags in Shopify

Shopify users who need custom robots directives may need to modify the site’s theme files.

The <head> section is commonly located within the theme’s theme.liquid layout file.

For example, a conditional rule could look like:

{% if handle contains ‘page-name’ %}

<meta name=”robots” content=”noindex”>

{% endif %}

The condition should be customized for the page or URL pattern you want to control.

Because theme modifications can affect the entire website, make changes carefully. If you are uncomfortable editing theme code, consider having a developer implement and test the change.

Implementing Robots Meta Tags in Wix

Wix provides SEO settings through its website dashboard.

A typical workflow is:

1. Open the Wix dashboard.
2. Choose Edit Site.
3. Open Pages & Menu.
4. Find the page you want to configure.
5. Open the page’s SEO settings.
6. Look for the advanced SEO or robots meta tag controls.
7. Select the appropriate robots directives.
8. Save your changes.

For directives that are not available through standard checkboxes, Wix may provide an option to add additional meta tags manually.

For example, a custom robots directive can be entered in HTML format where the platform permits it.

Always test the published page afterward to ensure that the expected directive is actually present in the page source.

What Is the X-Robots-Tag?

The X-Robots-Tag is an HTTP response header that provides search engine crawlers with instructions similar to those supplied by a robots meta tag.

The major difference is where the instruction is delivered.

A robots meta tag appears inside an HTML document:

<meta name=”robots” content=”noindex, nofollow”>

An X-Robots-Tag appears in the HTTP response header:

X-Robots-Tag: noindex, nofollow

This makes X-Robots-Tag especially useful for resources that are not ordinary HTML pages.

Examples include:

• PDF files
• Images
• Videos
• Documents
• Other downloadable resources

Because these resources may not contain an HTML <head> section, a traditional meta robots tag cannot always be used.

The X-Robots-Tag is configured at the server level, often through files such as .htaccess on Apache or server configuration files on Nginx.

How to Implement X-Robots-Tags

X-Robots-Tags are generally configured by modifying server settings.

The exact implementation depends on the server software you use.

Using X-Robots-Tag on an Apache Server

On Apache, you can place rules in the site’s .htaccess file or the server’s configuration file.

For example:

<Files ~ “\.pdf$”>

    Header set X-Robots-Tag “noindex, nofollow”

</Files>

This tells compliant crawlers not to index PDF files and not to follow links associated with those resources.

The rule can be adjusted to target other file types or directives depending on your requirements.

For example, you could create rules for specific extensions or directories rather than applying the instruction universally.

Before editing .htaccess, make sure you understand the existing configuration because a syntax error can affect how the website responds.

Using X-Robots-Tag on an Nginx Server

On Nginx, the directive can be added to an appropriate server configuration file.

For example:

location ~* \.pdf$ {

    add_header X-Robots-Tag “noindex, nofollow”;

}

This applies the specified X-Robots-Tag to PDF resources matched by the location rule.

As with Apache configuration, the exact placement depends on how your server is configured.

After making a change, test the affected resources to ensure the expected HTTP header is being returned.

Common Meta Robots Tag Mistakes to Avoid

Robots directives are powerful, but incorrect implementation can create serious indexing problems.

Here are some common mistakes to watch for.

1. Using a Meta Robots Tag on a Page Blocked by Robots.txt

If a URL is blocked by robots.txt, a search engine may not be able to crawl the page and see its meta robots directive.

For example, you cannot reliably use a noindex meta tag as the solution if the same URL is completely inaccessible to the crawler through robots.txt.

If your goal is to communicate a noindex instruction, make sure search engines can access the page well enough to discover that instruction.

2. Adding Unsupported Noindex Rules to Robots.txt

A common misconception is that you can simply add noindex to robots.txt.

Modern Google Search does not support noindex as a standard robots.txt directive.

If you need to control indexing, use an appropriate method such as a robots meta tag or X-Robots-Tag.

3. Removing Noindex Pages From Sitemaps Too Early

When a page has been deliberately set to noindex, removing it from an XML sitemap immediately may make it harder for search engines to discover the indexing directive.

In some situations, keeping the URL discoverable until it has been processed can help search engines recognize the change.

Once the page has been removed from the index, the sitemap can be updated according to your site’s normal process.

4. Forgetting to Remove Noindex From a Staging Site

One of the most dangerous mistakes is carrying a staging site’s noindex configuration into production.

Developers frequently prevent search engines from indexing staging environments while a website is being built.

That is sensible during development.

However, if the same directive remains after launch, important production pages may remain excluded from search results.

After moving a site from staging to production, check the robots directives carefully.

How to Check Your Website for Meta Robots Tag Issues

You do not necessarily need a specialized SEO platform to check robots directives.

Several methods can be used to identify problems.

1. Inspect the Page Source

Open the affected page and view its source code.

Search for:

<meta name=”robots”

Then check the value assigned to the content attribute.

For example:

<meta name=”robots” content=”noindex”>

If the page is supposed to rank but contains noindex, that may be an issue.

2. Inspect the HTTP Response Headers

For resources using X-Robots-Tag, examine the HTTP response headers.

You should look for:

X-Robots-Tag:

This can be especially important for PDFs and other non-HTML resources.

3. Check robots.txt Separately

Make sure the URL is not being blocked from crawling if you need search engines to discover a robots meta directive.

Remember that robots.txt and robots meta tags serve different purposes.

4. Use Search Console and Crawling Tools

Search engine webmaster tools can help identify indexing and crawling problems.

Look for issues involving:

• URLs excluded from indexing
• Pages blocked from crawling
• Unexpected noindex directives
• Sitemap indexing problems
• Crawling errors

Third-party crawling tools can also scan a website and identify pages containing unexpected robots directives.

5. Review the Page After Changes

After correcting a robots directive, do not assume the change has been implemented simply because you changed a setting in a CMS.

Open the published page, inspect the source, and verify that the expected directive exists.

For X-Robots-Tags, inspect the actual HTTP response.

This additional verification can catch configuration issues before they affect large portions of a website.

FAQs

When Should You Use the Robots Meta Tag vs. X-Robots-Tag?

Use a robots meta tag primarily for HTML pages when you want to provide crawler instructions through the page itself.

Use an X-Robots-Tag when you need to control non-HTML resources such as PDFs, images, documents, or other files.

X-Robots-Tag can technically be used for HTML responses as well, but a meta robots tag is often easier to manage when you are working directly with an HTML page.

Another advantage of X-Robots-Tag is that server-level rules can be applied to groups of files.

For example, you can configure a server to return:

X-Robots-Tag: noindex

for every PDF matching a particular file pattern.

This can be much more efficient than modifying each file individually.

Do You Need to Use Both Meta Robots Tag and X-Robots-Tag?

No.

In most situations, you do not need both methods for the same resource.

A single correctly implemented directive is generally sufficient.

Using both does not automatically make a crawler more likely to obey your instructions.

If both are present, however, you should make sure their instructions do not conflict.

For example, combining one directive that allows indexing with another that prevents indexing can create confusion when troubleshooting the page.

When possible, use the simplest implementation that accurately expresses your intended crawling and indexing behavior.

What Is the Easiest Way to Implement Robots Meta Tags?

For people using a content management system, an SEO plugin or built-in SEO settings are often the easiest option.

These interfaces allow you to change common directives without manually editing HTML.

The exact method depends on your CMS.

WordPress websites may use an SEO plugin, while platforms such as Wix and Shopify provide their own SEO controls and theme-level configuration options.

For custom websites, developers can add the directives directly to the HTML or server configuration.

Regardless of the method, always verify the final output after making the change.

Use Meta Robots Tags Correctly to Avoid Indexing Issues

Robots meta tags and X-Robots-Tags give website owners valuable control over how search engines interact with their content.

A simple instruction such as:

<meta name=”robots” content=”noindex”>

can prevent a page from appearing in search results, while directives such as nofollow, nosnippet, max-snippet, and max-image-preview provide more granular control over how search engines handle and present content.

The X-Robots-Tag extends similar controls to resources that do not necessarily contain HTML, making it especially useful for PDFs, images, and other files.

The most important thing is to use these directives intentionally.

Before adding noindex or nofollow, determine exactly what you want search engines to do. Make sure the page is not accidentally blocked by robots.txt, verify that staging restrictions have been removed after a site launch, and check the final HTML or HTTP response after making changes.

When implemented correctly, robots directives can help keep unwanted pages and resources out of search results while allowing your important content to remain discoverable and accessible to search engines.