How To Add Robots.txt File To your website
Are you an SEO Expert? then this article will help you learn how to make your website content visible for
search engines to crawl on with ease. One very important file to help you archive is what is called the
robots.txt file. so in this article, I will be walking you through How you can add
a robots.txt file to your website
What is robots.txt file?
A Robot.txt file is a file that you can add to your website to tell search engines what they are allowed to crawl on
on your website. i.e. it is a file that excludes pages or includes pages that you want the search engine to have access to on your website.
How To add Robots.txt File To your website
- Login to your WordPress dashboard
- Hover your mouse on the SEO panel
- Click on Tools
- Click on File Editor
- You will be taken to the place where you can create the robot.txt file
- You will see a message from Yoast plugin telling you that you don’t have a robots.txt file, that you should create one here
- Click on Create robots.txt file
- By default, you will see the following code (user-agent: *, Disallow: /wp-admin, Allow:/wp-admin/admin-ajax. Php)
Example of a good robots.txt file
User-agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/uploads/
Allow: /wp-includes/
Allow: /post-sitemap.xml
Allow: /page-sitemap.xml
Allow: /category-sitemap.xml
Disallow: /wp-admin/
Disallow: /refer/
Sitemap: https://www.yourdomain.com/post-sitemap.xml
Sitemap: https://www.yourdomain.com/page-sitemap.xml
Sitemap: https://www.yourdomain.com/category-sitemap.xml
Sitemap: https://www.yourdomain.com/sitemap.xml
Also see: Jiji Premium Ads Packages For Property/Real Estate Category
- After adding all of this click on save. that will be all!
Hints:
- the user-agent set to * tells search engines that these rules are applicable to all users
- The Allow specifies parts in your website that search engines are allowed to crawl and index
- The disallow specifies the parts in your website you don’t want search engines to crawl
- The sitemap below points to the sitemaps in your website: this is very important for Ask search engine or Yahoo and other search engines too to find the content
of your website with ease
In summary, robots.txt is an important file you must add to your website for search engines to easily find and index your website
Related:
Leave a Reply