Google 404 error re:robots.txt

This section is for posting questions which are not directly related to WYSIWYG Web Builder.
Examples of off topics: web server configuration, hosting, programming related questions, third party scripts.

Note that these questions will generally not be answered by the administrators of this forum.
Post Reply
khuston
 
 
Posts: 2
Joined: Tue Jul 10, 2018 8:54 pm

Google 404 error re:robots.txt

Post by khuston »

This site (byrobbie(dot)co(dot)nz has been live for 3 weeks and Google has just reported that one page is not indexed. (accessories). Google seems to be looking for a robots.txt file. I have put INDEX:follow on all pages under page properties and I cannot see how to generate a robots.txt file like my previous web builder (serif W*b***s). Is a robots file neccessary or am I looking in the wrong places?
Any help is appreciated.
User avatar
BaconFries
 
 
Posts: 5327
Joined: Thu Aug 16, 2007 7:32 pm

Re: Google 404 error re:robots.txt

Post by BaconFries »

I cannot see how to generate a robots.txt
WWB does not generate a robots.txt file, it is generally seen as a server implementation and left to you if you think you require one. Do you need one, only you can say if you need one or not!. Do you have anything you do not want any of the search engines to index? this could be pages (.html), folders that have images (.jpg) or still testing and don't want your site getting indexed yet. If not any of the previous then it safe to say you don't need one.

You can if you wish just upload a blank .txt file with no words in it, naming it robots.txt which the bots, spiders will find and try to read and find it empty but will still reconize that you have one.

As you have already added INDEX: follow then you should be OK as you are now saying to the search engines it OK to index this page(s). You can at a later date if you wish to write User-agent* information to the robots.txt but you will need to research this and add what you want yourself as I don't wish any comeback saying that your ranking it down or pages not getting indexed due to what I suggested.
khuston
 
 
Posts: 2
Joined: Tue Jul 10, 2018 8:54 pm

Re: Google 404 error re:robots.txt

Post by khuston »

Thank you baconfries for your reply. On further investigation with Google, I find the error refers to the last crawl of the former site (same name), last crawled prior to the WBB site going live. So the WBB site is OK. No need for a robots file, no page is excluded. Panic over! And a great forum!!
Thanks again....
Post Reply