Robots.txt - Site Promotion - Other - uCoz Community
You logged in as Guest
Username/E-mail:
Recent messages · Members · Forum rules · FAQ · SEARCH Registration · Log in


We are in the second round! Let's keep voting!

Page 2 of 2«12
uCoz Community » Other » Site Promotion » Robots.txt (website indexing)
Robots.txt
SunnyDate: Friday, 2009-02-06, 4:41 Am | Message # 1
Haunter
Group: Administrators
Messages: 2016
Awards: 121
Reputation: 139
Status: Online
A website with the standard robots.txt is indexed in the best possible way. We adjusted it in such a way that only pages with content are indexed, and not all existing pages (e.g. login or registration page). Therefore uCoz websites are indexed better and get higher priority in comparison with other sites where all unnecessary pages are indexed.

WE STRONGLY RECOMMEND NOT TO REPLACE THE DEFAULT robots.txt BY YOUR OWN. You may be sure, we do all possible for uCoz sites to develop better. Otherwise, what’s the use of uCoz?!

Here is how the default robots.txt looks

Quote

User-agent: *
Disallow: /a/
Disallow: /stat/
Disallow: /index/1
Disallow: /index/2
Disallow: /index/3
Disallow: /index/5
Disallow: /index/7
Disallow: /index/8
Disallow: /index/9
Disallow: /main/
Disallow: /admin/
Disallow: /secure/
Disallow: /informer/
Disallow: /mchat

There is a "quarantine" for each new website when no modification of the robots.txt file is possible. In case of good traffic the quarantine will last up to 2 weeks, for the sites with low traffic – 30 days. If you pay for any of the additional services quarantine will end immediately after the payment.

Robots.txt file is a system file. If you still want to substitute it by your own, create a text file using notepad or any other text editor and name it "robots.txt". Then upload it to the root folder of your site by means File Manager or FTP.

Address of the robots.txt file is http://your_website_address/robots.txt

Robots.txt during quarantine looks in the following way:

Quote
User-agent: UNetBot
Disallow: /a/
Disallow: /stat/
Disallow: /index/1
Disallow: /index/2
Disallow: /index/3
Disallow: /index/5
Disallow: /index/7
Disallow: /index/8
Disallow: /index/9
Disallow: /panel/
Disallow: /admin/
Disallow: /secure/
Disallow: /informer/
Disallow: /mchat

User-agent: *
Disallow: /

Informers are not indexed because they output information that ALREADY exists. As rule this information is already indexed on the corresponding pages.

All about robots.txt


I have accidentally corrupted robots.txt. What should I do?
Delete it. Our robots.txt file will be added automatically (the system checks whether a user has it, and if not – adds back the default file).

Websites with low traffic are in the quarantine for a month. This is too long. How can a website have good traffic if it is unknown to search engines?
This time is needed for you to make your website interesting, add unique content and design. Then search engines will index your website well and won’t ignore it because it’s empty.

Is it possible to shorten the period of quarantine for a website with low traffic?
Yes, you need to pay for one of the additional services.

If I attach a domain to my website which is in the quarantine, will the quarantine end?
Yes. There is no quarantine for attached domains.

Is there any use in registering a website in search engines if the quarantine hasn’t ended yet?
No, your website won’t be accepted.

Will the robots.txt file be replaced automatically after the quarantine has ended? Or should I update it manually?
It will be updated automatically.

Is it possible to allow the indexing of photo albums in robots.txt? Many people search by images.
It will hardly give any result, but you can if you wish.

How can I forbid the indexing of website mirrors?
Read about mirrors. You should write your host in robots.txt, and then it will be the main one.

Is it possible to delete the default robots.txt?
You can’t delete it, it’s a system file, but you can add your own. But as it is written above it’s better not to do this. During the quarantine it is impossible to add you own robots.txt.

What is "UNetBot"? uCoz robot?
Yes.

What should I do to forbid indexing of the following pages?
_http://site.ucoz.com/index/0-4
_http://site.ucoz.com/index/0-5

Add the following to the robots.txt file:
/index/0-4
/index/0-5

I have forbidden the indexing of some links by means of robots.txt but they are still displayed. Why is it so?
By means of robots.txt you can forbid the indexing of pages, not links.

I want to make some changes in my robots.txt file. What is the process of it??
You must download it to your PC & upload back via File Manager or FTP-client.


I'm not "man", "sir" or whatever. I'm female!
About signatures, screenshots etc...
 
l2viruzDate: Tuesday, 2009-05-19, 8:23 Am | Message # 16
Regular
Group: Checked
Messages: 57
Awards: 0
Reputation: 12
Status: Offline
What is robot.txt? and HOW CAN I FILE SIMILAR FILES LIKE THESE?
 
SunnyDate: Tuesday, 2009-05-19, 8:23 Am | Message # 17
Haunter
Group: Administrators
Messages: 2016
Awards: 121
Reputation: 139
Status: Online
http://en.wikipedia.org/wiki/Robots.txt

Quote (l2viruz)
HOW CAN I FILE SIMILAR FILES LIKE THESE?

What do you mean?


I'm not "man", "sir" or whatever. I'm female!
About signatures, screenshots etc...
 
TiaDate: Thursday, 2009-09-24, 6:03 Am | Message # 18
Curious
Group: Checked
Messages: 27
Awards: 0
Reputation: 0
Status: Offline
Sunny (or Darts), my site is 28 days old, but my robots.txt looks this way:

User-agent: *
Disallow: /a/
Disallow: /stat/
Disallow: /index/1
Disallow: /index/2
Disallow: /index/3
Disallow: /index/5
Disallow: /index/7
Disallow: /index/8
Disallow: /index/9
Disallow: /panel/
Disallow: /admin/
Disallow: /secure/
Disallow: /informer/
Disallow: /mchat
Disallow: /search

So...does it mean that my site is not in quarantine any longer?
If not, then what is the problem? I'm sorry...for being so annoying.

P.S. When i type www.mysitename.com/index/1 (2 or 3) it takes me straight to my main page. Does it mean that my main page is blocked (index/1 (2 and 3) are in the robots.txt)? or i just don't have those pages that's why it takes me to the mane page?

 
SunnyDate: Thursday, 2009-09-24, 6:48 Am | Message # 19
Haunter
Group: Administrators
Messages: 2016
Awards: 121
Reputation: 139
Status: Online
Yes, it looks like your site is not on quarantine any more. As for the problem... try to contact Tech. Support via your CP -> Help tab.

Quote (Tia)
or i just don't have those pages that's why it takes me to the mane page?

Yes, you don't have those pages.


I'm not "man", "sir" or whatever. I'm female!
About signatures, screenshots etc...
 
TiaDate: Sunday, 2009-09-27, 9:48 Pm | Message # 20
Curious
Group: Checked
Messages: 27
Awards: 0
Reputation: 0
Status: Offline
Sunny! I did... :

Answer: You should download your existing http://site_name/robots.txt, change it and then upload to the root folder of your site.
Answered by: Sergio | Received: 2009-09-26, 5:43 Pm | Answered: 2009-09-27, 5:19 Pm

hmmm....i thought i shouldn't touch robot's txt at all... (read your post, Synny, about robots.txt) now he told me to change it...???
I don't understand!!!!!!

 
TiaDate: Monday, 2009-09-28, 5:47 Am | Message # 21
Curious
Group: Checked
Messages: 27
Awards: 0
Reputation: 0
Status: Offline
Sunny, you closed my thread but i still didn't get the answer i'm looking for...
My original thread - http://forum.ucoz.com/forum/6-8045-1
So what am i supposed to do with robots. txt?

Quote (Sunny)
WE STRONGLY RECOMMEND NOT TO REPLACE THE DEFAULT robots.txt BY YOUR OWN. You may be sure, we do all possible for uCoz sites to develop better. Otherwise, what’s the use of uCoz?!

Quote (Tech Support)
Answer: You should download your existing http://site_name/robots.txt, change it and then upload to the root folder of your site.
Answered by: Sergio | Received: 2009-09-26, 5:43 Pm | Answered: 2009-09-27, 5:19 Pm

I don't get it...replace or not... If i need to replace it - where do i find this file (i can't find it in my File manager!)???

Quote (Sergio)
...change it and then upload to the root folder of your site.
...change what?

I also asked Tech Support if robots.txt was the reason why google was blocked by the server and he didn't answer...i thought indexing is prohibited only when your robots.txt is in quarantine???!!! Mine is not!

Please... explain!

 
SunnyDate: Monday, 2009-09-28, 7:44 Am | Message # 22
Haunter
Group: Administrators
Messages: 2016
Awards: 121
Reputation: 139
Status: Online
Tia, I closed the thread because you created three threads about one and the same matter which is against forum rules.

Quote (Tia)
So what am i supposed to do with robots. txt?

Yes, it is not desirable to change default robots.txt but if you need to change it you can do this. I don't think changing is necessary on your case.

Quote (Tia)
If i need to replace it - where do i find this file (i can't find it in my File manager!)???

To replace it open your robots.txt (by the address _http://site_name/robots.txt), click File -> Save as (in your browser), save, edit the file, then upload it to File Manager.

Quote
I also asked Tech Support if robots.txt was the reason why google was blocked by the server and he didn't answer...i thought indexing is prohibited only when your robots.txt is in quarantine???!!! Mine is not!

Yes, indexing is prohibited when a website is on the quarantine. Maybe at that time, when Google was blocked, your website was still on quarantine. Now it is not. And your website is indexed by Google, Yahoo!, Binq... If some of your pages are not indexed by Google it may be caused by many reasons and it is hardly because of the default robots.txt. I advise you to read articles on website promotion and search engine optimization.


I'm not "man", "sir" or whatever. I'm female!
About signatures, screenshots etc...
 
uCoz Community » Other » Site Promotion » Robots.txt (website indexing)
Page 2 of 2«12
Search: