uCoz Community » For Webmasters » Site Promotion » Indexing Policy & Robots.txt
Indexing Policy & Robots.txt
Sunny
Posts: 9296
Reputation: 456

Message # 1 | 10:41 AM
Website's Indexing Status


All uCoz websites have Indexing status that is displayed at the top of the Control Panel's main page (/panel/?a=cp). The parameter shows whether indexing by search engines is allowed for the website or not (whether the website is in quarantine).
The indexing status can show one of the two options: "indexing is allowed (quarantine is removed)":



Or "indexing is prohibited (the website is in quarantine)":



The status "indexing is prohibited (the website is in quarantine)" is assigned by default to all newly created websites.

Quarantine Removal Policy


A website can become available for indexing either automatically (if a premium plan is purchased) or upon the website owner's request. If the website does not have a premium plan and the user wants the quarantine to be removed, a request should be submitted from the website's Control Panel:



There will be a pop-up window with the info on the quarantine policy:



After the request has been submitted, the website will be checked automatically according to a number criteria: the website's age, presence of a custom domain name, content, verified phone number etc. On the basis of these criteria the system decides whether the quarantine should be removed. We cannot provide a more detailed description of the algorithm.

Note! If the quarantine removal was denied, the next request can be submitted no sooner than in 7 days.


Robots.txt


A website's robots.txt file is located at http://your_website_address/robots.txt. A website with the default robots.txt is indexed in the best possible way – we set up the file in such a way that only pages with content are indexed, and not all existing pages (e.g. login or registration page). Therefore uCoz websites are indexed better and get higher priority in comparison with other sites where all unnecessary pages are indexed.

That's why we strongly recommend not to replace the default robots.txt by your own.


If you still want to replace the file by your own, create a text file using Notepad or any other text editor and name it "robots.txt". Then upload it to the root folder of your website via File Manager or FTP. Note: while website indexing is prohibited, no modification of the robots.txt file is possible.

The default robots.txt looks as follows:
Quote

User-agent: *
Allow: /*?page
Allow: /*?ref=
Allow: /stat/dspixel
Disallow: /*?
Disallow: /stat/
Disallow: /index/1
Disallow: /index/3
Disallow: /register
Disallow: /index/5
Disallow: /index/7
Disallow: /index/8
Disallow: /index/9
Disallow: /index/sub/
Disallow: /panel/
Disallow: /admin/
Disallow: /informer/
Disallow: /secure/
Disallow: /poll/
Disallow: /search/
Disallow: /abnl/
Disallow: /*_escaped_fragment_=
Disallow: /*-*-*-*-987$
Disallow: /shop/order/
Disallow: /shop/printorder/
Disallow: /shop/checkout/
Disallow: /shop/user/
Disallow: /*0-*-0-17$
Disallow: /*-0-0-

Sitemap: http://forum.ucoz.com/sitemap.xml
Sitemap: http://forum.ucoz.com/sitemap-forum.xml



Robots.txt during the quarantine looks as follows:

Quote

User-agent: *
Disallow: /




Robots.txt FAQ


Informers are not indexed because they display information that ALREADY exists. As a rule this information is already indexed on the corresponding pages.


Question: I have accidentally messed up robots.txt. What should I do?

Answer: Delete it. The default robots.txt file will be added back automatically (the system checks whether a website has it, and if not – adds back the default file).


Question: Is there any use in submitting a website to search engines if the quarantine hasn't been removed yet?

Answer: No, your website won't be indexed while in quarantine.


Question: Will the robots.txt file be replaced automatically after the quarantine has been removed? Or should I update it manually?

Answer: It will be updated automatically.


Question: Is it possible to delete the default robots.txt?

Answer: You can't delete it, it's a system file, but you can add your own file. However, we don't recommend to do this, as was stated above. During the quarantine it is impossible to upload a custom robots.txt.


Question: What should I do to forbid indexing of the following pages?
_http://site.ucoz.com/index/0-4
_http://site.ucoz.com/index/0-5

Answer: Add the following lines to the robots.txt file:
/index/0-4
/index/0-5


Question: I have forbidden indexing of some links by means of robots.txt but they are still displayed. Why is it so?

Answer: By means of robots.txt you can forbid indexing of pages, not links.


Question: I want to make some changes in my robots.txt file. How can I do this?

Answer: Download it to your PC, edit it and then upload it back via File Manager or FTP.

I'm not active on the forum anymore. Please contact other forum staff.
Sunny
Posts: 9296
Reputation: 456

Message # 166 | 10:15 AM
Ed05, you just need to wait till Google re-indexes your website. The robots.txt file is ok: http://supercafe.ucoz.com/robots.txt
I'm not active on the forum anymore. Please contact other forum staff.
Ed05
Posts: 556
Reputation: 7

Message # 167 | 4:29 PM
Sunny, huh, thanks wink
!
Post edited by Ed05 - Friday, 2014-01-10, 4:29 PM
Pahis
Posts: 6
Reputation: 0

Message # 168 | 1:36 PM
I'm wondering why my site won't show up in Google, even though it has had over 10 000 individual visitors in less than a month. I have my own domain and the site address is this: http://pahamaa.info/

Google just says that cannot show any info because there's a robots.txt in the site.
Sunny
Posts: 9296
Reputation: 456

Message # 169 | 2:51 PM
Pahis, your website will be 30 days old in two days, and the robots.txt file will be unblocked. Then you will need to wait till Google reindexes your website.
I'm not active on the forum anymore. Please contact other forum staff.
Pahis
Posts: 6
Reputation: 0

Message # 170 | 3:11 PM
Quote Sunny ()
Pahis, your website will be 30 days old in two days, and the robots.txt file will be unblocked. Then you will need to wait till Google reindexes your website.


Ok, thanks. I was hoping that maybe those visitors would've been enough to unblock the robots.txt already smile Two weeks ago my site had 200 visitors at one day, and after that the site showed up in Google, but after few days, it disappered again.
Sunny
Posts: 9296
Reputation: 456

Message # 171 | 3:38 PM
Quote Pahis ()
Two weeks ago my site had 200 visitors at one day, and after that the site showed up in Google, but after few days, it disappered again.


It's not possible. If the robots.txt is blocked, it is blocked for the whole 30-day period.

I'm not active on the forum anymore. Please contact other forum staff.
doda
Posts: 1
Reputation: 0

Message # 172 | 11:13 AM
@Sunny Would you please just allow Adsense bot only to show ads

Quote

User-agent: Mediapartners-Google
Disallow:

Added (2014-03-04, 5:13 AM)
---------------------------------------------
Sunny, please just allow Adsense bot only to show ads

Post edited by doda - Monday, 2014-03-03, 11:59 PM
Sunny
Posts: 9296
Reputation: 456

Message # 173 | 11:18 AM
doda, it is not possible to change the robots.txt file till the quarantine period is over but

Quote Sunny ()
Is it possible to shorten the period of quarantine for a website with low traffic?
Yes, you need to pay for one of the additional services.


If you pay for any of paid services (that cost more than $2), the robots.txt file will be unblocked.

I'm not active on the forum anymore. Please contact other forum staff.
Dummy Dummy
Posts: 1
Reputation: 0

Message # 174 | 5:26 AM
Thanks!!!!!!
I AM NOT CREATING MULTIPLE ACCOUNTS!!!!!!!!!!!!!!!!!!!

I SHARE A HOUSE WITH MY SISTER!!!!!!!!!!!!!!!!!!!!!!
SG
Posts: 1
Reputation: 0

Message # 175 | 8:57 AM
is there a way to make the robots.TXT go away? when i search my web on google my website appears but the description say : A description for this result is not available because of this site's robots.txt – learn more. ???
Paradox
Posts: 3278
Reputation: 145

Message # 176 | 10:50 AM
SG, no, robots.txt is a standard file for any modern website, aside from the fact they're required as part of the system. Assuming you are not paying for a premium service Google is unable to index sites and content until a period of 30 days has passed since when you first created your site. Even when this 30 day period has passed it can take up to a week to completed indexing. smile
Web Applications Developer
Been here for 8 years and counting.
Visit me at alexmoloney.me.
serkangamis
Posts: 5
Reputation: 0

Message # 177 | 4:55 AM
Hi I changed the robots.txt file, but there are still disallow.
Please could you help. http://altin.ucoz.com/robots.txt site

Paradox
Posts: 3278
Reputation: 145

Message # 178 | 6:01 AM
serkangamis, your website is still within the standard uCoz quarantine period. Your site will be open for indexing once it has existed for a minimum of 30 days. smile
Web Applications Developer
Been here for 8 years and counting.
Visit me at alexmoloney.me.
serkangamis
Posts: 5
Reputation: 0

Message # 179 | 11:01 AM
robots noindex url :/
NiKeR
Posts: 11
Reputation: 0

Message # 180 | 0:24 AM
Hey I want to make a purchase but the problem is that here in Bulgaria we use BGN and my bank can't convert it to USD, so is there any way for me to do it via online payment in BGN? Or a website which is friendly to your system and can convert it to USD
uCoz Community » For Webmasters » Site Promotion » Indexing Policy & Robots.txt
Search: