uCoz Community » For Webmasters » Site Promotion » Indexing Policy & Robots.txt
Indexing Policy & Robots.txt
Sunny
Posts: 9296
Reputation: 456

Message # 1 | 10:41 AM
Website's Indexing Status


All uCoz websites have Indexing status that is displayed at the top of the Control Panel's main page (/panel/?a=cp). The parameter shows whether indexing by search engines is allowed for the website or not (whether the website is in quarantine).
The indexing status can show one of the two options: "indexing is allowed (quarantine is removed)":



Or "indexing is prohibited (the website is in quarantine)":



The status "indexing is prohibited (the website is in quarantine)" is assigned by default to all newly created websites.

Quarantine Removal Policy


A website can become available for indexing either automatically (if a premium plan is purchased) or upon the website owner's request. If the website does not have a premium plan and the user wants the quarantine to be removed, a request should be submitted from the website's Control Panel:



There will be a pop-up window with the info on the quarantine policy:



After the request has been submitted, the website will be checked automatically according to a number criteria: the website's age, presence of a custom domain name, content, verified phone number etc. On the basis of these criteria the system decides whether the quarantine should be removed. We cannot provide a more detailed description of the algorithm.

Note! If the quarantine removal was denied, the next request can be submitted no sooner than in 7 days.


Robots.txt


A website's robots.txt file is located at http://your_website_address/robots.txt. A website with the default robots.txt is indexed in the best possible way – we set up the file in such a way that only pages with content are indexed, and not all existing pages (e.g. login or registration page). Therefore uCoz websites are indexed better and get higher priority in comparison with other sites where all unnecessary pages are indexed.

That's why we strongly recommend not to replace the default robots.txt by your own.


If you still want to replace the file by your own, create a text file using Notepad or any other text editor and name it "robots.txt". Then upload it to the root folder of your website via File Manager or FTP. Note: while website indexing is prohibited, no modification of the robots.txt file is possible.

The default robots.txt looks as follows:
Quote

User-agent: *
Allow: /*?page
Allow: /*?ref=
Allow: /stat/dspixel
Disallow: /*?
Disallow: /stat/
Disallow: /index/1
Disallow: /index/3
Disallow: /register
Disallow: /index/5
Disallow: /index/7
Disallow: /index/8
Disallow: /index/9
Disallow: /index/sub/
Disallow: /panel/
Disallow: /admin/
Disallow: /informer/
Disallow: /secure/
Disallow: /poll/
Disallow: /search/
Disallow: /abnl/
Disallow: /*_escaped_fragment_=
Disallow: /*-*-*-*-987$
Disallow: /shop/order/
Disallow: /shop/printorder/
Disallow: /shop/checkout/
Disallow: /shop/user/
Disallow: /*0-*-0-17$
Disallow: /*-0-0-

Sitemap: http://forum.ucoz.com/sitemap.xml
Sitemap: http://forum.ucoz.com/sitemap-forum.xml



Robots.txt during the quarantine looks as follows:

Quote

User-agent: *
Disallow: /




Robots.txt FAQ


Informers are not indexed because they display information that ALREADY exists. As a rule this information is already indexed on the corresponding pages.


Question: I have accidentally messed up robots.txt. What should I do?

Answer: Delete it. The default robots.txt file will be added back automatically (the system checks whether a website has it, and if not – adds back the default file).


Question: Is there any use in submitting a website to search engines if the quarantine hasn't been removed yet?

Answer: No, your website won't be indexed while in quarantine.


Question: Will the robots.txt file be replaced automatically after the quarantine has been removed? Or should I update it manually?

Answer: It will be updated automatically.


Question: Is it possible to delete the default robots.txt?

Answer: You can't delete it, it's a system file, but you can add your own file. However, we don't recommend to do this, as was stated above. During the quarantine it is impossible to upload a custom robots.txt.


Question: What should I do to forbid indexing of the following pages?
_http://site.ucoz.com/index/0-4
_http://site.ucoz.com/index/0-5

Answer: Add the following lines to the robots.txt file:
/index/0-4
/index/0-5


Question: I have forbidden indexing of some links by means of robots.txt but they are still displayed. Why is it so?

Answer: By means of robots.txt you can forbid indexing of pages, not links.


Question: I want to make some changes in my robots.txt file. How can I do this?

Answer: Download it to your PC, edit it and then upload it back via File Manager or FTP.

I'm not active on the forum anymore. Please contact other forum staff.
altmp3
Posts: 6
Reputation: 0

Message # 151 | 6:45 PM
How do i pay and how much cost's ? and does the basic package applyes for all my ucoz websites ?
Quote
Is it possible to shorten the period of quarantine for a website with low traffic?
Yes, you need to pay for one of the additional services.
Post edited by altmp3 - Thursday, 2013-03-21, 6:57 PM
Paradox
Old Guard
Posts: 3284
Reputation: 145

Message # 152 | 8:19 AM
altmp3, open Control Panel > Paid services > Paid services. The basic package costs US $3.09. Premium services apply to the site for which you buy them for - so no, they do not apply to all your websites.
Jack of all trades in development, design, strategy.
Working as a Support Engineer.
Been here for 13 years and counting.
Glen
Posts: 37
Reputation: 0

Message # 153 | 3:21 AM
Hey There Sunny,
I have stupid question on the robots.txt file.
Where the heck is this file at? I have looked in the File Manager and its not there.
My web site has been up for over a year and it still don't automatically put one there.
Is there something else I suppose to do? I sure need this thing, these auto bots are hitting me
every day. Getting about 5 to 15 and tired of deleting them. Help please..

Glen

"I Have No Gills, Therefore I Fish"
"Love The Wife, Fish Hard, Fish Till You Die"
www.glenscustomrods.com
Paradox
Old Guard
Posts: 3284
Reputation: 145

Message # 154 | 7:03 AM
Glen, if you are using the default system robots.txt it does not appear in your file manager If you wish to make changes simply create a new robots.txt file using notepad with the appropriate adjustments and upload it to the root directory of your file manager. This will then replace the existing robots.txt file used by the system, to reset it back to standard just delete your custom file. smile
Jack of all trades in development, design, strategy.
Working as a Support Engineer.
Been here for 13 years and counting.
Sunny
Posts: 9296
Reputation: 456

Message # 155 | 8:50 AM
Glen, and to view the default robots.txt, just add /robots.txt after your website address in the browser address bar.
I'm not active on the forum anymore. Please contact other forum staff.
Glen
Posts: 37
Reputation: 0

Message # 156 | 10:23 PM
Ok, Thanks
Now to figure out how to get the robots from auto registering
onto my web site.

"I Have No Gills, Therefore I Fish"
"Love The Wife, Fish Hard, Fish Till You Die"
www.glenscustomrods.com
Paradox
Old Guard
Posts: 3284
Reputation: 145

Message # 157 | 1:17 AM
Glen, oh. Robots.txt only defines whether or not robots from search engines will index your site.

If you're encountering issues with automated bots registering on your site then you need to increase the difficulty of the captcha or provide some form of verification to check that your user is human. smile

Jack of all trades in development, design, strategy.
Working as a Support Engineer.
Been here for 13 years and counting.
Glen
Posts: 37
Reputation: 0

Message # 158 | 6:28 AM
Well tired the captcha, and that didn't do the trick.
Thanks for the help anyway.

"I Have No Gills, Therefore I Fish"
"Love The Wife, Fish Hard, Fish Till You Die"
www.glenscustomrods.com
LinearMan
Posts: 49
Reputation: 0

Message # 159 | 4:11 PM
My website 21stcenturystoa.ucoz.com was created on 11 May 2013, so is 42 days old. The quarantine period of 30 days has elapsed, but it is still not registered with Google. The information about the 30-day quarantine period doesn't say whether ucoz does the registration automatically after 30 days, or whether newcomers are meant to follow some unspecified procedure. Please clarify.
Xayaan
Posts: 558
Reputation: 13

Message # 160 | 4:56 PM
LinearMan, It takes 6 months for Google to index your website, As by the latest information on SEO optimization and Your website will be indexed on google IF you reguarly update your website with new content and if the flow of traffic or visitors are constant.

Quote (Sunny)
How can a website have good traffic if it is unknown to search engines?
This time is needed for you to make your website interesting, add unique content and design. Then search engines will index your website well and won’t ignore it because it’s empty.

My Skype : xayan123 | My Twitter : @Officialxayaan
uID : - Zayaan Ibrahim | Need a professional design? Check out my Portfolio
Paradox
Old Guard
Posts: 3284
Reputation: 145

Message # 161 | 2:52 AM
Quote (Xayaan)
LinearMan, It takes 6 months for Google to index your website, As by the latest information on SEO optimization and Your website will be indexed on Google IF you regularly update your website with new content and if the flow of traffic or visitors are constant.


Actually, once your site has been opened for indexing Google has the ability to crawl your site - it generally takes up to a month for Google to entirely index your site within search results so don't expect immediate results however little bit by little bit it will get there.

Your traffic and content does not actually contribute to whether or not it appears in search results. By updating your site content regularly Google's bot crawler is prompted to re-crawl your site more frequently thus keeping your search results current and up to date. Your standing within search results is decided using an equation which takes into account many factors such as content: relativity, currency, appropriateness, as well as other external factors such as the number of backlinks to your website, etc.

So to address your original question LinearMan, you don't need to 'register' as such to have your site indexed by a search engine. You just need to give its bots time to crawl your site and it will be indexed in its own time. smile

Jack of all trades in development, design, strategy.
Working as a Support Engineer.
Been here for 13 years and counting.
Glen
Posts: 37
Reputation: 0

Message # 162 | 11:38 PM
Hey linearman,
once your site gets out on the world wide web, be ready for the robots.
you are going to be flooded by them. people from other countries trying to post is your
web site with all kinds of garbage. I have blocked them from the forums but still
hadn't figured out how to keep them from registering. Deleting anywhere 10 to 30 every other day.
so aggravating, but just wanted to let you know, good luck with the web site.
Glen

"I Have No Gills, Therefore I Fish"
"Love The Wife, Fish Hard, Fish Till You Die"
www.glenscustomrods.com
moskvatv
Posts: 1
Reputation: 0

Message # 163 | 2:59 PM
My website was created about 1 year ago but robots file is till the same, why?

http://moskvatv.ucoz.com/robots.txt

How I can change it?
Thanks
Sunny
Posts: 9296
Reputation: 456

Message # 164 | 3:58 PM
moskvatv, you have a custom domain attached, therefore you should look at the robots.txt file by the following link http://www.moskvatv.com/robots.txt
I'm not active on the forum anymore. Please contact other forum staff.
Ed05
Posts: 556
Reputation: 7

Message # 165 | 2:03 AM
I have buy basic package 3 month ago and domain, now i removed that domain and return to primary one.But, when i search on google for my website i se that my website stil blocked in search bu google robot txt. why, i have basic package for 3 mont and the webisite is old over 6 ,7 month.

Attachments: 4823859.png (9.2 Kb)

!
uCoz Community » For Webmasters » Site Promotion » Indexing Policy & Robots.txt
Search: