Casio, Well, I tried loading your site before, which was on a mobile internet connection (very slow) but it still loaded rather quick. And the test servers at pingdom loads your site under a second, and my current connect the same. There shouldn't be a slower loading speed just because you're logged out, that makes no sense.
Does it happen at any particular hours/on any particular pages? Where you using different browsers or some particular one?
And are you and other using any particular ISP? (Internet Service Provider)