Bandwidth Exceeded, A New Record For Totse.info

DfgDfg Admin
edited September 2011 in Help and Suggestions
On August 29th 2011 Totse.info reached it's bandwidth cap which is 120GB per month. We started off from 9 GB back in Nov 2010 and then to 47 GB in Dec 2010 and today we just hit 123+GB. You said the error message because the traffic jump was unexpected and the delay the recovery process was mainly due to lack of communication since our IRC server is down but in any case it's a major plus for us. For all those haters that said that this place is dying and we're just waiting for the end, here you have the proof. We're actually exceeding our limits.


Vineman extended the cap and for that we're truly thankful. It's because of people like Vineman that places like Totse.info survive. We have so much ground to cover this month. We have plans which are already set in motion. The Totse you see today would get changed and turned into something that will benefit everyone and it will become the main pillar in your lives.

While the server was down, we were on TeamSpeak discussing various ways of reducing the load. I did some checking and after going through some sections, I found a way to optimize the website. We're no longer going to serve the mirror of Totse Text Files. We're also going to get a CDN for the images on the CMS.

Anyway, just enjoy the forums and have fun.

Comments

  • Darth BeaverDarth Beaver Meine Ehre heißt Treue
    edited August 2011
    Good thing this site is dying or can you imagine how much bandwidth we might have used.
  • RemadERemadE Global Moderator
    edited August 2011
    Like I said, if donations are ever needed, I'm able to give a bit. Thank you Vineman :thumbsup:
  • BoxBox Regular
    edited August 2011
    You saw* the error message...

    cool shit btw :cool:
  • edited August 2011
    Vineman - how much is hosting on your server? I'd like to preserve the old text files, and I'd like to put them onto an archive on my own webspace :) Totse.info and my mirror would also be able to link to each other, so that people can still read the text files and come here if they want.
  • DaktologistDaktologist Global Moderator
    edited August 2011
    I have most of them preserved on my server at home. Mostly from the archives i have found plus what is on here, the archives i found were mostly incomplete or were missing pages. I have slowly began with reconstructing the archive but it takes time, hopefully I will have a working archive soon.
  • ThirdRockFromTheSunThirdRockFromTheSun <b style="color:blue;">Third<em style="color:pink;">Cock</em>FromThe<em style="color:brown;">Bum</em
    edited August 2011
    I feel the need to bake a cake in a celebratory fashion. I better go read a guide on totse about how to make a cake!
  • easy-eeasy-e Acolyte
    edited August 2011
    I've got 500gb a month...haven't even come close yet.
  • Darth BeaverDarth Beaver Meine Ehre heißt Treue
    edited August 2011
    easy-e wrote: »
    I've got 500gb a month...haven't even come close yet.

    So what have you hit?
  • DfgDfg Admin
    edited August 2011
    easy-e wrote: »
    I've got 500gb a month...haven't even come close yet.

    Care to share the traffic stats?
  • edited August 2011
    trx100 wrote: »
    Vineman - how much is hosting on your server? I'd like to preserve the old text files, and I'd like to put them onto an archive on my own webspace :) Totse.info and my mirror would also be able to link to each other, so that people can still read the text files and come here if they want.

    It's whatever, my intention was never to make money off it, just to help lessen the server costs. Unfortunately the webhosting industry is a virtual monopoly and I can't compete, so it's not really been an effective plan. You can check out plans I put together here, but my hope was that I could utilize a sort of "build your own plan" model.
  • edited August 2011
    Are you talking about T2? I'd quite like to see your stats too, you guys seem to get a lot of guests online :)
  • easy-eeasy-e Acolyte
    edited August 2011
    I'll post them after work. I need to download a tool to analyse the IIS logs. We run a windows VPS...loads of fun :) If you guys have any tools you suggest let me know.

    What setting to you guys have for "active users"? Ours is only 30 days.
  • dr rockerdr rocker Regular
    edited August 2011
    Dfg wrote: »
    On August 29th 2011 Totse.info reached it's bandwidth cap which is 120GB per month...


    We're no longer going to serve the mirror of Totse Text Files. We're also going to get a CDN for the images on the CMS.

    Correct me if I am wrong but text files only take up server space, rather than bandwidth. That is, unless someone accesses them. Unless totse is hosted on some floating server that travels across the internet eating bandwidth as it moves and needs to eat more to carry the textfiles? What logs do you have of textfile access from google? It could be that we are seeing that amount of bandwidth usage from people accessing the text files to some extent. Is it that the text files are being removed so images can be stored?

    It is a one to think on.
  • RogueEagle91RogueEagle91 Regular
    edited August 2011
    Go us, and rock on Vineman for the (relatively) quick recovery.
    I've seen sites go through a billing cycle before it's back up due to bandwidth usage.

    Also, just checked my site. Have a storage cap, but no bandwidth cap. Still, only using maybe 1gb/month.
    My site is so lonely.
  • edited August 2011
    ^ Damn, that sucks :( Why don't we share links with you? Maybe that will help drive some traffic to your site :)
  • RogueEagle91RogueEagle91 Regular
    edited August 2011
    trx100 wrote: »
    ^ Damn, that sucks :( Why don't we share links with you? Maybe that will help drive some traffic to your site :)

    I'm not that concerned, honestly.
    It's less important the total as much as where the traffic comes from.
    I make more money from people within ~50miles of me, as it's for a business.

    Also, I haven't updated since december or january.
  • Darth BeaverDarth Beaver Meine Ehre heißt Treue
    edited August 2011
    dr rocker wrote: »
    Correct me if I am wrong but text files only take up server space, rather than bandwidth. That is, unless someone accesses them. Unless totse is hosted on some floating server that travels across the internet eating bandwidth as it moves and needs to eat more to carry the textfiles? What logs do you have of textfile access from google? It could be that we are seeing that amount of bandwidth usage from people accessing the text files to some extent. Is it that the text files are being removed so images can be stored?

    It is a one to think on.

    We use sitemeter and other third party tools to analyze our traffic. So instead of listening to your flying spaghetti monster server theories excuse us if we run the site according to reality.
  • DfgDfg Admin
    edited August 2011
    dr rocker wrote: »
    Correct me if I am wrong but text files only take up server space, rather than bandwidth. That is, unless someone accesses them. Unless totse is hosted on some floating server that travels across the internet eating bandwidth as it moves and needs to eat more to carry the textfiles? What logs do you have of textfile access from google? It could be that we are seeing that amount of bandwidth usage from people accessing the text files to some extent. Is it that the text files are being removed so images can be stored?

    It is a one to think on.
    http://www.sitemeter.com/?a=stats&s=sm7totse
    https://www.quantcast.com/totse.info

    The text files are dead weight. We have three mirrors for them already.
  • edited August 2011
    Dfg, may I suggest optimizing your code as well? I'm sure there are some things you could do to trim down the fat and make things more streamlined.
  • bornkillerbornkiller Administrator In your girlfriends snatch
    edited August 2011
    Kudos Vineman. You're the fucking shit......& WooHoo to us! :)
  • GoingNowhereGoingNowhere Global Moderator
    edited August 2011
    Vineman is awesome :) Well done sir!
  • edited August 2011
    It's because of times like this that I remember to recommend Vineman hosting to anyone who's looking for reliable webhosting. He's got some fantastic customer service up his sleeve :D
  • dr rockerdr rocker Regular
    edited August 2011
    We use sitemeter and other third party tools to analyze our traffic. So instead of listening to your flying spaghetti monster server theories excuse us if we run the site according to reality.

    A question must then have a hypothesis for it to become a theory. I proposed neither a hypothosis or theory, but clearly, you are quite capable of making your own.
  • Darth BeaverDarth Beaver Meine Ehre heißt Treue
    edited August 2011
    dr rocker wrote: »
    A question must then have a hypothesis for it to become a theory. I proposed neither a hypothosis or theory, but clearly, you are quite capable of making your own.

    Clearly you are someone with no people skills.
  • SlartibartfastSlartibartfast Global Moderator -__-
    edited August 2011
    How big are the text files anyway? If someone posts post a complete archive here i'll do my best to compress the files so we can put them up on some torrent/hosting site.
  • dr rockerdr rocker Regular
    edited August 2011
    Clearly you are someone with no people skills.

    If you say so. What is very clear is that for a team that apparently spends a shit load of time analysing traffic, you failed to see that your months bandwidth was about to run out.
  • Darth BeaverDarth Beaver Meine Ehre heißt Treue
    edited August 2011
    dr rocker wrote: »
    If you say so.

    Glad we agree on your people skills.
    dr rocker wrote: »
    What is very clear is that for a team that apparently spends a shit load of time analysing traffic, you failed to see that your months bandwidth was about to run out.

    I am sorry that your perception is flawed. I said we use tools to analyze our traffic. You said we spend shit loads of time doing it. You really should try to respond to what was said and not what you think was said. That and we misjudged by a whole 24 hours, sue us.

    Now if you wish to continue bitching about this topic please make a thread in B&M.
  • ShadyTrollShadyTroll Regular
    edited August 2011
    Thank God everything is better. I was annoyed I couldn't check totse when I was enjoying my coffee this morning. :mad:
    How big are the text files anyway? If someone posts post a complete archive here i'll do my best to compress the files so we can put them up on some torrent/hosting site.

    People might be less likely to read the text files if they have to torrent them.
  • DfgDfg Admin
    edited August 2011
    How big are the text files anyway? If someone posts post a complete archive here i'll do my best to compress the files so we can put them up on some torrent/hosting site.

    1.5GB in size, the problem is the files have an enormous 404 errors which harm us when it comes to Internet ranking and SEO. It's better to take them down and filter the html and then submit it again. Plus, the HTML pages are heavy. They're made up of images and they really do task your bandwidth and page loading speed. Hell, ask Facebook is you don't believe me, they use one image spirit for the whole page and it's 11KB in size. Totse.com page is around 239KB and the size varies between each file.


    But again, the major issue is the 404 errors that belong to each adbrite box in the articles. All, in all it's not worth it.
  • DfgDfg Admin
    edited August 2011
    Were you able to verify that it was legitimate usage and that it wasn't gigabit DoS?

    In July it was 109GB. We're getting a good deal of steady traffic and it will only increase from this point on wards, so yeah I am sure we weren't DDoS'ed. Otherwise I would see a spike in bandwidth usage and since we have third party monitors, I can check which thread/article got the attention.
  • RemadERemadE Global Moderator
    edited August 2011
    Are those 3 links all different archives? If so I'm impressed. Will download and mirror.

    Edit - of course they are. RemadE you dickhead.
  • edited August 2011
    Dfg - I think we should take the text files down and host them elsewhere, or at least link to them. I was thinking that they could be taken down, have the text stripped from them and then re-published.
  • DfgDfg Admin
    edited August 2011
    trx100 wrote: »
    Dfg - I think we should take the text files down and host them elsewhere, or at least link to them. I was thinking that they could be taken down, have the text stripped from them and then re-published.

    That can be done easily.
  • edited August 2011
    Sure it can, with a PHP script I think. You can strip all HTML tags and leave behind plaintext, just like they do on websites when they're trying to get rid of XSS attacks :)
  • DfgDfg Admin
    edited August 2011
    trx100 wrote: »
    Sure it can, with a PHP script I think. You can strip all HTML tags and leave behind plaintext, just like they do on websites when they're trying to get rid of XSS attacks :)

    Or you can hire Mexicans and force them to copy paste things.
  • RaggedOldManRaggedOldMan Regular
    edited August 2011
    Wow, well done Totse on breaking a record, that's pretty awesome! I just thought my phone was messing up when I couldn't access the site :facepalm: I should pay more attention to error signs.
  • thewandererthewanderer Regular
    edited August 2011
    I bet this has something to do with the Quick DMT article getting 75,000 hits in the last few weeks.
  • edited August 2011
    I bet this has something to do with the Quick DMT article getting 75,000 hits in the last few weeks.

    Yeah, that was some mad attraction to that one article. Does anybody know why it had so much attention? Was it to do with Stumbleupon or something?
  • thewandererthewanderer Regular
    edited August 2011
    trx100 wrote: »
    Yeah, that was some mad attraction to that one article. Does anybody know why it had so much attention? Was it to do with Stumbleupon or something?
    Most of the big hits are the drug articles. Stumbleupon had a big part in the success and probably a bit of luck to get it jumpstarted.
  • edited August 2011
    That's pretty cool. Are you using Stumbleupon right now? I'm just wondering how many articles we've got up on there... We should add a load more. I know there was a thread about this, but I definitely think we should make better use of it as it's a good platform to take advantage of.
  • Darth BeaverDarth Beaver Meine Ehre heißt Treue
    edited August 2011
    I bet this has something to do with the Quick DMT article getting 75,000 hits in the last few weeks.
    Considering that we used 109 GB in July and 123 GB in August I don't think that one article was the deciding factor.

    UwjH5.png

    As you can see there has been steady growth over the last several months. BTW what are you basing the 75,000 hits on that one article on?

    Keep in mind that image is about 8 days old.
  • thewandererthewanderer Regular
    edited August 2011
    BTW what are you basing the 75,000 hits on that one article on?
    When I was an editor, I could see the hits on every article. It was about 70,000 last I was able to check, and since then the stumbleupon link has 5,000 more views. The way it was increasing compared to the SU link, it's probably closer to 85,000. Clearly the best seller on the CMS with at least 5 times more views than the 2nd most viewed page. This might explain the jump of 50,000 unique visitors compared to last month which is shown on your above pic.
  • Darth BeaverDarth Beaver Meine Ehre heißt Treue
    edited August 2011
    When I was an editor, I could see the hits on every article. It was about 70,000 last I was able to check, and since then the stumbleupon link has 5,000 more views. The way it was increasing compared to the SU link, it's probably closer to 85,000. Clearly the best seller on the CMS with at least 5 times more views than the 2nd most viewed page. This might explain the jump of 50,000 unique visitors compared to last month which is shown on your above pic.

    Yes it could that makes some sense.
  • edited August 2011
    I just had a look at the Quick DMT article - it has 79302 views right now :)
  • edited September 2011
    When I was an editor, I could see the hits on every article. It was about 70,000 last I was able to check, and since then the stumbleupon link has 5,000 more views. The way it was increasing compared to the SU link, it's probably closer to 85,000. Clearly the best seller on the CMS with at least 5 times more views than the 2nd most viewed page. This might explain the jump of 50,000 unique visitors compared to last month which is shown on your above pic.

    Almost assuredly. Look at the top 4 highest viewed pages for August.

    2wnto3o.png
  • RemadERemadE Global Moderator
    edited September 2011
    - Looks at top pages visited
    - "how-and-when-I-fucked-my-mom"
    - :facepalm: well at least it draws them in...
  • edited September 2011
    Can we get a screenshot of what else is a good draw for our website? I'd like to see what else I need to write about, maybe it will give us some ideas.
Sign In or Register to comment.