Recent server issues

[The Bad Astronomer]

Folks-

I know that some of you are unhappy with the problems we've been having with the forum/website server. Jeff has been working closely with our webhost to isolate and solve the problem... which is, simply put, too many people accessing the site! Yes, we're a victim of our own success. We've been trying different incremental fixes to relieve the pressure, and we still have some ideas up our sleeves.

Let me assure you, we've been taking this issue very seriously. I live my life online, and I am personally aware of how important this forum is to the JREF. I read the forum as often as I can, though of course my time is limited (helping plan TAM 7 and TAM London turns out to be somewhat time-intensive). Jeff, of course, is also very busy trying to get this thing solved, so feel free to ask questions, but I'm asking that you please be patient with us.

I assure you that getting these problems solved is a top priority. At some point eventually we may need help from you, our community, but we'll work that out when the time comes.

For now, please be aware -- and this is coming straight from me, a guy who owes his career to the internet and the people who use it -- this forum and this community are highly prized by the JREF, and the problems will be fixed as quickly as we can make it happen.

[Hokulele]

That is good to hear. Thanks for the update.

[Zep]

Can I humbly suggest significantly over-engineering your next solution point. Plan for, say, twice or three times as much traffic per unit time and concurrent logins and visitors, etc.

Otherwise you will be right back here again very soon, doing this all again.

I hear Deep Blue is currently inactive. Make a bid! :)

[The Bad Astronomer]

Overengineering -- what web hosts call "room to grow" -- is in the plan.

[Miss_Kitt]

Phil -- sorry I didn't see this thread before I responded on the other thread!!

Apologies, Miss Kitt

[rjh01]

Just added the slow forum tag to this thread and another thread. If you look at the threads you will see
- Every few months a new thread has been created to discuss the issue.
- The problem started about the start of 2008 (13 months ago)
- If you read the threads Darat and Terry have spent a lot of time on the issue.

Other notes.
The forum has not grown at all over this period. This is partly due to the CT sub forum shrinking (maybe due to its success?). Shrinking means the number of threads created per month has gone down since the start of 2008. I suggest the lack of growth of this forum is also due to the topic at hand.

How much and how fast this forum will grow when the performance issues have been fixed can only be guessed at. I am eagerly looking forward to that date.

[arthwollipot]

Thanks for the update Phil. It's good to know the organisation values the forums highly. What (else) can we do to help?

[Zep]

Overengineering -- what web hosts call "room to grow" -- is in the plan.From an old computer scientist: ...why stop there?! :)

Seriously, I could arrange for a relatively monster server with Kenworth powered CPUs, mountains of RAM and heaps-o-disk to be made available solely for JREF, for possibly little or no money at all. But if the software ain't efficient and rugged, and the ISP network connection flows slower than treacle at the South Pole, there's no point. The forum will still seem like it is on a 486DX16 with 16MB RAM, Windows ME and terminal asthma.

As has been asked previously, Terry, what's the real problem? And is there anything we can possibly assist with?

[arthwollipot]

From an old computer scientist: ...why stop there?! :)

Seriously, I could arrange for a relatively monster server with Kenworth powered CPUs, mountains of RAM and heaps-o-disk to be made available solely for JREF, for possibly little or no money at all. But if the software ain't efficient and rugged, and the ISP network connection flows slower than treacle at the South Pole, there's no point. The forum will still seem like it is on a 486DX16 with 16MB RAM, Windows ME and terminal asthma.

As has been asked previously, Terry, what's the real problem? And is there anything we can possibly assist with?...please?

[tkingdoll]

As chillzero pointed out in the other thread, if you want Terry to see a question it might be best to PM him to point him here!

Re: the server size issue, it's clear to see what the requirements are if you look at the size of this forum relative to the rest of the internet.

From a forum ranking website (by metric), this forum is the 362nd largest forum on the internet. 362 out of hundreds of thousands. That's a big forum.

Compare this to the Bad Astronomy forum at number 1264, the Richard Dawkins or Skeptic Society forums which don't even rank (ETA the latter two may be because they haven't been submitted or crawled but if you visit them you can see the member and post counts for yourselves), and you can see just how huge it actually is.

Heck, this forum is bigger than the Penny Arcade forum. It's bigger than Mozillazine. It's just below the Doctor Who forum. It's massive.

http://rankings.big-boards.com/?p=all

Some of the biggies carry proper banner and skyscraper advertising, but not all do. I wonder, therefore, if it would be worth contacting the admins of some of those forums to ask then what their server spec is or what their costs are. They're meeting those costs somehow, so their knowledge might be useful.

[arthwollipot]

How the heck do you find out statistics like that???

[tkingdoll]

How the heck do you find out statistics like that???

I gave a link in my post :D

(the answer is, I believe they pull data from each forum's own measuring tools. There's an FAQ but it's not hugely detailed).

[rjh01]

I do not think that the total number of posts ever made is the best measure of how big a board is. For example we would be much bigger if we had not lost heaps of posts in the past. Forums grow or shrink over time.

A better measure is how many posts were made in a week. This is the link from the same site. Here is the link for that from the same site http://rankings.big-boards.com/?sort=week&filter=all,all&p=12. It shows we are 235 out of 1176 listed with 20,883 posts last week (subject to change over time). Compare that with the 14th biggest with 211,332 posts last week. 10 times bigger. In short we got potential to grow. On the other hand there are not too many other forums with a large general science sub forum or related subjects that are bigger than us.

I have been keeping stats of this over time.

[Alareth]

and this is coming straight from me, a guy who owes his career to the internet and the people who use it


I thought you owed your career to bad movie scripts and special effects?

[Darat]

Don't forget it is also not just the Forum, the server hosts both the Forum and the JREF website. (And both are generated by software that uses MySQL.)

ETA: For anyone wanting more detail about vBulletin and its reputation for needing hefty hardware: http://www.linuxbox.co.uk/vbulletin_performance_tuning.php .

[alfaniner]

From a forum ranking website (by metric), this forum is the 362nd largest forum on the internet. 362 out of hundreds of thousands. That's a big forum.

I guess that means skeptics talk a lot. I mean, a lot.

[Terry]

I do not think that the total number of posts ever made is the best measure of how big a board is.

Well, it depends. It isn't an unreasonable thing to look at if you're concerned about database performance.

[Terry]

As has been asked previously, Terry, what's the real problem? And is there anything we can possibly assist with?

vBuilletin makes some very inefficient queries, for one thing. Particularly it doesn't deal well with huge threads or members with huge numbers of posts. The other thing is the server needs more RAM (and ideally to be a 64-bit kernel so that a single process can grab a larger amount of RAM too). Almost invariably when the "no DB connections left" messages start, it's because we're thrashing swap. I've tried various things to tune the db to use VM better, but once one of those giant "runs for several hundred seconds" queries kicks in, we usually end up dropping people.

ETA: if we didn't drop people, it probably wouldn't recover on its own, and it would be hung until rebooted (or until someone restarted Apache and/or the database, at least).

[El Greco]

Why can't we copy what the biggest forums do ?

[JimBenArm]

You could try shoving a weiner in the warp drive. That sometimes helps...

[Toke]

362. biggest forum on the internet, sounds like it needs good hardware and good connections.
Might explain why it is not always avaible when I want to log on.

The forum does a lot of good against woo, so it would be worth expending the hardware on. Probably best value for the money.

[Klimax]

vBuilletin makes some very inefficient queries, for one thing. Particularly it doesn't deal well with huge threads or members with huge numbers of posts. The other thing is the server needs more RAM (and ideally to be a 64-bit kernel so that a single process can grab a larger amount of RAM too). Almost invariably when the "no DB connections left" messages start, it's because we're thrashing swap. I've tried various things to tune the db to use VM better, but once one of those giant "runs for several hundred seconds" queries kicks in, we usually end up dropping people.

ETA: if we didn't drop people, it probably wouldn't recover on its own, and it would be hung until rebooted (or until someone restarted Apache and/or the database, at least).

Does vBulletin have support for sever-side caching?Or at least Apache?This could decrease number of connection,but not sure how efficient with vB or Joomla it is/would be.

Or just new hardware,where new server would be dedicated for database and old one for website(PHP).

P.S:And I am ,I suspect, Mr.Obvious (Klimax)...

[arthwollipot]

ETA: if we didn't drop people, it probably wouldn't recover on its own, and it would be hung until rebooted (or until someone restarted Apache and/or the database, at least).Speaking of dropping people, why not cull old unused user accounts from the database?

Is there a query that you can use to determine the last time an account was used? I see by looking through the members list that there are a large number of accounts with 0 posts. If a script could be formulated such that {IF post count = 0 AND time last logged on > 365 days ago THEN delete the account} that would free up a modicum of space, wouldn't it?

[Terry]

Speaking of dropping people, why not cull old unused user accounts from the database?

Is there a query that you can use to determine the last time an account was used? I see by looking through the members list that there are a large number of accounts with 0 posts. If a script could be formulated such that {IF post count = 0 AND time last logged on > 365 days ago THEN delete the account} that would free up a modicum of space, wouldn't it?

I don't see how that would make a significant difference to the issues we're seeing.

[lionking]

From a forum ranking website (by metric), this forum is the 362nd largest forum on the internet. 362 out of hundreds of thousands. That's a big forum.



Before we start the self-congratulations, we are well below the Delta Goodrem forum.:(

[arthwollipot]

I don't see how that would make a significant difference to the issues we're seeing.Oh well. It was just an idea.

[Zep]

vBuilletin makes some very inefficient queries, for one thing. Particularly it doesn't deal well with huge threads or members with huge numbers of posts. The other thing is the server needs more RAM (and ideally to be a 64-bit kernel so that a single process can grab a larger amount of RAM too). Almost invariably when the "no DB connections left" messages start, it's because we're thrashing swap. I've tried various things to tune the db to use VM better, but once one of those giant "runs for several hundred seconds" queries kicks in, we usually end up dropping people.

ETA: if we didn't drop people, it probably wouldn't recover on its own, and it would be hung until rebooted (or until someone restarted Apache and/or the database, at least).Thanks, Terry. That does explain a fair bit. I appreciate your situation, and you have my full support!

Clearly the root cause is the vBulletin software. So just throwing more hardware at it (i.e. the Microsoft option) is unlikely to make a huge dent in it. That could be money and effort wasted just now.

The grizzled old coder here says that you will get far more bang for your buck by tracking down and "efficientising" the noticeable software roadblocks you have already identified. We have been this path before, hence the limited Search results, etc.

It seems clear that long topics and high post-count users are RAM hogs. That suggests vBulletin is making routine calls on MySQL that return giant tables in memory, possibly containing non-normalised data. (This would also show as high disk IO counts - disk thrashing.) My first "programmer analyst" question would be: Why are all these records back to the year dot needed every time in routine queries? Surely they would be expected to consume huge amounts of memory per instance, and yet it is highly unlikely all the records will ever be required for subsequent processing. Sounds like the vBulletin designers never allowed that these numbers could really get so high.

So I would look at reducing that memory demand in software before trying to accommodate it in hardware. If that can be done, you reduce RAM consumption per user and lower overall resource demand (i.e. RAM and disk IO) on your existing unmodified platform, allowing you to defer heavy-duty upgrades until things are calmer and planned. And paid for. :)

For example, would the most recent 1000 records be sufficient under most common circumstances? Or could a much smaller recordsize be requested, to be used as an index into one or two selected bigger records as required? (SQL programmers can stop laughing now! You suggest better code solutions, OK?) More to the point, can you re-code the guts of vBulletin at all anyway?

Good luck!

[rjh01]

vBuilletin makes some very inefficient queries, for one thing. Particularly it doesn't deal well with ... members with huge numbers of posts. <snip>

Ever thought about banning members with too many posts? But give them permission to create a sock puppet.

Or if our forum has too many posts then remove all old threads. Give a link to the archive on the main menu. The archive would have these deleted threads.

[Klimax]

Well,what about to start locking threads at 5000 range?There are some active.
Or even lower loke from 2000 or so...

[cj.23]

http://www.alexa.com/data/details/traffic_details/randi.org?site0=randi.org&y=p&z=1&h=300&w=470&c=1&u[]=randi.org&x=2009-02-06T10%3A07%3A51.000Z&check=www.alexa.com&signature=evvWUwzNIpiOgu0OGmkw19SbIx4%3D&range=max&size=Medium

has a lot of relevant data. Reach appears pretty steady, slight down on a year ago, but page hits are falling? 69.9% of hits are the forum. I can't interpret this data but you might be able to. :)

cj x

[legne]

I clicked on this thread and got a database error. How Ironic. =/

[Terry]

Thanks, Terry. That does explain a fair bit. I appreciate your situation, and you have my full support!

Clearly the root cause is the vBulletin software. So just throwing more hardware at it (i.e. the Microsoft option) is unlikely to make a huge dent in it. That could be money and effort wasted just now.

Not sure I agree that throwing more hardware at the situation is unwarranted.


The grizzled old coder here says that you will get far more bang for your buck by tracking down and "efficientising" the noticeable software roadblocks you have already identified. We have been this path before, hence the limited Search results, etc.


Of course, that's optimization 101. But this is effectively shrinkwrap software. We are in a world of hurt if we start changing vBulletin, and I for one am not up for supporting a fork of a licensed commercial product just because we happen to be able to read the PHP.


It seems clear that long topics and high post-count users are RAM hogs. That suggests vBulletin is making routine calls on MySQL that return giant tables in memory, possibly containing non-normalised data. (This would also show as high disk IO counts - disk thrashing.) My first "programmer analyst" question would be: Why are all these records back to the year dot needed every time in routine queries?

They aren't, but you have to look at all the posts in a thread to generate the thread display. Or at least look at an index containing all the posts.

[The Central Scrutinizer]

You could try shoving a weiner in the warp drive. That sometimes helps...

I've heard that reversing the polarity is more reliable.

[The Central Scrutinizer]

vBuilletin makes some very inefficient queries, for one thing. Particularly it doesn't deal well with huge threads or members with huge numbers of posts.

I would start by deleting anyone with more than 40,000 posts.

[El_Spectre]

I've heard that reversing the polarity is more reliable.

Thats crazy talk! Tachyons! we need more tachyons!

[Terry]

Well,what about to start locking threads at 5000 range?There are some active.
Or even lower loke from 2000 or so...

There are 8 threads that are open with more than 5k posts. I'll ask for them to be... TERMINATED!

[Klimax]

There are 8 threads that are open with more than 5k posts. I'll ask for them to be... TERMINATED!

:dig: or :alien003:

[temporalillusion]

I help run a vB forum that isn't quite as big as this one (not as many posts), but has about the same number of active users and will range from 350 to over 700 active users at any given time.

If you want you can send me a PM and I can share a few things we've done to help tune things up and get the max performance out of our server.. we're running on a single box with 2 Xeon processors, 2GB of RAM and mirrored SCSI drives and a throttled connection, and we rarely get slowdowns so I think we've done a good job of maximizing our resources without going to extremes.

Though eventually you get to the point that you need more than one physical server, but without knowing more details it's hard to be more helpful. Have you determined what the bottleneck is, sounds like it's mysql?

Anyway, if you are interested Terry shoot me a PM, or I can share in-thread if you don't mind the technical stuff in here.

In response to Zep, it's not quite as bad as all that, throwing more hardware if done properly can make a huge difference, there are some truly massive forums, I am a moderator on one (though I'm not really active anymore) that has 4-5000 active members normally, and lots more during really active times, so the software does scale, it's just a harsh reality of having hundreds of thousands of people hitting refresh every 30 seconds on a server where the pages are by necessity dynamically generated.

You are right though about identifying the bottlenecks, in my optimization quests I talked to a few vB users who had done just that; they'd basically gone through and rewritten some of the SQL queries to be more efficient and a few of the pieces of the software. That's tough though, not everyone has that kind of time (let alone skill, writing great queries isn't easy) to dedicate to their forum.

[amb]

Thats crazy talk! Tachyons! we need more tachyons!

Captain! I've giving you every ounce of power this tub can muster. The engines will blow at any minute! We need more Berilium crystals captain Kirk!

[Macgyver1968]

We could just get rid of all the crazy people...that would free up a lot of bandwidth. Although I think there would only be like..5 people left. :)

[KoihimeNakamura]

Hm. Well, I suppose I can't offer much, as the only other software I know of is PhpBB which is... not something I'd eagerly recommend.

[temporalillusion]

Invision Power Board is the other "big" commercial forum software out there, and I don't think it's much better or worse than vB with respect to efficiency and scalability.

I assume the low hanging fruit has been handled already, stuff like making sure XCache is installed and configured to run properly, analyzing the mysql instance and tuning the cache values so that most of the common queries are cached, making sure there's no memory bottleneck (RAM is comparatively cheap). You can use Yslow for Firebug to check the pages and make sure that the site is using the browser cache effectively, we made some Apache changes from that to improve the caching performance, reduced the raw # of hits to the server.

You could also look at using lighttpd (or there's another newer one that's supposed to be even better but the name eludes me) instead of Apache, much lighter and more efficient. We haven't done this one yet.

There also seems to be quite a few add-ons to vBulletin here, some research into what each one does and how much load it might put on the server might be worthwhile, in case there's one that's causing an inordinate amount of load.

One other thing we've done to reduce the load on our server is use a content delivery network for all the static resources. All the forum images, avatars, CSS and JS files can be fairly easily relocated to a content delivery network and the settings changed in the control panel. This reduced the # of hits to our server by 50% or more, it was a pretty huge difference. Won't help directly if the problem is MySQL, but will help indirectly by reducing the amount of resources the http server is using. SimpleCDN is the one we used, very easy to use if you use their mirroring functionality.

[amb]

Hey!! Speak for yourself. We the inmates have taken over the asylum. :D

[arthwollipot]

Invision Power Board is the other "big" commercial forum software out there, and I don't think it's much better or worse than vB with respect to efficiency and scalability.

I assume the low hanging fruit has been handled already, stuff like making sure XCache is installed and configured to run properly, analyzing the mysql instance and tuning the cache values so that most of the common queries are cached, making sure there's no memory bottleneck (RAM is comparatively cheap). You can use Yslow for Firebug to check the pages and make sure that the site is using the browser cache effectively, we made some Apache changes from that to improve the caching performance, reduced the raw # of hits to the server.

You could also look at using lighttpd (or there's another newer one that's supposed to be even better but the name eludes me) instead of Apache, much lighter and more efficient. We haven't done this one yet.

There also seems to be quite a few add-ons to vBulletin here, some research into what each one does and how much load it might put on the server might be worthwhile, in case there's one that's causing an inordinate amount of load.

One other thing we've done to reduce the load on our server is use a content delivery network for all the static resources. All the forum images, avatars, CSS and JS files can be fairly easily relocated to a content delivery network and the settings changed in the control panel. This reduced the # of hits to our server by 50% or more, it was a pretty huge difference. Won't help directly if the problem is MySQL, but will help indirectly by reducing the amount of resources the http server is using. SimpleCDN is the one we used, very easy to use if you use their mirroring functionality.I think we've just had a volunteer to provide technical assistance. :D Someone give this guy admin powers, quick!

[Soapy Sam]

I don't understand why the post count of an individual poster being high would affect anything, but if dropping mine to zero and keeping it there would help, go for it.
If that actually means deleting every post I ever made, let me save the valuable ones........ok, done.. Pull the control rods.

[temporalillusion]

Lol, I'm just regurgitating a lot of what our server admin guy and I researched out and decided was good, so I had help. I'm not a linux guru by any means (which I hate, but never seem to have time to remedy).

Though I did compile and install XCache from source, so I was impressed with myself.

[temporalillusion]

I don't understand why the post count of an individual poster being high would affect anything, but if dropping mine to zero and keeping it there would help, go for it.
If that actually means deleting every post I ever made, let me save the valuable ones........ok, done.. Pull the control rods.

The actual post count # that's displayed is just a field in the database and won't affect anything.

The number of records can have an impact though, I haven't researched it but I assume there's ways to archive older posts out to improve things; my forum has 1/3 the # of posts here so I haven't had to worry about that yet.

One other thing I forgot to mention is gzip compression; make sure it's enabled somewhere (either in the forum itself, or through apache with mod_deflate), but make sure it isn't enabled in BOTH places! :D I had that so every page the forum generated was being compressed twice, which of course uses up resources needlessly. :)

[shadron]

The only thing worse than being a paid IT with a knowledgeable boss must be being an volunteer IT with about 60 knowledgeable members.

Why don't y'all quit jostling his elbows? If he had to read all your stuff he'd never be able to find the time to type his own SQL.

<duck and cover>

[temporalillusion]

Lol very true, I was just offering up the things we've done that have worked for us, I'm sure Terry's looked at 90% of what we went through already, thought he content delivery network thing is not very common for vBulletin forums yet so it's not a well known option.

[Terry]

I don't understand why the post count of an individual poster being high would affect anything,

It doesn't affect anything until someone hits "See more posts by Soapy_Sam", at which point a very long-running query is generated which examines every post you've ever made, sorts them by date, and shows the most recent 400.

but if dropping mine to zero and keeping it there would help, go for it.
If that actually means deleting every post I ever made, let me save the valuable ones........ok, done.. Pull the control rods.

I'd rather not start throwing away data. If that does happen, it'll be because JREF has decided it is the appropriate course, and most likely that will be against my recommendation.

[Klimax]

One other thing I forgot to mention is gzip compression; make sure it's enabled somewhere (either in the forum itself, or through apache with mod_deflate), but make sure it isn't enabled in BOTH places! :D I had that so every page the forum generated was being compressed twice, which of course uses up resources needlessly. :)

I suspect that for now it should be turned off completely,unless bandwifth is really restricted.

And additionaly ( ;) ) I suppose that sooner or latery it is going to be either hardware or rewrite bad SQL queries.(But with proper documentation of changes it should not be difficult to upgrade later)

[Mitchell314]

I'm just shooting ideas, but if there's a problem with users having many posts, would it be effective to split up really old posts to an archive sock puppet? So if I have 5000 posts older than a year, what about some task makes a sock puppet as close as possible to me and cuts&pastes those posts to the sock puppet (which would be Mitchell314_ or something)? And if I get 5000 more old posts, push them on the old sock puppet? Probably not too practical but I'm just throwing stuff out there. Beats banning the old members. Then again...

[temporalillusion]

I suspect that for now it should be turned off completely,unless bandwifth is really restricted.

I guess it depends on what is limited, it's not just bandwidth, it takes apache less time to transfer the gzipped resource, quite a bit less time for raw HTML pages. Typically it's worth the extra CPU to compress them.

[ejk]

We could just get rid of all the crazy people...that would free up a lot of bandwidth. Although I think there would only be like..5 people left. :)

Please, no, don't make us leave!

[Psi Baba]

It doesn't affect anything until someone hits "See more posts by Soapy_Sam", at which point a very long-running query is generated which examines every post you've ever made, sorts them by date, and shows the most recent 400.
Would it help to simply disable that feature?

[arthwollipot]

Would it help to simply disable that feature?I find this feature useful for researching particular posts that I remember seeing, but not which thread they are in. I'm in two minds - in one sense I would not want to see it disabled, rather we should all self-regulate and not use it to find posts by people with a large number of posts. But I'm not naive enough to beleive that this could possibly work.

[PixyMisa]

vBuilletin makes some very inefficient queries, for one thing. Particularly it doesn't deal well with huge threads or members with huge numbers of posts. The other thing is the server needs more RAM (and ideally to be a 64-bit kernel so that a single process can grab a larger amount of RAM too). Almost invariably when the "no DB connections left" messages start, it's because we're thrashing swap. I've tried various things to tune the db to use VM better, but once one of those giant "runs for several hundred seconds" queries kicks in, we usually end up dropping people.

ETA: if we didn't drop people, it probably wouldn't recover on its own, and it would be hung until rebooted (or until someone restarted Apache and/or the database, at least).
Yeah, it's what I call the LAMP death spiral.

Your server is trundling along more or less happily, then you get a query that takes longer than usual for whatever reason. Other queries back up behind it, which means that the Apache/PHP processes involved are sitting around taking up memory while they wait. As more requests come in, Apache starts up more processes to handle them, which takes up memory that would otherwise be used to cache the data from the database.

That means that MySQL has to hit the disk to fetch its data instead of just grabbing it from memory, which means the queries run even slower, and in a matter of minutes you can go from everything running fine to the server being so overloaded that you can't even log in to fix things.

The solution is exactly what you are proposing to do: Throw hardware at it.

64-bit is nice but not critical; the Linux filesystem cache can grow to 64GB (I think) even on a 32-bit system, and while it's not quite as efficient as MySQL's own buffers (assuming you're using InnoDB), it gets the job done.

There's one other silver bullet you can use, and that's a Fusion-io ioDrive. They're not cheap (start at around $3000) but they're about 1000 times faster than a regular disk drive. I'm working on a system that handles about 1.5 million posts a day, and we do all sorts of crazy indexing (full text indexes, geospatial indexing) and one ioDrive handles everything we throw at it without blinking.

[Soapy Sam]

I googled these last time you mentioned them. Very nice bit of kit. Presumably the price of these will fall in the months to come?

[PixyMisa]

Actually, the price has gone up. They're selling them as fast as they can churn them out, and the closest competition is a year away from shipping, so for the moment they have the market all to themselves.

Worth every penny, though.

[PixyMisa]

Why can't we copy what the biggest forums do ?
Crash a lot?

[temporalillusion]

There's one other silver bullet you can use, and that's a Fusion-io ioDrive. They're not cheap (start at around $3000) but they're about 1000 times faster than a regular disk drive. I'm working on a system that handles about 1.5 million posts a day, and we do all sorts of crazy indexing (full text indexes, geospatial indexing) and one ioDrive handles everything we throw at it without blinking.

Nice! You'd need a big array of SSD drives to even get close to what that thing can do.

Do you use Postgres? MySQL is nice but its geospatial isn't fully implemented which has driven me crazy from time to time (having to load a bunch of records inside a bounding box, then go through them again to find the ones that are actually inside the shape I really want kind of thing).

[Soapy Sam]

Yeah, it's what I call the LAMP death spiral.

Your server is trundling along more or less happily, then you get a query that takes longer than usual for whatever reason. Other queries back up behind it, which means that the Apache/PHP processes involved are sitting around taking up memory while they wait. As more requests come in, Apache starts up more processes to handle them, which takes up memory that would otherwise be used to cache the data from the database.

Apologies if the following is stupid. Feel free to educate me. I know 0 about databases.
Is there no way to recognise a request that will take too long and simply reject it with a polite apology? I'm not asking for a solution to Turing's halting problem, just a recognition that (eg) If that eedjit has 15000 posts and I'm busy --> then call Copout. Better the originator of the request is frustrated than a dozen innocents are dumped in the cybersoup.

As for the hardware upgrade debate- we've raised money before for servers and can doubtless do it again. Given some good folk here are in the business, I'd hope we could take advantage of any savings they might be able to offer. Makes sense if we can give Terry what he needs at minimal cost to everyone.

[Darat]

Remember that vBulletin is a commercial product and whilst tweaking around the edges so to speak is relatively straightforward major rewrites of the core product are anything but.

[Soapy Sam]

Ah. So it's not yours to mess with?
I'm wallowing in my own iggerance here, darat.

[roger]

Sure, they can mess with it.

But then the next version comes out, and they have to throw away all their hard work, or try to figure out how to integrate it into the new version. It'd be a nightmare.

[Z]

I wonder, would there be a way to acquire a second server solely for archived posts, say, those older than two months, or something like that? Can the software shunt search queries to another server, and let the other server handle the query load and feedback?

Or, more to the point, is there a way to do this without each member of the tech staff at JREF going to get Master's Degrees in database administration?

[Darat]

Not built-in. Archive management is probably one of vBulletin's weakest features, it really doesn't let you do anything like this.

[Soapy Sam]

I understand Terry's reluctance to ditch data- and I remember the sobs and gnashing of teeth last time we did so- but it does sound to me like a regular trim would would keep things more manageable.

[Zep]

Not sure I agree that throwing more hardware at the situation is unwarranted.True, but I like the analogy of trying to drive fast with a flat tyre. You can certainly keep speed up by shoe-horning a turbo-charged dragster engine under the hood. But a far better solution would be to fix the flat first and use your existing engine as specified.

Of course, that's optimization 101. But this is effectively shrinkwrap software. We are in a world of hurt if we start changing vBulletin, and I for one am not up for supporting a fork of a licensed commercial product just because we happen to be able to read the PHP.Ah, I see. Not a vBulletin guru myself, but I still take your point.

[Re: big tables]They aren't, but you have to look at all the posts in a thread to generate the thread display. Or at least look at an index containing all the posts.Shouldn't have to do that if the thread is indexed wisely. Even just having post-numbers as an index of some sort for a thread allows selecting specific posts only to be returned for display. You don't need the whole thread returned in one query each time. Added advantage is that most activity will be at the end of most threads, so only the last few posts will need to be cached the most. As seen above, the SQL queries can stand to be reviewed! :)

But of course, there's always time and money issues to overcome too - appreciate that fully! Keep up the good work!

[technoextreme]

It doesn't affect anything until someone hits "See more posts by Soapy_Sam", at which point a very long-running query is generated which examines every post you've ever made, sorts them by date, and shows the most recent 400.

It seems like this is a five minute fix. All you have to do is disable that feature. Can anyone honestly say that they care what Soapy_Sam's first post was? Also, is the problem really that big of a deal as it is right now? So what if I can't make that post at that exact instant because someone is trying to figure out Soapy_Sam's first post.

[PixyMisa]

Nice! You'd need a big array of SSD drives to even get close to what that thing can do.
A big array. :)

Do you use Postgres? MySQL is nice but its geospatial isn't fully implemented which has driven me crazy from time to time (having to load a bunch of records inside a bounding box, then go through them again to find the ones that are actually inside the shape I really want kind of thing).
We're using MySQL. We do that sort of filtering on some stuff, but 99% of the time the bounding box is good enough for what we do.

[The Bad Astronomer]

There's one other silver bullet you can use, and that's a Fusion-io ioDrive. They're not cheap (start at around $3000) but they're about 1000 times faster than a regular disk drive. I'm working on a system that handles about 1.5 million posts a day, and we do all sorts of crazy indexing (full text indexes, geospatial indexing) and one ioDrive handles everything we throw at it without blinking.

Holy Haleakala! That's amazing. $3k is a fair bit of cash, but wow. Thing is, we have a web host, so they'd have to support it. Not sure they do, but I'll ask Jeff to look into this.

Bigger and better hardware is the route we'll probably take. I have no desire to prune out posts, and there's no need. Computers tend to get faster with time (give me Moore!), so I'm sure we'll find what we need. :)

[Skeptic Ginger]

I would start by deleting anyone with more than 40,000 posts.I suggest we clear the slates on all posters with over 17,000 posts. They can start fresh, no one would care. :)

[The Central Scrutinizer]

I suggest we clear the slates on all posters with over 17,000 posts. They can start fresh, no one would care. :)

Request denied.

[Skeptic Ginger]

Hah! As if you had any power.

[Klimax]

Holy Haleakala! That's amazing. $3k is a fair bit of cash, but wow. Thing is, we have a web host, so they'd have to support it. Not sure they do, but I'll ask Jeff to look into this.

Bigger and better hardware is the route we'll probably take. I have no desire to prune out posts, and there's no need. Computers tend to get faster with time (give me Moore!), so I'm sure we'll find what we need. :)

Hm.What specs of new server would be/are?

my guess/solution(hypotetical due to its price):
Core i7 965 EE
6GB RAM Corsair TR3X6G1600C8D
RAID 6 of 4 SCSI HDD (300GB each ,10k RPM)
(motherboard not specified)

or
Xeon Processor X7460
6GB RAM Corsair TR3X6G1600C8D
RAID 6 of 4 SCSI HDD (300GB each ,10k RPM)
(motherboard not specified)

Unfortunately I suspect that such perfect server would be away too expensive... :( (Don't know who would be able to afford such beast)
On the other hand,it would last for more then eight years.(three cores apache,three cores xSQL;same way memory)

[PixyMisa]

Technically, the server versions of the Core i7 aren't out yet, though there's nothing to stop you building a server out of a desktop CPU. But those configs don't have nearly enough memory.

I'm not sure how big the database is. If it's in the 4GB range, then the solution is pretty simple: A pair of quad-core servers, each with 8GB, one to run the database, the other to run Apache and PHP. That's the most cost-effective way to run this sort of system.

If the database is approaching 8GB, though, you'll want more than 8GB on that server, and that probably means moving to a dual-processor box, which is obviously going to be more expensive.

I don't know where the forums are hosted, but... Okay, now I do. Reliable, but not cheap.

[Klimax]

Technically, the server versions of the Core i7 aren't out yet, though there's nothing to stop you building a server out of a desktop CPU. But those configs don't have nearly enough memory.


Depends.I do not know how memory hungry apache with PHP is,but I would say that it would be so far enough.Since the only limitation for desktop PC regarding memory is lower number of modules and current memory density.And missing ECC and buffering for server role.

Thats why I included previous generation Xeon... ;)

BTW.Core i7 is already partially meant for database servers.(At least according to some reviews I read.)

[PixyMisa]

Yep, Core i7 is designed more as a server chip than a desktop chip; I'm eagerly awaiting the server versions.

Apache/PHP can chew through memory like nobody's business. In the lead up to the election, my web server was running 200-250 Apache/PHP processes, each using 10-20MB of memory. And that's after I did everything possible to reduce memory usage, on a server that doesn't run a lot of complicated PHP anyway; before that, the processes were commonly using up to 50MB each, and sometimes more.

That's why it's so valuable to split off the database onto a separate box: Even if Apache gobbles up all the memory on its own box, the database itself is unaffected. Doesn't solve the problem, but it goes a long way to mitigate it.

On the MySQL side: If you want good performance from your database, you want the whole thing in RAM. Particularly if you're dealing with things like full-text indexes, which can arbitrarily haul tens of thousands of pages off disk in response to the simplest query.

[Blue Mountain]

Well, I'm certainly in the company of giants here. My company runs a small commercial web site (we sell the service and use the web to provide it) that gets an average of 60,000 page views per month. Contrast that with this forum, which gets 2.5 million.

About the only thing I can add is you really do not want to use commodity desktop hardware for even the small website we host, let alone this forum. I have no idea what the JREF is using, but we've been using HP ProLiant servers for our sites. We've been very happy with them. Linux runs really well on these guys (including the RAID array because HP open-sourced the driver code.)

I also don't know what the JREF's budget looks like, or how much priority they give to the forum. Good servers aren't cheap--you're looking at $2,000 to $3,000 for even a low end system. So buying two of them (one for the forum software, one for the database) is certainly an investment. And that's before they end up on Terry's desk with a note from Phil asking how long it will take to get them set up and installed at the data centre. :eek:

If our site ever gets up to 2 million page views a month, I'd sure be looking at splitting the database off to a separate server. We'll be a while getting there, though.

[Jeff Wagg]

Thanks for all the input. We're looking at quotes now, and hopefully we'll be setting up a new server this week. :)

[arthwollipot]

Awesome. Good luck. I assume there'll be a bit of downtime?

[Soapy Sam]

Getting the cash for the last server upgrade was a fairly painless operation, though ready cash is admittedly tighter for many members these days.
Also it's clear we have enough members in various areas of IT that we could get some favourable deals.
I do hope the offers of assistance in that regard are taken up. I will certainly contribute to an upgrade fund, but I'd hate to think we end up paying top dollar for a machine we might have had at a discount. It's a buyers market these days- or should be.

[temporalillusion]

Take pictures of the new server! :D

When our forum outgrew a VPS solution and we had a drive to raise $$ for a new server, which went well, and we bought it. I took pictures for everyone.

I sent it off to the colocation facility then, which is far away from where I am (couldn't find a decent local colocation facility).

I then posted a picture of the building that it was going to end up at (found one on the 'net).

The funny thing was one day the server goes down, site's not accessible. Neither is the colocation company site! Uh oh. Call them, line is busy, busy, then get through but no answer, no voicemail, nothing. Now I'm worried.

I can't remember what led me to it but at one point I ended up at the local city website for where the colo was and right there is a "breaking" news story with a picture of a building.. the building my server was in... and there was FLAMES coming out of the top of the building!!

That would explain why the site wasn't up lol.

Anyway it ended up being a very minor fire on the roof, power was restored and the site came back up.

But it was quite fun to revive the new server thread with all the pictures and post a picture of the colocation building being on fire. :)

[Terry]

Shouldn't have to do that if the thread is indexed wisely. Even just having post-numbers as an index of some sort for a thread allows selecting specific posts only to be returned for display.

It is indexed on postid (and several other things). I said "or at least look at an index containing all the posts". Which you do.

[rjh01]

If our performance problems can be solved by spending < $5,000 then I say it would be easy to raise money for it. Just put out a request to us to donate money. Just make sure that it is plain that all money raised would be extra money to buy and maintain the hardware and software for this forum and no money would go elsewhere, e.g. to JREF.

JREF <> FORUM.

[Zep]

It is indexed on postid (and several other things). I said "or at least look at an index containing all the posts". Which you do.I took that to be the case. :)

How about one of these wot I get to play with daily: http://h20195.www2.hp.com/V2/GetPDF.aspx/5982-9830EN.pdf

64 processor Integrity system with 2TB of shared memory.

:boxedin:

[Blue Mountain]

I took that to be the case. :)

How about one of these wot I get to play with daily: http://h20195.www2.hp.com/V2/GetPDF.aspx/5982-9830EN.pdf

64 processor Integrity system with 2TB of shared memory.

:boxedin:

Meh. Go big or go home :D http://www.cray.com/products/CX1.aspx

Actually, what you have does look like a nice machine. How many VMs are you running on it? Which operating systems?

[Klimax]

Another idea.Move to archive about 2/3 of old threads from each section.

[tomwaits]

I'm certainly not an expert on any of this, but couldn't we just eliminate the search function and just keep the google search?

[geni]

I'm certainly not an expert on any of this, but couldn't we just eliminate the search function and just keep the google search?

No because the search function does a bunch of things google doesn't.

[tomwaits]

No because the search function does a bunch of things google doesn't.

Maybe, but I think a nice cost-benefit analysis would be good here. The search function may be able to do a few nifty things, but it's very slow and notoriously unreliable. I would be willing to give up any benefits the search function has if it would stop these server issues for good, since there is already a search function supplied by google.

[Terry]

I took that to be the case. :)

How about one of these wot I get to play with daily: http://h20195.www2.hp.com/V2/GetPDF.aspx/5982-9830EN.pdf

64 processor Integrity system with 2TB of shared memory.

:boxedin:

If you're buying, sure :)

[ElMondoHummus]

64-bit is nice but not critical; the Linux filesystem cache can grow to 64GB (I think) even on a 32-bit system, and while it's not quite as efficient as MySQL's own buffers (assuming you're using InnoDB), it gets the job done.


Wait... I'm afraid I'm missing something here, and if I am, I'm open to correction. But: How can the Linux filesystem cache grow to 64 gigs on a 32 bit system? If we're talking about cache residing in RAM, then the 4 GB limitation still applies because we're (presumably) talking about x86 hardware, and that limit is a physical one imposed by the addressing scheme, not an software one imposed by the operating system. So unless there's some funny RAM addressing tricks I'm unaware of, it still applies. Are there some tricks to get around the limitation? Or are we not talking about caching to RAM here?

Sorry for the digression, but I'm genuinely curious as to whether I'm misunderstanding this statement or not.

[PixyMisa]

Wait... I'm afraid I'm missing something here, and if I am, I'm open to correction. But: How can the Linux filesystem cache grow to 64 gigs on a 32 bit system? If we're talking about cache residing in RAM, then the 4 GB limitation still applies because we're (presumably) talking about x86 hardware, and that limit is a physical one imposed by the addressing scheme, not an software one imposed by the operating system. So unless there's some funny RAM addressing tricks I'm unaware of, it still applies. Are there some tricks to get around the limitation? Or are we not talking about caching to RAM here?
There are indeed funny RAM addressing tricks. It's called PAE (physical address extension). Although any one program can still only see 4GB of memory, the operating system can support up to 64GB in total.

This has actually been around since the Pentium Pro, so it's widely supported. It even works in Vista, if you know the right buttons to push, and don't have any non-PAE drivers installed (which you almost certainly do, which is why it's turned off by default).

[Cuddles]

Maybe, but I think a nice cost-benefit analysis would be good here. The search function may be able to do a few nifty things, but it's very slow and notoriously unreliable. I would be willing to give up any benefits the search function has if it would stop these server issues for good, since there is already a search function supplied by google.

The Google search cannot see the members only areas.

[six7s]

The Google search cannot see the members only areas.Oh, really?

Then how do you explain the following?

Google: Results 1 - 2 of 2 for "Aquarium of Blood Game". (0.44 seconds) (http://www.google.com/search?hl=en&safe=off&q=%22Aquarium+of+Blood+Game%22&btnG=Search)
JREF Forum
11 Feb 2009 ... Aquarium of Blood Game Thread. by Damien Evans. Today 06:17 AM Go to
last post. 23528, 738675. Humor · Continuation-squared: The Three Word ...
forums.randi.org/forumindex.php - 58k - Cached - Similar pages -

Members Only - JREF Forum
Aquarium of Blood Game Thread. by Giggywig. Today 06:39 PM Go to last post. 23428,
724562. Humor · I think I have you now... by The Drain ...
forums.randi.org/forumdisplay.php?s=&daysprune=&f=20 - 30k - Cached - Similar pages -
More results from forums.randi.org »
In order to show you the most relevant results, we have omitted some entries very similar to the 2 already displayed.

[Ashles]

I'm up for chipping in if new hardware would help.

I would have thought a fair amount could be raised pretty quickly.

Surely at least 300+ forum members would be happy to donate at least $10.

Between 4 and 6pm GMT every day I see the same "Too many connections" error message (or whatever it specifically says).
I would pay to not see that any more.

[Klimax]

Oh, really?

Then how do you explain the following?

Google: Results 1 - 2 of 2 for "Aquarium of Blood Game". (0.44 seconds) (http://www.google.com/search?hl=en&safe=off&q=%22Aquarium+of+Blood+Game%22&btnG=Search)

Simple.It indexed forumindex.php,which contains some info about last thread per section including members-only.If you tried to search for any threads inside the members-only area or content of them,you would find that google does not see them.

[Soapy Sam]

I don't believe I ever found anything I set out to find using the Search utility.
Anyone else similarly underwhelmed?

[arthwollipot]

I don't believe I ever found anything I set out to find using the Search utility.
Anyone else similarly underwhelmed?I've used Search on a number of occasions. Sometimes it finds what I want, sometimes it doesn't.

[rjh01]

So can we ask Darat to throw away the search index? Just keep Google and the user ids? If you want to do a search of members only area then your only options are
1. use tags
2. new posts
3. Search on user ids?

[six7s]

Or edit the robots.txt file (or whatever) and let Google index the members area

[rjh01]

I do not like the idea of Google indexing the member's area. Because non members will then do searches that come up with threads in the members only area. Then be able to read the thread via the cached link (http://209.85.173.132/search?q=cache:wz2fHGe57GAJ:forums.randi.org/showthread.php%3Ft%3D134768+%22Recent+server+issue s%22&hl=en&ct=clnk&cd=1&gl=au). Either that or get an error when they click on the link.

Look how old that cached page is! (done 5 Feb and it is now 19 Feb!)

[six7s]

Or they could simply register and read the whole damn lot

Do you have a point?

[rjh01]

I do not think it very good for the forum's reputation if people attempt to get to the forum via google and then get an error message. Not much point in having a members only area if you do not have to register to read it.

[six7s]

Is that a convoluted way of saying 'no'?

[Cuddles]

Or edit the robots.txt file (or whatever) and let Google index the members area

I think the reason Google can't index the members only sections is because Google spiders can't log in. In order to make everything visible to Google, it would all have to be visible to everyone.

I don't believe I ever found anything I set out to find using the Search utility.
Anyone else similarly underwhelmed?

I use it all the time when moderating. It's not perfect, but it works fine the vast majority of the time.

[six7s]

Or edit the robots.txt file (or whatever) and let Google index the members areaI think the reason Google can't index the members only sections is because Google spiders can't log in. I think, in the absence of substance in your statements, that there's a distinct possibility that you're wrong

http://www.randi.org/robots.txt
User-Agent: *
Disallow: /latexrender/
Disallow: /forumlive/ajax.php
Disallow: /forumlive/attachment.php
Disallow: /forumlive/calendar.php
Disallow: /forumlive/cron.php
Disallow: /forumlive/editpost.php
Disallow: /forumlive/global.php
Disallow: /forumlive/image.php
Disallow: /forumlive/inlinemod.php
Disallow: /forumlive/joinrequests.php
Disallow: /forumlive/login.php
Disallow: /forumlive/member.php
Disallow: /forumlive/memberlist.php
Disallow: /forumlive/misc.php
Disallow: /forumlive/moderator.php
Disallow: /forumlive/newattachment.php
Disallow: /forumlive/newreply.php
Disallow: /forumlive/newthread.php
Disallow: /forumlive/online.php
Disallow: /forumlive/poll.php
Disallow: /forumlive/postings.php
Disallow: /forumlive/printthread.php
Disallow: /forumlive/private.php
Disallow: /forumlive/profile.php
Disallow: /forumlive/register.php
Disallow: /forumlive/report.php
Disallow: /forumlive/reputation.php
Disallow: /forumlive/search.php
Disallow: /forumlive/sendmessage.php
Disallow: /forumlive/showgroups.php
Disallow: /forumlive/subscription.php
Disallow: /forumlive/threadrate.php
Disallow: /forumlive/usercp.php
Disallow: /forumlive/usernote.php
Disallow:/forumlive/vbimagehost.php


User-Agent: Slurp
Disallow: /forumlive/

User-Agent: Referer Karma/2.0
Disallow: /forumlive/




In order to make everything visible to Google, it would all have to be visible to everyone.Maybe you ought to read up on White Hat and Black Hat SEO (http://www.google.com/search?hl=en&lr=&as_qdr=all&q=%22white+hat%22+%22black+Hat%22++SEO&btnG=Search) and then think again

[PixyMisa]

I think, in the absence of substance in your statements, that there's a distinct possibility that you're wrong

http://www.randi.org/robots.txt
robots.txt has nothing to do with it.

To retrieve the members-only pages, the Google spider would have to log in to the forum, just like anyone else.

[Klimax]

I think, in the absence of substance in your statements, that there's a distinct possibility that you're wrong

http://www.randi.org/robots.txt





Maybe you ought to read up on White Hat and Black Hat SEO (http://www.google.com/search?hl=en&lr=&as_qdr=all&q=%22white+hat%22+%22black+Hat%22++SEO&btnG=Search) and then think again


You are wrong.First even if restriction would be lifted from robots.txt,indexer would have to login,since vB does require it before sending content of area.There is no way around.
Then robots.txt is in www.randi.org,so does not apply to forums.randi.org,so unless there is transparent redirect to /forumlive/ then it does not matter for this forum.
And last, community forums and regular forums are distinguished only by associated permissions and attributes in database,not by/in php files!(applies for initial proccessing,once info from db retrieved,php will differentiate)

And forums.randi.org does not have any robots.txt associated...

[six7s]

Thanks...

I have just swallowed a rather nourishing and surprisingly tasty slice of humble pie

[Miss_Kitt]

Let me say, on behalf of several in the Forum:

What?!

But that's okay, we assume the tech-savvy are arguing about something meaningful.

Just tell us how much money to send where...

Thanks, MK

[Klimax]

Let me say, on behalf of several in the Forum:

What?!

But that's okay, we assume the tech-savvy are arguing about something meaningful.

Just tell us how much money to send where...

Thanks, MK

:D
Money:50.000$ to randi.org and myself...

[cj.23]

Thanks for all the input. We're looking at quotes now, and hopefully we'll be setting up a new server this week. :)

That is fantastic news. I have just had the longest period I have not been able to log on - every attempt over a twelve hour period failed, and I have finally got on board at 2.26am when I need to go to bed! Oh well! Anyway congrats on the new server, and hope all goes well during the install. :)

cj x

[Darat]

There will be an update on the problems very soon (days).

[PixyMisa]

Was that it?

Server is running much faster now, but it's not time yet...

[Darat]

Not yet - there will be some downtime while the live data is transfered to the new server.

[PixyMisa]

Okie dokie. :)

I thought it was a surprisingly smooth transition...

[Klimax]

There will be an update on the problems very soon (days).

Looks like you and Terry are starfleet engineers.Always working faster then their estimates... ;)

[Darat]

Not me - all the credit belongs to Terry.

[Doubt]

Okie dokie. :)

I thought it was a surprisingly smooth transition...

Well, it appears to have been a very smooth transition.

I think they need to go back and muck it up once or twice. Otherwise they may give us unrealistic expectations for future changes. They cannot go on building our hopes up for future transitions by having things work out this well! No good can come from that!

:D

[Klimax]

Terry,it looks like you are starfleet engineer!You worked faster then your estimate!!!

And server is fast!!!:eek::cool::D
What specification of server is?
Anyway congratulation. Wish every server transfer/upgrade/sw upg went this smooth...

[A W Smith]

oops its happening again

[Terry]

I'm still tweaking the server configuration. Bear with me, there might be issues today while I get all the wrinkles ironed out.

[DoubtingStephen]

I can tell you that this is getting a great deal of attention from Terry and he is very well suited to this task.

[Cleon]

I can tell you that this is getting a great deal of attention from Terry and he is very well suited to this task.

Sorry! You can have him back when he's done.

[HawaiiBigSis]

I can tell you that this is getting a great deal of attention from Terry and he is very well suited to this task.

And we truly appreciate him giving up yet another Sunday to try to make this work.

[arthwollipot]

Absolutely! I noticed that the UserCP was much faster to load this morning.

[rjh01]

Where am I? It looks like the JREF forums. However it does not feel like it. Everything is so much faster, so something is wrong. Will it stay that way?

[amb]

Scotty,[Darat] I need more power. Captain, She's giving all she's got. Anymore, and the engines will blow! :p

[Cuddles]

The big problem I can see here is that with the forum working properly, people are going to start using it more. This will inevitably cause more trouble down the line, and any later fixes will be just as temporary as this one. In order to sort this out once and for all, I recommend just banning everyone, deleting the forum database and leaving the poor server to go about its business without being bothered by this internet thing.

[Darat]

Terry,it looks like you are starfleet engineer!You worked faster then your estimate!!!

And server is fast!!!:eek::cool::D
What specification of server is?
Anyway congratulation. Wish every server transfer/upgrade/sw upg went this smooth...

Quad Processor Quad Core Intel 7320 - 2.13GHz (Tigerton) - 4 x 4MB cache,
8 GB FB-DIMM Registered 533/667, 74GB SATA Raptor 10k, 73GB SA-SCSI 10K RPM, 1x Terry managment system

[JimBenArm]

I told you shoving a weiner in it would make it work better. But does anyone ever listen to me? Nooooo!!!!!!!!!

[Blue Mountain]

Quad Processor Quad Core Intel 7320 - 2.13GHz (Tigerton) - 4 x 4MB cache, 8 GB FB-DIMM Registered 533/667, 74GB SATA Raptor 10k, 73GB SA-SCSI 10K RPM, 1x Terry managment system
You should have held out for an upgrade to the management system; the current 1x model has been under considerable strain lately.

Great work, Terry! I'm responsible for the servers in a small I/T business, which includes a website that gets 1/40 the traffic this one does. And it keeps me busy.

[Klimax]

Hm.What specs of new server would be/are?

my guess/solution(hypotetical due to its price):
Core i7 965 EE
6GB RAM Corsair TR3X6G1600C8D
RAID 6 of 4 SCSI HDD (300GB each ,10k RPM)
(motherboard not specified)

or
Xeon Processor X7460
6GB RAM Corsair TR3X6G1600C8D
RAID 6 of 4 SCSI HDD (300GB each ,10k RPM)
(motherboard not specified)

Unfortunately I suspect that such perfect server would be away too expensive... :( (Don't know who would be able to afford such beast)
On the other hand,it would last for more then eight years.(three cores apache,three cores xSQL;same way memory)

Quad Processor Quad Core Intel 7320 - 2.13GHz (Tigerton) - 4 x 4MB cache,
8 GB FB-DIMM Registered 533/667, 74GB SATA Raptor 10k, 73GB SA-SCSI 10K RPM, 1x Terry managment system

Interesting.I was not that far away... :D (For sure I was nearer then most of psychics predictions...)
This should keep server-side software for some years operating.

But I am afraid that one component(bolded) will wear down fast and its starting condition is not so great..

[tomwaits]

Whatever it is you did, it worked like a charm. The forum is moving quite smoothly.

[rjh01]

The big problem I can see here is that with the forum working properly, people are going to start using it more. This will inevitably cause more trouble down the line, and any later fixes will be just as temporary as this one. <snip>

I think you are right. The forum stats from the new server era onwards will be interesting. I wonder how long before another thread is started complaining about the slow forum due to the increased traffic?

[El_Spectre]

I think you are right. The forum stats from the new server era onwards will be interesting. I wonder how long before another thread is started complaining about the slow forum due to the increased traffic?

It's a good point. In a related note, someday, we all will die. Perhaps I should go dig a hole and lay down...

[ElMondoHummus]

I told you shoving a weiner in it would make it work better. But does anyone ever listen to me? Nooooo!!!!!!!!!

They were talking about computer servers, pervert, not... er... uh... is it bad taste to name family members here?? :o

[ElMondoHummus]

Quad Processor Quad Core Intel 7320 - 2.13GHz (Tigerton) - 4 x 4MB cache,
8 GB FB-DIMM Registered 533/667, 74GB SATA Raptor 10k, 73GB SA-SCSI 10K RPM, 1x Terry managment system

Wait... you replaced Terry with another Terry?? This wasn't part of the deal! I strenuously object!!! I want the original Terry!!!!

;)



... or are you saying that Terry only runs at 1x? :D


:boxedin:

[Hindmost]

I'm still tweaking the server configuration. Bear with me, there might be issues today while I get all the wrinkles ironed out.

Thanks Terry...

glenn:)

[temporalillusion]

Very smooth and professional transition!

[Blue Bubble]

May I ask what the old server configuration was ?

And is it up for grabs ? ;)

[PixyMisa]

Quad Processor Quad Core Intel 7320 - 2.13GHz (Tigerton) - 4 x 4MB cache,
8 GB FB-DIMM Registered 533/667, 74GB SATA Raptor 10k, 73GB SA-SCSI 10K RPM, 1x Terry managment system
I know that configuration. :D

We have one of those at my day job, with the same hosting company (excellent pick by the way, I've been using them since 2006).

Ours has the exact same CPUs but 96GB of memory.

[Klimax]

I know that configuration. :D

We have one of those at my day job, with the same hosting company (excellent pick by the way, I've been using them since 2006).

Ours has the exact same CPUs but 96GB of memory.

You maximalists...

[Darat]

...snip..

Ours has the exact same CPUs but 96GB of memory.

Is that all.... :)

[Darat]

May I ask what the old server configuration was ?

And is it up for grabs ? ;)

See this link (http://www.jupiter-ace.co.uk/) and no.

Seriously I don't know the spec of the old server for certain, it was upgraded a couple of times.

[The Bad Astronomer]

Just to let y'all know, we found a minor domain name server issue that is now resolved, but may take a few hours to propagate around the planet. You may experience a temporary inability to access the main site, but I don't expect it to last long.

[rjh01]

Well, if no one else is going to say it, I will.

Thank you The Bad Astronomer for keeping us informed.

[amb]

Seconded. Only I believe Phil Plait is a very good astronomer. So why this Bad Astronomer name tag? It's like calling Darwin The Bad Zoologist. :)

[Travis]

Seconded. Only I believe Phil Plait is a very good astronomer. So why this Bad Astronomer name tag? It's like calling Darwin The Bad Zoologist. :)

Well one-with-the-pleasantly-distracting-avatar Phil is bad in certain ways.;)

He was in the skepdude calendar after all.

[temporalillusion]

Seconded. Only I believe Phil Plait is a very good astronomer. So why this Bad Astronomer name tag? It's like calling Darwin The Bad Zoologist. :)

Oh back before Phil got all famous and important (:D) he had his Bad Astronomy website where he'd talk about bad astronomy in movies, or debunk stupid people's crap (moon landing hoax, etc). So of course a Bad Astronomy website requires a Bad Astronomer to man it.

I must admit that waaay back then I had never even heard of people that denied the moon landings.. when I started reading the BA site and found out about that kind of thing it started me on the road to critical thinking and such.

So I can say that the BA site and the BA himself has had a positive impact on at least one person :)

[six7s]

Since the upgrade, everything has been loading v quickly

Thanks Terry!

However...

In the last 16, maybe 20 hours or so, I have been getting a few time-outs on this site (and no others)

I dunno why...

Jus' thought I'd mention it here

[Lrrr]

I've experienced the same thing, right around my lunch time.

[Toke]

Same here, like before the changeover it happens in the evening from 2000-2300.
(it´s 20.32 now)

[Darat]

Are you seeing a database error page?

[Toke]

no

[six7s]

Are you seeing a database error page?No, I'm not either...

Rather, the relevant tab in my browser simply 'hangs'

[HawaiiBigSis]

That happened to me yesterday too.

And I had a couple of incidents today where graphics didn't load. I had the little x-boxes where anything other than words would have been.

And some slow-loading incidents, but eventually we got there. But no "database unavailable" errors.

[six7s]

No problems re loading/hanging for the last five hours :)

:confused: One curiosity: the favicon is absent in both my (NN 9.0.0.6) address bar and tabs ( I mentioned this to Terry a while back... maybe he knows why)

[temporalillusion]

I have seen database errors, if I see it again I'll jot down what it said (though vB should generate an email for every DB error, if that option is turned on anyway).

[rjh01]

I think the server was not responding for about 30 minutes until about 30 minutes ago. I know it was not me because I checked new posts and no one posted anything for about 30 minutes.

But I do agree everything is much better recently.

[arthwollipot]

I got timeout messages for a bit there, but no database errors. Working now.

[Klimax]

I got just minutes ago standard non-vB error message (408/9 server busy)
And every day cca (GMT+1) 17-23h forum is loading slowly.Other times it is good.(23-7 untested)

[rjh01]

OK, can I be the first to say it? We need a bigger server.

[Klimax]

OK, can I be the first to say it? We need a bigger server.

Not correct.

Let me be first to say it:We need TWO servers.

[rjh01]

OK I concede the point. Maybe one for members area and the other for the public area?

[Klimax]

OK I concede the point. Maybe one for members area and the other for the public area?

And crash them both? ;):D

Seriously,one for webserver and one for database.

[Skeptic Ginger]

It happened to me late nite before last, for a few minutes. It would have resulted in a lost post had I not had it saved on Notepad. When I went to post the screen hung as has been described then a couple times simply timed out. This went on for about 10-15 minutes then everything returned to normal.

[six7s]

About half an hour ago - for maybe 2 minutes, I encountered three 'The connection has timed out' errors

[Darat]

See this thread (http://forums.randi.org/showthread.php?t=139814) for some information regarding recent performance issues.

[six7s]

Possible server downtime approx 00:15 to 00:22 GMT

[amb]

Is this the resurrection thread? :)

[rjh01]

See this new thread for recent discussion http://forums.randi.org/showthread.php?p=4816716#post4816716

[six7s]

When trying to load forums.randi.org/showthread.php?t=124603&page=313 (http://forums.randi.org/showthread.php?t=124603&page=313)

I keep getting the following error:
Fatal error: Maximum execution time of 300 seconds exceeded in /home/forumsr/public_html/includes/class_bbcode.php(344) : eval()'d code on line 15

The previous page - forums.randi.org/showthread.php?t=124603&page=312 (http://forums.randi.org/showthread.php?t=124603&page=312) - loads as per normal

ETA:

This happens both when I'm logged in and a 'guest'

[Toke]

I believe you are talking about "the evidence for the new testament".
I have a similar problem with it, I assume the stupidity there have finally reached critical mass. :)

[six7s]

I believe you are talking about "the evidence for the new testament".
I have a similar problem with it, I assume the stupidity there have finally reached critical mass. :)It was my fault! Using (too) many nested quotes, I br0ke teh forum softwhere!!11!!

[rjh01]

Congratulations six7s on breaking the software. Only good people do that sort of thing. I hope you do not do that sort of thing again.

[Toke]

It was my fault! Using (too) many nested quotes, I br0ke teh forum softwhere!!11!!

I am impressed:D, try not to do it again.