Archive for the 'Server' Category

Page 2 of 2

Major server problems.

Published on 27th of March, 2007 in Server. 5 Comments

[17:38 GMT] It seems crazed daemons have generated several gigabytes of log files. These have been deleted, so the ‘disk full’ errors are gone. Still working on the other issues though.

[23:23 GMT] The `servers` table has been repaired, and the online status system is operational again.

[23:17 GMT] Hmm. Seems the MySQL server got upset – “Table './avonline/servers' is marked as crashed and should be repaired.” That’d be why the online status system is “down or overloaded.”

[22:48 GMT] It seems that the cron daemon decided not to start at boot. This has screwed up all the stats. Gah!

This server is having serious issues. Random processes have taken to segfaulting, it thinks it’s been up for a negative amount of time, and it’s apparently executing, on average, 0.08 microqueries per second. Or roughly two per year.

Daemons refuse to start for no good reason too.

This can be traced back to someone deciding it would be a good idea to screw around with /dev. I have updated the kernel and uninstalled udev. The former made the machine bootable (into something other than into a single user system), and the latter took the load average down significantly. It’s still unreasonably high though.

I have no idea what’s wrong with it, and frankly I’m tired of messing with it. Anyone have any ideas?

Stats site 404s

Published on 21st of March, 2007 in Server and Statistics. 0 Comments

[17:20 GMT] This is now resolved.

I am aware that stats.katharineberry.co.uk is giving 404 errors when you try and view data for certain sims; mostly private islands. This is happening because the links to those pages didn’t previously exist. As part of the database load reduction yesterday, I have removed the checking whether these links should be generated in the first place; this has resulted in a roughly 10,000% speed improvement on generating that page.

No data has been lost where those 404s were, it’s just sending you to a page that never existed, but wasn’t linked to before.

I have an idea for fixing this without having to hammer the database, and (assuming it works) will implement this tonight.

I’m also aware that nobody actually cares enough for it to be worth me making these posts in the first place.

[UPDATED] Server issues resolved

Published on 20th of March, 2007 in Server and Statistics. 1 Comment

[21:30 GMT] The issues are resolved, I think. Woo!

[21:15 GMT] Most parts of the stats gathering are up again; still working on the individual sim pages (sim graphs are fine) and the average stats graph. I think it might be dumping data too… >.<

[20:59 GMT] HUD Thingies have been re-enabled. Work on the database to avoid excessive load is ongoing.

[20:31 GMT] The problem is not currently occuring, but the system causing it to happen has been disabled; that is, the one that regenerates the pages on the stats site. I’m working on it.

[19:20 GMT] The problem has been located and is being investigated.

[19:03 GMT] The problem is happening again. Still looking into it.

[18:54 GMT] After several hours of maintenance, the server is back up. “Maintenance” meaning updating things and reconfiguring things.

Performance data during the downtime has been lost, but SL data has been logged successfully. This shows that the performance logger is bugged; this will be looked into. (It’s supposed to keep the data whenever it’s recieved, regardless of whether it can be processed yet.)

I’ll keep an eye on it and see if it breaks. Fingers crossed it’s been restored now! >.>

Server Problems

Published on 19th of March, 2007 in Server. 1 Comment

This server is having… issues. MySQL and FTP related issues.

Due to the incessent crashing, I have written a script that runs every ten minutes and checks the MySQL server. If it crashed then it’s restarted. As I type this I’m watching an updated MySQL compile in the other window, and will be updating various other services too.

How to check my server’s health in two steps:

  1. Visit http://blog.katharineberry.co.uk. If you get a WordPress error about databases, the MySQL server crashed.
  2. Visit http://katharineberry.co.uk. Check the “load average” figures for “this server”. If the first number is around 10-11, something’s broken (it shouldn’t really be above 2). If the first number is significantly lower than the other two the server is just recovering. If it’s significantly higher than the other’s it’s just about to break.

[07:26 GMT] “ctype-eucjpms.c:8472: internal compiler error: Segmentation fault”

So much for that then.

Another crash

Published on 16th of March, 2007 in Server. 1 Comment

The server just crashed rather horribly for unknown reasons related to the FTP server. And then crashed again for good measure. Seems to be okay.. I’ll be checking it for a while.

Oops…

Published on 14th of March, 2007 in Server. 0 Comments

Looks like the site crashed while I wasn’t looking. Probably because of this post.

Anyway, it’s back up now, and in the meantime you were able to admire MySQL errors here, the lack of any useful information here, and WordPress’ pretty “Database error” message on this site.

I noticed and brought it up again after 74 minutes of downtime, which I highly doubt anyone noticed.

Brief Update

Published on 26th of February, 2007 in Development, Life, School, Second Life and Server. 2 Comments

This is a brief update just to make sure this site doesn’t die like I predicted it would when I started it.

I haven’t posted anything recently, largely due to depression. And I’m not doing much better now, but at least I can actually work on things, which is a significant improvement over the state I was in last week. Which is good, I suppose. I’ll start posting here properly when my lives sort themselves out.

So, things I’ve done since my last post:

  • Cried a lot. :(
  • Started a new project-type-thing, phpsimcaps. It’s a library that allows you to interact with sim-caps using PHP. So far it can only request map layer data and upload sounds and textures. I’m having difficulty getting ImageMagick to convert to JPC at a decent quality though – any ideas? Also, animation upload isn’t going to work until I’m in the mood for working out what LL are doing to the animation to make it play properly. (Currently you can upload animations, but they won’t play back). I might make a separate post about this some other time. Maybe.
  • Upgraded my server with the extra RAM and HD space it needs (no more random segfaults or disk space errors, woot! :D )
  • Improved my stats site in order to make pages load much faster without putting any real load on the server. (Compared to having to locate and average tens of thousands of numbers every visit)
  • Thought about life, got depressed, and sat in a corner doing nothing. And I went to school sometimes too, where I sat slumped over my desk doing nothing.

So that’s my update for however long. I’ll post again when I feel better.

Another Collapse

Published on 10th of February, 2007 in Development, Life and Server. 0 Comments

Today my MySQL server collapsed with errors about being too busy, while everything else was really slow. I redid some code to work differently and restructured, so it’s happier now. I’m also now collecting a few other SL related numbers. I guess I’ll do something with them some day.

I collapsed too, after what was supposed to be a harmless comment from Cirr. I don’t feel like doing anything and I can’t think.. I’ll be absent for a while now. Meh.

Server collapse

Published on 9th of February, 2007 in Server. 1 Comment

[22:30 GMT] I have disabled Apache’s logs and removed MySQL’s logs, freeing up a significant amount of disk space. Still planning on upgrading.

The server running this blog, stats.katharineberry.co.uk, services.katharineberry.co.uk, katharineberry.co.uk, vps.tslemporium.com, tslurl.com, and various others, collapsed earlier, with messages like sed: couldn't write 1 item to stdout: No space left on device.

I have deleted 100MB of files, which will serve as a short-term temporary solution. The stats gathering and online status will have been nonfunctional during the downtime.

I will be adding more hard drive space ASAP. And more RAM too, since there are a few memory issues.