Strange Downtime Issues

Friday 01/23/2009  –  Category: Uncategorized  –  No Comments

Flickr Original users and everyone else--my server hiccuped sometime yesterday afternoon which caused most of the running apps to die, resulting '503 service unavailable' errors. I've looked at my logs and charts over the past day and I can't tell what exactly happened yet...my merb processes totally disappeared and php-cgi instances stopped responding. I'll be continuing to investigate what caused the hiccup...sorry for the inconvenience!

As a side note, I really need to write some monit scripts to notify and restart services if they start acting up again.

Thanks to this guide and this post, I was able to setup Ubuntu 8.10 (Intrepid Ibix) on a Dell Poweredge 2400 in a RAID1 configuration.

I ran into more than a few hurdles along the way:

  • The first iso I burned was corrupted...I suspect it was because I had paused/resumed the download a few times. Grr.
  • On my first install attempt I just installed onto one hard drive.  Upon booting up, I got this message:

Loading...
Gave up waiting for the root device.
Alert /dev/disk/by-uuid/cf921ca07-7fa3-4031-a5de-44099fd5b82a does not exist.

  • After I found that first guide I properly partitioned and formatted all 5 drives in a RAID1 configuration.  After booting, I got this message:

Gave up waiting for root device. Common problems:
...
ALERT! /dev/md0 does not exist. Dropping to a shell!

Google yielded this second post that suggested to add rootdelay=40 to your grub kernel line so /boot/grub/menu.lst looks like:

kernel   /boot/vmlinuz-2.6.27-4-server root=UUID=401eb0e1-f624-4d86-a1a4-47374ba9a556 rootdelay=40 ro quiet splash

  • To get to the grub bootloader screen, hit ESC and you should see your partitions.  Hitting 'E' will bring up a prompt to edit the command.  Hit enter to accept and the hit 'B' to boot from that partition. Once you've booted successfully, make sure you edit /boot/grub/menu.lst because the grub editor didn't seem to save my changes.

Phew.  So after a few hours of trial and error, googling and fixing, I finally got things up and running.

The End.

(...or, how MRTG is too much trouble to setup)

chart2

My server was having spiking load averages this past week and I had no idea was going on; my top output didn't have any processes that stood out and I didn't have any cron jobs running either.  A friend suggested I use a monitoring tool like MRTG to keep track and graph my performance.  After downloading and compiling I couldn't for the life of me figure out how to get MRTG fully working (following the guide here).  I was able to get the free memory and TCP connection graphs working but had no luck with the CPU usage graph--the one I really wanted to see.

After a few hours I finally gave up on MRTG and started looking for other solutions: Cacti, Zenoss, Nagios...I eventually ended up choosing Zabbix because it looked more lightweight and less intimidating to setup...and most importantly it generated graphs of relevant data.

Some gotchas during the setup process:

  • The install script installs the PHP admin interface into /etc/local/zabbix (or something like that), creates an apache conf file that aliases whatever you set your web dir to be, and tries to restart Apache. Since I'm using nginx this obviously didn't work so I ended up copying /etc/local/zabbix to a public web directory.
  • It isn't completely apparent that you have to go to Administration > Installation to setup Zabbix for the first time
  • Setup items to monitor under Configuration > Items
  • Setup graphs under Configuration > Graphs

I've recently transitioned to using a  separate machine work but wanted to be able to access my home lappy in case I needed anything.  Here's how I got stuff working:

Set up Screen Sharing (from the helpful guide at Lifehacker)

  1. Make sure Screen Sharing and Remote Login are enabled in your Sharing preferences pane
  2. Configure your router to forward port 5900 and 22 to the IP of your local machine
  3. (Optional) Create a custom domain at dyndns.com to point to your IP
  4. From your other machine, create an SSH tunnel for the VNC connection: ssh -L 5900:127.0.0.1:5900 user@remote_machine_ip
  5. Still on your other machine, switch to Finder and select Go > Connect to Server and connect to vnc://localhost
  6. You should be connected!

Mount your remote filesystem with Macfuse and Macfusion
The built-in Screen Sharing app doesn't support drag-and-drop between windows so we'll have to set up an alternative for that.  Macfuse and Macfusion enable you to mount a remote filesystem that shows up as just another volume on your system.

  1. Download and install Macfuse and Macfusion
  2. You should be able to easily setup a connection and mount your SSH filesystem through the Macfusion GUI.

Listen to your iTunes Library
I didn't really want to copy all my music to the other machine so I found an app called Simplify Media that lets you share your iTunes library over the internet.  Make sure you have your library shared, install the client on both machines, and sign on with the same screenname.  Your remote library should now show up under the Shared section of the sidebar.

Ending Notes
Your performance may vary depending on your internet connection on both sides.  With Screen Sharing, I definitely wouldn't be able to be productive working on the remote machine due to the refresh lag but for quick and simple tasks (like retrieving a file from the Trash) it works well enough. Simplify Media works as advertised but can have some lag in streaming between tracks.

This is a pretty basic concept, but for some reason I forgot the power of parentheses:

To select people with first name equal to "Jason" and last name equal to "Tom" or "Ting":

SELECT * FROM people WHERE
first_name='Jason'
AND (last_name='Tom' OR last_name='Ting')

 Page 7 of 14  « First  ... « 5  6  7  8  9 » ...  Last »