Server performance — solving disk bottlenecks
In February we added lots of disk space to the Eclipse.org server farm. There were two reasons for this:
- We were running out.
- Our current disk arrays were doing too much, hurting disk performance.
How do you know if your disk systems are too busy?
- look at the activity lights on the drives. Two of our 7-disk array’s lights were mostly on all the time.
- look for excessive IO Wait time using top. Clear indications are high wa% as well as processes in the D state.
Disk bottlenecks at Eclipse.org occur because so many files need to be read and updated at any given time that the disk heads are constantly moving (seeking).
How do you solve a disk seek bottleneck like this?
- Avoid going to disk altogether by adding RAM [1]. Unused RAM is used for write buffers and file cache. Adding 8, 16 and even 32 GB of RAM can make a sluggish system run like new.
- Add disk spindles [2]. One disk can only seek so many files at a time, so the more disks in your array, the faster the seek times.
Our budget was quite tight, and I needed to solve both the low disk space and high disk seek problems with minimal money. I opted for a 16-disk array of 750G SATA drives, but no additional RAM. Although SATA drives are typically slower than their high-end SCSI counterparts, they are much, much cheaper, allowing you to get lots of disk space and awesome seek performance from a higher number of drives.
Since implementing the new array, disk wait time went from a whopping 50% to 12% on our file server, and from 40% to 2% on the build server. Money well spent!
[1] If performance is critical, consider using a ramdrive for frequently accessed files. Beware of writing there, though.
[2] Solid-state drives will definitely help out here, as soon as they become more accepted and available in entreprise-class hardware and environments.
Posted March 13th, 2009 by Denis Roy in category: Uncategorized
You can skip to the end and leave a response. Pinging is currently not allowed.
Leave a Reply
You must be logged in using your Eclipse Bugzilla account to post a comment.
