OS Quest Trail Log #58: Snow Edition

Snowy Trees

The first two months of the year are just about history and there hasn’t been much activity on the Quest. I can’t blame the record snowfalls for that. And the good news is the ground is again visible in places. There haven’t been any changes to my Ubuntu Home Server, but it has gotten some heavy use and I’m reaching an inflection point where I’ll have to make a decision. Then there was the recent replacement of my Motorola Droid with the Verizon iPhone. As if the iPhone didn’t give me enough Apple products, I also just added a 13″ Macbook Air to my collection.

Ubuntu Home Server

My Ubuntu server has been rock solid, even after having a bookcase fall on it. Yea, that’s right, it was next to a wire rack that was a overloaded with books and stuff. I recognized the potential problem and had just pull the first books off when it collapsed and knocked the running server over. The server kept right on running, although I did shut it down for the cleanup. It seemed no worse off after the near disaster.

The only other minor problem had came a couple days ago when I noticed file copies from the server were extremely slow. Everything seemed OK but I went ahead and rebooted it. After the reboot I did noticed that both the OS mirror and the data drive RAID array where doing a rebuild. In both cases it looked like they had temporarily lost a drive as the status indicated a faulty drive was being rebuilt. Performance was good during the rebuild process, file copies were back to normal. The re-sync happened quicker than I expected and even the large data array finished the re-sync by the next morning. This wasn’t caused immediately by the collapsing bookcase as they were a couple weeks apart and all was fine during that time, but I did find one drive’s power cable became loose. This happened to be an extension cable I had questions about since it’s connect was heavy and wasn’t a tight fit, so I a added buying a replacement to my shopping list.

I’ve also pretty much filled the home server to capacity, using 85% of my 13 TB of available space. Added to this is I’m pretty well maxed out my available space for backups. So it time to make some decisions. I do have the hardware necessary to add storage to the server, but it would be in the external cage that had originally been with my WHS v1 box, I’d prefer to avoid needing a external box for storage, although it remains an option. So that leaves reducing my drive usage which is seeming like a better idea the more I consider it. Currently my video library is on the server in uncompressed form, while nice an convenient it’s overkill for most videos. But encoding them in a new format will take time. Currently the backup is split between my old Windows Home Server box and my Drobo, both of which are at capacity. I do have spare drives I could use but that would require me to do and manage the backups manually, which means I might as well not do them.

The recent release of the Windows Home Server 2011 Release Candidate solidifies things for that option, and it doesn’t seem so bad. Drive management will still be more complicated but performance over WHS v1 should be better. My existing hardware RAID controller provides the redundancy for the data I need to be reliably available, while my video library and other non-essential files can be on non-redundant drives. If the drives fail I can wait to recover the files from the backup. The Windows Home Server backup, while it seems lame (2TB limit on drives to be backed up) on the surface may actually work for my video library which doesn’t change much. Worst case is my video library gets manually backed up to bare drives like it did in the past. So high on my to do list is to install the Windows Home Server 2011 Release Candidate on my test machine so I can take it for a spin.

Despite a couple hiccups Ubuntu has been reliable but WHS may be better suited to my needs and my existing hardware can handle it. Ubuntu’s requirements are considerably less than WHS, but I sized the server in preparation for WHS so there won’t be any need to add horsepower. While Ubuntu appeals to the geek in me, Window Home Server will win in the ease of use category which is the primary concern for my daily data server. I still have virtual and test machines for running Ubuntu.

I downloaded the release candidate last night and hope to install it on a test server within the next few days.

Macbook Air

I gave into my impulses and bought a Macbook Air a couple weeks ago. It’s the 13″ model, with the base 2GB of RAM and a 128GB SSD drive. It’s a strange machine, the specs are low-end but it performs better than any other laptop I’ve used, even those with new processors. The SSD drive makes all the difference. I found myself running a Windows 7 Virtual machines, a couple different browsers (with multiple tabs each) and a half dozen other apps, all without a problem or delay switching between them and using the foreground app. In this case none of the apps where running background processes (except maybe an occasional sync to another PC). The Air easily switched between the apps and had the horsepower for the app I was using. Switching between Spaces is also nice and brisk.

The Macbook Air quickly became my “couch PC” due to it’s light weight. (I’m using it now.) I haven’t done any traveling in the short time I had it but with it’s long battery life and light weight I suspect I’ll be likely to carry it more than more previous laptops and will be more likely to use it. The nearly instant on also makes it feasible to use when I only have a couple minutes.

On the Agenda

I’m looking for a simple and low cost backup solution for my parent’s PC. Currently they back up to Mozy for offsite backup and data is copied to a second local drive. After a recent PC problem it became apparent that I needed an image type backup. Recently they had a PC problem and when restoring from Mozy the files were duplicated (due to the default to rename existing files on a restore). Cleaning this up was a hassle, especially since I couldn’t just delete the directories and replace them. It would have been nice to have a Home Server type backup where I could grab the entire directory tree and be sure to get every file that was there at the time of the backup.

I may just give them a small Windows Home Server that just does backups but that another computer that they need to keep running. When I built the PC I included an internal hard drive to use for backups so I may try to find some software that can use that. But I need more than a simple image, I need it to have some history as a problem may not be noticed before the backup image is overwritten. What I need is Time Machine for Windows. Any suggestions?

Between looking for the backup alternative and testing out Windows Home Server 2011 I should have most of my free time occupied. I’ve also been playing around with Mac software again. I’m writing this with Mars Edit (I typically use Windows Lie Writer) but so far this is the only new piece of software I’ve come across that I might want to keep using.

Hopefully things will be picking up on the Quest and things will be more active here over the next two months than they were for the last two.

Ubuntu Home Server: The OS Install and RAID Configuration

Cooler Master HAF32 Case

My Ubuntu Home Server caseAs I previously mentioned, I’ve decided use Ubuntu for my Home Server now that Windows Home Server will lose the biggest reason I loved it. Thanks to some well placed vacation I had five days to explore the possibilities. By Tuesday the Ubuntu Home Server had all my files and it was serving all my files. The WHS still handled backups, but that’s all. Since then I’ve settled on a configuration that seems to work for me and have moved 7 TB of data to the server so far.

I used Ubuntu Server 10.04.1 which is the Long Term Support version and will be fully supported until April 2015. This means I won’t be forced into an upgrade by a obsolete OS anytime soon. I’m looking for stability over cutting edge technology so I probably won’t be looking for a full OS upgrade anytime soon.

I’m using Samba to serve files to my Windows and OS X computers. This seems to be the weakest link in the Ubuntu Home Server chain. But since I’m only serving files within my house I’m avoiding most of the complexity.

I will lose the ability to do PC backups directly to the server, unless I find some replacement software. I’ll also lose the ability to run offsite backups to KeepVault from the server. So for now a scaled down WHS v1 will remain to handle these backup functions.

RAID Configuration

The RAID configuration was the nut I had to crack first. Linux doesn’t work well  with the fake RAID offered by my motherboard controllers (based on my research, not experience) so I didn’t consider it. So first off I broke the mirror that was set in the BIOS for the OS drives. From now on, everything would be set in Ubuntu.

The first time I installed Ubuntu Server it said that some SATA configurations had been found and asked if I wanted to activate the RAID. I answered “No”. I did several installs and was only prompted the first time so it must have seen traces of the old mirror on the drives.

I spent some time testing Ubuntu’s software RAID using mdadm and didn’t have any problems. I did a completely fresh install between each RAID configuration and deleted the partitions each time. Everything was done using the standard Ubuntu Server 10.04.1 DVD. I didn’t have to use any additional drivers. I did apply any post-DVD release updates and patches but stuck with the pre-configured Ubuntu repositories.

After I got comfortable with the software RAID I decided to try using the LVM (Logical Volume Manager) so that I could easily expand the drive in the future. This was easier than I thought, but I’m getting ahead of myself.

In all my configurations the two 320GB drives where mirrored to handle the root partition (the OS). So as I talk about the configurations I tried I’ll be ignoring this mirror.

I configured all the arrays during the installation phase. This seemed the easiest thing to do since it handled all the mount points.

I have two RAID controllers, an Intel controller on the motherboard with 6 ports and a 3Ware controller with 4 ports. (The 320GB drives are on the motherboard’s 2 port Gigabyte controller which is a third controller) All the drives I’m using are 2TB drives, but the 3ware had the same model Hitachi drives, while the Intel had two Samsung, two Western Digital and two Hitachi. The Hitachi’s are 7200rpm drives while the others are 5400rpm drives. I was concerned the different drives would cause a problem, but so far it’s been OK. I also haven’t found any information that this would be a problem. Still, If I had enough similar drives I would have used them. On the other hand, if I have to buy enough drives to match them up I’ll have an insurmountable budget problem.

My first configuration had two RAID 5 arrays, one for the drives on the Intel controller and one for the 3Ware controller based drives. This made sense to me since the performance within the array would be the same.  So I lost two drives for data protection, one in each array. I wasn’t using LVM at this time and I mounted the 6 TB (usable) 3Ware array as /home and the the 10TB (usable) 3ware array as /shares. The problem with this is I couldn’t come up with a good way to split the files. I’d either have wasted space or run out of room in the future. This is when I started looking at LVM.

But rather than using LVM to combine the two different arrays in one volume group I re-installed and configured one big RAID 5 array and mounted it as /home. This seemed to work well and I only lost one drive to data protection.  But the problem this has was the time it took to resync/rebuild the array. I didn’t actually wait for it to finish, but is was slow and if the time stayed consistent I figured it would take over two days. With 10 drives in the array I figured it was possible I’d lose two drives before the array could be rebuilt. Unlikely, but possible and Mr. Murphy likes to visit.

So my third and final install was a single RAID 6 array with one hot spare. RAID 6 provides two parity bits so in effect I’d lose two drives to data protection. It’s possible that the writing of the parity bits would slow write performance but this might not be any worse than RAID 5. With the hot spare I lost another drive to data protection. So I’d lose the equivelant of three drives to data protection. But I figured the hot spare could take over when the first drive is lost and not require me to do anything. If a second drive was lost before the rebuild finished then I’d still be covered. Since I used duplication for everything on WHS this is still an improvement. Plus I can add more drives without losing more drives to data protection.

Installation

The installation was well documented in the Ubuntu docs so I won’t repeat it all. The Advanced Installation procedures provide all the information needed to configure software RAID. I did enable booting while degraded for the OS drive. Also, it wasn’t obvious to me from the instructions, but the LVM configuration is during the same session as the RAID configuration and I couldn’t save the RAID configuration until the LVM volume was configured (well, I could save if I wanted to skip the LVM configuration).

In my configuration all ten 2TB data drives are configured as a single RAID 6 array with one hot spare. That RAID array is then configured as one single logical volume which is mounted as /home.

I used LVM up front as I think this will give me more flexibility in adding drives. I can add drives without being forced to add them to the existing array. I may never need the flexibility, but it doesn’t seem to add any complexity.

I let the Ubuntu installer install SAMBA and OpenSSH. I’ve never installed SAMBA so I don’t have any pre-conceived notions about how I want it installed. I do plan to install a full LAMP stack but since I want that done my way I’ll do it manually.

I’m still playing around with Samba so will leave leave my discussion about that  for another time (besides, this is long enough now).

Post Installation

The initial build (aka sync) of the array takes a long time. Mine took roughly two days. The drive can be used during that time and i n fact I had my file copies running almost continuosly for the first day. The performance I saw seemed fine (was actually faster than my WHS) although it probably extended the sync time. Also be aware that the sync time doesn’t depend on the amount of real data. It just build parity for all the bits, whether or not they contain a file. It’s not like WHS file duplication which just copies a file to a second drive.

Ubuntu (actually I think any Debian based Linux) schedules a RAID check to run on the first Sunday of every month. I noticed this running and based on it’s scheduled start time it took about a day and a half. I didn’t have a noticeable impact on performance (based on my usage of the server, not any actual benchmarks). But for me a Sunday is actually a heavy usage day for me since this is a home server so I’ve removed the task from the cron schedule. To do this I commented out the line in /etc/cron.d/mdadm (by adding a # at the beginning). I didn’t want to delete it as it’s something I may want, just with a different schedule.

Open Questions & Resources

The screen shots below show today’s disk usage and RAID status:

filesystem

mdstat

I still need to go through the process of expanding a drive volume. I also want to go through the process of replacing a drive in the RAID array, I did disconnect and reconnect a drive to test the RAID but it was the same hardware.

I’ve had some issues with Samba. Once, it seems the only way to stop and restart it is to kill the processes and then restart them (seems risky) or reboot. None of the procedures I’ve found work for me. Luckily, now that it’s configured I don’t need to make changes. But in the beginning I was rebooting each time my config changed (including adding shares). My use of Samba is limited to file sharing.

I’ve also had an occasional problem with Samba if there’s a problem at a PC as it’s copying files to the server, For example, a copy was in progress when I lost power and my unprotected (by UPS) iMac died mid-copy. I had to use terminal to connect to the server and then use sudo permissions to delete the directory that was being copied to. My ID didn’t have permissions (I could have just modified permissions, but the copy was bad so I deleted it).

Other than that I’m happy with performance. Knock wood, cross-fingers, wave rubber chicken, etc…

My biggest problem is that the amount of data on the server has gotten ahead of my testing and ability to recover quickly if there’s a problem or if I decide an Ubuntu Home Server isn’t the way to go. I no longer have the ability (or hardware) to keep all the files on another server that I can quickly switch to. So I’m more committed to this path than I was to Vail.

Some additional resources I found helpful:

This is another Ubuntu software RAID article, but based on Ubuntu 9.10. Includes some screenshots so you’ll have an idea of what to expect.

Still on my todo list is a review of the disk SMART tools discussed here.

While not mentioned in this article, I use a CyberPower UPS on the server and their Linux software is here.

The OS Quest Trail Log #57: Year End Party Edition

Santa’s finished his deliveries and is tipping back a few. Time for me to wrap things up for the year and get ready for the next one. Not much has changed since the last trail log but I figured it was as good as time as any to wrap things up.

Power Upgrades

It’s still in the box, but I did get a new UPS after my recent power problems. I bought the CyberPower CP685AVRLCD Intelligent LCD Green UPS which I’ll use for my Mac Mini with attached Drobo. Because of it’s location I wanted one with the outlets facing up. I also like the LED readouts to show capacity, load and other stats. This model also has line conditioning so the UPS will compensate for variations in power quality. I’ve had good luck with the reasonably priced CyberPower brand so stuck with it. Like I said, it’s still in the box but the ups does support Macs. The weather report is for lots of snow here on the east coast so I should probably hook it up as soon as I post this.

Wireless Upgrades

I’ve been having problems with the wireless bridge I use to connect to my workbench. The problem is the PC’s on the workbench side of the bridge frequently fail to renew their DHCP address so lose network connectivity. I say fail to renew because when I turn the PC on it gets an address and connects. Sometime later it will lose the address, I’m assuming it’s related to the renewal but maybe not. Happens on every PC I used and the address typically drops after I’m away from the PC for some time.

Two things happened shortly before I noticed this problem. The first was my power problems, and the wireless bridge is not UPS protected. The second, and my personal pick as the cause, is I changed up my network configuration and added a new router. This takes the Netgear router that connects to the bridge out of the loop as a connection to my WAN and as my DHCP server.

So I’ll be ditching the Netgear router and will try out a couple D-Link DAP-1522 Xtreme Wireless Bridge/Access Point. Hopefully I’ll get some time to set these up before the new year.

Ubuntu Home Server

My Ubuntu Home Server is still chugging along and serving files as quick as I consume them. The RAID has been stable and the server has been stable so I’ll start looking to expand it’s use beyond serving files. First step will be to install Apache and the other assorted software to set up a test server for myself.

I’ll also need to start researching setting up remote access, although this will primarily be a dynamic DNS problem. Since there’s no GUI all my connections will be via SFTP or SSH. I may consider making the web server available from outside my house, but the potential security issues has me concerned. My gut reaction is it won’t be worth the hassle.

Windows Home Server

I shut down my Windows Home Server when I went away for Christmas, and when I powered it up one drive came up as missing. I think this is a return of my motherboard SATA port problem and not a real bad hard drive. I had re-used the ports to try and add space cheaply. They tested well, so I hoped the cleaning and reseating resolved the problems. At this point, duplication is not turned on for anything on the server.

Since the Windows Home Server is now used exclusively to backup files (and PCs) I simply went through the drive removal process and ran the backup database repair. The lost files were recovered when I ran the backup again.

As for the PC backups, they were all lost according to the consistency check. Not a major problems for me since it’s not a primary data backup for me.

On Tap

Since looking ahead seems to be the thing to do this time of year I’ll do the same.

This past year saw a lot of PC builds. Not so many in the year ahead. Maybe some minor upgrades but all the major bases should be covered. I still haven’t decided what to do in the living room. I’ll want to get video to my TV in some way, I might build a HTPC to handle the task.

I’ll keep looking into a new Home Server. I think the Ubuntu Home Server will be the best choice for me, if not for most other WHS v1  users. But that tends to fit in with what I’m itching to work on in 2011. Web development, design & programming. It’s been a long time since I did any programming for fun and I’m hoping to use some web projects to get back into it. We’ll see.

Happy New Year

I might still have one more article in me before the year ends, but I suspect this will be it. So have a happy new year.

Ubuntu Home Server: The Beginning

My Ubuntu Home Server caseAs I previously mentioned, Windows Home Server is all but dead to me now that Microsoft is killing the Drive Extender technology. So the search was on for an alternative. As the name of this post implies, I’ve made a decision. Here’s how I came to it.

Just days before Microsoft’s announcement I wrote about how I use Windows Home Server. My primary use is for file storage. Every file of any significance is on it, including my iTunes and Video libraries. Music and Video are on the server simply as files, there’s not management/streaming software actually running in the server. I also use it for PC backups but since all my data is on the server it’s not a critical piece. I’ll want to do something in case I need to rebuild my PC, but it doesn’t have to be something the server does.

The other requirement is that it needs to work with my new hardware. Moving Windows Home Server V1 to the new hardware is a viable option, at least for the next year. So anything I looked at was compared to WHS v1.

I’ve also been researching ways to manage my music and video libraries, which is currently done with iTunes but I haven’t seen anything, for any OS, that has me excited. So at this point what I want is flexibility.

So to to sum up my requirements:

  • Lot’s of file storage. I currently have 5TB worth of data and expect dramatic growth in my video, music and photo libraries. My new hardware can hold 10TB before data protection under WHS. (I turn on duplication for everything)
  • Good data redundancy. I doesn’t count as a backup, but hard drive failures will occur at the worst possible time so I want the files fully accessible should a hard drive fail. The new hardware has 12 drives, the odds are against me.
  • The files need to be accessible from both Windows and Macs. Might as well throw Linux in there too.
  • No new hardware needed.

I looked at Amahi Home Server and it seems interesting. But I decided not to test it out. It is on my list to look at if I get a chance, but I’m not willing to commit to a full blown managed server. It has a lot of features, many I won’t use. It seems to have a level of complexity I don’t want.

I decided to go with a plain old Linus server running Samba for file sharing. Since my web server is Ubuntu I’m most familiar with it and decided to stick with it. My main concern was how to do RAID, or RAID like data protection. Only 4 of my 12 SATA ports are on a card that can do real hardware RAID. The rest are on the motherboard which does fake RAID. (Fake RAID is when the PC’s cpu provides the RAID processing, and not hardware on the controller.) I’ve never really trusted Motherboard based RAID but I had been using it to mirror the Vail OS in testing. As it is, most Linux distro’s are flaky (at best) with fake RAID.

So RAID through the motherboard BIOS was out which meant the best I could do via hardware was a 4 drive RAID 5 array using my 3ware card. But my research found good things said about software RAID in Linux, using mdadm. This was a flexible solution that seemed reliable, so I moved forward with it.

I spent some time over the long weekend installing Ubuntu Server using different RAID array configurations and breaking the arrays to make sure they would rebuild without a problem. Everything seemed fine. The only minor hitch was when I broke the OS mirror I needed to reboot before I could add the drive back in. Removing or adding the drive resulted in a not found error. That is until I rebooted which seemed to cleanly remove the drive and I could re-add it.

So as it stands now I installed Ubuntu Server 10.04.1 into what I think I want as my configuration. I picked 10.04 rather than the newer 10.10 because Ubuntu 10.04 is the long term support version and I won’t be forced into an upgrade 18 months down the road. I have until April 2015 before it end of life’s.

I used software RAID (mdadm) all around. The root and swap partitions are mirrored using the WD 320GB. I ended up putting all ten of the 2TB drives in a RAID 6 array. RAID 6 can survive two drive failures as there are two parity bits. In what may be overkill I included one hot spare.

Considering I had planned to lose half my space to duplication under Windows Home Server this is an improvement. Currently 30% of my space is for protection but the percentage will decrease as I add drives. My testing showed that resyncing the array takes a long time. The initial sync for my 13TB array took over a day (with the server under pretty heavy use). So the hot spare will get the rebuild going without me needing to do anything (at least in theory) and being able to lose two drives simultaneously makes me feel more secure.

I also installed LVM and created on logical volume for the data array. The theory is I can easily add drives to increase the space, although my testing hasn’t gone far enough to prove this.

As I write this tonight all my non-video data is on the Ubuntu Home Server and the RAID array finished it’s initial syncing a little while ago. I start copying the video files tonight but that’s going to take about a day to copy. All the data is still on the Windows Home Server in case I need to go back, but I’ve blocked access so I don’t save data there by accident. I have two more days off so can spend some additional time testing things out. But so far things look good and I plan to continue down the Ubuntu Home Server path. Next on the agenda is a PC backup solution that will be similar to Windows Home Server.