Edit: that turned into kind of a story. You can skip ahead to the "*"s if you just want the question. Everything else is colorful backstory, but hey, sometimes I’m entertaining…
I’m in kind of a weird spot right now, moving all my stuff into a (very small) rental so we can sell our home and get started building a new one.
So, making things on the primo is going to be challenging, the cup Turner stuff is all packed away somewhere, and the pid speed control is on hold because I just spent all my pocket money pre-ordering the oculus quest 2 (I know, I know, Facebook…barf). I just built a pc in March (ryzen 3950, 5700xt, nvme, 32gb ram…sweet) but the motherboard took a crap, or at least that’s what I think it is, and it’s not much fun to CAD or game on my Laptop.
Well, that leaves me sitting here thinking about setting up the home network on the new house, which leads to thinking about safe file storage. Sucks getting bored. Once upon a time, I used a bunch of old stuff I had lying around (dell dimension 4600+random assortment of odds and ends) to play with FreeNAS. I was super impressed with myself that I got it working, but it was mostly just play. Then after a few months of posting myself on the back I got a little cocky and moved all my data to it so I could upgrade to windows 8 with a clean install. You know how this story has to end. The FreeNAS motherboard failed and I spent 2 months of free time trying to figure out how to get my data back. It is supposed to be as easy as replacing a broken part (wasn’t happening there) or moving the drives to another freenas and booting up (either I want able to do this or I did and something else broke…I’m not even sure anymore). Next failure was when I built my current machine. I knew my drives were on borrowed time, but hadn’t had any issues and I kept getting busy with stuff like building a cnc, for example. Then when I was trying to move my data over (finally redundant, raid 1). I got the dreaded bad sector. I got lucky with most of the important stuff, but whatever. Found out today my oldest kid has almost 1tb of her files (graphic designer) on an external HD with no backup. Apples and trees, lol.
So, what would y’all tell me to consider? At a minimum, I want a space to backup files periodically. Is a Nas or server worth the effort over just buying a bunch of hard drives and passing them out? I’d also really like to get rid of Dropbox for moving files back and forth (wife, kid, and I pass files around, plus I’ll use it to load gcode on the Linux box that drives the cnc). Then I could use my Dropbox accounts for backing up the stuff I REALLY don’t want to lose.
I’d like to pretend that I have a ton of movies and music to send out to tvs and whatever, but we stream Netflix and Spotify, so I’d be lying. My boys like to play a bunch of steam games, and some of them have pretty big downloads. They need to build new machines, too. Does it make sense to lean on network storage or just keep putting more hard drives in new builds? FWIW, i gave myself 1tb on my pc and I’m shocked at how much I have on it already.
Feel free to comment on any part of this, all of it, none at all, or something else entirely.
Here’s my 2p worth, that’s about 2.2c worth, so an extra 10% free
I’ve worked in IT since the early 80’s so I’ve seen things come and go and have tried lots of thing out, some at my own expense and some at work expense. The two big changes for backup are
The bandwidth we have available to us now. When I started, I had a VT100 terminal with dial up modem 1200 baud download and 300 baud upload. A baud is approx 1 character a second. Since all we had was 80x25 character displays, this wasn’t that bad a limitation, I could still play Rogue so life as good.
We now have massive bandwidth in comparison, in the UK most people can get 35-70Mb/sec, though some people are outliers with 2Mb/sec and 1Gb/sec. Happily I am in the 1Gb/sec category.
The cost of disk storage is dirt cheap and is only going to get cheaper for spinning drives, though the industry would like you to move to solid state drives as that drives up their profit margins.
These two factors drive different options for you and how you can think about backup storage.
You can continue to use DropBox or any of the other types of online storage, iCloud, OneDrive, Google Drive, etc etc. There are loads of these type of systems around. These normally work by syncing the contents of a local folder on a local PC into the cloud. This means you have your local copy on your local device(s) and a copy in the cloud. You work locally and get the local speed and you have copies of your data in the cloud. You can hook up more devices and get more local copies, so as you write out a local file to you PC, it gets copies to DropBox and then back to a iPhone or a tablet or whatever. This works well and I have a free DropBox account with 5GB as I can share stuff with friends or with customers who want that way of working.
The main downside (as I see it) is that DropBox wants to push you to a premium rate and wants you to buy more storage. Thats their business model and they need paying customers to support the freeloading customers like me.
The upside is that you don;t have to worry about networked storage, working out SMB configurations or whatever as you download a simple piece of software and it just works. The costs are reasonable and it just works for many scenarios. However as you need more disk space, this can be expensive. I’ve just checked the DropBox prices in the UK and 2TB is £96/annum (around $105). That’s this year, next year and the following year etc etc. The team (or family) account is £10/month/user for 5TB which strikes me as a lot to pay out.
You pay for simplicity and for somebody else to worry about backing up.
You can buy one in. Synology, QNAPP and others do networked storage devices that will hold 1-12 disks and act as a network file storage. You can get whatever size you like and configure the (larger) versions with all sorts of different RAID configurations to give you more redundancy at the expense of using more disk, or faster access speed at the expense of less redundancy or whatever. My personal view is that many of these are under powered at the low end of the market (£300+) as they use smaller CPU’s and have less memory to save costs. I had a Synology two disk box (DS210+?) and it was nice but I struggled with the network performance with backing up from my Mac using Chronos backup software. The company and I spent a lot of time trying to debug the rubbish performance before they came to the conclusion that Synology doesn’t play as nicely with Mac’s as with PC’s. This was 2-3 years ago so things might have changed.
You can get a lot of storage this way, a six drive box will give you many TB’s to play with all in a redundant configuration. As @jeffb3 has already commented elsewhere, you get other toys to play with such as containers and the like. This is probably not that interesting to you.
You can also do automatic backups from your network storage to a cloud storage system such as Amazon S3 or Glacier. This is cheap (but slower) storage and can be thought of as a backup of a backup. In my view a backup of a backup is a good idea and doesn’t cost that much if you value your data.
You can also build your own, such as the FreeNAS one you have already done. I have old Dell Xeon server with 6 x 4TB Iron drives. This has been faultless for a year now and provides excellent performance to my Mac, to the ESXI server, and provides Nextcloud, a cloud based system DropBox type system. The user groups on FreeNAS are very aggressive though and are 100% clear that for production or proper use, you should have a proper motherboard that supports ECC memory. Most desktops do not have ECC memory and are vulnerable to bit flipping.
FreeNAS gets it’s speed by caching as much data as possible in RAM and using the disks carefully. AMD is not as well recommended and they don’t recommend virtualising it under VMWare for real use. However once you get past that, you end up with a rock solid system with great redundancy, great speed and (I think) an easy to use system. However you do need to understand some newish disk concepts about setting the disks and partitions up correctly. Many people do not want to spend the time to set this type of system up and that’s quite understandable. I work in IT for my living so whilst this is not what I normally do, it’s in my comfort zone. YMMV.
You can simply expand your PC/Mac out with extra drives This is probably the easiest option. A 4TB hard disk from Amazon is £85/$100. Given you only have a 1TB drive at the moment, this gives you plenty of headroom to grow. The only thing that will start to est that space up is videos, a high res long film might be 3-5GB. You can get an awful lot of video in 3GB. We have a 3GB drive on a Mac we use for iTunes and it’s got hundreds of films on it. Still lots of space to go.
CNC files are tiny in comparison. Word files are small, there is almost nothing that will compare to a video file for size (and lets ignore the data streams from the LHC or from radio telescopes, that’s very specialised and in the PB size range). I store data for a business (traffic analysis for London drivers) and after four years I have got to the giddy heights of 2TB. That’s every traffic incident (jam light out, accident, crash) for every 3-6 minutes and it still fits on a USB external drive (it’s backed up BTW). The data sets produced from this are somewhat larger though, hence the need for 6 x 4TB drives. I think this is unusual and in the realms of the professional user and for most people, this is overkill.
The other thing to think about with large hard disks is file management, it’s very easy to lose data in the masses of all the files you have. Ask me how I know Sometimes restricting the space you have available is a good idea as it forces you to think do I actually need it?
Anyway in answer to your question, I’d probably get a bigger hard disk for my PC/Mac, get an external USB drive and hook it up for backups and look at a simple, cheap and reliable service similar to drop box for off line backup and storage in case your house burns down.
Apologies for the long reply, I hope it was useful to understand more of the thinking.
Our company refuses to sell NAS boxes. We’ve had too many issues with them in the past. From end uses not understanding the reboot/reset settings, to boards just up and dieing and the vendor saying “sorry about your luck”. Also, if you do go multi terabyte NAS, what are you going to back it up to? I’m currently using an old windows 8.1pro box with storage spaces. It has a half dozen drives varying in size from 512Gb to 2Tb in a software raid. Even if the computer craps out, the files are still accessible since they’re just ntfs formatted drives. Not the weird ext3 or reiserfs striped stuff you find on a nas. This is also built into windows 10.
I was shown a model for corporate data that was pyramid shaped. The top was a small amount of super critical data. The bottom was a huge chunk of stuff that had very little importance. I like to think of my own storage this way.
At the top, is things I would be very upset to lose. Important legal documents. Curated photo albums of my kid’s growing up. Things like that. Those go in cloud storage. There is a local copy too, but I am happy to pay for extra storage to know I don’t have to worry about those thing if my house burns down.
Next is my qnap nas. I have 4 2TB disks in RAID5. I put all out photos and videos there. We have stuff we want to access locally, like my software projects or documents we are working on. We also have a bunch of ripped dvds that we never watch, but those will be the first to move onto an external drive if we need the space. We back up my computers there, but I’m very picky about what gets backed up. I don’t want your MyDownloads going there, or anything from your program files.
I have a few 500GB and 1TB usb 3 drives that I move around big stuff on. If I lost these, I would not care. The drives are worth more than the data. They do end up having a lot of duplicate information.
I used to have my big desktop PC with the raid in it. But I measured the electricity it was using and it was basically 1/5-1/4 of my house’s electricity. Having a big CPU and GPU on whenever you are moving files is a big waste, IMO.
I have been super happy with my qnap. I love Linux though, so I prefer ext3 to ntfs . It just works. I don’t use any of their q services though. They have a bunch of qphotos and qvideo sharing stuff. I just use smb and ssh and the http interface. The biggest thing is, my wife has zero tolerance for my projects when she has to use them. She is super smart, a very talented engineer. But I have to update the shortcuts on her computer if I change something because she just won’t want to deal with it. That’s why we have the google storage and the nas. They have very reliable interfaces. I just make sure the nas has the same IP and I don’t cause her a lot of stress.
We’ve had very few problems though. One drive died on the nas. I replaced all of them with larger ones. No real tests of the system as a whole.
I’ve never completed it or hack, spent decades trying to as well.
My fingers still know all the key strokes after all this time as well. I learnt a lot about UNIX trying to break into it and copy save files over. That’s when we found the developers would put the inodes into the save file so if you copied the save file, it changed the inodes around. We had all sorts of tricks to break into it and play it. The administrator then locked us out in working hours, so we broke into his VT220 and hot wired the display so we could capture his password. Well I didn’t of course as I would kill myself on the flyback voltage, but the electronic people did. I worked in a place where high end HP oscilloscopes were everywhere, people were developing next generation digital telephone exchanges (circa 1984) and so things like hot wiring terminals was pretty easy.
I think that, fortune and cookie were the only games at the time.
Which game was the Xen inm hack or Rogue? That always got me at the lower levels and I ended up discarding all my iitems to get away and then usually died. This would be followed by a screaming “Nooooooooooo”.
Never completed Adventure either, though I know there are walk throughs.
Yeah, I’ve still never finished it either. Always ended up getting killed by some ridiculously out-of-place high level monster that shouldn’t even be on that level. I wonder if there is a save-and-continue work-around somewhere.
As for the topic at hand, I’ve just been using an old Dell R520, 530, 5something? With small mirrored boot drives and a set of 4 2tb drives in a raid5. Used to mess with FreeNAS years ago, and I liked it, but it just seemed to complicated for just simple file hosting for me. The really nice part about FreeNAS is the ability to have it replicate to other FreeNAS servers, kinda like the big EqualLogic units Dell/EMC makes.
Still a lot to chew on. I’m leaning towards Jeff’s model, if I can pull it off. I’m willing to pay for cloud storage, but I think the stuff I absolutely don’t want to lose is probably smaller than the space I have available to me.
Complicating the issue is that I use windows and Linux, my younger kids use Chromebooks (their data needs aren’t really a consideration right now, though) and the oldest kid uses Mac.
I’ve also been reading about Linux file servers. I think it probably doesn’t solve my problem very well, but it made me remember that pre pandemic I gather footage of our local shooting club at our events and make it available to them for download so they can review or share with friends etc. That generally eats up two of my dropboxes for at least half of the year. Would any of these solutions solve that problem as well?
I let my kids have one of my Linux boxes for school work, steam, browser games, etc, but I guess minecraft isn’t available on Linux so it just sits. I think I’ll try to scare up some cheap drives and play with freenas again. Don’t worry, I won’t be trusting this with any data at all, much less important stuff. But maybe if I can put some stuff on it and move the drives to something else and still get the data back, I’ll feel better about using it. The “for realsies” solution (whatever it turns out to be) will be all new hardware, i promise. But at least I’ll be able to try it out and see how we can actually use it.
I also told my oldest kid it won’t be ready any time soon and she should make a backup her next priority.
Do you care if it’s public? You can just load it into imgur or youtube. google photos has unlimited storage if you let them downsample it to “high quality”. If you need it private… Not sure I should be helping, haha. I think hosting nextcloud at home with a ddns and opening a port is the next best thing, but there are security issues opening up any service like that.
A few guys have expressed concerns over YouTube, even as unlisted videos, even though they’re ok with being in short clips of the montages I put together, so by and large that’s out. Honestly I think that’s just because I don’t put the clips with lots of failures or poor shooting in those, lol. Those are more speed to be what the shooting club is about, and some motivation for our friends to come join us.
Thanks for the Google photos tip, I’ll try that. Anybody who wants higher quality can just give me a thumb drive. Hopefully we’ll get back into it soon more that the weather is cooling off. We shut down early this year over covid concerns and lost March / April. June is usually pushing it, anyway.
Ah, there’s more information. FreeNAS is good but you need to put some time in up front to make it run the way you want it to, but when it’s setup, it runs very well. Not touched mine in months now. Not logged on to it, changed it. It died (or rather it failed gracefully) when we had a major power cut and the UPS eventually couldn’t keep it going after a few hours on battery, but it started up again like a trooper. It runs off a USB key so you don’t lose a precious disk slot. I’d still look at el cheapo cloud storage as a further backup.
The key I use for backups is what would the CFO say if I lost all the IT, would she be interested in company bank statements, govt docs, medical records, iTunes films? No, all she would be interested in would be pictures and videos of the kids. That’s currently 500GB and growing. That fits on a cheap USB external disk but I also mirror it to my brothers home computer system 300 miles away and he does the same. I also mirror it to AWS as well as there’s no such thing as too much paranoia when it comes to personal photos and the CFO
I’m dreaming of setting up some kind of kubernetes distributed private torrent-like cloud storage yadda yadda.
If I could make a small rPi box with a drive or two, that could be hooked up using ethernet (even better with POE) - I could leave them on 2-3 locations(friends and family). Then they would be connected, keeping mirrors and cooperating on serving the best bandwith possible, and staying online in case of outages.
Not gonna lie, I got in quite a bit of trouble over that first failure. I think the fact that I’d been talking about replacing old drives for a while and it never being high enough up on her list is the only reason I don’t walk with a limp today.
I’ve seen something somewhere that will aggregate all those free online storage solutions into one big pot, with the local machine doing encryption before uploading. I’ve thought about that a few times, but so far my home server seems to work alright. I do have some stuff on my google drive, and in dropbox. Almost all my photos are backed up by google, and I do the google rewards app, so that pays for the extra gdrive storage so my photos are backed up full resolution. Shaneh used backblaze to back up her work desktop and laptop.
Well, pretty close. A Baud (bd) is defined as one symbol/s, which can be a bit or more. Given the quality of the phone lines at that time, a nibble (4 bit, 16 states) was maybe possible, but mostly it was just one state change/s, and that is a bit/s then. That’s also what modems of that time (and even acoustic couplers, btdt) could handle. Today’s Gigabit cables can normally differentiate 4 states (i.e. voltages or pulse widths/positions, or combinations) per wire(pair), but only if they are good…
I would not dare to recommend any product, just mention that here two synology DS 1813+ equipped with 32 TB each have been working flawlessly for years. One failed recently all of a sudden, dead, quiet, no light, no sound. After getting a new power supply which did not alter the status I found out that it was a broken contact in the power supply cable. So, at least, I’ve got a spare PSU here now
I run a multi-tiered storage setup with two different forms of backup.
My desktop has a large amount of storage for local file access… pictures, movie creation/editing, games. This system uses an online ‘real-time’ based backup using Crashplan. I pay for the service. It provides unlimited storage/recovery and uses online encryption.
I have a server sitting in the shop with 25T of storage (only about 3TB used). It’s running ESXI and using raid-6 with 2 hot-spares. It houses all of my movies and music on an NTFS share and uses plex to share it out to my other devices. I also run a dozen other VMs on it doing multiple things. Nothing on this server is extremely important. I have a script that runs once a week that pushes all of the music and movies up to a glacier storage in AWS. The glacier storage is pretty cheap, but you get charged access fees. The initial push when I first set it up cost me about $25. If the drive ever fully crashes and I have to download, it will cost me quite a bit and take a long time, but it’s a risk I’m willing to take. Worst case, I’d have to spend the time to re-rip all my movies and music.
If I can ever find a truly useable backup solution that takes advantage of glacier and doesn’t cost an arm and a leg, I’ll probably move to that. Especially if it offers a two-tiered approach where it will backup my desktop both to a drive on my local network AND glacier. That was if I need to restore a file, I can restore from local, but if local takes a dump, I can then pay to pull form glacier. I honestly haven’t looked in to this in over a year. There might be something out there at this point.
Your rpi setup is something I toyed around with. I had two setup with usb drives. I ran a nightly rsync that synced between the two. The idea was to setup my parents to backup to one at their house and a second one to backup my stuff here.
. It just works. I don’t use any of their q services though. They have a bunch of qphotos and qvideo sharing stuff. I just use smb and ssh and the http interface. The biggest thing is, my wife has zero tolerance for my projects when she has to use them. She is super smart, a very talented engineer. But I have to update the shortcuts on her computer if I change something because she just won’t want to deal with it. That’s why we have the google storage and the nas. They have very reliable interfaces. I just make sure the nas has the same IP and I don’t cause her a lot of stress.