Optimal setting for How Much Disk Space to Use

[samccarthy]

I use MultiServer and do my backups, 8 servers worth, to a couple of NAS units. I have found myself running out of room, or the Grooming runs out of room and I have to manually fix this. When I setup a new backup set it asks How Much Disk Space to Use. (Use at Most). I have searched high and low for the optimal setting here, but cannot find it.

 

Here is an example

My NAS is 1.2 tb and my backups are set to keep 30 days

Server 1 may use 75gb over 30 days

Server 2 may use 100gb over 30 days

Server 3 may use 375gb over 30 days

Server 4 may use 500gb over 30 days

 

If I set them like this I invaribily run into problems.

 

Can I just leave them at the default of 1.2tb or 99%?? If I do that what happens with my drive space. Will Retrospect keep my backups for 30 days and groom and just use the space that is needed or do I need to set these limits so things don't get out of control?

 

Any assistance would be appreciated.

[Mayoff]

Please see http://kb.dantz.com/article.asp?article=9629&p=2

[samccarthy]

Great article on Grooming, but it did not really answer my question. Any other insight?

[Mayoff]

I always use 99%

[samccarthy]

With using 99% and 30 backups maximum, is Retrospect smart enough to police itself and only use the disk space it actually needs for the 30 Backups? The easiest thing for me is to leave them all at 99% and let Retrospect take care of what it really needs instead of me having to guess at whether I allocated the right amount of space.

[Mayoff]

From the document I linked above:

 

If you set Grooming to "10", Retrospect will automatically keep as many Snapshots as possible, as long as you have enough available disk space on the destination backup drive. When the grooming operation completes, all but the last 10 Snapshots per volume will be removed. It is not unusual to have more then 10 Snapshots per volume on the backup if grooming has not activated for a long time. Keep in mind, if you delete a file from a hard disk after your second backup, it can not be groomed out of the backup until you have done at least 12 backups of the disk.

 

 

Grooming does not stop at "30" and then groom. You could have 40 or 50 per volume until the disk fills.

[samccarthy]

OK, so if I understand this right, WHEN it grooms, it will keep 30, but will basically fill up the disk until full, then groom. So, If I have 4 servers going to this NAS putting all at 99% may cause me problems as they will all go to fill up the disk space that they all share. If I give each their own Maximum, then I shouldn't have to worry about one stepping on the other. Is this correct?

[Mayoff]

yes, that sounds right.

[cmcfarling]

I've been wrestling with grooming for a while now. My conclusion at this point is that it doesn't work well. My current version is 7.5.387 MultiServer. I have a similar setup to sammccarthy. In my case the backup destination is a 7.5 TB RAID. I've got 7 backup sets all storing their data on that volume. Based on the documentation and how things are worded in the UI, it seems to me that you should be able to setup each backup set to use 99% of the destination volume. In my case the setting is "Use at most 7607 G or 99%". At most is the key phrase there.

 

Knowlege base article 9629 (referenced above) states:

"Don't groom too often. Grooming will automatically remove data when the destination disk fills up."

 

So it seems that we are at least being told that Retrospect is smart enough to groom data based on the amount of free space, or lack there of, remaining on a drive. In my experience so far this does not work though. My 7.5TB volume recently got down to 10MB (yes megabytes) of free space. I let it go just to see what would happen. After 5 or so days Retrospect never groomed out anything on its own. I ended up wiping out one of the backup sets, including all of the backed up data, and starting that one over from scratch.

 

I'm still in the process of trying to find a workable grooming solution that I can rely on not to fill the drive. All of this has come after mulitple talks with tech support. Initially I had set the grooming policy for each backup set to "Retrospect defined policy". After several weeks of letting that run, grooming was not working properly. That's when I was told by a tech that grooming is not really intended for enterprise level backups. Well if that was really the case then EMCInsignia has done a pis* pour job of explaning in their marking materials who should and shouldn't be using grooming. Actually though, I suspect that was either one techs opinion on the matter, or a general support response, now that customers are reporting many grooming related problems, that they think may hold them over until they can make it work as advertised. It was also recomended by tech support that the "Retrospect defined policy" not be used and that the "Groom to remove backups older than XX" be used instead. I specifically asked if switching from the former to the latter would cause any problems and was assured that it would not. I then did just that but Retrospect failed to groom. I then tried manually grooming and 2 of 3 backup sets that I was attempting to groom failed.

 

I could go on an on with details but the gist is that grooming is not reliable in my opinion. I'm sure there are users out there that are successfully using it without problems but it seems to have some definite shortcommings for a lot of people.

 

If indeed grooming is not intended for "enterprise level" backups, then EMCInsignia needs to make that clear before people shell out the substantial cost of the software.

 

Chris

[Mayoff]

Quote:

---

 

. That's when I was told by a tech that grooming is not really intended for enterprise level backups.

 

 

---

 

You do realize that Retrospect is an SMB product and not an enterprise level product. We have never marketed Retrospect for the enterprise. EMC sells Networker for that market.

 

Right now I have a grooming operation running on my backup server. It has almost 7000 segments to groom. It has been running for close to 24 hours and hasn't reached the 1/2 way point yet. I plan to let it finish, and I do expect it to complete successfully. The backup set is about 3 TB in size.

 

I have never tried (and EMC has never tested) grooming a set that is 7.5 TB in size.

[cmcfarling]

Yes I do realize that, and we're not an "enterprise" level business. We've got 1 location, 25 employees, 5 Windows servers and 1 Apple XServe. In total there is about 4 - 4.5 TB of physical storage that Retrospect backs up. This is hardly an enterprise operation. I got that from a tech I talked to who informed me that I was running an enterprise level backup. I think it was just another way of putting the blame on my setup rather than the software.

 

According to KB 9629, "We recommend keeping the total number of sessions in the backup set to under 4 or 5 thousand." So far I have been well under that limit. I haven't found any published limits on backup set size. Considering that inexpensive 1TB single drives are available today though, it would stand to reason that any backup app touted as a "server" version should be able to handle multi-terabyte backup sets.

 

Your 3TB is on par with the stuff I'm doing. I too am in the process of trying to groom a backup set. Here's the progress so far:

 

- backup set is 4.43TB

- Retrospect's storage folder (where the rdb files are stored) contains 13432 rdb files

- yesterday at 9:01pm I started a manual groom operation

- at 9:47pm the groom failed:

Grooming Backup Set Production2 failed, error -1101 (file/directory not found)

You must recreate the Backup Set's Catalog File.

 

- started catalog rebuild at 10:36pm

 

The rebuild is still in progress and is only a little over 25% done. Assuming it finishes without error, I'll attempt to groom again.

 

Unfortunately this is not the first time this scenario has happened. The all encompassing "corrupt catalog" problem has happened before. The only way to fix it last time was to completely trash the backup set and start over. That is one of my complaints with Retrospect, the catalogs seem to be very fragile and can become corrupt for who knows what reason. Others have expressed similar concerns on this forum as well.

 

Also just to note, I never said I had a 7.5TB backup set. I said I had a 7.5TB volume where backup data is stored.

 

Chris

[Mayoff]

My concern is the total number of segments: 13432

 

That is a lot of segments and it may be too much for grooming to realistically handle in some situations. It is more then double the number of segments I have in my set.

 

1) What disk is the catalog file saved on?

2) Is the backup data on a local disk or a NAS/Network volume?

3) How much free space is on the C drive?

4) What is the percent of disk fragmentation on the backup drive?

[cmcfarling]

Quote:

---

too much for grooming to realistically handle in some situations

 

---

 

That is a bit ambiguous. Does EMC have any hard data to expound on that?

 

1) H drive (local disk)

2) E drive (local disk, SCSI attached RAID)

3) 28GB, which I have monitored and verified is not the issue (i.e. the C drive is not running out of space)

4) According to Disk Defragmenter (this is a W2K server):

Total frag - 43%

File frag - 87%

 

I can predit that EMC would point to #4 and say that "oh obviously your disk is too fragmented". I have serious doubts about the relevance of that. Other than a short period where I gave Diskeeper a try about 4 years ago (and gave up on it), I've never defragged any servers. In 7 or so years I've never seen disk fragmentation cause problems.

[Mayoff]

You are right that the only obvious point for concern would be fragmentation, but I don't expect that to cause the -1101 error.

 

Typically we see this error because one of the backup segments is missing or damaged. When you do a catalog rebuild for "all disks" does Retrospect report any segments are missing?

 

How much physical RAM do you have on the server?

How many total files exist within the backup set itself?

 

Like I said in a prior post, a backup set with 13432 RDB files is very large, and larger then many of our customers typically have when using grooming. I can say for a fact that EMC has not tested backup sets with that many segments.

 

I did speak to a customer recently who had a chronic problem of grooming failures due to his backup set being too large. When he split his backup data into 2 different backup sets, he stopped having grooming problems.

[Mayoff]

One other note, when you have a backup set with 13432 segments, the grooming process is going to take several days, time that backup will not run.

 

On my own server, I estimate 7000 segments could take 2 days to groom.

 

By keeping the backup sets smaller (keeping fewer snapshots and splitting data between sets), grooming will be much faster and more stable.

[cmcfarling]

At 1529GB into the recatalog process there are no reported errors or warnings yet.

 

2.25GB of RAM in this machine

Can't check the total file count at the moment as the backup set is locked

 

I understand what you're saying about the number of segments. That is actually my intent, to parse this backup set down. But unless I want to wipe it out and start over I have to get a successfull groom completed in order to get rid of some of those segments.

 

If I can make that happen then I was going to set a limit to the amount of space the backup set can consume on the backup volume. As mentionend previously, the backup set is configured to use at most 99% of the volume. I'm going to change that to around 30%, or ~2.3TB. That brings up another question. Let's say I config the backup set in this way so that it is limited to using just 2.3TB of the 7.5TB volume. Also, the grooming policy is set to a defined number of snapshots, 10 just for examples sake. Which one takes precedence? As long as <2.3TB are used, Retrospect should adhere to the grooming policy and keep at minimum 10 snapshots. However what if the backup data reaches the 2.3TB limit and only 8 snapshots have been saved? Will retrospect try to keep going until it saves 10 snapshots even if it takes 3TB? Or will it go against the grooming policy and only save 8 snapshots in order to honor the size limit?

 

I'm curious, how long has your backup set with 7000 segments existed? It seems that with the way it's implemented the segment count could continue to grow indefinitely, even with grooming. As long as one file exists in a segment, that segment will exist. So over time I could see many segments being shrunken down but not completely eliminated. All the while new segments are constantly being created. It seems to me that EMC should be doing more testing on this and come up with some more concrete numbers to give to customers.

[cmcfarling]

Update...

 

The recatlog finished without error after 43 hours, 6 minutes. At that point the catalog stats were:

4543G for 292541 files, 975 sessions, 105 snapshots, 13432 rdb segments

 

At 1/6/2008 8:15am I started a groom operation (initiated via the Actions window). The groom completed successfully after 22 hours 30 minutes. At that point the catalog stats were:

1043.5G for 87642 files, 444 sessions, 105 snapshots, 4711 rdb segments

 

Yay, I finally got this thing groomed. That seems to only be half the battle though. See more trials and tribulations here:

 

Grooming fails to start in some cases

http://forums.dantz.com/ubbthreads/showflat.php/Cat/0/Number/105159/an/0/page/

 

 

Note that Retrospect groomed through 13432 segments, and nearly 4.43TB, in 22.5 hours. Mayoff, you mentioned that after 24 hours your 7000 segment, 3TB groom operation was not even half way done. Wonder why? This machine is no speed demon. It's a Dell Poweredge 1650 with 2.5GB RAM.

 

Chris

[cmcfarling]

BTW...

 

Quote:

---

You do realize that Retrospect is an SMB product and not an enterprise level product. We have never marketed Retrospect for the enterprise.

 

---

 

The argument could be made that Retrospect is being marketed to the enterprise. From the product web page:

 

"EMC® Retrospect® 7.5 for Windows is designed to deliver automated, reliable, cost-effective protection for small and medium businesses (SMBs) and the distributed enterprise."

[Mayoff]

Distributed enterprise is actually an odd concept. It really means the small offices that are part of an enterprise. Basically you can treat a "small office" in an enterprise like a small business due to the limited amount of data that the branch office would contain.

[MRIS]

Just returning to a much earlier question, about backup set sizing, multiple backup sets to the same target disk etc... here are some answers and an important point.

 

The important point: Retrospect does not perform low-disk-space-detection.

 

This means retrospect can't tell that there is only 6 MB left on a disk and thus start grooming.

 

What it can do, however, is know the size of the backup set, and the maximum size of the member volume allocated to it. When the backup set size reaches the maximum member size it then starts grooming to make room for the backup being taken.

 

What does this mean?

 

If you have 7 backup sets going to a 7.5TB NAS, then you should carefully allocate space to each backup set, eg:

 

Backup Set A : 1000GB

Backup Set B : 1000GB

Backup Set C : 1000GB

Backup Set D : 1000GB

Backup Set E : 1000GB

Backup Set F : 1000GB

Backup Set G : 1000GB

 

total allocated space: 7000GB.

This leaves 500GB free on the NAS unit for later.

 

Thus having 7 backup sets going to the one disk, it is INCORRECT to allow each set to use 99%.

 

In this example, you should allow each set to use a maximum of 13% instead.

 

In my experience, you must always leave about 10% of the space (thats the 500GB in this example) as working-overhead. Do NOT allocate all the space, make sure that the sum of the parts does not equal the total space.