Grooming Removes Snapshot Listings, But Not Files

[7e7ded5c-1194-4c60-bde8-107d31a92b2b]

I've got a weekly Groom process that is *supposed* to groom all my backup sets (stored to harddrive), and remove anything older than 7 days (nightly backups)

 

However, I'm seeing that the Snapshots for the backup sets show 7 days of data, but the files still exist on the hard drive. From what I can tell, 75% of the drive's contents is being taken up by .rdb files older than 7 days.

 

I have tried to manually remove the files to free up space on the drive, and for the subset of backups I've done this for, Retrospect hasn't complained.

 

Two questions on this part:

 

A ) Is grooming supposed to remove the snapshot from the backup listing, and remove the .rdb files from the drive?

 

B ) Is there any other way to force the removal of the .rdb files from disk through the application?

 

 

In addition, how does recycling work when working with .rdb files on a harddisk? Do I need to be using the recycling method to force override of the 7-day-old media to increase free space on the drive?

Edited April 21, 2011 by glasnt

[Lennart_T]

First of all, it's 7 snapshots, not 7 days.

 

Since you (probably) still have files older than 7 days still existing on the clients, there is no surprise that rdb files older than 7 days still exists.

The default size of an rdb file is 614,408 kB. Grooming may remove part of the file. The file isn't deleted until every part of its contents is groomed out. If any part contains an old file that still exists on the client, the rdb file is (of course) retained.

 

A ) Yes and Yes, but there are cases where the rdb files are retained, see above.

 

B ) Recycling. But that wipes out ALL backups.

 

Recycling wipes out all backups and all rdb files.

 

It seems as you have that much data, so grooming can't remove as much as you might want. If that is a problem, buy a larger hard drive. Hard drives have never been cheaper.

 

If you have a 20GB file that is updated every day (such as a database file), you will have 7 generations of that file in your backup set . That's 20*7=140GB.

Edited April 21, 2011 by Lennart Thelander

[7e7ded5c-1194-4c60-bde8-107d31a92b2b]

First of all, it's 7 snapshots, not 7 days.

 

Since you (probably) still have files older than 7 days still existing on the clients, there is no surprise that rdb files older than 7 days still exists.

The default size of an rdb file is 614,408 kB. Grooming may remove part of the file. The file isn't deleted until every part of its contents is groomed out. If any part contains an old file that still exists on the client, the rdb file is (of course) retained.

 

A ) Yes and Yes, but there are cases where the rdb files are retained, see above.

 

Most of the .rdb files (say 80-90%) have not been edited in the last day, and there are ~%60 that have not been edited in the last week. There are files going back to the start of retrospect backups that aren't being removed.

 

 

B ) Recycling. But that wipes out ALL backups.

 

 

Isn't there a way of trying to get Retrospect to realise that files AA00001 through AA00XXX are 'old tapes' and to re-use them?

 

 

Recycling wipes out all backups and all rdb files.

 

It seems as you have that much data, so grooming can't remove as much as you might want. If that is a problem, buy a larger hard drive. Hard drives have never been cheaper.

 

If you have a 20GB file that is updated every day (such as a database file), you will have 7 generations of that file in your backup set . That's 20*7=140GB.

 

I haven't done the maths, but it's a lot of data. We've already got tens of terabytes of hard drives already (RAID 10), and I've been told that we can get more hardware if required, but the amount of data being backed up is so large, that it's not reasonable to purchase enough drives to store it all. As in, the storage capacity of the servers we are backing up is larger than the backup server itself. So, given that retrospect can't work out that if I want 7 x

"Current Drive Usage" as my limit for storage, then I've got to give it the total size of the drive, and hope it re-uses the .rdb files?

 

Also, re: the entire process itself: I'm to understand that grooming is supposed to just remove the files inside the .rdb files, but it doesn't actually delete the files until what condition? if any .rdb file contains a file that still exists on the system -- so if I have a full system being backed up, the original series of .rdb files won't be groomed until every file has been edited? removed?

[7e7ded5c-1194-4c60-bde8-107d31a92b2b]

Also, given you're saying if a file doesn't change, it stays put; but I'm finding that I'm getting data outputs of the daily backups that match the used space on my system. That is, instead of 1 full backup and increments, I get full backups every day. Unless there's a setting for that to happen, I don't understand why its happening

[Lennart_T]

Most of the .rdb files (say 80-90%) have not been edited in the last day, and there are ~%60 that have not been edited in the last week. There are files going back to the start of retrospect backups that aren't being removed.

OK, that's interesting. More about that later.

Isn't there a way of trying to get Retrospect to realise that files AA00001 through AA00XXX are 'old tapes' and to re-use them?

Only if AA00XXX is the LAST file. That's when you do a recycle.

I haven't done the maths, but it's a lot of data. We've already got tens of terabytes of hard drives already (RAID 10), and I've been told that we can get more hardware if required, but the amount of data being backed up is so large, that it's not reasonable to purchase enough drives to store it all. As in, the storage capacity of the servers we are backing up is larger than the backup server itself. So, given that retrospect can't work out that if I want 7 x

"Current Drive Usage" as my limit for storage, then I've got to give it the total size of the drive, and hope it re-uses the .rdb files?

Retrospect never reuses rdb files. New backup are always gives the next higher number. See below about the old rdb files.

Also, re: the entire process itself: I'm to understand that grooming is supposed to just remove the files inside the .rdb files, but it doesn't actually delete the files until what condition? if any .rdb file contains a file that still exists on the system -- so if I have a full system being backed up, the original series of .rdb files won't be groomed until every file has been edited? removed?

The old rdb files are not kept in their entirety. As soon as a part of an rdb file is groomed out, the rdb file is reduced in size.

 

Also, given you're saying if a file doesn't change, it stays put; but I'm finding that I'm getting data outputs of the daily backups that match the used space on my system. That is, instead of 1 full backup and increments, I get full backups every day. Unless there's a setting for that to happen, I don't understand why its happening

OK, Suppose Retrospect really back upp all files every day. Suppose you have 100GB on a client. Then you get 100GB backed up every day. After 7 backups the oldest are groomed out. So you can have 1/7 capacity for old files or about 14%. But you claim you have 60%.

 

You can go to Backup set properties. It's the pane (tab?) to the right of "Snapshots", I can't remember its name. There you can see how much is actually backed up for each client (and volume).

[7e7ded5c-1194-4c60-bde8-107d31a92b2b]

OK, that's interesting. More about that later.

Only if AA00XXX is the LAST file. That's when you do a recycle.

Retrospect never reuses rdb files. New backup are always gives the next higher number. See below about the old rdb files.

The old rdb files are not kept in their entirety. As soon as a part of an rdb file is groomed out, the rdb file is reduced in size.

 

OK, Suppose Retrospect really back upp all files every day. Suppose you have 100GB on a client. Then you get 100GB backed up every day. After 7 backups the oldest are groomed out. So you can have 1/7 capacity for old files or about 14%. But you claim you have 60%.

 

You can go to Backup set properties. It's the pane (tab?) to the right of "Snapshots", I can't remember its name. There you can see how much is actually backed up for each client (and volume).

 

From memory, I have full 618MB .rdb files going back to the start of retrospect time. There are some in there that have less, but most are huge. Is it expected that I should have a full 'set' of the original backup, then only the diffs from then on (hopefully less than the 618mb per snapshot?) The actual snapshot listing only has the last 7, but I have sessions running back to the start of backups, regardless of how much I try and groom.

[Lennart_T]

From memory, I have full 618MB .rdb files going back to the start of retrospect time. There are some in there that have less, but most are huge. Is it expected that I should have a full 'set' of the original backup, then only the diffs from then on (hopefully less than the 618mb per snapshot?) The actual snapshot listing only has the last 7, but I have sessions running back to the start of backups, regardless of how much I try and groom.

That is really strange. What does your log say after running a groom script? (Or after running grooming manually)?

 

If you don't have a groom script or groom manually, Retrospect will groom only after filling the destination to your specified limit. I think that defaults to 99% of the disk capacity.

 

Here are two screenshots from my setup. The first is from an old backup set that has been groomed numerous times. All files are smaller than 614MB, some are missing (there are gaps in the numbering sequence).

 

The second is from a backup set that was recycled last week: All files are 614MB, except the last ones from each client. (Every new client always starts fresh with a new rdb file.)

 

[7e7ded5c-1194-4c60-bde8-107d31a92b2b]

That is really strange. What does your log say after running a groom script? (Or after running grooming manually)?

 

If you don't have a groom script or groom manually, Retrospect will groom only after filling the destination to your specified limit. I think that defaults to 99% of the disk capacity.

 

Here are two screenshots from my setup. The first is from an old backup set that has been groomed numerous times. All files are smaller than 614MB, some are missing (there are gaps in the numbering sequence).

 

The second is from a backup set that was recycled last week: All files are 614MB, except the last ones from each client. (Every new client always starts fresh with a new rdb file.)

 

 

We have a weekly grooming script, but all backup sets are set to 99% capacity (the default). Would it be an idea to set a limit of (retention) x (max storage size) per backup set to reduce filesize? I know the sum of all the maximum backup sets will exceed our current storage, but then at least the files should reuse themselves, yes? I think I tried this on a test backup, but it was asking me for new media from the overnight run. :/

 

The logs usually say 'grooming was successful', but comparing the before and after drive usage for the collection of backup sets, barely any data has been removed, if any.

 

I'm going to have to also check how much we are trying to backup vs disk usage, current storage capacity of backed up devices, and compare that to the storage we have on the backup server. As a side note, do you know an easy way to export any of the information collected from the retrospect clients? e.g. the data in the volumes, clients, backup set listings? I can manually collected this data, but it would be far simpler if I could export it from Retrospect.

 

Thanks for all your help so far

[Lennart_T]

The logs usually say 'grooming was successful', but comparing the before and after drive usage for the collection of backup sets, barely any data has been removed, if any.

The log should tell how much data that is groomed out.

EDIT: So would your e-mail notification, if you got that turned on:

Script: Groom Desktops
Date: 2011-04-25
Groomed 53,0 GB from Backup Set WinDesktops.

I'm going to have to also check how much we are trying to backup vs disk usage, current storage capacity of backed up devices, and compare that to the storage we have on the backup server. As a side note, do you know an easy way to export any of the information collected from the retrospect clients? e.g. the data in the volumes, clients, backup set listings? I can manually collected this data, but it would be far simpler if I could export it from Retrospect.

No I don't. You could do some screenshots, but that would be pixels, not "spreadsheet-readable" data.

 

Thanks for all your help so far

My pleasure.

Edited April 25, 2011 by Lennart Thelander

[7e7ded5c-1194-4c60-bde8-107d31a92b2b]

The log should tell how much data that is groomed out.

EDIT: So would your e-mail notification, if you got that turned on:

Script: Groom Desktops
Date: 2011-04-25
Groomed 53,0 GB from Backup Set WinDesktops.

 

I just ran a one off groom, and I got a similar log file. It groomed 41.2GB out of a 214.5GB backup set, so that's ok. Checking the file makeup, it's got a full set of .rdb 618MB files from day-0, and most files since then are partials, so that looks ok.

 

I can't check the main log from the main groom, because retrospect lost that log (more than 100 events ago, or something), but it's weird that the groom doesn't seem to be taking much effect when it's run as per usual.

 

Might have to go along the 'more space' route, after checking how much we're actually backing up. Over 5 days, we have 1.1TB more disk space in use, which doesn't make sense though :/

[Lennart_T]

I just ran a one off groom, and I got a similar log file. It groomed 41.2GB out of a 214.5GB backup set, so that's ok. Checking the file makeup, it's got a full set of .rdb 618MB files from day-0, and most files since then are partials, so that looks ok.

That sounds promising.

 

I can't check the main log from the main groom, because retrospect lost that log (more than 100 events ago, or something), but it's weird that the groom doesn't seem to be taking much effect when it's run as per usual.

OK, there should be a new grooming this weekend, right? Set up e-mail notification, so you can keep track of what is happening, even if the log is lost.

 

Might have to go along the 'more space' route, after checking how much we're actually backing up. Over 5 days, we have 1.1TB more disk space in use, which doesn't make sense though :/

Right, checking what gets backed up is the way to go. Maybe it's large database files? Databases that are backed up another way (normally by creating "dump" files and backup them instead of the running database.

[008e3f8f-ac2d-4494-8f82-24d5c4ae8d5b]

On a sidenote... If you remove a client from your script, its backed up contents will not be groomed from the backup set. You have to manually delete the snapshots before they can be groomed out.

 

Furthermore, did you create/recycle your backup set with your current version of Retrospect? In the past we have had some grooming issues that seemed version related, so most of the time we do a recycle after upgrading Retrospect.

[7e7ded5c-1194-4c60-bde8-107d31a92b2b]

Right, checking what gets backed up is the way to go. Maybe it's large database files? Databases that are backed up another way (normally by creating "dump" files and backup them instead of the running database.

I've upped the log retention to ensure I can see things from the weekend. (Should see stuff from 48 hours before still)

 

On a sidenote... If you remove a client from your script, its backed up contents will not be groomed from the backup set. You have to manually delete the snapshots before they can be groomed out.

 

Furthermore, did you create/recycle your backup set with your current version of Retrospect? In the past we have had some grooming issues that seemed version related, so most of the time we do a recycle after upgrading Retrospect.

As far as I know, we started with Retrospect 7.7 in January, with no previous versions.

[7e7ded5c-1194-4c60-bde8-107d31a92b2b]

Ok, I'm part way through my next groom (issues with windows updates meant this hasn't been running, because the retrospect launcher isn't working to restart retrospect on reboot, sad panda), and I'm getting a heap of -1101 errors: "Grooming Backup Set [xyz] failed, error -1101 ( file/directory not found)". I'm guessing this is related to the files I removed that weren't being updated, etc.

Removing .rdb files from explorer is bad, mmkay?

 

I'm hoping a re-catelog might fix it, but I"m thinking I might need to do a full backup of all the failed sets to get them back in a usable state, amiright?

[Lennart_T]

Yes, removing rbd files manually is very bad.

 

Yes, since you have NO idea which client files were in the removed files, you should (must?) do a "recylce" backup to backup all files again.