version 10 and supposed grooming improvements

[davidduff]

i'm wondering about grooming under version 10.

 

as a preface, i will say that i could never really get grooming to work the way i wanted it under versions 8 (where i mostly didn't experiment much due to a variety of other problems and issues) and version 9 (where my experiments met with poor results leaving me thinking that i must be doing something fundamentally "wrong" in terms of how the program was designed to work).

 

my simplistic starting assumption is that with grooming, retrospect should be able to work somewhat like the way time machine does in that you could allocate some storage to it, let it start backing up and it would save everything (every version, every incremental backup, etc.) until it ran out of space, and then when the allocated space was depleted, it would start deleting the oldest, unused stuff to make way for new stuff. with a modest number of machines (3-4) and what seemed to me to be a modest number of backups (roughly 2x/week for a few months), i would find that my media sets would take a really really long time to groom (like ~3-4 days) under version 9.

 

so now i've installed version 10 and i'm trying it out. my first attempt is to groom an existing media set with 1TB of storage and a few million files. so far it's been running almost 5hrs. as with previous versions, there are NO meaningful progress indicators of any kind during grooming, which is quite frustrating (edit: see followup note below about progress reporting). but if i had to guess, i would say that version 10 does not look like it is going to provide any kind of huge improvement.

 

the web page describing mac version 10 says "IMPROVED! Automatic Disk Grooming Tell Retrospect how many backups to keep, and it will automatically remove older backups when it needs more space on disk-based storage. This technology allows Retrospect to perform faster, smaller Smart Incremental backups forever." this sounds like exactly what i'm looking for.

 

questions:

 

is there something i need to do in order to see this "improved" behavior and in particular to see faster grooming? for example, do i need to create new media sets?

 

is there some kind of first-time cost when v10 reads v9 catalogs before this wonderful new feature kicks-in?

 

any other advice on how to get "smart incremental backups forever" w/o having to recycle backup sets and hopefully without needing multi-day groom operations?

[davidduff]

i know performance numbers (even crude ones) are meaningless w/o more context...

 

the above is on an intel 2.4GHz core2 duo processor. it has only 6GB of ram but does not appear to be doing a lot of paging.

the media set is on an external firewire drive. catalog is on the system (SATA) drive.

mac os x 10.6.8 server

 

i'm using "groom to retrospect defined policy" option on the backup set in question.

[Maser]

I think it's supposed to be "improved" in that it's supposed to be more stable. In my (limited) tests with Retro 10, I haven't seen a whole lot of difference in grooming over Retro 9 -- but I have yet to put it to pace against my large, many-millions-of-files media sets yet.

 

 

What *mostly* reduced the amount of time (in Retro 9) to Groom my large media sets -- was upgrading the CPU on the system. Going from a core2duo to an i5 -- cut my Groom times in half.

[davidduff]

i actually have a replacement machine with quad-core i7 standing by waiting to replace this machine, but i thought i would try head-to-head comparison of v10 vs. v9 on the same hardware while i had a chance to do so.

 

as of now the groom op that i started yesterday has run for 20 hrs. it has consumed about 17 hours of cpu time (that's the time used by RetroEngine process and this has been the only job running since i installed v10 yesterday).

 

the media set in question is only 1TB in size. i say "only" - i realize 1TB is a lot of data. but the laptops i'm backing up have 500GB or 750GB drives now, usually over half full. so by my standards, i don't have the feeling that a 1TB media set is large. on the contrary, i would have thought that 10TB media sets and up would be common for users who are backing up machines for small/medium businesses.

 

if i were not facing these horrendous performance problems with retrospect (and i had confidence in its reliability), i would probably dedicate 4-6TB of storage to it. so that's what i mean when i say only 1TB.

 

one thing i should add - as of today, the RetroEngine process is competing for resources with the new RetroISA process on my backup server. i believe RetroISA scanned all the drives attached to my server and is continually re-scanning them looking for changed files. so while RetroEngine has consumed ~17hrs, RetroISA has consumed ~10 cpu hrs. i think i need to look into making some mods to the RetroISA config file to exclude certain of my disks from scanning -- however this seems like a questionable design - it looks like going from v9 to v10, we replaced the slow/inefficient on-demand scanning process with a different highly inefficient process of continuous scanning. isn't there a better way?

 

as a sanity check, i'll float these questions out there:

 

- how many machines are people backing up with retrospect server? any small/medium businesses out there backing up ten or more?

- how large are your media sets?

- is anyone really using retrospect grooming in any serious sustainable workflows? if so, how? how does it perform? what is your hardware setup like?

- anyone know what the limiting factor is on grooming performance? RAM? speed of access to catalog file? speed of access to member file system?

 

mysteriously, when grooming, my machine seems to use a ton of CPU, but (oddly) does not seem to do a ton of disk i/o. what is grooming actually doing? is there some general rule of thumb regarding how long grooming should take? am i seeing some kind of unforeseen resource contention or something? bad interaction with some other Mac OS X process (such as spotlight)? some strange quirk of mac os x server? would Retrospect benefit from more RAM if it were available?

 

some further thoughts on grooming...

 

it has been a slow process getting to the meat of this issue - going back to early versions of retrospect 8. for a long time, i assumed the problem was just due to the idiosyncrasies of my own system: somewhat old non-server-grade hardware plus i was using Drobo's for my backup storage. i didn't want to blame retrospect for known Drobo performance problems. then for a while after that, i wondered if i was trying to use Retrospect in some crazy or unreasonable way - such as letting my media sets grown outside retrospect's design limits.

 

however, in the case i describe above, the media set is what i think would be considered at most "medium sized" by reasonable objective standards. the mediaset is stored on a regular firewire hard drive (not Drobo) and the catalog file is stored on the internal SATA system drive. this is a freshly installed version of retrospect v10 which i installed based on the marketing claim of "IMPROVED! Automatic Disk Grooming".

 

i know that there are others who are using retrospect in different ways - for example, forgoing the use of grooming and doing frequent/regular rotation of backup sets. if people are able to get what they need from retrospect this way, that is great for them. this is not particularly interesting to me as a solution. on rare cases, i want expect retrospect to be able to reach back into a past state of my machine to recover some data. constant media set rotation essentially defeats this abiltily. without this ability, it feels that i might as well just fall back to using simplistic disk imaging-type software for backup.

 

i welcome any further discussion of grooming. if you are using it and not having probelms, please tell me. maybe i can figure out why it performs so badly on my system.

[davidduff]

one further note: i complained earlier that grooming doesn't display progress indicator. this was certainly true in v8 and v9.

 

as of now in version 10, retrospect is displaying "grooming segment 1835 of 2209" in the status window. so at least that's something.

 

otoh, the progress bar underneath is a static candy cane pattern. no movement and it does not track progress through the segments, even if movement through segments does not track exactly with grooming progress, at least it would be better than nothing to have it progress from start to finish by tracking segments completed. also the "grooming segment k out of m" message goes away sometimes (replaced by some other message such as "analyzing catalog data" or something), so if you look at the status at any point in time, you may or may not be able to tell how far along it is.

 

if i click on the "log" tab (vs. the "summary tab") looking at the grooming activity, it shows nothing. it doesn't even show the time the operation started! from past experience, even after completion, there will be nothing logged. it seems to me like more detailed logging would definitely help here. how many bytes/segments/files or whatever got processed? what was the rate in files/bytes/segments or whatever per second in each major phase of the operation? logging something like that might help me identify where my bottleneck is.

 

by linear extrapolation (which may be wildly inaccurate, i realize), it looks like i'm getting it is processing a bit under 100 segments per hour, and thus will probably end up taking an entire day of wall-clock time to complete grooming of this modest media set. and that's assuming that processing my 2209 segments represents the bulk of the grooming operation and is not merely a first of several passes or something.

[Maser]

- how many machines are people backing up with retrospect server? any small/medium businesses out there backing up ten or more?

 

Yes -- I back up approximately 55 clients spread out over 5 media sets (4 sets of Macintosh clients, 1 set of Windows clients) -- I also backup two servers -- each to their own media set)

 

- how large are your media sets?

 

 

In general, my Mac sets are (size-wise) about 600G each with about 1-3M files. My server sets are somewhat smaller and my Windows set it much smaller.

 

- is anyone really using retrospect grooming in any serious sustainable workflows? if so, how? how does it perform? what is your hardware setup like?

 

Yes! I have my client media sets all set to keep 60 backups. One server set keeps 90 backups and the other 180 backups. Each weekend, I groom one media set (because of how long it takes).

 

The last groom I ran groomed 92G from one Mac set and took 19 hours -- and this is on an i5 Mac mini. This was one of the longer grooms since moving to the i5 mac mini, though. I don't keep log records that far back to know if the groom of the same set took longer 5 months ago. This particular Media Set has (currently) the most files in it -- 3M.

 

I do a weekly groom of my Windows set and my 90-backup server set. Together, that groom takes 1.25 hours. The windows set is about 400G with 300K files, the server set has only 30K files, but is about 500G

 

 

- anyone know what the limiting factor is on grooming performance? RAM? speed of access to catalog file? speed of access to member file system?

 

There are two factors that affect this:

 

1) Number of files in the media set

 

2) Speed of the CPU

 

 

You can have a 1TB media set -- with only 400K files in it. That will groom *much* faster than a 500G Media set that has 3 Million files in it.

 

I tend to reboot my engine machine on Friday afternoon (when I remember) so it's a bit more cleared out before the grooms run. I don't think that makes much of a difference, though.

 

 

I, too, am interested to see what happens with Retrospect 10. My intent is to document a few more weekly grooms before upgrading to 10 (shortly) for comparison.

[Maser]

one further note: i complained earlier that grooming doesn't display progress indicator. this was certainly true in v8 and v9.

 

as of now in version 10, retrospect is displaying "grooming segment 1835 of 2209" in the status window. so at least that's something.

 

 

The "grooming segment" status is really the *only* status information you get -- that's the status that occurs when the actual grooming of the media set files is taking place. The non-indication-of-activity in the status *prior* to that is when the temporary file that is created that contains *what will be removed* is being generated. Unfortunately, there's no way (and probably never can be a way) to get a status on that progress of generating the list of what is going to be removed. It would be nice, but I don't see how you could estimate that for a progress bar...

[davidduff]

in the end, grooming took 25:42:57. it ended in failure. the only message that was visible in the activities window was "execution incomplete" with a little red "x" next to it.

 

if i click on the Log tab in the activities window, nothing at all is displayed. btw, i think this (not showing anything in the log tab) is a bug.

 

if i click on the view menu in the retrospect app and select Log (apple-L), then i see that grooming failed with error -2242 "catalog file duplicated or ambiguous". it says i need to recreate the backup set's catalog file (i think this is strange wording for what the UI refers to as "rebuild").

 

so i'll do a rebuild (which will take 10-15 hrs), then i'll retry grooming again.

 

but so far, this doesn't feel like any kind of big improvement - in either functionality or performance - in version 10 vs. earlier versions.

[Maser]

When you have a media set you can't groom (based on experience) what you should try:

 

1) rebuild the catalog file.

 

2) Do a "copy media set" script and copy everything from the possibly-bad media set to a new media set.

 

3) See if you can groom the new media set.

 

 

For a core2duo CPU, grooming a media set with "a few million files" sometimes took me a full 24 hours. The more files, the longer the groom.

 

But if your catalog file can't be rebuilt for some reason or the "copy media set" script doesn't work -- that indicates other problems with that set and you may just want to start with a new set and use that old set for restoring as necessary.

[Lennart_T]

Where is the catalog file stored?

It should NOT be on the same volume as the data files. Updating the catalog file needs lots of headroom, which you don't have if you run backups until the (same) volume is full.

 

I haven't used grooming on the Mac version, but have used it on Windows. A five year old dual core Xeon processor copes just fine with grooming multi-million file backup sets with a bit over 1 TB of data.

 

We schedule grooming every weekend to avoid filling the backup volumes to the limit.

[davidduff]

catalog file was on separate disk. disk space was not an issue on the drive containing the catalog file.

 

i have done a full rebuild. media set is functional again.

 

i notice last few backups show "using instant scan" in the logs. i don't see evidence in the logs that an (auto-)groom has taken place since the rebuild. i'll initiate one manually and see how it goes.

[mastercam]

I'll get into details later...the quick:

 

110+ machines

9 Backup sets from 1.4 to 2.0 TB each set.

Grooming have been fine...it's the catalog rebuilds that take 18-22 hours.

 

I'd be more impressed if the single instance storage would be better(I don't know how)....or if we stopped ordering 750+GB drives...but that won't be happening soon!

[davidduff]

thanks for the input. very interesting, mastercam and maser. when you get a chance, here are some additional details that would be useful:

 

do the details you provide above related to retrospect 9 or 10? (i assume 9)

 

have you had a chance to test retrospect 10 yet? if so, have you noticed any changes? i.e., any have you seen any improvements to performance (specifically wrt grooming) under v10?

 

anything interesting to report about your system config? what type of high-volume storage are you using for your media sets? for your catalog? are you using SSD or RAID?

 

you mention size of your media sets in terms of bytes. what about # of files?

 

regarding "grooming has been fine"... can you elaborate? are you using "groom to retrospect defined policy" or are you using grooming to keep around a fixed number of backups for each source? do you let grooming happen "on demand" or do you invoke it as a separate operation? how long do grooming ops take to complete?

[Maser]

Details I provided were with Retro 9. I have only tested Grooming with Retro 10 on some test media sets -- not my production media sets yet. I have been gathering some data on my grooms for the past few weeks so I'll have some comparision data when I install the 10 engine (when I get my code -- I don't want to install it in trial mode...)

 

My backups are stored on a Pegagus Thunderbolt RAID. My engine machine is now an i5 Mac Mini. Really -- increasing the speed of your CPU for your engine machine is probably the only thing that will make grooming faster (all other things being equal). Moving from FW800 to Thunderbolt had no affect on the grooming speed..

 

I think I mentioned above how many files were in my media sets. The sets with more files (3M files) take much longer than the sets with fewer files (27K).

 

I retain a fixed number of past backups (60 for most sets, 90 for one, 180 for another). I groom my two small media sets every Friday night and one large media set each Saturday night (these are scripted grooms.)

 

The amount of time *really* depends on the number of files in the set. In my larger post above, my "B" set -- took 19 hours. My "C" set took 10 hours. (My "D" set will be groomed this weekend...) My B Media set has 3.3M files, my C Media set has 2.1M. My D set has 2.8M files, so I expect it to take closer to 19 hours than 10 hours to run its groom...

[Monafly]

I'll weigh in here with another set of data points.

Current configuration: RS 10, Mac Mini (2 GHz core 2 duo w 4GB ram), external 3TB firewire drives.

I groomed a disk set with approx 2000 file and 1.5TB in about an hour (didn't pay close attention).

Next groomed one with 400,000 files an 60GB in a few hours (again, didn't pay close attention).

These both worked just fine.

Finally tried grooming a third set w/ nearly 4 million files and 3.5TB size. In 20+ hours it was closing in on completing (via the "grooming xxx of xxx" message it only had a few hundred segments left to go) when I left for the day. Next morning this media set was no longer visible to RS. No log entries to indicate what had happened to it. The backup set showed in the media window, but as it was locked, I attempted to unlock it only for RS to say "an error occurred". After a support call, it was decided to rebuild the catalog. 24 hours later the catalog was rebuilt, but now the proactive script that uses this disk set isn't backing up for no apparent reason. The is still an active support issue and I'm waiting to hear back from RS as to what the next steps are.

[Maser]

were there any errors in the log after you rebuilt the catalog? Can you do a non-scripted backup to the media set?

[davidduff]

quick summary of my experiments to date with grooming under retrospect 10:

 

a few times, i had media sets that seemed otherwise healthy, but when i tried to do a groom, problems were found and i had to fall back and do a full catalog rebuild. hopefully, that's not a recurring issue but instead is either a one-time v9-v10 transition issue or something. i will monitor ongoing behavior and see.

 

after rebuild, i can now get groom to complete successfully on a few of my media sets. here are some performance numbers:

 

mediaset A: ~13M files. 1.5TB. catalog size: ~22GB. FW800 drobo storage (slow)

mediaset B: ~4M files. 1TB. catalog size: 7.5GB. separate FW800-connected single hard drive.

mediaset C: ~13M files. 3TB. catalog size: 53GB. iscsi drobopro storage (somewhat fast).

 

results for mediaset A:

groom took 21.5hrs - "success" (but see note below)

rebuild took 19:02*

 

results for mediaset B:

groom took: (can't find the number here, but i think it was ~8-10hrs - i will edit this next time i groom this set)

rebuild took 7:18*

 

results for mediaset C:

groom took: <not yet attempted>

rebuild took 18:44

 

results marked with * above were done on a slower CPU with less memory so are not directly comparable.

"*" indicates ops done on core2duo 6GB ram.

no "*" indicates ops done on quad-core i7 with 16GB of ram.

 

all groom operations above are "groom to retrospect defined policy", which (perhaps stupidly/stubbornly) i am still continue to try to get to work -- for some definition of "work".

 

here's a big question: after full rebuild, followed by a lengthy groom (on mediaset A above) logs indicated that a successful groom had completed. a total of 1.6GB was groomed (about .1% of the total size of the mediaset). WTF!? (so in about 78000 cpu-intensive wall-clock seconds, i freed up 1.6 GB - roughly dial-up modem speed - not good).

 

the real question or issue is: how is this supposed to work? when i initiate a groom, i don't get to tell retrospect how much it should try to cull from the media set - so does it have some default target, e.g. 10%? apparently not based on the results above.

 

does it merely figure out the minimum amount of stuff it needs to cull to fit the existing space? well that might make sense for an "in-line" groom operation (my term), where retrospect knows (maybe) how much it wants to add to the mediaset as part of the current backup operation, but that can't be the way a manual groom works, since obviously there was enough space before, there's going to be enough space afterwards, even if nothing gets culled.

 

does a manual groom merely cull snapshots on a schedule basis and then delete whatever files from the mediaset that may no longer be accessed from snapshots? this seems to match the data. here's the retrospect (v8) manual on the topic: "When the backup drive fills up, or when you run a scripted or manual groom operation, Retrospect uses its own grooming policy to delete old backups. At a minimum, Retrospect’s policy retains two backups for each source, saving the last backup of the day for each source from the two most recent days on which each source was backed up. Given enough space in the Media Set, Retrospect keeps a backup of each source for every day in the last week, a backup for each week in the last month, and a backup for each previous month."

 

so i think this means that if i am using groom to retrospect-defined policy (GtRDP), that a manual groom may generally accomplish little or nothing in terms of recovering space. it allows me to assess performance and confirms the health of my media sets, but is otherwise mostly useless. the only way using GtRDP can accomplish anything useful is if it invoked in-line with a backup.

 

so for example, i'm trying to backup a user's machine over a network, and if one of them happens to be the unlucky one, their backup will trigger an inline groom, which may take 10+ hours to complete, after which their backup will finish if they've bothered to stay connected for that long. and at the end of this process, enough stuff will have been deleted from the mediaset to fit the new backup, but there's no guaranty that the next user to come along won't have the same experience.

 

in other words, GtRDP, unless i'm missing something GtRDP is really not useful for what i am trying to accomplish. it does not provide basic TimeMachine-like behavior of a simple "rolling backup set", or at least it does it so slowly as to be non-useful from an operational perspective.

[ferthalangur]

It would be fantastic if someone at Retrospect, Inc. took a few hours to write a good Knowledge Base article on what Retrospect stores and how grooming is supposed to work, functionally. I don't think we need enough information to be able to reverse-engineer the Retrospect software. However, it would be useful to have some detail about what is supposed to happen when Retro. grooms a Media Set to X backups. What happens when I delete a Past Backup and then groom the Media Set? What happens when I delete a source and then groom the Media Set?

[Maser]

I think there is some info, but having done grooming ever since it was first announced, I think I can answer some of this:

 

1) Groom to "X" backups. (Say X=5 here...) If you run more than 5 backups for a source, you will only see the last 5 in "Past Backups". When you *groom* the media set, you will remove the unique data that is in the media set for the older-than-the-most-recent-5-backups. Fairly simple. If you have X set to 5, but haven't run a groom, if you change X to 10, you'll see a quick relookup activity run so that there would now be 10 Past Backups listed for the source (they are there, but haven't been groomed yet, so you can get them back). If you change X back to 5, then the first 5 backups will be hidden again.

 

2) If you manually delete a Past Backup and then Groom, it still grooms to whatever your current groom settings are. In the example above, if you have X=5, but have done 10 backups, but you manually delete one of the 5 *visible* Past Backups it will *not* refresh to bring "backup 6" back to the list -- that one is already marked to be groomed. So if you run a groom at that point, it will delete the first 5 backups (the non-visible ones) *plus* whatever backup you manually deleted. It will *not* only just groom the one backup your removed if you check the box to "groom now". (It would probably be nice if it did, though...) (EDIT: If you mistakenly "delete" a Past Backup, the only way to get it back -- if you haven't groomed yet -- is to rebuild the catalog file, I believe...)

 

3) If you delete a source -- the backups stay in the media set. You have to manually delete the visible past backups for that source and *then* run a Groom script -- so it will delete the visible and non-visible backups for that source. If you do not manually delete the visible past backups, any other groom just deletes the non-visible backups.

 

4) Groom to defined policy -- there's a KB article about this somewhere (or it's in the manual). Something along the lines of keeping X number of days, then one backup a week for Y months and then one backup a month beyond that. I don't use it because I want "X" backups

[Monafly]

Update on my earlier post regarding missing media sets after running a grooming operation (w/RS10). All of this is using disk media sets.

I'm finally back to full functionality. To keep backups running, I brought in the alternate, off-site disk and it's media sets generally ran ok. I had to rebuild the catalogs for the troublesome media sets numerous times (like 3 or 4). Some of these took around 24 hours to finish and when done I'd find that while the media set seemed fine, there were no members recognized. RS just ignored my attempt to find them. During this whole process, at least on other media set that had not been involved in the grooming started acting errantly-such as suddenly scripted backups to it would require manual intervention because it "needed media". This was rebuilt to no avail, and eventually I had to resort to recycling the media set which wouldn't completely go away until I had deleted all evidence of the set from the disk and deleted the script. I started it over from scratch and it now works ok.

I have suspicions that the config file was corrupted but I didn't go so far as to start over from scratch with that, although I was close to reaching that level of frustration.

During all of this RS support generally said to rebuild or repair the catalogs, which of course was what I was doing, but it wasn't always working.

[ferthalangur]

Thanks Maser. That is very helpful.

[mastercam]

thanks for the input. very interesting, mastercam and maser. when you get a chance, here are some additional details that would be useful:

 

do the details you provide above related to retrospect 9 or 10? (i assume 9)

 

have you had a chance to test retrospect 10 yet? if so, have you noticed any changes? i.e., any have you seen any improvements to performance (specifically wrt grooming) under v10?

 

anything interesting to report about your system config? what type of high-volume storage are you using for your media sets? for your catalog? are you using SSD or RAID?

 

you mention size of your media sets in terms of bytes. what about # of files?

 

regarding "grooming has been fine"... can you elaborate? are you using "groom to retrospect defined policy" or are you using grooming to keep around a fixed number of backups for each source? do you let grooming happen "on demand" or do you invoke it as a separate operation? how long do grooming ops take to complete?

 

My error...I am using Windows 7.7. Although I believe this is a consistant issue across the board.

 

Number of files range from 6-8 million.

[henry-in-florida]

The "grooming segment" status is really the *only* status information you get -- that's the status that occurs when the actual grooming of the media set files is taking place. The non-indication-of-activity in the status *prior* to that is when the temporary file that is created that contains *what will be removed* is being generated. Unfortunately, there's no way (and probably never can be a way) to get a status on that progress of generating the list of what is going to be removed. It would be nice, but I don't see how you could estimate that for a progress bar...

 

Steve,

 

Is there a way to quit the groom while it's running?

Is there a way to rebuild the catalog while the groom is in an active state?