Disk Media Sets Have Wrong Free, Capacity Columns

[BHIAdmin]

Hi,

 

After upgrading to v. 9 (Mac 10.6.8) I noticed that the columns for media sets (all of mine are the 'disk' type) are incorrect:

 

Used = Used

Free = Assigned Capacity (i.e. the 'Use at most...' setting when making/editing a disk set) OR some completely different number

Capacity = Used + Free (obviously, but since Free is wrong, it is wrong as well)

 

Capacity should be the assigned capacity for the set, like as it was displayed in v. 8. This led to apparent large capacity jump across all of my media sets (which did look nice at first). I have each set on a drive figured to the GB of what they are allowed to use, in disk space. So, this error throws off my notification for media sets that are an incorrectly-allocated size or in need of a manual grooming.

 

The media set columns show up in several places in the admin console (like when manually running a script), and they are incorrect in those places as well (beyond the general Media Sets panel).

 

I don't have any other type of sets, so I haven't seen this error other than for disk sets.

 

 

Here's an example of the funkiness: Hard drive with total formatted capacity of 319.73 GB (Retrospect thinks its capacity is 297.8) with only two media sets assigned to use no more than 148 GB each (50% of drive)

 

Media Set A shows: Used=146.5 GB, Free=113.4 GB, Capacity=259.9 GB

Media Set B shows: Used=38.2 GB, Free=112.9 GB, Capacity=151.1 GB

 

The Used numbers are more or less correct at 184.7 GB (though Disk Utility shows 198.45 GB) but what's with the Free and Capacity numbers? Set B is close, but Set A is certainly not.

 

 

Is anyone else seeing this? Shouldn't Capacity be the assigned 'Use at most...' number or percentage for a given disk set? BTW, changing the 'Use at most...' setting doesn't affect the column calculations at all (odd, I think).

 

Larry

[Guest Steve Maser]

 

Media Set A shows: Used=146.5 GB, Free=113.4 GB, Capacity=259.9 GB

Media Set B shows: Used=38.2 GB, Free=112.9 GB, Capacity=151.1 GB

 

The Used numbers are more or less correct at 184.7 GB (though Disk Utility shows 198.45 GB) but what's with the Free and Capacity numbers? Set B is close, but Set A is certainly not.

 

 

Is anyone else seeing this? Shouldn't Capacity be the assigned 'Use at most...' number or percentage for a given disk set? BTW, changing the 'Use at most...' setting doesn't affect the column calculations at all (odd, I think).

 

Larry

 

 

Set B was created after A did backup of about 100G, right?

 

If so, the full capacity of B could only be the maxim of the free disk space remaining when you created it. The numbers make sense to me.

[BHIAdmin]

Set B was created after A did backup of about 100G, right?

 

If so, the full capacity of B could only be the maxim of the free disk space remaining when you created it. The numbers make sense to me.

 

Sure doesn't make sense to me. Set B was created after A; but, what does the capacity of the drive when the media set's member was made have anything to do with what it is supposedly limited to via the 'Use at most' setting, especially if other set members on the drive expand over time? In v. 8 I had the Percentage Used column yellow or in the red for several media sets that were near the limit of their 'Use at most' setting (and the Capacity column, I swear, always showed the 'Use at most' sum for all members). Then after upgrading, all my sets miraculously went down to less than half (green) relative to their new capacity that should never be reached. Most sets now show the 'Use at most' setting under the Free column!

 

All of my sets have only one member, so the Capacity for the set should match that of the one member. Regardless the Capacity of a given set should be the sum of all member's 'Use at most' setting (No?).

 

In v. 8 when I changed a member's 'Use at most' setting, the Capacity column for the member updated to show that, as well as the Capacity for the set. It does not do that now for any of my many sets except for the one new disk media set that I created after upgrading to v. 9, to handle the new client on-demand backups. Only that new 'v.9' set displays properly and updates its display as expected when changing the 'Use at most' setting of it one member.

 

I think this is a serious GUI bug. One that I am not ready to rebuild (from scratch) all of my media sets to fix. However, regardless of the displayed Capacity, sets are being auto-groomed relative to their 'Use at most' setting (though my catalogs seem to become corrupted).

 

Is anyone else seeing this disk media set display issue after upgrading to v. 9?

 

Larry

[twickland]

Is anyone else seeing this disk media set display issue after upgrading to v. 9?

We were seeing a somewhat different but possibly related issue in Retro 8.2. Whenever I attempted to adjust the "use at most" figure for an existing disk media set to use a lesser amount, I always ran into problems. Sometimes the change would not take. The new figure would appear in the Media Set> Members window, but when I switched to a different window and back again, the capacity figure would sometimes be different. If the new capacity figure remained stable, we would always find that shortly thereafter, the media set's catalog became corrupted. The media set was configured to groom, but was not yet at capacity (either with the original or the new capacity figures) and no automatic grooming occurred. I didn't investigate the issue further, deciding it was simply safer not to tweak the allowed capacity figure for an existing set.

 

Assuming that, per your other posts, you've already tried rebuilding the catalog, the issue you're experiencing may be due to a bug in Retrospect, or it might be a corrupt configuration file. The only sure test would be to check it under a clean, virgin configuration. To do so, quit the console, stop the engine, and drag the files Config80.dat and Config80.bak to the Desktop or somewhere else safe. Restart the engine, relaunch the console, locate the media set's catalog, and see if the behavior remains the same. To be even more clean in your testing, you may want to skip locating the existing catalog and go directly to rebuilding the catalog from the media.

 

If you do perform any testing, please report back with your results.

Edited December 15, 2011 by twickland

[Guest Steve Maser]

Sure doesn't make sense to me. Set B was created after A; but, what does the capacity of the drive when the media set's member was made have anything to do with what it is supposedly limited to via the 'Use at most' setting, especially if other set members on the drive expand over time? In v. 8 I had the Percentage Used column yellow or in the red for several media sets that were near the limit of their 'Use at most' setting (and the Capacity column, I swear, always showed the 'Use at most' sum for all members). Then after upgrading, all my sets miraculously went down to less than half (green) relative to their new capacity that should never be reached. Most sets now show the 'Use at most' setting under the Free column!

 

All of my sets have only one member, so the Capacity for the set should match that of the one member. Regardless the Capacity of a given set should be the sum of all member's 'Use at most' setting (No?).

 

Larry

 

 

I think I get your point, but I think you misunderstand how it functions.

 

 

Say you have an empty external 500G (and I'll use round numbers for this example) hard drive you are using to store your media sets.

 

While the external drive is empty, if you create 3 new Media Sets to be located on that drive -- all 3 media sets will have a Capacity setting of 500g -- even thought it would be impossible to fill all three sets with 500G of data simultaneously.

 

If you back up 150G of data to each of those 3 sets, you will see that Used is 150G, but "Free" is still 350G -- for each set. But there's only 50G left on the external drive.

 

If you made three *new* sets at this point (using the default settings) the capacity of those 3 new sets -- would each be 50G only. Even if you tried to set the % to "100" -- it wouldn't let you use more than the available free space of the disk.

 

 

The Percentage "Use At Most" setting is *only* used when setting up the media set initially *or* when modifying the capacity of the Media Set by clicking the pencil icon in the Members tab for the media set to "resize" it.

 

"Use At Most" is not a *dynamic* setting (which is what I think you are thinking it should be, maybe?)

 

 

All I can say is that if the Capacity values for media sets changed between moving from 8.2 to 9.0 -- then I missed it entirely. It's been quite some time since I set up my media sets (most were set under 8.0 or 8.1), but all I can say is this is what my media sets look like:

 

 

 

 

and my external drive is a 4.5 TB RAID (with only 640G of free space on it). Which is why I groom some sets weekly and other sets monthly -- at this point I have a good handle of how much data I back up and how much data gets groomed off for my clients when I run grooms.

 

 

 

If I'm missing your point, then I apologize, but what I indicate above is how I've always seen it work.

Edited December 15, 2011 by Steve Maser

[BHIAdmin]

If I'm missing your point, then I apologize, but what I indicate above is how I've always seen it work.

 

Thanks for your input. To test your assertion, in case I was missing something, I created a new disk media set with the following disk setup:

 

 

Backup 3

 

Disk Utility:

Capacity = 999.86 GB

Used = 382.25 GB

Free = 616.61 GB

 

Retrospect (under Sources section):

Capacity = 931.2 GB

Used = 356.9 GB

Free = 574.3 GB

 

 

Backup 6

 

Disk Utility:

Capacity = 999.86 GB

Used = 445.92 GB

Free = 553.94 GB

 

Retrospect (under Sources section):

Capacity = 931.2 GB

Used = 415.3 GB

Free = 515.9 GB

 

 

 

Made new disk media set named 'Test Set' and added two members of 50 GB each, one on each of the two above drives, which also currently contain other media sets.

 

 

Here are the results

 

 

After set creation with first 50 GB member...

 

 

After adding second 50 GB member...

 

 

And, the set members...

 

 

 

So, you should now be able to see what I am talking about. Disk media set's capacities are the sum of 'Use at most' setting, with no indication of remaining actual free disk space, since that is accounted for when creating the member. What happens later after some editing is when the bug shows up, or maybe it was during the upgrade, then the numbers get wonky.

 

 

I will try to recreate some sets and add existing members, to see if that fixes things. Then, I'll test messing around with the 'Use at most' setting for the Test set and backup some media, etc. to see if the bug presents itself.

 

 

Larry

Edited December 15, 2011 by BHI_Ad_Min

[Guest Steve Maser]

So, am I reading this right?

 

What you posted above is *not* showing a bug. You added two members -- which you set at 50GB each -- to a media set which now has a capacity of 100G total.

 

If you start using this media set it should (I believe) fill up the member on "backup 3" and then start filling up "backup 6" -- until you've reached 100G total space for the entire media set.

 

Because you explicitly set the members to use 50G, the media set doesn't know about any other free disk space -- it only knows what you told it the maximum amount of disk space that the media set *could* use. If you filled up both external drives so there was only 2G free space left on them -- it will still report the Capacity of the media set to be 100G.

 

 

Maybe I'm missing something obvious...

[BHIAdmin]

What you posted above is *not* showing a bug.

 

You are correct. My point with the Test Set was that this is the expected behavior with newly minted sets and members, noting that the Used, Free and Capacity columns work as expected, relative to the cumulative member's 'Use at most' setting. However, ALL of my disk media sets have always been created this way, with a specific 'use at most' setting that is never the free disk space available at time of member creation, excepting the last member allocated to a particular drive where that member's 'use at most' completely allocates the total drive's space. There is definitely something wrong, because none of those old media sets display correctly now, after migrating to v. 9.

 

I have tried deleting the catalog set, then trying a rebuild, but there is no way to indicate what added members should 'use at most' in the rebuild dialogs (GUI oversight?). This results in a set with members that exhibit the same whacked numbers as before, seeming unrelated to to any 'use at most' setting or even free disk space. However, the old 'use at most' setting now bizarrely appears under the Free column, as I indicated before.

 

So, there appears to be no way to rebuild a catalog with 'use at most' settings for its members. This is maybe where the bug comes in. Changing the 'use at most' after rebuild for a member does nothing to the set's Capacity column, though it does with newly minted sets (expected behavior).

 

 

Also of note, these messed-up sets have single members, and I can not delete them in the members panel. The '-' icon is greyed-out. If I just try to re-add the member to a re-created set with the same name, I just get the red error text stating that it already belongs to a disk set (even though that set no longer exists). Bummer, that would be the easy fix.

 

 

 

I am trying to find a workaround to this issue that will:

 

1) Recreate the set members with appropriate 'use at most' settings.

 

2) Maintain the actual existing backup files in current members, since I need those backups to continue to grooming correctly, keeping X number of backups, etc.

 

3) Somehow have the files on the old members become the files recognized under the new member.

 

Then I would be back in business. Kind of a member rebuild, instead of a set rebuild. Anyone have any ideas on how to do that?

 

 

 

My recommendation to the Retrospect engineers is that when rebuilding or creating a set, allow a 'use at most' setting to be chosen during the process for members (obviously this happens now when creating a set), then save that info to a file in the member's directory location. That member setting is now available, even if an admin has to trash their configs and start over from scratch (where I'm at now). Then on a set catalog rebuild, the setting is parsed and offered as the default for 'use at most' for that chosen member, which can be overridden then, if needed.

 

If there is already something saved with a member's files that indicates the 'use at most' setting (which I suspect that there is since it is showing up incorrectly in the Free column) I suggest that the rebuild code has a bug that is placing it in the wrong attribute for the member (Free instead of Capacity).

 

 

I will be submitting a bug report now, referencing this forum topic.

 

 

Larry

[BHIAdmin]

Further testing results:

 

I created a new disk media set and member and backed up a Mac to it...

 

 

'Use at most' for set's only member: 50 GB

 

 

Displayed set columns:

 

Used = 0 B, Free = 50 GB, Capacity = 50 GB

 

 

 

Backed Up: 9.4 GB

 

 

Displayed set columns:

 

Used = 9.4 GB, Free = 50 GB, Capacity = 59.4 GB

 

 

Obviously, there is a problem here. Capacity should not increase by the total backed up size every time! Some code is adding Used to the Capacity column when it should be subtracting it from the Free column. At least the whacked numbers I am seeing somewhat make sense now.

 

 

Larry

Edited December 15, 2011 by BHI_Ad_Min

[Guest Steve Maser]

I can confirm what you outlined above. I'm seeing that issue for pre-9 created disk media sets as well (if I erase them and reset their capacity), too.

 

Nice catch!

Edited December 16, 2011 by Steve Maser

[Guest 1TallTXn]

Further testing results:

 

I created a new disk media set and member and backed up a Mac to it...

 

 

'Use at most' for set's only member: 50 GB

 

 

Displayed set columns:

 

Used = 0 B, Free = 50 GB, Capacity = 50 GB

 

 

 

Backed Up: 9.4 GB

 

 

Displayed set columns:

 

Used = 9.4 GB, Free = 50 GB, Capacity = 59.4 GB

 

 

Obviously, there is a problem here. Capacity should not increase by the total backed up size every time! Some code is adding Used to the Capacity column when it should be subtracting it from the Free column. At least the whacked numbers I am seeing somewhat make sense now.

 

 

Larry

This is exactly what I'm seeing too.

The 'Free' column remains at what I set the media set "use-at-most" as.

 

This is true of both 8.x created sets and 9.x created sets.

 

I did not have this issue under 8.x but do now under 9.x

 

I suspect it is just a display issue (simply calculating the column values incorrectly), in which case, the media set's aren't expanding with every backup, but as it's displaying the same way when selecting the individual members, I cannot confirm this.

 

Please fix this!

[BHIAdmin]

This is exactly what I'm seeing too.

The 'Free' column remains at what I set the media set "use-at-most" as.

 

This is true of both 8.x created sets and 9.x created sets.

 

I did not have this issue under 8.x but do now under 9.x

 

I suspect it is just a display issue (simply calculating the column values incorrectly), in which case, the media set's aren't expanding with every backup, but as it's displaying the same way when selecting the individual members, I cannot confirm this.

 

Please fix this!

 

 

Sorry, I forgot to mention that support did get back to me after reporting the bug on Dec. 12, 2011 with:

 

 

This is a known issue and we're working on a fix (hopefully) for the next release.

 

 

So, here's hoping it will be fixed in the next update. I think it is more than a cosmetic bug, though. My guess is that the fields in the GUI are assigned correctly, just that the underlying calculation is incorrect. I'm also guessing that same miscalculation is causing problems with automated grooming, too. But, I don't have any data to back that up.

 

 

Larry