Disk Grooming versus Media Timeout

[Dustin.Loftis]

Hello there.

 

We are having a problem with Retrospect SBS, Multi Server and Single Server, all version 7.5, all with the latest patches and drivers.

 

We use SATA hard disks to run our clients' backups, generally using two disks and swapping them on and offsite.

 

The problem I am encountering is this:

 

BackUpSet_A is the drive in the system. It has only a few GB free, but backups are not failing; grooming is working.

 

I remove BackUpSet_A and insert BackUpSet_B. For reference, let's say BackUpSet_B is half-full. It really doesn't impact this scenario either way.

 

My backup scripts are set to back up to both drives, so that they will back up regardless of which drive is in the system. The media timeout is set for two minutes, so the scripts trying to back up to the off-site set will timeout when it is not found.

 

The problem is this: When the script tries to back up to BackUpSet_A, it initiates a groom. The grooming system attempts to locate the missing backup set, and prompts the user to insert it.

 

This dialog does not time out. It also prevents all other scripts from running. It even keeps Proactive Backups from occuring. The groom dialog will still be sitting there a week later when I swap the drives again.

 

The only solution that will get the backups running when the offsite set is missing is to report the set missing/damaged, or forget it completely.

 

Is this something that can be addressed in an update? If the media sets are supposed to time out, they should time out regardless of whether grooming is needed or not.

[Mayoff]

That is interesting, and could be a bug. I will report this to our engineers for review.

[Dustin.Loftis]

In addition, if I change the settings on a backup set to maintain only two copies of each backup, it appears it will not groom out SQL or Exchange databases automatically.

 

Is there a reason for this?

[Lennart_T]

What happens if you schedule a "Groom" script a few minutes before the actual backup script? Do you get the proper timeout then?

[Dustin.Loftis]

No, the script does not timeout, and all activity after the scripted groom never happens.

 

I changed out the drives on the 5th, and a groom started the next day, which stopped all activity until I returned on the 12th. There were no backups for an entire week because of this issue.

 

Additionally, you cannot click "cancel" on the grooming prompt, it just sits there. You have to choose a "missing member" or force close Retrospect to move on.

 

 

 

[eppinizer]

Dustin Loftis,

 

Additionally, you cannot click "cancel" on the grooming prompt, it just sits there. You have to choose a "missing member" or force close Retrospect to move on.

 

This is a recently found issue and is being worked on. At the moment there is not ETA for a fix.

Edited May 12, 2008 by Guest

[Dustin.Loftis]

Checking to see if there has been any mention by the dev team on this.

[Dustin.Loftis]

7.6 Does not resolve this problem. We've changed the backups for all of our clients to Proactive only to prevent this issue from crippling them, but when the proactive backups fill up, we cannot script a groom to occur on weekends because of this issue.

[mmsadmin]

Don't hold your breath. I reported this problem back in December and still have no resolution. I need to have a FULL backup every single day, with an off-site backup as frequently as possible. The only way for me to accomplish this is to have 2 separate backup scripts and 2 backup sets that alternate running every other day, and swap the hard drive every day. If the wrong hard drive is ever plugged in, the backup doesn't get done, period. This is a really bad thing. One can't expect to be in the office every single day, and a business can't afford to be without a daily backup, so I rely on the email notifications to let me know if I ever forgot to swap the drive. I then have to remote into the retrospect server and reconfigure the script to allow the backup to continue that night. If I am not able to do that, the backup doesn't run at all. I'm really surprised that there isn't a new post about this every day. Are we really the only ones backing up to hard drives for off-site storage? Is everyone else so perfect that they never miss swapping the hard drive on any single day? If only the Media Timeout function would actually work, I wouldn't have this problem. I actually regret purchasing Retrospect for this very reason (and a few others).

 

Proactive backup solves this problem, but causes other problems that are even worse. I require an email notification when backup is complete to let me know if there is a problem. With proactive backup and backing up individual exchange mailboxes, I have to weed through hundreds of emails each night to see if everything was backed up (because proactive sends a separate email for each mailbox). The proactive backup also takes much longer on my system than a regular backup, for whatever reason. But the quantity of e-mails received after each night requires too much administration and leaves too much room for error. And so I cannot use the proactive backup.

 

Which leads me into the reason why I have to do individual exchange mailbox backups in the first place. I have a public folder that is too large to be backed up. The only way for me to exclude this folder from the backup is to do individual mailbox backups. And, of course, there is a bug with this as well. When a mailbox is deleted from exchange, retrospect still tries to back it up, even after its purged, causing the backup to report as failed. The mailbox has to be manually de-selected from the configuration. It also doesn't automatically back up new mailboxes when they are added to exchange (when doing individual mailbox backups). The only mailboxes that are ever backed up are the ones that are polled from exchange when the license is first installed. The only way to get new mailboxes to show up on the list, and get backed up, is to re-install the exchange license. So I have to reconfigure the backup every time someone is fired, and remember to re-install the exchange license every time someone new is hired. It's an administrative nightmare.

 

All in all, when I spoke to tech support about my problems, He insisted that my installation was corrupted. However, even after reinstalling several times, I've still got the same problems. I was left with a very bad taste in my mouth from this.

 

http://forums.dantz.com/showpost.php?post/72935/

[mmsadmin]

So after I re-examined the "Media Timeout" feature in Retrospect today. The description of the feature is as follows:

 

'Terminate execution after waiting this interval when required media is unavailable.'

 

According to this description, if I have 2 backup sets configured as a destination, and one is not available, the script would then terminate and the backup would not occur. So according to the description, the function is actually doing what it says. Although when I've tried to use it, it actually never timed out and terminated the script.. It just continued wait for user input indefinitely. And so I do believe there is a bug in this feature, but I don’t think fixing that bug will solve the problem we have because we want the script to continue; not terminate.

 

No end-user or administrator can be expected to always guarantee the swapping of media on any specific day. If the person is not able to get to the office one day or has an emergency situation that precludes them from reconfiguring the backup, the program should still be backing up the system if there is any destination backup set available at that time. If the regular backup would handle destinations in the same manner as a proactive backup, it would.

 

In other words, if multiple backup sets are configured as a destination, then the program should back up to whatever destination sets are available to it at that time and continue normally. This would allow administrators to implement an effective off-site rotation strategy while guaranteeing that there will always be a full backup of the systems in the event of a failure to swap media.

 

I really hope that EMC will find a solution for this. Even if its in the form of an option or preference, it would allow me to sleep better at night knowing that I am always going to have a backup. Right now it's an administrative nightmare having to verify that the right backup drive was swapped in every day. If ever I am unable to for whatever reason, there is no backup that night, which is unacceptable.

[mmsadmin]

Now that I've re-read this thread and thought about it even more, I realized that fixing the bug would partially solve the problem. If the media timeout feature actually terminated the script and allowed further scheduled items to execute, I could accomplish what I want by creating 2 identical backup scripts to run every day, each with it's own dedicated backup set assigned to it. However, this would cause me to receive a "backup failed" notification or something of that sort every day. But it would still be more acceptable than not having a backup at all. Ideally my original suggestion would probably work better, but I'll take whatever I can get.

[Mayoff]

1) I have reproduced the media request timeout problem with a disk backup set and grooming turned off. This is a bug (in my opinion) that I am hoping we can try to fix.

 

2) Engineering is taking the position that grooming should ONLY be used when all members of a backup set are online during the groom attempt. Swapping of member in the middle of a groom attempt is unsupported.

[mmsadmin]

Thank you Robin,

 

With regard to #2, I don't think anyone has described this as being an issue. Obviously in order to groom a backup set, it (and all associated members) need to be present. However, if a backup script attempts to groom a backup set that is offline, it should time out and continue with the rest of the script. In the original posters scenario, I believe each hard disk he was using for backup was a single-member backup set.