Recommendations D2D2T backup setup

[Joriz]

I’m trying to achieve an automated backup setup in Retrospect but I feel the setup needs some optimisation to run more error free. Please share your findings/recommendations. I’m running Retrospect version 13.

The purpose is a disk to disk to tape backup with 3 fileservers running FreeNAS as source and 3 different tapesets which I want to take offsite every week.

• The disk to disk backup scripts can run every week day during the night
• The disk to tape backup scripts can run during the weekend

I have 3 tapesets with each 2 members which gives me +/- 14TB uncompressed in total for each tapeset.
The tapesets are named:

• OFFSITE A
• OFFSITE B
• OFFSITE C

The idea is to rotate them every week. The tapeset should contain all the data of the week. So incase of a disaster on May 5th,  I can use tapeset OFFSITE A to restore the data of that week.

An example for May 2020:

Disk media sets
For each fileserver i made an individual disk media set.

• FILESERVER 1 DISKSET
• FILESERVER 2 DISKSET
• FILESERVER 3 DISKSET

The capacity of each diskset is 8TB and is stored on a HP SAS raid enclosure running RAID6 with a total useable capacity of 30TB.

The diskset grooming option is set to: no grooming

Sources
Each fileserver has many file shares using SMB protocol. Each file share is added as SMB share with the Administrator account as source.

Scripts
Disk to Disk backup
For each fileserver I made an individual script. The script type is set to “Backup”. In each script I selected the SMB shares for the corresponding server as source. As Media Sets I selected the corresponding Media Disk Set.

• FILESERVER 1 backup to disk
• FILESERVER 2 backup to disk
• FILESERVER 3 backup to disk

The scripts schedule options are set as follows:

• Start: 10PM
• Repeat: weekly
• Every: 1 week (on Mon, Tue, Wed, Thu, Fri)

Disk to Tape backup
For the Disk to Tape backup I made an individual script for each tapeset. The script type is set to “Copy Media Set”:

• Backup to tape OFFSITE A
• Backup to tape OFFSITE B
• Backup to tape OFFSITE C

Each script has the Disk media set of all 3 servers as source. As destination the corresponding Tape media set is selected.

• Backup to tape OFFSITE A
  • Source:
    • FILESERVER 1 DISKSET
    • FILESERVER 2 DISKSET
    • FILESERVER 3 DISKSET
  • Destination:
    • OFFSITE A

And so on for each tape set..

The scripts schedule options are set as follows:

• Start: 08:00AM
• Repeat: weekly
• Every: 3 weeks (on Sat)

I’m running this configuration for a month now but I think some additional configuration need to be done. Some things I have noticed:

• The used diskspace for each Disk mediaset (disk to disk backup) keeps growing while the data on the source (the fileserver) is not.
• I’ve been reading about grooming so I set the grooming options for each disk mediaset to keep 1 backup and “Performance-optimized grooming”.

After setting this option I started grooming but the used diskspace didn’t drop. For example the total used diskspace of the source is around 1TB while the Disk media set is showing 2TB used.

Because the Disk mediasets grow, 2 LTO7 tape are not enough as destination while the data should fit uncompressed:
Fileserver 1: 2.42 TiB used
Fileserver 2: 5.81 TiB used
Fileserver 3: 0.73 TiB used 

Tape recycle
Should I recycle the tapeset everyweek before doing the disk to tape backup to prevent Retrospect asking for new media? This means a fullbackup every weekend which can take a lot of time and can cause tape wear? What is recommended?

Errors during disk to disk backup
I often see errors during disk to disk backups for a list of files. The files are not open by a user when this happens. Also the Administrator account is used to read the data so I would not expect a permission issue. Often when I manually initiate the same script after the error occurred it runs error free…
    can't read, error -1.100 (invalid handle)
    can't read, error -1.101 (file/directory not found)

The script ends with a big red mark. So I’m not sure if the copied correctly and don’t want to find out when a disaster happens. The script doesn’t re-run automatically when an error occurred for example.

Disk to tape speed
The disk to tape backup speed is around 6 – 6.5GB/min. I’m wondering if this is normal and if not what kind of optimisations can be implemented? The file sizes are mixed but even with large files (200-300GB per file) the speed is not going up.

The datasets are stored on a HP P2000 SAS raid enclosure. With BlackMagic Disk speedtest it is showing 380MB/s read and 213MB/s write.

The tape library is a Neo T24 SAS LTO7 with HP LTO7 tapes.

The SAS HBA is an ATTO H680. The tape library and the HP enclosure are each connected to an individual SAS port.

Restore speeds are very fast, around 10-11GB/sec.

The Mac Pro is old but is not showing any high cpu / memory loads during backup:
Mac Pro Early 2008 / El Capitan / 2 x 2,8Ghz Quad Xeon / 8GB memory
 

[Nigel Smith]

Easier stuff first...

3 hours ago, Joriz said:

I often see errors during disk to disk backups for a list of files. The files are not open by a user when this happens…
    can't read, error -1.100 (invalid handle)
    can't read, error -1.101 (file/directory not found)

This is usually either disk/filesystem problems on the NAS (copy phase) or on the NAS or target (compare phase), or networking issues (RS is more sensitive to these than file sharing is, the share can drop/remount and an OS copy operation will cope but RS won't).

So disk checks and network checks may help.

But if a file isn't backed up because of an error, RS will try again next time (assuming the file is still present). RS won't run again because of the errors, so you either wait for the next scheduled run or you trigger it manually.

3 hours ago, Joriz said:

Often when I manually initiate the same script after the error occurred it runs error free…

Think of it this way -- if you copy 1,000 files with a 1% chance of errors, on average 10 files will fail. So on the second run, when only those 10 files need to be copied, there's only a 1-in-10 chance that an error will be reported.

3 hours ago, Joriz said:

So I’m not sure if the copied correctly and don’t want to find out when a disaster happens.

Easy enough to check -- are the reported-to-error files from the first backup present in the backup set after the second?

Now the harder stuff 😉 

3 hours ago, Joriz said:

The disk to tape backup speed is around 6 – 6.5GB/min.

Is this overall? Or just during the write phase? How compressed is the data you are streaming (I'm thinking video files, for some reason!)? You could try your own speed test using "tar" in the Terminal, but RS does a fair amount of work in the "background" during a backup so I'd expect considerably slower speeds anyway... A newer Mac could only help here.

4 hours ago, Joriz said:

• I’ve been reading about grooming so I set the grooming options for each disk mediaset to keep 1 backup and “Performance-optimized grooming”.

I'm confused -- are you saying you back up your sources nightly, want to only keep one backup, but only go to tape once a week? So you don't want to off-site the Mon/Tues/Wed night backups?

Regardless -- grooming only happens when either a) the target drive is full, b) you run a scripted groom, or c) you run a manual groom. It sounds like none of these apply, which is why disk usage hasn't dropped.

4 hours ago, Joriz said:

The used diskspace for each Disk mediaset (disk to disk backup) keeps growing while the data on the source (the fileserver) is not.

If someone makes a small change to a file, the space used on the source will hardly alter -- but the entire file will be backed up again, inflating the media set's used space. If you've set "Use attribute modification date when matching" then a simple permissions change will mean the whole file is backed up again. If "Match only file in same location/path" is ticked, simply moving a file to a different folder will mean it is backed up again. It's expected the backup of an "in use" source is bigger than the source itself (always excepting exclusion rules, etc).

At this point it might be better to start from scratch. Describe how your sources are used (capacity, churn, etc), define what you are trying to achieve (eg retention rules, number of copies), the resources you'll allocate (tapes per set, length of backup windows (both for sources and the tape op)), then design your solution to suit.

You've described quite a complex situation, and I can't help but feel that it could be made simpler. And simpler often means "less error prone" -- which is just what you want!

[Joriz]

Thanks Nigel. I understand what i have explained is complex. Let me try to describe what i want.
The goal is to have offsite backups on tape. In case of a disaster, i need to restore all the most recent data of the 3 fileservers.
I have 3 tapesets, each with 2 LTO7 members which i want to rotate every week.

Sources:

3 fileservers sharing files over SMB.

Destination:

Disk mediasets for D2D backup

Tape mediasets for D2T backup

 

The disk to disk backup can be run every night during the week if a restore from disk is needed.
The disk to tape can be run in the weekend because it needs more time to backup. On monday morning i remove the tapes from the library to store them offsite.

The tapesets have 2x LTO 7 tapes. This should be enough for at least 1 backup of the 3 fileservers.

 

[DavidHertzberg]

Joriz,

First read pages 14-15 of the Retrospect Mac 13 User's Guide. 

Quote

Performance-optimized grooming makes certain decisions about what data to remove, only removing outdated backup information that it can quickly delete. On a technical level, this faster grooming mode only deletes whole RDB files instead of rewriting them.

That's why grooming isn't doing anything to reduce the size of your disk Media Sets.  If even one source file backed up to an RDB file is still current, then performance-optimized grooming won't delete the RDB file.  You should be using storage-optimized grooming unless your disk Media Sets are in the cloud—which you say they aren't.  (It seems the term "performance-optimized" can trick administrators who aren't native English speakers, such as you.)

There's a reason performance-optimized grooming was introduced in the same Retrospect Mac 13 release as cloud backup.  It's because rewriting (not deleting) an RDB file in a cloud Media Set  requires downloading it and then uploading the rewritten version—both of which take time and cost money.

[Nigel Smith]

15 hours ago, Joriz said:

Let me try to describe what i want.

You still haven't stated size of shares to back up, expected churn on each, retention policies (ie how long must a backed up file be available for) and much more.

I find it helps to think like you're asking a really nit-picky contractor to do the job -- and he'll give you exactly what you ask for, no more and no less, and take your money and leave you in the lurch if that wasn't what you actually wanted 🙂

And you'll have to compromise. For example:

15 hours ago, Joriz said:

The goal is to have offsite backups on tape. In case of a disaster, i need to restore all the most recent data of the 3 fileservers...

The disk to tape can be run in the weekend because it needs more time to backup. On monday morning i remove the tapes from the library to store them offsite.

These are mutually exclusive -- a server room fire on Saturday will mean your disaster recovery will be from a week ago. To me that's acceptable -- it's a very small risk, we have no compliance issues to worry about, and if our server room catches fire we have a lot more problems than the loss of a week's worth of data -- but it may not be to you.

Edited May 5, 2020 by Nigel Smith
Poor English!

[DavidHertzberg]

Joriz,

First of all, there's something unusual about one fact in your OP in this thread.  There you state you are "running Retrospect [presumably Mac] 13", whereas in this post 8 months ago you stated you were using Retrospect Mac 16.1.2.  Did you go back 3 major versions—assuming you continued working at the same company—in the last 8 months, and if so why?

Second, I agree with Nigel Smith that your provisions for offsite backup are inadequate for a company that probably can't afford to lose up to a week's worth of data in the event of a flood or fire in its server room.  I think you should run a disk-to-tape Copy Backup script every day as soon as the disk-to-disk Backup script is finished, and then store the tapes belonging to the current OFFSITE Media Set outside the server room—possibly in your boss' office.  You would bring the tapes back to the server room at 8 a.m., and put them back into the tape library device while the disk-to-disk Backup script is running.  That way if the server room is flooded sometime during the rest of the day, you'll still have tape backups as of 8 a.m..

A Copy Backup script should run faster than a Copy Media Set script, since it can be set up to "Copy most recent backups for each source" (see pages 161-162 in the Retrospect Mac 13 User's Guide).

[Joriz]

21 hours ago, Nigel Smith said:

This is usually either disk/filesystem problems on the NAS (copy phase) or on the NAS or target (compare phase), or networking issues (RS is more sensitive to these than file sharing is, the share can drop/remount and an OS copy operation will cope but RS won't).

So disk checks and network checks may help.

But if a file isn't backed up because of an error, RS will try again next time (assuming the file is still present). RS won't run again because of the errors, so you either wait for the next scheduled run or you trigger it manually.

I think we can exclude that either the disk/filesystem on the NAS is causing this. They all run a ZFS filesystem which is a very reliable. ZFS is very sensitive to read and write errors on the disks, so it would give some alerts if disks were faulty. 
A data scrub and smart test is scheduled twice a month. A data scrub will read back all the data and validates if the data still matches the checksum.

I’m using Zabbix for monitoring the network and servers. The Mac Pro running Retrospect and the tape library are included in the monitoring.
Zabbix didn’t report any major issues with the network, for example switchports going up/down or high error rates on the switchports while I sometimes see spikes in the response time/ICMP loss to the Retrospect server.

The Retrospect server is not in the same room/switch as the NAS.
ACTION: I’m going to move the Retrospect server to the same room/switch as the NAS.

21 hours ago, Nigel Smith said:

Easy enough to check -- are the reported-to-error files from the first backup present in the backup set after the second?

The files are present in the first backup and can be restored. So it looks fine but weird that Retrospect reports this error.

21 hours ago, Nigel Smith said:

Is this overall? Or just during the write phase? How compressed is the data you are streaming (I'm thinking video files, for some reason!)? You could try your own speed test using "tar" in the Terminal, but RS does a fair amount of work in the "background" during a backup so I'd expect considerably slower speeds anyway... A newer Mac could only help here.

This is the number I read during the write phase of a disk to tape backup.

NAS1 and 2 have all kinds of data: Office files, raw audio, video, picture files.
NAS3 is storing vmware backup files made by veeam (with compression)
I’m not using software compression in Retrospect while the tape library has hardware compression enabled. I’ll check the options of the storage enclosure if something can be optimised. know what "tar" is but i don't exactly understand what you mean with speedtest using tar. Do you want me to create a large tar file on the volume and check the speeds?

21 hours ago, Nigel Smith said:

I'm confused -- are you saying you back up your sources nightly, want to only keep one backup, but only go to tape once a week? So you don't want to off-site the Mon/Tues/Wed night backups?

Regardless -- grooming only happens when either a) the target drive is full, b) you run a scripted groom, or c) you run a manual groom. It sounds like none of these apply, which is why disk usage hasn't dropped.

During weekdays I run the disk to disk backups and during the weekend I run the offsite backup. The offsite backup backups the most recent backup of the week. The offsite backup is too slow to run every weekday and I have only 3 tapesets available.

22 hours ago, Nigel Smith said:

If someone makes a small change to a file, the space used on the source will hardly alter -- but the entire file will be backed up again, inflating the media set's used space. If you've set "Use attribute modification date when matching" then a simple permissions change will mean the whole file is backed up again. If "Match only file in same location/path" is ticked, simply moving a file to a different folder will mean it is backed up again. It's expected the backup of an "in use" source is bigger than the source itself (always excepting exclusion rules, etc).

At this point it might be better to start from scratch. Describe how your sources are used (capacity, churn, etc), define what you are trying to achieve (eg retention rules, number of copies), the resources you'll allocate (tapes per set, length of backup windows (both for sources and the tape op)), then design your solution to suit.

You've described quite a complex situation, and I can't help but feel that it could be made simpler. And simpler often means "less error prone" -- which is just what you want!

The option "Use attribute modification date when matching" is enabled while "Match only file in same location/path" is not.

I understand it might be better to start from scratch and the details you are asking about capacity, churn, retention,…) are important to give advice.

The thing is, i’m very limited in budget, it is actually zero. My boss wasn’t using any backup strategy with offsite backups at all in the past. He doesn’t care, but luckily I do. The only backup in place was based on file replication between multiple FreeNAS servers, onsite, all in the same rack (yes I know…)

What I’m trying to do is implement a 3-2-1 backup strategy with the limited budget (=0) i have.
So I worked the opposite way and asked myself the question: what equipment do I currently have (storage, tapes, backup software, backup server, tape library) and how can I use them in the most efficient way to implement a 3-2-1 backup strategy without additional costs.

I totally agree with your way of working. Every project should start with defining the needs. But it doesn’t work here sadly enough…

About 8 months ago I tried to implement a backup strategy the same way but I failed. So I tried to backup the data directly from the fileservers to tape over the network which didn’t work well.
Recently a friend of mine was decommissioning old storage enclosure. He gave me 2x HP P2000 raid enclosure with disks for free. 

Because of the COVID outbreak I had more time available so I decided to mount the enclosures in the serverrack and stress test them for 1 week. 1 enclosure is now connected to the Retrospect server, the other one I kept for spareparts.

After reading manuals and youtube tutorials about D2D2T in Retrospect I was able to setup the configuration I described in my first post, which is probably not the best solution…

What I currently have is:
-    1x 30TB storage connected to Retrospect
-    6 x LTO7 tapes
-    3 FreeNAS fileservers to backup

Fileserver 1
Total disk capacity: 8TB
Used: 2.42TiB
File types: Office files, raw audio, video, picture files
Total data change each day is max. 10GB

Fileserver 2
Total disk capacity: 8TB
Used: 5.81TiB
File types: Office files, raw audio, video, picture files
Total data change each day is max. 20GB

Both fileservers replicate their data and have snapshots available with a 2 week retention.

Fileserver 3
Total disk capacaity: 8TB
Used: 0.73 TiB
File types: veeAm vmware backup files (full and incremental with compression)
Total data change each day is max. 50GB
veeAm is using a 10 restorepoint retention

Because I have 6 tapes available, I created 3 tape sets with 2 tapes each.
With 3 tape sets I can span max 2 weeks retention for offsite backup.

21 hours ago, DavidHertzberg said:

That's why grooming isn't doing anything to reduce the size of your disk Media Sets.  If even one source file backed up to an RDB file is still current, then performance-optimized grooming won't delete the RDB file.  You should be using storage-optimized grooming unless your disk Media Sets are in the cloud—which you say they aren't.  (It seems the term "performance-optimized" can trick administrators who aren't native English speakers, such as you.)

There's a reason performance-optimized grooming was introduced in the same Retrospect Mac 13 release as cloud backup.  It's because rewriting (not deleting) an RDB file in a cloud Media Set  requires downloading it and then uploading the rewritten version—both of which take time and cost money.

Thanks. I changed the option to storage-optimized and initiated groom

18 hours ago, Nigel Smith said:

These are mutually exclusive -- a server room fire on Saturday will mean your disaster recovery will be from a week ago. To me that's acceptable -- it's a very small risk, we have no compliance issues to worry about, and if our server room catches fire we have a lot more problems than the loss of a week's worth of data -- but it may not be to you.

Edited 5 hours ago by Nigel Smith
Poor English!

For the limited amount of available tapes, restore from a week ago is fine. We had no offsite backup before and my boss still doesn’t care.

I’m sorry for my English.

9 hours ago, DavidHertzberg said:

Joriz,

First of all, there's something unusual about one fact in your OP in this thread.  There you state you are "running Retrospect [presumably Mac] 13", whereas in this post 8 months ago you stated you were using Retrospect Mac 16.1.2.  Did you go back 3 major versions—assuming you continued working at the same company—in the last 8 months, and if so why?

Second, I agree with Nigel Smith that your provisions for offsite backup are inadequate for a company that probably can't afford to lose up to a week's worth of data in the event of a flood or fire in its server room.  I think you should run a disk-to-tape Copy Backup script every day as soon as the disk-to-disk Backup script is finished, and then store the tapes belonging to the current OFFSITE Media Set outside the server room—possibly in your boss' office.  You would bring the tapes back to the server room at 8 a.m., and put them back into the tape library device while the disk-to-disk Backup script is running.  That way if the server room is flooded sometime during the rest of the day, you'll still have tape backups as of 8 a.m..

A Copy Backup script should run faster than a Copy Media Set script, since it can be set up to "Copy most recent backups for each source" (see pages 161-162 in the Retrospect Mac 13 User's Guide).

This was a typo, I’m sorry. My Retrospect version is 16.6.0 (114)
I don’t have enough tapes and time to run a D2D and a D2T every day.

[Nigel Smith]

Quickly, before addressing the other points properly...

54 minutes ago, Joriz said:

I’m sorry for my English.

I was complaining about my poor English, not yours! I had to edit that bit to make it clearer, and wanted to put a reason in case David or yourself had already quoted the then edited post.

It seems even my explanations need explaining... 🙂 

[Nigel Smith]

2 hours ago, Joriz said:

I think we can exclude that either the disk/filesystem on the NAS is causing this.

Reasonable argument. ZFS does do a lot in the background -- it may even be that that's causing Retrospect to throw those errors. Your network change certainly won't hurt but, assuming the number of errors is a small percentage of the files being backed up, I'd be inclined to ignore them and trust RS to get the files on the next run. Maybe monitor, and see if there's any consistency in which files error -- a certain share, during similar times, particular file types, etc.

Speed-wise, my feeling is that you are getting a reasonable write speed given the age of the hardware and the average file size. But that is just a gut feeling, with nothing solid behind it. Using RS does involve some overhead and a test using "tar" to write directly to tape would remove that, giving you an indication of "hardware speed".

More importantly, ~6GB/min write speed implies ~4GB/min overall (write and compare) -- even if we play safe and call it 3GB/min that's 180GB/hour, a good figure to work with.

If we say the three servers hold 3, 6.5 and 1TB respectively (allows plenty of margin for error) that's 16 hours, 36 hours, and 6 hours for full backups. So if you start the tape run after your Friday night D2D has completed you'll easily be done before Monday's D2D is due to start.

Your total daily churn is ~80GB, so less than an hour including scanning. Will easily complete after a nightly D2D.

Weekly churn is ~560GB (using 7 days to allow for extras), so allow 4 hours. Will easily complete after Friday's D2D.

Tape-wise, two LTO-7s per set gives a set capacity of 12TB without compression (again, to give us plenty of wiggle room). Easily enough for a full backup plus 2 weeks of churn.

That suggests the following:

Nightly D2D backups continue as you are doing now. Use groom scripts set to keep 4 backups, run these before Friday's D2D scripts so you can always go back at least 5 days to eg restore a deleted file.

First Friday after the D2D has finished, run a "Copy Backup Script", all backups, to Tape Set 1, recycling the tape media. Leave those tapes in and Mon-Thur of the next week run a nightly "Copy Backup Script", most recent backups, adding on to what's already in the tape set.

Second Friday, do the same with Tape Set 2. Third Friday, same with Tape Set 3. Fourth Friday, back to Tape Set 1.

So you'd always have the last 5-9 days of restore points of the P2000 and a rolling 21 days on tape, basically unattended apart from swapping tapes on a Friday and monitoring the logs for problems.

You can easily vary the above to suit -- only have 2 sets of 3 tapes so you go longer between recycles but have less redundancy. Add 3 more tapes to go longer but keep the redundancy. Only copy "most recent" backups to tape (again, sacrificing redundancy for an increase in capacity), or groom the P2000's disk sets more aggressively. A lot will depend on how people work and what you expect to have to recover from -- my most frequent request is for files "lost" some time ago, so I keep a lot more than someone who was just expecting to restore the most recent version of a failed drive.

There are many ways to skin the backup cat, and I'm not claiming the above is in any way definitive. With luck someone (David? Lennart?) will be along to improve it, and I'm sure you'll find a better way of doing it yourself. But hopefully it'll give you a good starting point from which to roll your own solution.

[DavidHertzberg]

Joriz,

Sorry for the delay, but I needed to correctly run a test of a Copy Backup script early this morning.  The test copied the last Recycle backup of my MacBook Pro to an extra Media Set that fortunately happened to be occupying spare space on the portable USB3 hard disk drive that has been my destination for Backup runs since early Saturday morning (my scripts switch between 3 such portable HDDs weekly).  The successful test run was itself a Recycle, since I had used up the destination Media Set's space on previous tests.

The purpose of the test was to get timing for a Copy Backup script versus the Backup script that copied the same data.  The Copy Backup script took 30 minutes to copy 65GB of files.  The original Backup run, by contrast, took 190 minutes to Backup the same files to the destination Media Set that was the source for that portion of the Backup run.  The Copy Backup copying rate was 2.4GB/minute, which is about 7 times the speed of the 0.361GB/minute copying rate of the Backup run.  That is what I expected, since my previous experimentation has shown there's a lot of processing work that the Client program does on a Backup run—and my MacBook Pro is is a "client" on my LAN (which has a 6GB/minute speed—allowing 20% overhead for TCP/IP headers).  The same Saturday morning Backup run also did a Recycle backup of two drives that are internally mounted on my Mac Pro "backup server" machine; the copy rates were 2.0 GB/minute for the old HDD drive and 2.3GB/minute for the SSD drive.

Therefore I simply don't believe that "The disk to tape can be run in the weekend because it needs more time to backup [my emphasis]", as you wrote in this up-thread post.  You wrote in your OP "For the Disk to Tape backup I made an individual script for each tapeset. The script type is set to 'Copy Media Set'".  I think your disk to tape scripts are actually Backup scripts, whose sources are the same FILESERVERx SMB shares that are the sources in your disk to disk scripts..  They probably run somewhat slower than your disk to disk scripts because their destination is a tape drive (the source and destination of my test Copy Backup script is the same portable USB3 HDD, so that should produce a somewhat equivalent slowdown).

I need to go to bed now, so I'll later add suggestions to this post about how to do a Monday-through-Friday Copy Backup script run after each disk to disk Backup run, as I suggested up-thread.

 

 

[Joriz]

18 hours ago, Nigel Smith said:

Speed-wise, my feeling is that you are getting a reasonable write speed given the age of the hardware and the average file size. But that is just a gut feeling, with nothing solid behind it. Using RS does involve some overhead and a test using "tar" to write directly to tape would remove that, giving you an indication of "hardware speed".

I will test using tar once the system is available again and post the results. Something i forgot to mention is that i changed the raid configuration on the enclosure from raid6 to a raid10 array. There was no improvement so i set it back to raid6. Probably the Mac Pro is causing this then.

Also yesterday i ran a D2D and a D2T script at the same time. For the P2000 enclosure this means writing and reading at the same time, something a 7200RPM mechanical HDD doesn't like. It had no impact at all on the D2D/D2T speed. The D2D was running at 4.5GB/min (the source was a SMB share over a 1Gbit link, so speed is ok) and the D2T at 6.6GB/min.

18 hours ago, Nigel Smith said:

That suggests the following:

Nightly D2D backups continue as you are doing now. Use groom scripts set to keep 4 backups, run these before Friday's D2D scripts so you can always go back at least 5 days to eg restore a deleted file.

First Friday after the D2D has finished, run a "Copy Backup Script", all backups, to Tape Set 1, recycling the tape media. Leave those tapes in and Mon-Thur of the next week run a nightly "Copy Backup Script", most recent backups, adding on to what's already in the tape set.

I like the idea of the Copy Backup Scripts during the week. I have to change the Copy Media Set scripts to Copy Backup Scripts. For the Copy Backup Script i have to create a script for each D2D media set because i can only select 1 source while multiple are possible when using Copy Media Set scripts.

When i put this in a schema i get something like this. Starting from May 1st.

3 hours ago, DavidHertzberg said:

Sorry for the delay, but I needed to correctly run a test of a Copy Backup script early this morning.  The test copied the last Recycle backup of my MacBook Pro to an extra Media Set that fortunately happened to be occupying spare space on the portable USB3 hard disk drive that has been my destination for Backup runs since early Saturday morning (my scripts switch between 3 such portable HDDs weekly).  The successful test run was itself a Recycle, since I had used up the destination Media Set's space on previous tests.

It's fine David, no worries.

3 hours ago, DavidHertzberg said:

Therefore I simply don't believe that "The disk to tape can be run in the weekend because it needs more time to backup [my emphasis]", as you wrote in this up-thread post.  You wrote in your OP "For the Disk to Tape backup I made an individual script for each tapeset. The script type is set to 'Copy Media Set'".  I think your disk to tape scripts are actually Backup scripts, whose sources are the same FILESERVERx SMB shares that are the sources in your disk to disk scripts..  They probably run somewhat slower than your disk to disk scripts because their destination is a tape drive (the source and destination of my test Copy Backup script is the same portable USB3 HDD, so that should produce a somewhat equivalent slowdown).

I must have made a calculation error then. The disk to tape scripts are Copy Media Set scripts whose source are the disksets (3).

I appreciate your effort, thanks.

 

[Nigel Smith]

15 minutes ago, Joriz said:

I like the idea of the Copy Backup Scripts during the week. I have to change the Copy Media Set scripts to Copy Backup Scripts. For the Copy Backup Script i have to create a script for each D2D media set because i can only select 1 source while multiple are possible when using Copy Media Set scripts.

You could also do the same using Copy Media Set scripts -- see the User Guide, particularly p120, for a comparison of the two. Indeed, now that we're grooming the disk set a Copy Media Set script might be the better option. TBH, I don't really understand the difference between the two -- try them yourself and see which is more suitable 😉 

18 minutes ago, Joriz said:

When i put this in a schema i get something like this. Starting from May 1st.

Looks good. My main concern would be that you're re-writing all the tapes every rotation, reducing their life-span considerably. Adding another tape to each set would increase the time between re-writes enormously, from every third week for each set to every 30th week! You could even stagger it so one was done every couple of months, giving you the ability to go back 6 months for any data restore -- but you sell it to the boss as a cost benefit, spend a little more now to save in the longer term.

[Joriz]

34 minutes ago, Nigel Smith said:

You could also do the same using Copy Media Set scripts -- see the User Guide, particularly p120, for a comparison of the two. Indeed, now that we're grooming the disk set a Copy Media Set script might be the better option. TBH, I don't really understand the difference between the two -- try them yourself and see which is more suitable 😉 

You are right. If grooming does its job well, the outcome would be similar. Using Copy Media Set would mean less scripts.

As far as i understood. A Copy Backup script gives you more backup options for the selected mediaset (Copy most recent, copy selected, copy all), while a Copy Media Set script only allows to select the mediaset.

[Nigel Smith]

There may also be differences in how and what you can restore. They both match "files in source to file already in the destination", so I don't know what the practical difference is.

I haven't used either often enough to really know the pros and cons of each and, being stuck at home, don't want to risk banjaxing a work system that I can't then reset remotely just to run some tests. Perhaps someone who has used both in real life will chip in...

 

[Nigel Smith]

18 minutes ago, Nigel Smith said:

I haven't used either often enough to really know the pros and cons of each

Perhaps a word of explanation, since the above makes it sound like we don't "off-site".

We do, but we use yet another method -- backing up the backup files! Yep, I just copy the .rdb files to tape and take those away. The downside is that a restore from these takes considerable time -- you have to restore all the .rdb files for the set, maybe re-catalog it, before you can restore single file. The upsides are that it is really easy to do the backups and, crucially, these can be done using a another machine while regular client backups are taking place.

It works for us because our off-sites are one-part disaster recovery but nine-parts archive. I've never had to use them for DR and only once in the last five years have I had to go back to them to restore data from an old backup set (anything recent is handled using the onsite disk sets). But we have to have them because of retention policies, and this is an efficient way to handle a lot of data using relatively low-spec equipment with minimal impact on daily client backups.

It also followed on neatly from my old RS6-based kludge for D2D2T using Internet backups...

Our backup system re-spec has been put on hold because of COVID-19 -- after that's done I expect we'll opt for a more "normal" methodology similar to what's described above.

[Joriz]

24 minutes ago, Nigel Smith said:

Our backup system re-spec has been put on hold because of COVID-19 -- after that's done I expect we'll opt for a more "normal" methodology similar to what's described above.

What kind of hardware are you actually using? Like a Mac Pro or a Mac Mini (with thunderbolt to SAS ?)

To backup the catalog files i use an Amazon S3 bucket.

[DavidHertzberg]

Joriz (if Nigel Smith is a good boy he can read this too 🤣 ),

Let me start out by guessing that you didn't institute off-site backup because you wanted to play with tape drives, but because your employer seems to be a business  that has the responsibility of promptly recovering from the disaster of the server room flooding.  (If I emphasize flooding, it's because U.S. companies have been in the habit of putting their server rooms in the building basement underneath the cafeteria—with predictable results.  Besides, Belgium borders on the North Sea.)  Nigel Smith, from his statement in another thread, seems by contrast to be an administrator for an educational or research institution—where it's the responsibility of the researchers to do their own backup during COVID-19 Work From Home.

The reason I suggested daily Copy Backup scripts up-thread is because I thought they would run faster than Copy Media Set scripts, and you seem to be concerned with adding to the total elapsed time of your disk-to-tape runs.  In connection with this, I'm not sure from your OP whether you run disk-to-disk Backup scripts on Saturdays or Sundays; from the 10 p.m. start time of those scripts it sounds as if you wouldn't want to run one on Saturday or Sunday night—unless you have employees using the fileservers over the weekend.  I'd guess you do, though, and therefore currently have the requirement that a disk-to-tape script run starting at 8 a.m. on Saturday complete before 10 p.m. on Saturday. 

But if you switched to daily disk-to-tape runs, they'd be incremental and therefore run faster than your current weekly disk-to-tape runs—which seem to be the equivalent of Recycle Backups.  You can make sure an incremental disk-to-tape script runs after its corresponding disk-to-disk script run, by designating the same Activity Thread for both scripts.   Page 225 of the Retrospect Mac 16 User's Guide says "By assigning activities to the same activity thread, it ensures that they will run one after the other."; an Activity Thread is assigned for a Backup script in a "Use ..." popup at the bottom of its Summary tab that is shown—but not explained—in the screenshot on page 94 of the UG, and is assigned for a Copy script by a "Use ..." popup at the bottom of its equivalent Summary tab.  Caution: do not let a Copy Backup/Media-Set script execute in a different—or "Any"—Activity thread from its corresponding Backup script; I discovered a couple of years ago that doing this will allow a Copy Backup/Media-Set script with a particular Media Set as a source to execute while a Backup script with the same Media Set as destination is still running—which I thought was a wonderful feature until the head of Retrospect Tech Support pointed out that a Backup script doesn't update the Catalog File of its destination Media Set (which the Copy Backup/Media-Set script reads) until the end of the run.  But so long as you observe this precaution, you are guaranteed that a daily Copy Backup/Media-Set script will not start executing until its corresponding daily Backup script is finished.  Therefore you have a full 24-hour day for the combination of the two corresponding scripts to run.

Up-thread I suggested using daily Copy Backup scripts instead of Copy Media Set scripts, because I thought they'd run faster.  But now I'm not so sure, and I don't have the capability of running a test for it.  That's because, contrary to what the UG implies on pages 137-138, my test the other night indicates that "the most recent backup [my emphasis] for each source contained in the source Media Set" doesn't work as of Retrospect Mac 16.6.  The source Media Set for my Copy Backup test had been backed up with Recycle last Saturday morning and then backed up withe the incremental No Media Action on Sunday and Monday mornings, but the Copy Backup test copied the entire source Media Set instead of only the last incremental Backup  The Match source files against the Media Set and Don’t add duplicate files to the Media Set options didn't have any effect because I had specified Recycle in the Schedule for the script.

As for tape wear, I'm afraid that—because of the fundamental non-random-access nature of tape drives—each successive Copy Backup run would probably cause Retrospect to read its way down the last tape mounted on your LTO-7 tape drive to get to the point where it should start writing.  Possibly leaving the tape in the drive from one night to the next would prevent it from doing this, but that would only be true if Retrospect remembered the name on the label of the tape from the last run—and therefore didn't have to rewind to read the label—but I don't think it trusts the drive user to that extent.  The Linear Tape File System might add that capability, except that Retrospect doesn't use LTFS for Backup operations.  And besides, I suggested that you unload the current tape from the drive after each Copy Backup run—and temporarily store it in your boss' office so that it would be in a separate room from the backup disks.

[Nigel Smith]

1 hour ago, DavidHertzberg said:

Possibly leaving the tape in the drive from one night to the next would prevent it from doing this, but that would only be true if Retrospect remembered the name on the label of the tape from the last run

This happens -- I don't know if it is Retrospect that does the remembering or the tape library, but RS knows which tape is in the drive, what's in the other slots etc without a re-scan. Indeed, even if you quit and re-launch RS it seems to assume the last-known state is the same until proven otherwise. I just trust RS and the library to do their thing during the week, my only "hardware intervention" is to swap tapes and then do a single re-scan of the library on tape-change day.

I don't know if tapes are rewound between sessions when left in the drive, but it doesn't matter much in terms of tape life. I know I mentioned tape life above, but there's a lot more wear in completely re-writing at least one tape every time -- even then, you should be averaging at least 200 full writes, which is >10 years using the above scheme 🙂  It's more a way of shoehorning an extra 3 tapes out of the boss, and look at the advantages that brings in terms of backup retention and reduction of the overall backup window required.

[DavidHertzberg]

Joriz and Nigel Smith,

In my first job as a programmer in the mid-1960s, I worked as a consultant with one U.S. Air Force-written project cost and schedule control application that ran for hours on a "big iron" IBM 7090 or 7094—using multiple half-inch tapes for I/O because those machines generally didn't have then-incredibly-expensive disk drives.  I was paid—indirectly by the U.S. Navy—to personally supervise those runs, which were made at one of the computer service bureau offices belonging to my employer.  Thus I became quite familiar with factors affecting tape life.

The IBM 729s and 2400s we used then and the LTO-7 drive Joriz is using have one thing in common; they make read and write scans linearly up and down the tape.  The tape is moved by rollers that touch the tape and it passes over a read-write head (a second read head for LTO does read-after-write verification) that even for LTO also touches the tape.  It's the repetition of high-speed linear scans that wears out tapes.  (As I write this I am using Retrospect Mac 6.1 to back up my ex-wife's old Digital Audio G4 onto a DAT drive—which uses helical scanning with a rotating head instead of linear scanning, but DAT-based Digital Data Storage was abandoned because it couldn't keep up with LTO's capacity increases.)  AFAIK LTO tape wear doesn't depend on whether the read-write head is writing when the drive does those linear scans, but at least the read-write head has to be reading at least a servo track—except for rewinds—because that's the only way the read-write head can position itself for non-direct-access as a result of those scans.

I just spent the best part of 24 hours doing comprehensive tests of both the Copy Media Set and Copy Backup features of Retrospect Mac 16.6.  At first I didn't think they worked, but that was because I had placed the Media Set member 1-CopyTestDestination in the spare space on my portable USB3 HDD—which wasn't large enough to hold a complete copy of 1-Media Set Red that occupied the rest of that HDD.  After I did a Rebuild of CopyTestDestination to place 1-CopyTestDestination on the larger spare space of my "backup server"'s SSD boot drive, both Copy Media Set and Copy Backup functioned as pages 116-122 of the Retrospect Mac 16 User's Guide say they should—but see my post directly below.  (Switching my destination from USB to boot drive halved the speed of the script runs—even though it straightened out the function testing, so you'll have to do speed testing with your tape drive , Joriz.) 

The "Copy most recent backups for each selected source" option on the popup menu of the Sources tab for a Copy Backup script functions exactly as it states, Nigel Smith.  I tested by first running  a Copy Media Set script after running a Recycle Backup of three source drives and a Favorite Folder onto Media Set Red, and then interspersing No Media Action Backups of one source drive with No Media Action Copy Backup script runs using the "Copy most recent backups for each selected source" option.  However I also tried a No Media Action Copy Media Set run after a No Media Action Backup run; the “Match Source Media Set to destination Media Set” and “Don’t add duplicates to the Media Set” options caused that Copy Media Set run to copy only the latest backup of that source to Media Set Red, exactly as did the Copy Backup run with the "Copy most recent backups for each selected source" option.  Thus Joriz can use either Copy Backup with the "Copy most recent backups for each selected source" option or Copy Media Set for daily tape backups.

 

[DavidHertzberg]

Joriz and Nigel Smith,

While searching the Retrospect Mac Cumulative Release Notes for a possible answer to another administrator's problem, I just noticed this for 17.0.0:

Quote

Transfer: Fixed issue where transfer backup set transferred more snapshots than required (#8404)

(Transfer Backup Set is the old name for Copy Media Set; it is still used in Retrospect Windows, which is why the common-code Engine uses the term.)

In my early tests mentioned directly above, I was seeing the same problem using Retrospect Mac 16.6.  Because the problem went away when I changed the disk drive location of 1-CopyTestDestination, I thought it resulted from my original test setup.  Instead it evidently resulted from an intermittent bug, .

Joriz: If you can't get your penny-pinching boss to shell out the equivalent of US$69 for an upgrade to the Retrospect Mac 17 version of Desktop Edition—which is the Edition I assume you are using, you'll have to use a Copy Backup script with schedules specifying No Media Action for your daily tape incremental backups.  Leave the “Match Source Media Set to destination Media Set” and “Don’t add duplicates to the Media Set” options check-marked.   I think you'll want to pick the "Copy most recent backups for each source" option from the popup, but see the next paragraph.

Nigel Smith: Maybe this bug was why you wrote "I don't know what the practical difference is."  Though it's probably not useful for Joriz, the popup options for Copy Backup—when it's working as designed—include "Copy most recent backups for each selected [my emphasis] source".   Apparently that lets you specify which backup onto a selected source Media Set should be copied, using the Browse button on the selection dialog shown for that option.

P.S.: Corrected the Joriz paragraph, which I had messed up.

Edited May 11, 2020 by DavidHertzberg
P.S.: Correct the Joriz paragraph, which I had messed up

[Nigel Smith]

On 5/10/2020 at 2:22 AM, DavidHertzberg said:

Nigel Smith: Maybe this bug was why you wrote "I don't know what the practical difference is."

I wrote that because p118 of the guide seems to describe "Copy Media" and "Copy Backup" as two methods of achieving the same goal, though "Copy Backup" also has options to restrict/select what gets copied. Which makes me wonder why they aren't the same thing with expanded options. Which makes me think I'm missing something, either in what gets transferred, what can be restored from the resulting set, or in how it is done.

Since the first two are generally important (data integrity) and the last may impact time required (and therefore backup windows needed in Joriz's scheme) and I can't run tests at the moment, I haven't a clue which is better suited.

[DavidHertzberg]

On 5/11/2020 at 4:52 AM, Nigel Smith said:

I wrote that because p118 of the guide seems to describe "Copy Media" and "Copy Backup" as two methods of achieving the same goal, though "Copy Backup" also has options to restrict/select what gets copied. Which makes me wonder why they aren't the same thing with expanded options. Which makes me think I'm missing something, either in what gets transferred, what can be restored from the resulting set, or in how it is done.

Since the first two are generally important (data integrity) and the last may impact time required (and therefore backup windows needed in Joriz's scheme) and I can't run tests at the moment, I haven't a clue which is better suited.

Nigel Smith and everybody else,

They aren't supposed to be "the same thing with expanded options."  Page 120 of the Retrospect Mac 17 User's Guide says (as does page 137 of the Mac 16 UG) :

Quote

Copy Backup scripts are different from Copy Media Sets scripts in a number of ways:

▪They copy only active [my emphasis] backups; Copy Media Sets scripts copy all [my emphasis] backups.

▪They provide different methods for selecting which backups get copied, such as the most recent backup for each source contained in the source Media Set; Copy Media Sets scripts always copy all backups.

To answer the questions "What is an 'active backup'?" and "why does Copy Backup offer 'Copy all backups' as a choice in the drop-down?", here's a 2016 post by "Eppinizer"—IRL Jeff of Retrospect Tech Support—answering those questions in two separate paragraphs:

Quote

Your active backups depend on your current grooming settings. If you have grooming disabled then your active backups will be the most recent backup of each source. If you have grooming enabled, we will keep the number of snapshots (backups) in the catalog necessary to carry out your grooming policy. For example, if you have "Groom to keep this number of backups" set to 3, we will keep 3 copies of your recent backups in your catalog, and consider them active.

While I admit this is a bit confusing, and I actually had to test the behavior to be certain, the user's guide is indeed correct. Copy backup scripts will only copy active backups, meaning that with this drop-down option selected, we will copy all of your active backups.  I'll preemptively agree with you that this is misleading, and have already logged a bug to correct the wording here.

 I just looked at my test runs in Activities; my early test runs were of a Copy Backup script.  The second of those runs copied all backups of "David's MacBook Pro" from Media Set Red, not merely the latest No Media Action incremental Backup.  So I think that the Release Note I quoted in this up-thread post is wrong, and I've left voicemail messages for both the head of Tech Support and the head of North America Sales asking them to re-check it.  In the meantime I now think that Joriz should either run a test himself, or use Copy Media Set for his incremental disk-to-tape backups and rely on the “Don’t add duplicates to the Media Set” option to restrict the copying to the most recent backup whose Destination was the Media set he is using as the Source.

[Joriz]

Thanks David, Nigel;

I currently have the disk to disk backups running with grooming set to 4 backups. I have also created a grooming script to run before the disk to tape backup starts.

One thing i still need to investigate is the disk to disk backup of the veeam backup files. The churn rate is higher then expected as Retrospect is copying all the data instead of doing an incremental during the weekdays disk to disk backups. Probably is veeam "changing" the incremental backup files of previous backups too and Retrospect recognize them as a new files and include them in the incremental backup. So i have to find something for that.

I will also do some test with Copy Backup scripts and Copy Media Sets scripts later. I can't go into the office right now as i'm ill (non covid)

 

[DavidHertzberg]

In regard to the churn rate of disk-to-disk backup of your Veeam backup files,  Joriz:

First, make sure you have left un-checked  "Match only file in same location/path" in the Options->Backup->Matching tab of Scripts for your Backup script.  Per pages 98-99 of the Retrospect Mac 16 User's Guide, this will make sure that any separate copy of an incremental backup file will not be backed up again.

Second, I suggest you consult the Veeam Backup and Replication forum of the Veeam Community Forums.  Here, for instance, is a 2016 thread in which the OP asks "Till now Retrospect is managing the tape library but we need a possibility for outsourcing out of date Veeam backups we don't need anymore but we don't want to delete finally. Is it possible to create a job in Retrospect which writes the related backup files from Veeam onto, with the possibility to restore them again to it's original destination if needed?"  There is a Tape sub-forum there, but the OP in the thread I linked to in the second sentence of this paragraph posted down-thread (spelling mistakes are his, not mine) about what is probably also your single-tape-drive situation:

Quote

I guess that it is not a good idea to use a tape library or a tape drive with two seperate applications. We have quite a lot of servers we have to backup with a normal backup application because they are not virtualized. Internal we discuss the possibility to use a second tape drive in our tape library, one exclusive for Retrospect and the other exclusive for Veeam. But we are nit sure if this is workable.

Third, I have to ask whether you have considered using the Retrospect Virtual application instead of Veeam.  I don't know anything about R.V., and have been forbidden to discuss it on these Forums.  However I'm pretty sure it's cheaper than Veeam, even though your boss —and apparently you—is used to Veeam.  R.V. only runs on a Windows machine, but you must have one of those to run Veeam.  R. V. doesn't seem to have a tape backup capability of its own, but—considering it is marketed by Retrospect "Inc."—it probably has destination disks that can be backed up onto tape by Retrospect non-V. Mac.

 

[Nigel Smith]

13 hours ago, DavidHertzberg said:

To answer the questions "What is an 'active backup'?"

etc...

And this is my confusion: p120

Quote

• They provide different methods for selecting which backups get copied, such as the most recent backup for each source contained in the source Media Set; Copy Media Sets scripts always copy all backups.

...which implies that you can select whether to have all, some, or only the most recent while Copy Media is always all, and p121

Quote

▪  Copy most recent backups for each source
▪  Copy most recent backups for each selected source
▪  Copy selected backups
▪  Copy all backups

...where "backups" is plural, even for a single source.

While I realise no manual is big enough to explain every single someone might come across, "Copy the most recent backup for each source" wouldn't take up much more virtual ink if, indeed, that is what happens.