Manually queueing multiple jobs in Retrospect Desktop?

[jhg]

If I try to manually initiate a job while another job is running to the same backup set, Retrospect rejects the attempt.

I would expect that it would just add the job to the "Waiting" queue instead.

Is there a way to explicitly add a job to the "Waiting" queue?

[Lennart_T]

Does it have to be manually? Can it be a script? If "yes", create a script and schedule it to run once at a time in a few minutes from "now".

[jhg]

What I want to do is run 3 backup scripts one after the other.  Can a script execute other scripts?

[DavidHertzberg]

jhg,

I am a Retrospect Mac administrator, but I've got a couple of suggestions:

First, if running 3 scripts as separate Immediate operations, explicitly designate the same Execution Unit for all 3 scripts.  Look on page 93 of the Retrospect Windows 15 User's Guide  (you don't specify your version of Retrospect) about switching into Advanced Mode to designate the Execution Unit, then proceed from there.  I think that will put the operations into the "Waiting" queue; the equivalent does so on Retrospect Mac.  OTOH Retrospect Windows may have special limitations for Immediate operations; Retrospect Mac eliminated Immediate operations in 2009, treating Toolbar-button submissions as scheduled-immediately script creations.

Second, if trying to run 3 backup scripts as if they were one script, I assume that you would simply put all the Sources into a single script if they were disjoint volumes or Subvolumes.  Since therefore they must overlap, consider defining the overlapping Sources as Subvolumes.  Subvolumes are discussed beginning on page 420 of the UG, and —since they are only defined to Retrospect—I don't see any prohibition on defining them so that they name the same folder—provided that (at least on the same volume, and preferably globally) you use different names for each Subvolume.  Page 175 of the UG says "Click and drag to rearrange the list order" of Backup Sources.

Third, a script cannot execute other scripts.  Administrators have occasionally asked for this facility, but Retrospect Inc. has evidently decided that its programming complications would outweigh its usefulness.

P.S.: As DovidBenAvraham clarified recently in Note 4 in the Wikipedia article,  "a Subvolume must be specifically defined to the Retrospect application as a name for a filesystem folder" ; changed 3rd sentence in second substantive paragraph accordingly.

Edited September 13, 2019 by DavidHertzberg
Added OTOH qualification for Retrospect Windows to end of first paragraph; added P.S. about change clarifying Subvolumes

[mbennett]

What David Hertzberg said.

At the suggestion of Retrospect support, I now schedule multiple jobs in multiple scripts one minute apart.  You can easily run dozens at a time, and the maximum number is tunable in Preferences.

You can't run two jobs at the same time from the same source.  You can run multiple scripts to the same destination, but for the sake of clarity and organization you will want to use unique destinations.

I believe (but haven't tested this) that you could schedule two scripts to run with the same source, one minute apart.  The second one will wait until the first one is complete.  Try this and let us know.

If you have a script and want to run it immediately, highlight it in Manage Scripts then click the third icon from the left on the toolbar, a little scripty looking sheet of paper with a lightning bolt next to it.

[DavidHertzberg]

mbennett and anybody else,

You currently can't run multiple scripts executing at the same time to the same destination Backup Set (Media Set in Retrospect Mac terminology).  I submit pairs of such scripts to Retrospect Mac 15.6.1 (105) occasionally, and the second script submitted always goes into the "Waiting" queue.  You will be able to do that some time this side of the indefinite future, once Storage Groups come out of beta.

You can't run multiple scripts executing at the same time from the same-named Source.  However since Retrospect Windows 12.6 (and Retrospect Mac 14.6) you can run multiple scripts executing at the same time from the same source volume if all of the scripts except one name different Subvolumes on the same source volume (ignore what it still says on page 421 of the Retrospect Windows 15 User's Guide; a "new" 7 November 2017 entry in the cumulative Release Notes overrides that).  As I said in the second suggestion in my preceding post, there doesn't seem to be any prohibition on defining multiple Subvolumes (Favorite Folders in Retrospect Mac terminology) as the same real folder—including the root folder of a real volume.

You can indeed schedule two scripts to run with the same source, one minute apart.  I routinely run two such scripts scheduled 5 minutes apart every day, because until less than a month ago I needed to run a "sacrificial script" before the "real script" to try to eliminate -530 errors for the "real script".  I schedule the "sacrificial script" 5 minutes before the "real script", because the "sacrificial script" uses No Files as the Selector (Rule in Retrospect Mac terminology)—and therefore normally completes in 4 minutes.  However if the "sacrificial script" starts to run more than 1 minute after its scheduled time, the "real script"—which has the same source and destination—always waits to start.

P.S.: As DovidBenAvraham clarified recently in Note 4 in the Wikipedia article,  "a Subvolume must be specifically defined to the Retrospect application as a name for a filesystem folder" ; changed last 2 sentences in second substantive paragraph accordingly.

 

Edited April 19, 2019 by DavidHertzberg
Revised last sentence of third paragraph, because realized that I frequently have same-source-same-destination script waiting; added P.S. about change clarifying Subvolumes

[DavidHertzberg]

mbennett and anybody else,

"I dipped into the future, far as human eye could see".  That's because Retrospect Inc. engineers put some of their new or revised Knowledge Base articles onto the website on 4 March, even though those KB articles are "last updated March 5, 2019".   Anyway the last sentence in the first paragraph of my immediately-preceding post now has to be significantly qualified.

This KB article now says "With Storage Groups, you can run parallel backups to the same disk destination with a single ProactiveAI script.  Scheduled scripts support Storage Groups as destinations, but the backups run on a single execution and not in parallel."

However the basic concept is that "Under the hood, a Storage Group is a container for per-volume backup sets. This architecture is why Retrospect allows you to keep the same workflows that you are accustomed to for backup, restore, transfer, grooming, and catalog rebuild, while providing far better performance and simplicity. You can treat the Storage Group like a Backup Set that allows simultaneous writes to it."  The remainder of the KB article explains that the containerization is less transparent for the Retrospect Windows 16 GUI than it is for the  Retrospect Mac 16 GUI.  Eat your hearts out, Retrospect Windows administrators.

For all of us, nevertheless, "The architecture for Storage Groups allows simultaneous operations to the same destination because each volume is a different backup set under the hood. However, this workflow also prevents data deduplication across volumes."  BTW, those different backup sets must be either on disks or in the cloud; sorry, no Storage Groups for you tape enthusiasts.

[Nigel Smith]

As well as another "What David said...", whilst you can't use the finish of one script to trigger the next, you can set scripts to run sequentially after a single trigger-action by using Run Documents. See page 234 of the Windows guide but, basically, if you open several runs documents at once then the scripts contained will run in alphabetical order. So name them "A - First Script", "B - Second Script", "C..." etc, set them all to use the same execution unit just to be sure, and you should be good.

Personally though, I'd just use scripts as others have suggested. Set them to trigger at 1-minute intervals in the order you want, set them all to the same execution unit, and they'll run consecutively.

[DavidHertzberg]

Just because I mentioned Storage Groups in the first paragraph of this up-thread post, I want to clarify what Storage Groups can do—now that they are out of beta in version 16.  As this post in another thread says, they seem to have been developed primarily to deal with a problem administrators have in record-keeping when they have Proactive script(s) that back up a large number of "clients".

jhg could use a Storage Group as the destination for a non-Proactive Backup via a script—but apparently not via an Immediate operation (see the quote in the last sentence of this paragraph) .  However I doubt whether he/she would want to, considering that the backup of each source volume would be stored on a different Backup Set within the Storage Group—with no deduplication between the source volumes.  And, as the Knowledge Base article says, "Scheduled scripts support Storage Groups as destinations, but the backups run on a single execution and not in parallel."

[x509]

On 2/6/2019 at 6:27 PM, mbennett said:

What David Hertzberg said.

At the suggestion of Retrospect support, I now schedule multiple jobs in multiple scripts one minute apart.  You can easily run dozens at a time, and the maximum number is tunable in Preferences.

You can't run two jobs at the same time from the same source.  You can run multiple scripts to the same destination, but for the sake of clarity and organization you will want to use unique destinations.

I believe (but haven't tested this) that you could schedule two scripts to run with the same source, one minute apart.  The second one will wait until the first one is complete.  Try this and let us know.

If you have a script and want to run it immediately, highlight it in Manage Scripts then click the third icon from the left on the toolbar, a little scripty looking sheet of paper with a lightning bolt next to it.

I have several backup scripts that use the same resource (source drive) and which are scheduled to start exactly one minute apart.  I want to confirm that that approach works just fine on R15.61 on a Windows 10 Pro machine..

The practical impact is to "batch process" all the scripts together, and minimize total execution time.

 

x509

PS:  I've been away from this forum for a long time, for reasons that I don't want to explain but are "understandable."  I will try to do several catch-up posts in the next few days to document all my issues with scripts and selectors.

[DavidHertzberg]

Assuming, x509, that "I want to confirm that that approach works" means "I want someone else to confirm that that approach works" rather than "I want to confirm that that approach has worked for me", I've done the following—using Retrospect Mac 16.0.2.101:

Every morning except Saturdays at 3:05 a.m. I schedule a No Media Action (Retrospect-Mac-speak for Normal) backup of my MacBook Pro "client"'s only drive, but 5 minutes before that "real" script is scheduled I still schedule—out of habit although it's no longer necessary— a "sacrificial" script that does the same backup for that same "client" with the No Files Rule (Selector).  x509 doesn't pay me enough money  to stay up until 3 a.m., so last night before I went to bed I re-scheduled this morning's "sacrificial" script for 3:04 a.m.. My BPH awakened me around 3:40 a.m., upon which I booted my MBP followed by my Mac Pro "backup server".  The "sacrificial" script" immediately started running, while the Activities panel (the Retrospect Mac equivalent of merged panels from all the Activity Monitor tabs except History)  on my Console showed a yellow warning icon—which when clicked said "Waiting for Media Set White" (which I would have named Backup Set White if I had been running Retrospect Windows)—scheduled before the "real" script.  As soon as the "sacrificial"  script had finished running in about 4 minutes, the warning icon disappeared and the "real" script—with the All Files Except Cache Files Rule (Selector)—began to run.

This is the behavior I've had since I started scheduling "sacrificial" scripts with Retrospect Mac 12 in February 2017.  So, barring any unlikely scheduling difference between application variants, I predict "that that approach works" for Retrospect Windows too.