Jump to content

Malcolm McLeary

  • Posts

  • Joined

  • Last visited

Profile Information

  • Location

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

Malcolm McLeary's Achievements


Newbie (1/14)



  1. Retrospect can leverage Amazon S3 Storage for cloud backups but has anyone tried installing the Retrospect Linux client on a Linux instance running at AWS and pulling back backups rather than simply trust the AWS infrastructure? Perhaps its more than trusting the resilience of AWS infrastructure ... perhaps its more of a case of wanting a copy of the data "in-house" in case I want to host elsewhere. One of the points Retrospect make about their Cloud Backup option is that you are not locked into a given provider as you can always "transfer" your backups to another S3 Compatible Storage provider. I'm sure that may take a while, but it can be done. In this case I'm thinking about keeping backups of a hosted machine "in-house" such that the data could be restored to another service provider should I choose.
  2. No ... just saying that I've never had much luck getting DRD to work. I've found its generally simpler to just reinstall the OS. The DRD process is designed to use FAT32 so its not going to have any files bigger than 4G anyway. Yes its a "waste" to use 32G when you could get away with 8G or less but nobody makes "small" flash drives these days. Just use the whole media.
  3. David, I have been very clear in that its the "Desktop Experience" requirement that I have issues with, which could have and should have been changed a long time ago. I don't need a version by version summary of under the hood changes since v7.7. I have always said Retrospect has some great features under hood. Yes ... presently Backup Set Encryption is optional and disabled by default on a Set by Set basis. It really needs to be a global Security Preference and perhaps mandatory for Cloud Backup Sets. That is fundamentally why my "exploit" works ... once one has access to the Retrospect Management Console you can add a Cloud Backup Set on any S3 Compatible target and have the "backup server" diligently use it (without encryption). Being a cloud service 2FA is industry best practice if not essential. Yes I was noting that there is a Heroku presence in APAC and you identified how simply Retrospect can make it happen. Still I'd prefer self hosting on premise. Drop Bears are a story passed around to keep visitors away, however some of the most deadliest creatures on the planet do live here.
  4. David, I can't explain the "stagnation" between v7.7 and v17.0 as I stopped using Retrospect at v7.7 when the writing was on the wall that EMC intended to kill the product, only to return recently to see next to nothing had changed. Don't get me wrong, Retrospect has some great features under hood ... shame about the look and feel (of the Windows version). Presently I've posted my concerns about the security of the Management Console here to illicit peer group thoughts and advised APAC Sales, but the next step is Tech Support although I have already raised a feature request for 2FA without indicating my underlying concern ... just that its best practice for cloud services. I must say that my previous post is not simply theory ... its pretty straight forward to demonstrate that once you have access to the Management Consoles you can add a Cloud Backup Set and the add a script or modify a script to send company data to that Backup Set. You can then recatalog that Backup Set on a separate machine and hence "restore" anything. Its irrelevant that the Management Console can't access data ... it can instruct a "backup server" to save data to a "foreign" Cloud Backup Set. The weak link is access control. Apart from 2FA the situation could be improved/resolved by enforcing backup set encryption where ONLY the "backup server" knows the secret key. This exploit works because the additional "foreign" Cloud Backup Set is not encrypted by default. Strong passwords are a myth ... no-one cracks strong passwords ... users are tricked into revealing them or they are stolen. A "random" 6 digit PIN which changes every 30 seconds and is generated by an independent device is a simple to implement fix. Leveraging a hardware token like a Yubikey would gain massive street cred. I have raised a support case about the Management Console having issues with time zones ... status is hours behind reality most likely because APAC is in the future compared to the US. This is quite different to simply "response" and "refresh". AWS has a presence in APAC ... and so does Heroku https://blog.heroku.com/private-spaces-sydney-region but realistically I'd prefer to run a Management Console on premise ... not in the cloud. Presently Retrospect is hard coded to post status updates to console.retrospect.com ... it would be better if this was user definable (aka self hosted). Drop Bears are real ... if you have ever gone camping in the Aussie bush you may have heard the males at night ... truely frightening. 😉
  5. Sorry for the confusion over "Does a Drobo NAS include S3 as a service ... ?" ... I was using the same terminology as is used on a FreeNAS box. There are Knowledge Base articles which describe how to use Docker to run MinIO on Synology or QNAP ... not seen anything regarding Drobo. It would be good if it was builtin just like it is in FreeNAS. Anyway ... I don't take anything Marketing says for granted until I've actually seen/experienced it. "Marketing" would have you believe that Retrospect for iOS works with Retrospect for Windows v17 because there is a download link with all the other v17 downloads and its covered in the v17 Users Guide.
  6. David, Realistically Retrospect does a great job performing backups to S3. What I was asking was if Drobo provides an S3 Compatible service ... can it be used as a "private" S3 Target. That is what I can do with FreeNAS as it has MinIO built in. Various Retrospect KB discuss installing MinIO on platforms so as to use them as an alternative to Amazon S3. Does Retrospect for Windows v17 listen on port 22024 ... yes it does ... just do a port scan and it will respond. Retrospect for iOS uses this port to connect to the MacOS engine and will also connect to the Windows version and work, but but then it breaks. According to Tech Support the API behind port 22024 has been changing since v7.7 but Retrospect for iOS hasn't been updated since 2017 and in fact hasn't been tested against later versions ... although its listed with the v17 downloads and is included in the v17 User Guide. See page 486 of the Retrospect for Windows v17 Users Guide. UAC really has nothing to do with a given service listening on a given port ... what the service then tries to do may cause "conflicts". The backup engine does not listen on port 497 ... do a port scan and see. 497 is the port the client listens on. The backup engine listens on port 22024. For remote backups to work the client can't simply broadcast its existence ... it needs to "call out" to the "backup server" at its IP address or FQDN on 22024 and then wait for the "backup server" to call back on port 497 (just as it does a LAN after discovery). Clearly the remote user will need to have port 497 port forwarded through their firewall. Node.js and Ruby are very common coding languages and supported an MacOS, Windows and Linux. The programming language is not really the issue ... its more about accessing the API behind the listening port. The API should be based on industry standards rather than some in-house propriety protocol. I'd suggest very few Administrators are using Retrospect for iOS or the new Retrospect Management Console. Retrospect for iOS may work for MacOS installs, but Retrospect changed the API in the Windows versions so its now broken. Retrospect Managements Console is most likely the future but it has issues. I've come back from the "wilderness" and have higher expectations based on experience with other products ... not backup related ... see what Ubiquiti is doing with monitoring/management of their network infrastructure. They have iOS and Android apps, web based consoles which can be cloud or locally installed plus CLI on most devices. My biggest beef with Retrospect for Windows is that it still requires a "Desktop Experience" to operate and that prevents it from being taken seriously in any environment larger than home users or small business.
  7. Its been said that the Retrospect Management Console is adequately secured and besides it doesn't contain any actual data. Further its "mainly" a reporting tool which passively receives status information from "backup servers". Apart from the performance aspects of this thing running in the US, I'd prefer to be running it "in-house" because I believe that presently there is a fairly significant security flaw which could be exploited to exfiltrate data on a large scale. Please tell me I'm wrong ... The attack vector requires first gaining a username/password combination for Retrospect Management Console. Given that accounts are compromised all the time lets assume that an Advanced Persistent Threat (APT) exists for a given organisation and they have the means to obtain credentials for what is an external publicly visible system which does not enforce 2FA. Now although the Retrospect Management Console does not contain any data, it does have the ability to edit backup scripts and even create new scripts. For these new scripts, or existing scripts, you can set up "Other Destinations" ... including Other Cloud Storage. So having gained access to the Retrospect Management Console (as its only secured by username/password) the actor simply adds an extra Amazon S3 destination with a new 20 char Access Key and 40 char Secret Key and have Retrospect send out "everything" ... essentially untraceable as the Access Key and Secret Key don't provide any information as to which Amazon Account owns the S3 Bucket. Then the actor simply Recreates the Catalogue for this exfiltrated Storage Set, on another machine running Retrospect and restores whatever they want. It seems that the actor can even cover their tracks by deleting the created script after they've gotten what they want. There is no need to mount an attack on Amazon or any encrypted backups which may be stored there. There is no need to penetrate the target site and copy data directly or remove encrypted backup media which may be laying around ... just add a Cloud Storage Backup Set to one or more existing Scripts (or add one) and have Retrospect diligently do all the work. Who is going to notice that "extra" Backup Set or that "extra" script in a large installation? The only problem I see at the moment is that you can't turn off verification on this new Cloud Storage Set so the additional download may be noticed but as most organisations download way more than they upload this overhead may go unnoticed. Just putting it out there that any cloud based service which is only secured by username/password is just asking to be compromised and in this case becomes an avenue for massive data theft.
  8. I have often said ... "the only backup you ever really need is the one you don't have". Hence I tend to backup everything, except the things I know I don't need, rather than backup just the things I think I may need or may get "lost". In this case I'm being overly cautious (and just testing things out) as realistically the only stuff I need to backup are; /home/ /backup/Backups/ Database backups are stored in /backup/Backups/ so short of complete disk failure everything I need is on a disk in the machine. Retrospect's job is to maintain backups off the machine. Its not unreasonable to define the sub volumes I really want to backup to elsewhere as presently I use SAMBA to publish appropriate "shares" and back them up as if the machine was a fileserver. Its just that using the Retrospect client I can backup the whole machine quite easily. Whether I should is a different question. To rebuild the machine I just boot from the CentOS install image, install, update, add a few packages stored in /home/sysadmin/ configure FileMaker Server and restore the most recent backup from /backups/ (which is a separate drive which may have been restored).
  9. I'm using a new install of Retrospect for Windows v17.0.2 and I have created a Custom Selector so as to NOT backup files in some specific directories because I know the content changes every 5 minutes and I don't need a backup of them. Linux path /backup/FileMaker/Progressive/ and subfolders Linux path /home/FileMaker/Progressive/ and subfolders Linux path /opt/FileMaker Server/FileMaker/Data/Progressive/ and subfolders Depending on my configuration only 1 of these will be in play and contain files. The backup runs but records hundreds of errors because my progressive backups "change" during the backup. --Log-- +Normal backup using Immediate - Lenovo at 07/07/2020 9:09 AM (Execution unit 1) * Resolved container Lenovo to 3 volumes: /backup on Lenovo /home on Lenovo / on Lenovo 07/07/2020 9:09:41 AM: Finished scanning backup set data files To Backup Set Lenovo 002... - 07/07/2020 9:09:40 AM: Copying /backup on Lenovo While scanning volume backup, Folder /backup/FileMaker/Progressive/IncrementalBackup_2020-07-07_0900/Additional/RC_Data_FMS/PacificStone/Files/PacificStone/Secure/01/, Scanning incomplete, error -1101 (file/directory not found) Folder /backup/FileMaker/Progressive/IncrementalBackup_2020-07-07_0900/Additional/RC_Data_FMS/PacificStone/Files/PacificStone/Secure/02/, Scanning incomplete, error -1101 (file/directory not found) (snip) 07/07/2020 9:13:38 AM: Found: 102,106 files, 395,298 folders, 59.7 GB 07/07/2020 9:13:42 AM: Finished matching 07/07/2020 9:14:04 AM: Selector "All Files Except Progressive Backups" was used to select 81,592 files out of 102,106. 07/07/2020 9:14:16 AM: Copying: 0 files (zero KB) and 0 hard links 07/07/2020 9:14:29 AM: Building Snapshot... 07/07/2020 9:14:29 AM: Copying properties for 395,298 folders 07/07/2020 10:54:17 AM: Finished copying properties for 395,298 folders and 0 files 07/07/2020 10:54:27 AM: Copying Snapshot: 2 files (132.5 MB) 07/07/2020 10:54:30 AM: Snapshot stored, 132.5 MB 07/07/2020 10:54:30 AM: Comparing /backup on Lenovo 07/07/2020 10:54:32 AM: 257 execution errors Duration: 01:44:52 (01:44:43 idle/loading/preparing) It seems that the File Selector is being applied too late in the process. I would have thought that if I've identified paths NOT to be included in the backup then by definition they'd be excluded from the scan. Why waste time scanning paths (and recording errors for files) which are going to be excluded?
  10. David, There is a difference between running a "web server" and simply leveraging web technologies. A "web server" in most peoples minds conjures up being able to access fully formatted and structured pages of information but it doesn't have to be that way. The "service" could respond simply with XML or JSON formatted data to an appropriately crafted request (which also contains an Access Token previously obtained). The "service" could process appropriately formatted XML or JSON data which is in the payload of a http POST. Commonly if the host does have a "web server" then it handles the routine "web" traffic, but appropriately crafted requests are passed to the "service". This way the "service" can leverage an existing installation of Apache, IIS, nginx, etc. I suspect the Retrospect Management Console works like this where the "engine" does an http POST to the "console" periodically to give it status updates and polls for "commands". It would be appropriate if port 22024 on the "engine" worked like this as well where the Retrospect for iOS sends a http POST and gets back appropriate data which it processes, caches and presents. Security is provisioned by using https rather than simply http. If you keep it all "in house" then self signed certificates are fine, but if you get standard browsers involved when rendering the "console" then you need to have properly signed certificates. This can be problematic for Home users but seldom an issue for Business users as its best practice to buy a wildcard certificate and use it everywhere its needed. An alternative is Let's Encrypt https://letsencrypt.org which is an option provided by most NAS vendors. None the less it can be tedious because you need to have DNS entries and expose port 80 and 443 to the internet so Let's Encrypt can validate the endpoint. Anyway, Windows 10 can run IIS ... its simply not installed by default. MacOS has Apache and so does Linux (even if its not part of a minimal install). Retrospect don't need to include a "web server" ... its most likely already there, BUT if they persist in maintaining backward compatibility with Vista/7/8, etc, for a "backup server" then things probably get complicated, however I'm only talking about a "backup server" ... not clients.
  11. David, Full circle, and back to where I started. Retrospect is attractive because the clients are treated equally (MacOS, Windows and Linux). I've been in the "wilderness" and although Synology does have a neat solution it conveniently overlooks MacOS ... however that's a trend because Apple provide TimeMachine so many vendors just provide a TimeCapsule compatible service. It works, but that is not what I want. As an alternative to TimeMachine, Synology also provides Synology Drive which is sort of an on premise version of DropBox, GoogleDrive, OneDrive, etc but with the ability to schedule backup tasks. Its a client but more active, however I'd prefer to have centralised management ... hence Retrospect. I also tend to use CCC rather than TimeMachine. Web (or browser) based management (e.g. Retrospect Management Console) doesn't necessarily mean in the cloud. Sure it is now, but I'm suggesting that it should be able to be hosted anywhere ... localhost, separate host or cloud service. Just like when you leverage Cloud backups (e.g Amazon S3) you need to provide a "Path" which starts with the FQDN of the server it should be possible for "Management Preferences" to include the FQDN of a Management Console as a variable and not just be hard coded to "console.retrospect.com". FreeNAS includes S3 as a built in service (based on MinIO) ... just turn it on ... so instead of an Amazon FQDN I can simply enter the the FQDN of my server (which may be onsite, different building or offsite). Does a Drobo NAS include S3 as a service or does it just do SMB, AFP, and NFS? Anyway back to Retrospect for iOS ... it works over port 22024 which may or may not use SSL and it may or may not be http based (but it should). If it was http based you could put a reverse proxy in front of it and hence not have to open a hole in the firewall for it to work. The listener on port 22024 appears to be passive in that the iOS App sends it commands for it to respond with details or do something. Sounds like the basis for a Management Console to me. The actual Retrospect Management Console (hosted) receives https traffic from Retrospect for MacOS and Retrospect for Windows. Presumably it queues up commands and managed machines poll the queue periodically as it works without inbound firewall adjustments. So fundamentally the architecture I desire is where the Retrospect "engine" runs as a faceless background service and listens on port 22024 (as it does today) for commands from the iOS App. The Retrospect Management Console listens on a different port (could be https 443 as it does today) but is essentially a custom web server such that a browser on local host or a remote host can access it. It could get its details from the "engine" by sending commands to port 22024 (just like the iOS App does). The "engine" could actively send the "console" status details as it does now but not hard coded to "console.retrospect.com" ... it could be localhost or a specified host. The existing Retrospect for Windows UI could be retained for local management if desired/required by abstracting the UI such that it communicates with the "engine" via the same mechanisms as the Remote Management Console (and doesn't need to be left running). Is there precedence for this? Sure ... just look at FileMaker Server. Its comprised of multiple faceless background services which are task specific. The clients (i.e. FileMaker Pro for MacOS/Windows) and FileMaker Go (for iPhone and iPad) access hosted databases on port 5003. Web clients can access hosted databases on port 80/443 as there is a service which actively (on the fly) converts layouts into html/css/javascript and talks to the database engine on behalf of the clients. There is also ODBC access on port 2399. The Administration Console can be accessed either from localhost or a workstation via a browser on port 16000. A very different port is used here such that it can be easily firewalled if you want to limit access. Customised access is possible via two published APIs ... the Data API and the Admin API ... both are accessible via 80/443 such that "anyone" can build their own app be it for accessing a hosted database or to monitor/manage the server. Further you can leverage Zabbix to monitor/manage the whole host and even manage FMS by scripting its CLI. For scaleability you can add additional "worker" machines which essentially just run the webdirect service such that web clients are offloaded from the "master" machine. Each of these "worker" machines handle the browser client interaction and communicate with the database engine on their behalf. Sounds like a lot of moving parts, but having multiple moving parts is much better than a monolithic application which requires a "Desktop Experience" to run because a given part can fail and be restarted without impacting the whole. You can even have a "helper" service which checks that all the services are running and restart them if necessary. Further ... moving from a "Desktop Experience" to say CentOS (without GUI) or a NAS (without screen, keyboard or mouse) is straight forward because none of the services depend on a GUI ... you don't even need a GPU. Although Retrospect for iOS is listed with the rest of the downloads for Retrospect for Windows v17 it hasn't been updated since 2017 and the documentation only mentions "partial" support for Retrospect for Windows 7.7. Perhaps the documentation has never been updated. Perhaps it did work with Retrospect for Windows in 2017, but 3 years later it appears that "partial" is now "broken". Is it unreasonable to expect that if if its listed with v17 it should work with v17? Tech Support Advice ... check port 497 is open.
  12. David, Thanks for the feedback. I have logged a Support Case ... would like to provide the Case No but nothing gets listed in my Portal even though I get an acknowledgement email and a link (which just gets me to the login screen). Shouldn't really discuss competing products, however I use Hyper Backup to backup the Synology itself. The product I haven't mentioned is "Active Backup for Business" ... https://www.synology.com/en-au/dsm/feature/active_backup_business If a Drobo NAS was to have Retrospect integrated, then I'd see this as a product the marketing boys would want to benchmark against. I'm actively looking at Retrospect again because it does have advantages over the competition. Its just unfortunate that some capabilities I'd like to be able to leverage are incomplete. The iOS App for example would give local staff visibility of status without having to physically access the backup server. Similarly with the Retrospect Management Console as its also at arms length of the backup server. It would be nice for both to have "read only" access just for monitoring. The FileMaker Server 19 for Linux Developer Preview does have the advantage in that its been running on CentOS at AWS for several years so the base is solid, its just that now they have made it available for on premise installation there are lots of new variables to be concerned about and there are some differences between the Cloud option and on premise so capabilities are being added such that it is feature equivalent to the MacOS/Windows versions. Hardware is probably the biggest variable, I'm testing various VMs which are themselves hosted on various NAS platforms including Synology and FreeNAS, while others are going down the Docker path. Many new deployments to the previous AWS only. Although Claris is owned by Apple they are a separate business unit and expected to perform as if independent. To a degree Claris (aka FileMaker) is similar to Retrospect in that both started doing a Mac only product then diversified as the Windows market was bigger. A big difference is that FileMaker for Windows looks and behaves the same as FileMaker for MacOS. There are some platform specific differences, but by and large they are the same product. Claris at one stage managed many products but then became FileMaker Inc with only 1 product. Recently reborn as Claris (again) with the opportunity to expand the product portfolio. Occasionally Apple tosses them a curve ball like "Claris Connect" where the parent bought a capability and assigned it to Claris to make it work. Claris Connect is not about FileMaker its about connecting a diverse range of apps via APIs. Where Retrospect can with some effort talk to Slack, Claris Connect promises to make this much easier to do and not be limited to just Slack ... basically anything in the ecosystem. Currently my biggest issue is cost and the fact that it relies on US Hosts. No doubt Retrospect has staffing challenges with respect to dev/test, which is why I believe they are making it harder than it needs to be by supporting legacy platforms. Ok ... they can't drop support completely, but I think the client is where legacy support should be focused. Customers don't have to upgrade their entire fleet of workstations/servers if they don't want to, they just need to maintain the "backup server" if they want to run the latest release and in return Retrospect just need to focus on Retrospect for Windows working on currently supported OSes (which in turn only runs on supported hardware) and legacy support is available by ensuring the various platform clients work with the current version of Retrospect. Just looked at the 3 Windows 10 "backup servers" I'm working with and they are all Windows 10 Release 1909. All three are fully patched and say they are waiting for 2004. I don't run Retrospect on any Windows Server machines directly. They are also fully patched and all running 2019. Don't have any Vista/7/8 machines at all. I'd have to create a VM to test that and I don't have any motivation to go there.
  13. The Web Hooks documentation may be what I'm looking for with respect to API documentation. https://github.com/retrospectinc/datahooks but it refers to v15 ... is it still supported in v17 (and beyond)? If the iOS App leverages web hooks in order to do its thing, the the answer is probably Yes for MacOS but No for Windows.
  14. Is there a definitive statement of the compatibility of Retrospect for iOS? All the information I have found is quite dated. Appears to be working with v17 for MacOS but currently not working with v17 for Windows, although it did for a couple of connections but now just “Connecting …”. Same thing on my iPad. Have tried with both a local server and an offsite server ... appropriate firewall rules have been implemented. Perhaps its confused that I’ve asked it to talk to 1 x MacOS backup server and 2 x Windows backup servers concurrently. The UI supports adding multiple servers so I expected this to work. Also, are the details of the API the iOS App uses documented somewhere? Is it https based but using port 22024? If it is a standard http based API I’d like to build my own “Management Console” with FileMaker (as the existing Retrospect Management Console is less than satisfactory). I could then have my server "checkin" with various backup servers and present consolidated stats reports. Wait … I have an old copy of v16.6 Solo running on a Microsoft Surface … it works with the iOS App! Does this mean Retrospect broke v17 for Windows. Scratch that ... v16.6 for Windows is unreliable as well ... works once, but then gets listed in Pending as "Connecting ..." but doesn't ... even after a reboot.
  15. David, I'd answer "No" and "Probably not, but one can hope" to your questions. It has potential, but Retrospect Management Console doesn't really work. It should only be available for testing ... a Limited Preview. I'm currently trialing a Developer Preview of FMS 19 for Linux and so far I can't break it. Its as stable as the release versions for MacOS and Windows ... feature for feature its the same product (accepting a few capabilities yet to be implemented) running on a different OS. IMHO Retrospect need to do same. Not only leverage common code, but also the exact same UI and workflows. Deploy the client as appropriate and run the server on what is most appropriate for the environment. The experience should be the same. I'd be interested in what they do with a Drobo NAS. I've used a Drobo for personal storage for years but only a direct connect version. Yes, the Windows market is bigger but unfortunately the users are generally comfortable with using slapped together, poor quality software ... its simply consistent with the OS. It doesn't have to work ... it just has to sell. I gave EMC a chance when they took over Dantz Development Corp. but it was apparent to me that they were a Windows company and product quality was going to suffer so I moved on. Perhaps I should forget about this little misadventure back to Retrospect and stick with Synology based options.
  • Create New...