Yet Another Case for Viable Back-Ups and Testing

Mistakes Happen
By Charles Parker, II

InfoSec has the distinct tendency to be very taxing and stressful, at the most inopportune times, field to work in. There are the usual deadlines, budgetary constraints, labor hour limitations, internal politics, vendors calling and/or emailing, and the inevitable compromise or successful phishing campaign at 3:50 Friday afternoon or 3:30 Tuesday morning.

Murphy’s Law has been very active in InfoSec for some time. These moving parts must be considered and scheduled to continue the forward movement while maintaining the in-depth defensive posture against the attackers from across the globe.

This balancing act is manifested with the user multi-tasking. The human experience only has so much attention to applying to all the projects. With a greater number of projects, there is less attention to each applied. With this, all it takes is one oversight and there may be a massive time-consuming issue to resolve.

One area of operations that has become increasingly important is the back-ups. Back-ups have been very useful and a beneficial tool on many different fronts for the business and Admins, e.g. a user deletes an email or sets of emails, hardware errors, users being ransomware victims, and other use cases.

In general, this is a prudent practice and an industry standard. The Admin never knows when the data would be needed. This protocol is simply important. Not to utilize a back-up protocol is, at the least, bordering on negligence.

With the back-up methodology, there are many factors to take into consideration, including the timing and media. Also, as important is the testing. Without a robust test periodically, there is no guarantee the back-ups are viable. Testing is not always done though.

At times, the Admin simply is too busy and accepts the output from the back-up application stating the back-up was perfectly acceptable. Although this report may provide an artifact stating all is fine, there may be an error. The dependence on this may provide the background for significant oversight and error.

GitLab
An issue was noted recently with GitLabs back-ups. GitLab is like GitHub, except with an alternate focus of lab work. With this instance, an employee deleted a directory located on the incorrect server.

This was clearly an accident and not a case of malicious insider misfeasance. The SysAdmin was at work later in the evening, and in the fatigued state inadvertently deleted a directory on the wrong server. Within this directory was a folder holding 300GB of live production data, which was supposed to be backed-up.

The SysAdmin realized the oversight when there were only 4.5 GB of data remaining. At this point, the SysAdmin was thinking of the back-ups and hopping these were still working and inviable.

Although this would have been a great use of the back-ups and a victory, there, unfortunately, were issues. This use case involved live data. The prior viable back-up was completed six hours previously, so there was a gap. To add an issue to this, GitLab utilized five back-up formats. None of these continued data or was set-up initially.

Oops.

Lessons Learned
The application of insurance is to protect against an event with a low chance of occurring that would have a large impact if realized. This was one of those cases. The back-ups are a form of insurance. With a catastrophic, epic failure, the business operations would simply cease or nearly so. The business would need to use paper again to do much of anything.
The users and Admins may not put a mass amount of thought into this until the back-ups are needed. At this point, it may be an emergency to get these in place and working.
The business needs to have regular back-ups scheduled and tested regularly. Without these, the Admin is merely hoping and placing their reputation on a report.

About The Author
Charles Parker, II began coding in the 1980s. Presently CP is an Information Security Architect at a Tier One supplier to the automobile industry. CP is presently completing the Ph.D. (Information Assurance and Security) in the dissertation stage at Capella University. CP also is an adjunct faculty at Thomas Edison State University. CP’s interests include cryptography, SCADA, and NFC.He has presented at regional InfoSec conferences. Charles Parker, II may be reached at [email protected] and InfoSecPirate (Twitter).