I was told to make backups, not test them. Why does that make you look so worried?

On Call Each week at work creates memories many are happy to forget, but some are willing to share with fellow Register readers in On Call, our Friday column that tells your tales of tech support.

This week, meet a reader we’ll Regomize as “Lionel” who recounted a story from a moment in his career that saw him serve as “senior developer/L2 help desk/the guy taking care of hardware for a mainframe software development team.”

Lionel’s role sprawled into that range of responsibilities “because almost everybody else on the team thought that Intel servers, PCs, and laptops would rob you of your soul.”

But the 80-strong team of mainframe devs Lionel worked with couldn’t escape the modern age entirely because the server that backed the code they created was an Intel box.

“Source code was stored on an SMB share on a large Intel server on which a backup was taken daily onto an 8 mm tape drive,” Lionel told On Call. “We even kept weekly and monthly backups, just for good measure.”

One day, the chap who maintained the backup box – let’s call him “Richard” – decided he’d had enough of sullying himself with such chores after having done the job for around two years.

Lionel was given the gig and kicked things off by asking Richard when he last conducted a test restore.

The answer was “never” because Richard had been asked to make backups, not test them. But he assured Lionel that the tapes used for backups could all be found in a cabinet next to the server.

Lionel asked how Richard knew backups were working and was shown a backup software log file that included the line “Backup completed successfully.”

Unconvinced that this was proof of successful, recoverable backups, Lionel asked Richard if he had ever verified the backups.

A blank stare ensued.

Lionel therefore tried to restore. The tape drive quickly complained that the tape he inserted was blank.

Then, as now, tape was finicky stuff so Lionel wondered if perhaps the tape drive needed cleaning before it could read the backup. Richard produced a cleaning tape he had inherited from a predecessor and had proudly used every month since … without realizing it was only supposed to be used five times.

And Richard had handled the backups for at least two years.

Lionel ordered fresh cleaning tapes, and watched in horror as a first pass produced a thick brown smear of dirt and left the drive still unable to read tapes. He tried a couple more tapes and eventually spotted one that carried a warning that it could only be used ten times before the vendor could not guarantee readability. Of course, Richard had never replaced the tapes.

After more experimentation, Lionel realized no amount of cleaning would allow the drive to read a tape.

Arrr! Can a sailor’s marlinspike fix a busted backplane?

User said he did nothing that explained his dead PC – does a new motherboard count?

Tech support fill-in given no budget, no help, no training, and no empathy for his plight

Devs sent into security panic by ‘feature that was helpful … until it wasn’t’

By now more than a little worried about the company’s data, he bought a new and expensive replacement that thankfully managed to complete a restore.

Of course, Lionel was not thanked for this feat. Instead, his managers asked about the unexpected and substantial spending on the replacement tape drive.

Lionel explained that the company may have gone without backups for a year or two, and to illustrate why he held up one of the tapes he’d tried to coax back to life.

“I held it up against the light. It had been worn so thin that you could see through it.”

Which made it impossible for his managers to object to his expenditure.

“This was a profound learning experience that has stayed with me ever since,” Lionel told On Call. “Unfortunately, even today, test restores are sadly very uncommon, in my experience. Perhaps because it’s called ‘Backups’ and not ‘Restores,’ leading people to the mistaken belief that restores are only necessary when everything else fails and their world is ending.”

“Fortunately, as long as that belief prevails, skilled and competent technical IT people will never be without a job,” he concluded.

What’s the longest period of time you’ve seen bad backups go undetected? And what happened when the time came to restore? Refresh your memory of data recovery romps, then click here to send On Call an email so we can share your story on a future Friday. ®