Reinstall OS in a CyberSecurity Emergency

Disclaimer: This content is the result of my having survived several emergencies of varying effect sizes. Most of these ideas are rules of thumb, and not canned answers. As in any emergency your best outcome will be if you keep calm, consider many options, and flow from plan to backup plan as required.



The front-line defense of IT personnel to any security problem is a clean OS install with backup data restored from a clean source. This is the main work-around when none other exists and when the source of the problems is unknown.

While this process initially seems quite onerous, with a little practice and preparation it is not too difficult. Most IT professionals, whether working for a big organization or working in a computer repair shop, know how to do this kind of recovery.

If your machine is immediately re-infected after a clean install, then you made some mistake in some step. Your install media may not have been clean, particularly if you used a tainted network server or tainted USB. Your backups may have become tainted via the computer that was originally infected. Some critical mistake could have been made in configuring the machine or the firewall.

As a knowledgeable user, it is important to notice at which step of the recover the infection re-appeared. This key information will help you recover your machine again in a cleaner fashion.

When exposed to a new series of cyber attacks, it may be necessary to clean your machine numerous times while you iteratively improve your configuration and protection methods. During this time, you may notice clear patterns in attack vectors which can be prevented via critical changes to your machine configuration.

It is incumbent on IT professionals to share whatever information they have about attack vectors. It is important to share this information along with the confidence that one has in it's correctness. One should endeavor to distinguish between hypothesized attacks, work-arounds that are demonstrated to be effective, and the conditions under which the attack appears to be most effective.

The most mature report of an attack would be complete with source code and a known technical fix for the exact software vulnerability. Before that level of exactness is known, it is important to communicate hypotheses and work-arounds. In this process, we must do our best to avoid fear-mongering, avoid over-selling of solutions, and avoid minimizing bad user experiences. Communication, precision, and practicality are vital.

See cyber attacks for an example of how this information can be conveyed.

Organize Your Backups

It used to be sufficient to backup data to a USB device. However, due to the recent prevalence of USB trojans, it is currently very difficult to keep a USB backup clean. (With a lot of patience, a clean USB backup can be obtained, but it takes a good deal of technical skill to achieve that at the moment.)

There are a number of backup options which will be covered in detail later. We will list general categories of backups, here. As always the fidelity of your recovery process will be improve greatly if you make two backups to two different types of media. This will leverage the power of two choices, and improve your chances of having one clean and safe backup. One can also leverage the principle of off-site backup, to provide options in the case of a physical failure such as a natural disaster, disk failure, or a physical attack.

The backup options are as follows:

  1. USB attached hard-drive
  2. fixed media, such as CD-ROM, DVD-ROM, SD-card, etc.
  3. thunderbolt attached hard-drive
  4. firewire attached hard-drive
  5. network backups, such as network attached storage (NAS), cloud services, and the old Linux stand-by remote network drive

Carefully peruse your machine for files to save. The clean install of your OS will require formatting your hard-disk. So any files that you do not backup will be lost forever.

Whether you take your machine to IT for a reinstall or whether you do it yourself, you, the user, are the only person who know where all your files are located. You are responsible to make sure they are correctly selected for backup.

Organize Your Clean Install Media

CD-ROM or DVD-ROM is the cleanest source of install media. This is because there are read-only and can be certified free of infection.

USB thumb-drive install media can also be mostly free of infection, but it takes a great deal of technical skill and knowledge of the USB attacks to produce a USB install drive that is clean (more on this later). If you succeed in doing so, isolate and secure this USB device for the future. The simplest method is to grab an envelope, put the drive in the envelop, seal it, and sign your initials across the line of the seal.

At present, it is best to avoid USB install media, but difficult to do, as many machines no longer have CD/DVD drives. One could consider purchasing an external CD/DVD drive, but be wary of ones that attach via USB. It is not clear whether the USB DVD drives can become infected with a USB trojan. It seems that any USB device that has write-able memory may become infected.

Network install of binaries can work well, if the boot disk used to activate the network install is clean and the network server is clean. IT should be extremely careful to observe whether there is a pattern of infection that indicates either the boot disks or the network binaries are compromised.

Remember that you can always create your own clean install media, even using an affected machine, by downloading the source from the OS web-site, verifying the md5sum fingerprint against that published online, and using dd to write a clean image of the install source to your media. (This method can work using a USB thumb-drive, if it is done extremely carefully, more on this later.)

Consider borrowing clean install media from your local sysadmin. They probably have a stack of CDs with various versions of operating systems. If you have clean install media, consider loaning it to a friend.

Recall that licensed software may be helpful or unhelpful here. The licensing may help certify the cleanliness of the install media. Or the license, if it is limited, may prevent you from reinstalling a fresh copy of your OS. Please consult your IT professionals if you encounter this problem.

Remember that you will also need to re-install the software that you routinely use for work: MS Office, Adobe PDF, scientific utilities, graphics and image software, etc. It is important to also collect clean install media for this software. The install choices are the same: fixed media, network downloads, USB drives. It is becoming increasingly common to rely only on network downloads, which is incredibly vulnerable to DNS attacks and tainted binaries.

Organize the (Re-)Configuration of Your Machine

Record or save any important configuration files or scripts. This includes: firewall configuration, email address books, password key-chains, etc.

If you forget to save your configuration information, you will need to reconfigure your machine. There is lots of online documentation that aids will configuring machines for various purposes. Fortunately, PCs used as terminals attached the the internet are the easiest to configure.

Advanced configuration scripts and files are typically necessary only for servers, routers, and other infrastructure or cluster machines.

When in doubt, and when possible, reconfigure the machine from scratch. There could be some hidden vulnerability in your previous configuration that needs to be addressed. Finding that vulnerability will require careful attention to configuration.

Avoid Too Much Standardization

Standardization is an important engineering principle that can aid with consistency of IT service. It is also a vulnerability which can be exploited in a cyber attack. If the attackers know the standard configuration that IT employs, then they can devise an attack that compromises all the machines with that standard configuration.

However, standardization is critically important for providing software and security updates to users once a vulnerability has been identified and fixed with a patch.

In all cases, it is critical that users be aware of both the importance of and vulnerabilities of standardization. A standard OS image may work well for keeping a computing lab clean, but key personnel and leaders may require tailored security solutions that do not rely on standardization.


Homework

  1. If you are in a leadership position, brainstorm ways to implement a national reporting system for cyber incidents. Brainstorm also the vulnerabilities of such a reporting system (i.e. hoax reports, panic-ed user reports, mis-information, etc.). Is there a way to leverage the social network and low-tech solutions for reporting?
  2. If you are in an IT position, consider ways to roll-out independent security solutions which are user-centric. This means that we should not rely only on remote management and uniform updates to machines.
  3. If you are in a position to teach, consider offering instruction (full courses, workshops, or single lectures) on some aspect of security that you understand well (i.e. physical-, financial-, psychological-, or cyber-security).
  4. If you are a member of the trusted social network, consider educating yourself on any aspect of security and share information and solutions with friends and neighbors. If you are short on time, split this tutorial material with a friend and have them provide you the cliff-notes version of half the tutorial.
  5. If you take your computer to an IT professional to be cleaned, attempt to learn something about identifying software and security problems, so that you can detect and/or prevent them in the future.
  6. Consider using a cell-phone recovery method that is analogous to this laptop re-install approach. Most cell-phones can be re-set to factory install and settings.


© 2015-2021 Intrepid Net Computing. All rights reserved.