Five security lessons learned from Office 2010

I continue to be impressed with the changes coming in Office 2010 (currently in beta). Previously, I explained how Microsoft drew on real-world usage data to craft the beta suite’s updated UI. Security is another priority for the upcoming release, and while the improvements there aren’t as readily apparent, for developers they’re equally noteworthy.

The Office suite is complex and feature-rich, making it a ripe target for malware and other security exploits. Given its vast and diverse user base, however, for Microsoft to remove existing features in the name of security would be out of the question.

[ See why InfoWorld’s Neil McAllister likes Office 2010 and why Randall C. Kennedy hates it. | Keep up on business apps with InfoWorld’s Technology: Applications newsletter. ]

Instead, the Office 2010 product team concentrated on developing a new security strategy. It studied past vulnerabilities to learn how they were exploited and what could be done to prevent similar abuses in the future. The result was a new, multilayered security model based on five essential principles — ones any application developer would be wise to remember:

1. Validate all user input before acting upon it Any good programmer knows input validation is crucial. Unverified input can lead to buffer overruns, code injection attacks, and any number of other software flaws. But too often coders think only of validating form fields, text input boxes, and other UI elements. What about documents?

One way Office 2010 avoids unwanted security surprises is by prevalidating documents against a library of schemas of documents with known qualities — good and bad. The idea is that Office will be able to recognize specific document types and engage proactive security measures before it starts to interpret their code — for example, by disabling macros when it encounters documents that match the characteristics of known Word macro viruses.

2. Look for random flaws and atypical use cases Well-crafted use cases are crucial to software testing, but not even the wiliest of QA engineers can think of everything. The nastiest bugs can arise from unexpected areas — and sometimes all it takes is a few unexpected bytes to trigger them.

That’s why the Office team uses a technique called “file fuzzing” to detect unanticipated use scenarios. Fuzzing takes documents that are known to be valid and changes them at random — for example, by swapping parameters around, changing the contents of fields, or simply introducing random garbage data into the middle of the file.

Ideally, applications should be able to handle fuzzed files gracefully. In the worst-case scenario, however, opening a fuzzed file leads to a crash — and that’s a red flag for a possible security exploit. According to Microsoft reps, the Office 2010 beta can successfully handle fuzzed files 10 times more often than Office 2007.

3. Don’t trust third-party libraries Parsing Office’s own XML-based file formats is easy enough. But in practice, real-world Office documents contain much more than just ASCII data. They might also contain images, audio, video, or even code objects embedded from other applications.

Years of testing Office 2007 led Microsoft engineers to an unexpected conclusion: Most of the critical bugs they found weren’t the fault of Office code per se, but were the result of flaws in the core image-processing libraries used to render JPEGs, GIFs, and other graphics.

For Office 2010, Microsoft has switched its image-processing libraries to Windows Imaging Component, which is based on pluggable codecs. For developers of other applications, however, the lesson is clear: Don’t assume that even a widely used library will be free from defects. Test the code you draw from other sources as well as your own.

4. Don’t rely on users to enforce security Too many desktop applications use a security model that boils down to asking the user: “The action you are about to take poses a security risk. Do you want to go ahead anyway (Y/N)?”

This approach is not only lazy but dangerous. What makes a specific action a security risk? Just how risky is it? And for that matter, what does “security” even mean in this context? Lacking any meaningful guidance, the average user will just click Yes.

Offering more detailed warning messages doesn’t help, either. Users who issue commands expect the computer to obey their commands. They don’t want to argue, and they don’t have time to read essays about software vulnerabilities.

Instead, Office 2010 takes a proactive approach to security. The main goal of processes like file validation is to push more security choices into the background, rather than confronting users with dialog boxes. By making basic, rational decisions behind the scenes, in advance of user interaction, the new Office apps help take the guesswork out of vulnerability management.

5. Prefer degraded user experience to outright denial When older versions of Office encountered security risks, users really had only two choices: They could proceed as usual — succumbing to the threat — or they could not open the file at all. Neither choice was ideal.

Office 2010 tries to offer a middle ground whenever possible, by giving users a degraded experience rather than denying risky actions outright. For example, if Word 2010 encounters a document containing dangerous macros, it might open the document with macros disabled. The user still gets a warning message, but the document opens anyway. Text is visible, but enabling macros requires extra confirmation steps.

This philosophy of “meeting the user halfway” is one of the most important concepts underlying the new Office security model. As long as security procedures create a tug of war between users and software, users will always find ways to “win” — thus, subverting the very security measures designed to protect them. By offering a gradual, multilayered security model, Office 2010 tries to make security a partnership with the user — and that’s a lesson that any application developer can take to heart.

This article, “Five security lessons learned from Office 2010,” was originally published at InfoWorld.com. Follow the latest developments in security, Microsoft Office, and application development at InfoWorld.com.