Watch out, rogue users — the data detective is on the case

Data is king! Big data analysts aren’t alone in making this proclamation; we IT pros can drink the Kool-Aid too. In fact, we often depend on the data trail to pin down causes and find solutions for our peskiest problems. However, it’s easy to forget raw data tells only part of the story — and the conclusions we draw from the information may have unforeseen repercussions.

This story of data’s reach and limitations dates back to the late 1980s, when our department worked with users in a software development environment. Our users were a mixed group of very talented and intelligent developers, along with those who didn’t log in very often, such as salespeople.

[ Get a $50 American Express gift cheque if we publish your IT story. Send it to offtherecord@infoworld.com. | Think your workday is bad? Check out these dirty IT jobs. | Get your weekly dose of workplace shenanigans by following Off the Record on Twitter and subscribing to the Off the Record newsletter. ]

I was one of the system administrators, as well as a systems programmer. For our platform, we used Digital Equipment’s VAX/VMS, and most users had computer terminals in their offices that tied into a multi-user system. The admins asked users to log out at the end of the day, and most — but not all — complied. This became a problem when we had to reboot the system.

Although we requested they log off, we knew some users’ projects would be compromised if they lost their command history upon a system reboot. Still others simply needed or wanted to stay logged in, which we could also abide. But then you had a segment of users who forgot to log off or weren’t running anything relevant, unnecessarily tying up system resources. That last group also made it difficult for the admins to determine how our after-hours system reboot affected the users overall.

A solution to the rescue

To minimize this conflict, I worked on a convenient solution that happened to be a rather fun project. I’d found free software that would kill idle user processes, and with the program code, I could customize the software to be a little more intelligent.

I modified it to check for terminal I/O activity and to determine if a user was running a program. I didn’t want to risk killing any practical user processes, such as a file-editor session that might cause a user to lose changes to a file that had not been saved. Therefore, if a user ran any program or was in a text editor, the program would leave them alone.

Our goal was to minimize user logins at the end of the day, not to get rid of them entirely; we didn’t want to antagonize anyone either. Therefore, I was determined to make the job killer easy to circumvent. If someone was dead-set on staying logged into the system by running a program, that was fine. But I’ve seen users go to great lengths to circumvent this kind of tool — for example, by creating a completely useless and continual subprocess loop that would prevent their session from ever going idle, in direct opposition to the goal of freeing up system resources.

As for system reboots, we always gave advance warning and would place a courtesy call to anyone still logged in before cut off transmissions. As expected, the job killer reduced the number of users in the system — and our hair.

Catching the culprit … or not

We ran into a snag. Once this utility was in place, overall it seemed to work well. However, on a few occasions it mysteriously died. Everything checked out fine as I narrowed down the possibilities. Eventually I came to a conclusion: In all likelihood, someone with privileged access had shut down the process killer.

To find the wrongdoer, I used a trick I’d heard about at a recent computer symposium. I had learned that one could dump the command buffer for any current user on the system to see what commands were issued. Often you could see every command the user issued on their terminal since logging into the system.

It was limited only by the total size of the collection of commands because the size of the command history area in system memory for each user was fixed. If the user logged out, the command history would be gone. For once, it worked to our advantage that some users did not want to log off. The next time my process died, I looked at the command buffers for several users and found the command that stopped my process. I had caught the person red-handed and presented a printout of these commands to my boss.

We confronted the user and asked him not to do this, but he said he had never entered those commands and someone else must have used his account to do the deed. It wasn’t worth pressing the issue any further at that point, so my boss gave the user the benefit of the doubt and gently asked him to keep his terminal and password secure. Whether or not that user was guilty, we didn’t see the problem again.

Besides being a fun project, this endeavor taught me a lesson: Just because I could show an action originated from a particular user account does not always prove it was done by that particular user. More evidence may be required to tie an event to an actual person. Proceed cautiously — it could be their job on the line.

Send your own IT tale of managing IT, personal bloopers, supporting users, or dealing with bureaucratic nonsense to offtherecord@infoworld.com. If we publish it, you’ll receive a $50 American Express gift cheque.

This story, “Watch out, rogue users — the data detective is on the case,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.