GRAND FORKS – An email a stranger sent to the North Dakota University System’s computer security officer on the morning of Feb. 7 was the first sign that there had been a security breach.
The stranger had received a warning from Google the night before saying that an NDUS server had tried to hijack his account.
It so happens that that server, physically located at the University of North Dakota in Grand Forks, contained the private information of more than 290,000 students and employees.
News of the attack quickly spread as NDUS warned potential victims of the security breach. Later, it would fire three employees, including the security officer. But a security audit and other documents (CLICK LINK) obtained by The Forum shows that the problems went beyond just those employees to a flawed corporate culture in which security did not appear to be a top concern.
The audit, performed by the state’s Information Technology Department this past spring and summer, found security plans that were vague, people charged with security who didn’t think it was their job and no team in place to respond to security breaches.
In fact, after the UND breach was discovered in February, a team had to be cobbled together at the university because the NDUS computer security officer apparently didn’t have the technical skills needed, the audit said.
The UND team was led by another security officer but he struggled to identify confidential data on the server because he wasn’t familiar with it, the audit said. Though NDUS is supposed to have one information technology department, he was responsible for IT security only at UND.
Since the audit came out in July, there have been two other security breaches at NDUS institutions but none as big as the one at UND.
Asked about the damning tone of the audit, NDUS Chief Information Officer Lisa Feldner stifled a rueful chuckle before answering, “It wasn’t good.”
The former state CIO, she came to her NDUS position in June 2013, four months before the breach apparently began. She said Friday she had been trying to deal with deficiencies she found.
“We were in the process of restructuring internally and gotten some things taken care of ... just didn’t get there in time,” she said. “But I feel confident we’re really cleaning things up now.”
The IT security officer contacted by the stranger was Richard “Dick” Jacobson, who worked from an NDUS IT office in the MultiBand Tower in Fargo.
Until 2012, he was the security officer for North Dakota State University in Fargo. That was the year NDUS merged IT departments at its various institutions into one system-wide IT department, as the Legislature insisted.
The merger appears to have not gone well, at least where security was concerned.
As the designated IT security officer for NDUS, Jacobson should’ve been one of the top if not the top security official in the organization, the audit said. Yet he played almost no role in mounting a response to the breach. Jacobson told NDUS executives in protesting his firing that he offered his help but the audit suggests that he didn’t have the skills to do so. Instead the response was led by Brad Miller, the IT security officer for UND.
Jacobson, who couldn’t be reached for this story, said he believed he was responsible for security policy, not security operations, meaning the execution of those policies. Auditors, though, found security policies to be very vague and incomplete. They asked Jacobson what he thought were the current best practices in computer security and he “struggled” to answer, saying medical problems prevented him from getting much training.
More damning, auditors said Jacobson allowed confidential data to accumulate on the breached server. It was only supposed to act as a secure transfer point as data was transmitted to and from other servers, meaning the data should’ve been deleted after the transfers. He said he didn’t have the access to delete the data but NDUS executives said he should’ve worked with others to fix the problem.
The audit also found a huge problem with how the NDUS IT department was organized.
“This merger has yielded an organizational structure that is confusing and in some cases, redundant,” it said. “Positions are ill-defined. Duties overlap among positions; conversely, there are gaps in responsibility. This has led to hard feelings, confusion, and misplaced expectations among employees.”
All employees with some responsibility for IT security worked within the infrastructure and operations department, which appears to be responsible for keeping NDUS computer network up and running. Brad Miller was in that department, as were various systems administrators.
Jacobson was not. Instead he reported directly to Rosalinda “Rosi” Kloberdanz, an assistant CIO who oversees a division responsible for enhancing education with programs such as the ODIN library catalog. She admitted to auditors that she should’ve kept closer tabs on Jacobson, but they concluded that she probably didn’t have the security skills to do so.
Asked how it is that she ended up supervising Jacobson, she explained that he had a reputation for being difficult to supervise and, since they had worked together at NDSU, she thought she could “handle” him.
Jacobson has said he planned to retire in October of this year. He would’ve been two years from retirement at the time of the merger, and it seemed his supervision by Kloberdanz was never meant to be a longstanding arrangement.
But as a consequence of the arrangement, Jacobson and Miller, the only dedicated security officers in NDUS, didn’t work together very much.
At the time of the security breach, the server, one of more than 600 within NDUS, was already more than a year past the end of its lifecycle.
End-of-lifecycle is an IT term that means a system is so old that the vendor no longer provides updates. Auditors said the server at UND was running an operating system “without new bug fixes, security errata, product enhancements, and technical support” since February 2012. That is, newly discovered security vulnerabilities would no longer be fixed.
The responsibility for updating the server lay with William “Bill” Walker, the senior systems administrator. He could not be reached for this story but he did tell NDUS executives that he had been asking for approval to replace it since 2010, but no one got back to him.
In addition, because the breached server was used to transfer data among other servers, he said replacing it would have required coordination with about half a dozen departments to avoid interrupting their business. “The process was made more difficult due to no one department taking owner ship of the (redacted) server.”
Walker was fired after NDUS executives concluded that he had the authority to upgrade the server instead of just pushing it up the chain of command and forgetting about it.
The third employee fired was Marvin “Marv” Hanson, who was responsible for the hardware in the NDUS computer network. The audit said he was involved in discussions about the server’s vulnerability but did nothing.
Hanson declined comment when contacted by The Forum, citing the ongoing appeal to get his job back.
In a way, the firing of Walker and Hanson was a way to get rid of employees seen as embodying the culture of insecurity in the IT department.
Their inaction did not seem to have contributed directly to the security breach because the server itself was never actually under attack. The breach happened because an NDUS employee in a department responsible for software development had downloaded, presumably by accident, malicious software. This malware allowed a hacker to record the password needed to access the server.
Feldner, the CIO, said intrusion countermeasures installed throughout the NDUS computer network could have blocked traffic from computers known to be used by hackers. But she also said that wouldn’t have worked if the hackers been clever and masked their origins by using innocent computers, which appears to be what they used the NDUS server to do.
The audit and NDUS executives condemned both Walker and Hanson for failing to take responsibility for security and blaming others. That is, they harmed the culture of the IT department.
Walker was warned about the server vulnerability by a subordinate, who would’ve seen that the warning didn’t result in any action. “Walker’s reports (meaning subordinates) have received the message that nothing will be done when concerns are escalated to management,” the audit said.
Auditors say they discovered that Hanson was a vindictive boss who demoted an employee who called attention to a problem unrelated to the security breach. It appeared Hanson was angry that the employee told Hanson’s superior about the problem as well. Hanson denied that happened.
“An environment has been created where concerns, such as concerns for the security of the (redacted) server, are stifled due to a very reasonable fear of retaliation. Hanson’s conduct created significant risk for (the IT department),” the audit said.
Feldner said reforms to improve security are ongoing.
Among other things, she said her department is assembling a true IT security team that will work together under one department, the security policy is being rewritten, intrusion countermeasures are being installed, employees now are required to change passwords every 90 days, and all NDUS employees will receive training each month on security and be tested on their knowledge.
Asked if other IT employees have been disciplined, including the supervisors of the three that were fired, Feldner said only that “we are working with others on improvement.”