2016-09-30

The scandal of Hillary Clinton’s “home brew” email server, as it is played out over more than a year and a half, has served as a Rorschach test for her supporters and opponents. In her critics’ eyes it’s just another example of the Clinton family taking ethical shortcuts and playing by their own set of fast-and-loose rules; her supporters say it’s another example of the hysterical near-insanity that motivates her attackers in which, after millions of dollars in investigations, congressional hearings, FBI interviews and more, the scandal has amounted to little more than a whopping nothing-burger.

But until last week, the American public had never really had the chance to know how it all happened.

Then, last Friday, the FBI released the final batch of what amount to nearly 250 pages of interview notes and reports collected during the course of its investigation. Agents interviewed officials ranging from former Secretary of State Colin Powell to CIA officers to the IT staffer who first rented a minivan to drive the server from Washington to the Clintons’ home in New York. The files also include the FBI’s forensic investigative process and never-before-seen details of the staff decisions that led to the server, the mechanics of Clinton’s email system, and the confusing and balky State Department processes that led a technophobic Clinton to embrace her own BlackBerry. The FBI interviewed both those who supported her and those who questioned her decisions, as well as plenty of disinterested public servants who had no allegiance or beef with her either way. While the interviews were not technically conducted “under oath,” lying to federal agents is itself a crime, as is obstruction of justice—so they open a uniquely candid window into how the decisions around Hillary Clinton’s email server unfolded. They may be as close to the actual truth as we may ever get.

The interviews—taken together and reconstructed for this article into the first-ever comprehensive narrative of how her email server scandal unfolded—draw a picture of the controversy quite different from what either side has made it out to be. Together, the documents, technically known as Form 302s, depict less a sinister and carefully calculated effort to avoid transparency than a busy and uninterested executive who shows little comfort with even the basics of technology, working with a small, harried inner circle of aides inside a bureaucracy where the IT and classification systems haven’t caught up with how business is conducted in the digital age. Reading the FBI’s interviews, Clinton’s team hardly seems organized enough to mount any sort of sinister cover-up. There’s scant oversight of the way Clinton communicated, and little thought given to how her files might be preserved for posterity—MacBook laptops with outdated archives are FedExed across the country, cutting-edge iPads are discarded quickly and BlackBerry devices are rejected for being “too heavy” as staff scrambled to cater to Clinton’s whims.

Whereas President Barack Obama has long publicly cultivated his geek persona, embracing new technologies, trying new tools and generally trying to prove his tech savvy, Hillary Clinton comes across in the FBI interviews as a disengaged tech user who sees the communication tools as little more than a means to an end. She has, according to multiple aides, never even learned how to use a desktop computer. Clinton regularly pumped those around her for help with her devices—even those, as her long-time aide Philippe Reines joked to the FBI, whose job had “zero percent” of their responsibilities focused on IT. Reines, whose name is redacted in the FBI files but whose identity is easily discernible, “likened it to your parents asking for technical help with their phone or computer.”

Except that what Clinton turned to others for help with wasn’t an Amazon purchase or reading CNN.com: She needed help managing a massive trove of communications about the inner workings of the nation’s diplomacy and national security. Over the course of five years, those emails lived first in her Chappaqua, New York, basement, then later in a data center in New Jersey, then they were FedExed across the country and possibly copied onto a thumb drive before being printed out, sorted and handed back to the State Department in 12 bankers’ boxes. The boxes soon found themselves at the center of an FBI investigation and led ultimately to the biggest controversy to shadow Clinton during the 2016 presidential campaign. But it all started with the strange home server. This is its story.

Note: FBI Form 302 reports are interview summaries by trained agents. Quotes in the following account are taken from the reports and usually represent the agents’ summaries rather than the verbatim words of the interviewees.

***

1. The rules

It all began, according to the story told by Hillary Clinton and her aides, as “a matter of a convenience" after Clinton was named Barack Obama's first secretary of state. As Huma Abedin—Clinton’s most loyal aide—explained to the FBI during an interview in April 2016, the State Department told Clinton’s team during its January 2009 transition to Foggy Bottom that its tech experts didn’t allow personal email accounts to be installed on government-issued devices.

Clinton didn’t like the idea of carrying around two devices—one for official government work and one for personal or political correspondence, which is discouraged on government accounts. So she opted instead to carry only a single device, a personal BlackBerry, linked to a newly registered email account on a private domain, clintonemail.com, that was run off a recycled server from her unsuccessful presidential bid the year before.

The decision to create a dedicated email setup for the former First Lady had evidently begun earlier, at the conclusion of her unsuccessful 2008 presidential bid, even before it was clear that her victorious opponent, Barack Obama, would enlist her help in a key role in his Cabinet.

At the end of 2008, it fell to a longtime Clinton family aide named Justin Cooper to figure out how Hillary Clinton should receive email. Cooper, an American University alum who had joined Bill Clinton’s White House in the Office of Science and Technology Policy, finished the presidency working on the Oval Office operations staff. He and another aide, Doug Band, were among the close aides that President Clinton asked to move to New York to help set up his post-presidential life. Cooper helped edit the president’s autobiography and then stayed on, attending Fordham Law School on the side, while helping to manage the day-to-day administrative details of the Clinton family, a job that included many of their financial details like credit cards and, according to a 2006 New Yorker profile of the former president, managing his bags and gifts while he traveled the world.

In 2008, the Clintons had two primary email domains: wjcoffice.com, which was mostly a legacy domain that auto-forwarded emails to more modern accounts, and presidentclinton.com, which was used by the staff for their email accounts. The email domains were run off of a basic Apple server in the Clintons’ basement in Chappaqua that Cooper had purchased while Hillary was still running for president. Hillary Clinton, though, had never had an email account on either domain; Hillary@presidentclinton.com, after all, would have seemed at first presumptuous and, by late 2008, a mean joke of an email address to assign her.

During her tenure as U.S. senator from New York, she had used two different emails tied to her BlackBerry, hr15@mycingular.blackberry.net, which later became hr15@att.blackberry.net when AT&T took over Cingular. There’d been little thought to archiving her emails—AT&T didn’t save users’ emails—and they disappeared whenever she changed devices, which she did frequently. Cooper had purchased her a MacBook laptop in 2008, but he didn’t think she’d ever used it.

Cooper says he knew that the Apple server was already outdated by late 2008—Apple didn’t seem to integrate well with BlackBerry’s mail system—and as Hillary Clinton’s presidential campaign was being dismantled, Huma Abedin suggested Cooper talk with one of its IT staffers, Bryan Pagliano, about using some of its leftover computer equipment to update the Clinton family server. Pagliano agreed to build a server setup for the Clintons and began to rummage around, collecting equipment, network switches and the various components of a modern email system from the leftovers at Clinton’s Arlington campaign headquarters. At the time, as Pagliano later told the FBI, he didn’t realize Hillary Clinton would even have an account on the server—he thought it would just be for the use of the existing team of Bill Clinton’s aides.

At the same time, Abedin and Cooper were discussing what to do with the former first lady’s email. Cooper and Abedin—who had long served much the same purpose for Hillary as Cooper had served for Bill—discussed between them the advantages of a “covert email domain versus a domain including the Clinton name.” Abedin ultimately “blessed off” on using a new domain, @clintonemail.com, to handle the senator’s email. On January 13, 2009, almost two months after Clinton accepted Obama’s nomination, Cooper used an Internet registrar named Network Solutions to register that domain. The next week, Hillary Clinton resigned her Senate seat and was sworn into office as the nation’s 67th secretary of state.

The Clinton team’s arrival in Foggy Bottom was harried, like all administration transitions, but came with the extra complication that she intended to install two deputy secretaries of state. That decision required physically rejiggering the seventh floor of the State Department headquarters that houses the leadership’s offices, a historic, wood-paneled area known internally as “Mahogany Row.”

In Clinton’s early days in office, there were various conversations among Clinton, her team and career State Department officials about her preferences and how to set up communications to aid her. There was a crucial complication: BlackBerrys—the tools that Clinton and her aides had come to rely upon in the Senate and on the campaign trail—weren’t allowed inside Mahogany Row. This section of the State Department was technically considered a “Sensitive Compartmented Information Facility,” government-speak for an eavesdropping-proofed room. The assistant secretary of state for diplomatic security, Eric Boswell, later stated he never received any complaints about Clinton using her personal BlackBerry inside the secure area, but that among the State Department team there was some “general concern” that Clinton’s team might use the BlackBerrys that they’d relied on so heavily during the campaign. His team made clear that the devices were prohibited.

Yet something was going to have to change: Hillary Clinton, after all, didn’t know how to use a desktop computer. A BlackBerry was her lifeline. As Cheryl Mills told FBI agents later, “Clinton was not computer savvy and thus was not accustomed to using a computer, so efforts were made to try to figure out a system that would allow Clinton to operate as she did before DoS.”

State Department officials presented Clinton’s team with a memo on January 24, 2009, outlining various options, suggesting if Clinton wanted to check her email, she would have to either physically walk out of her office to use her BlackBerry, or the State Department could set up a dedicated computer for her. Jake Sullivan, a foreign policy expert from the campaign who had grown to become one of Clinton’s most trusted advisors, immediately saw problems with the proposal and echoed Mills’ concern: “Clinton did not know how to use a computer, so the suggestion of a stand-alone computer for Clinton was not an appropriate solution.”

That same day, Lewis Lukens, the department’s deputy assistant secretary for the Executive Secretariat—the unit that oversaw the logistics for State’s leadership—sent an email asking about the possibility of setting up a “living room” outside the office’s secure area where the new secretary could check her email. There was a model for this; something similar had been done for Colin Powell.

Instead, after much back and forth and various proposals, the solution turned out to be a simple one. During her tenure as secretary of state, Hillary Clinton—who was known to her security detail by the code name Evergreen—would deposit her BlackBerry into a desk drawer at the Diplomatic Security station outside her office when she arrived on the seventh floor. The practice of leaving the BlackBerry at the guard station, known as Post-1, was technically a security violation—the desk was considered inside the Mahogany Row secure area—but it seemed to those involved an appropriate compromise. To use it, she’d leave her office and wander, often visiting State’s eighth-floor balcony.

In the days after she was sworn in, Hillary Clinton also contacted her predecessor, Colin Powell, to ask how he had managed his information flow as secretary of state from 2001 to 2005. In his early weeks, Powell recalled, he’d “received several security briefings that restricted his ability to communicate.” He’d questioned the NSA and CIA on “why PDAs were anymore of a risk than the television remote controls.” He never got a convincing answer. And so, he advised Hillary Clinton “to resist restrictions that would inhibit her ability to communicate.” But he told her to choose wisely and not to create an unnecessary paper trail. He said if it became “public” that Clinton had a BlackBerry and she used it to “do business,” her emails could become “official record[s] and subject to the law.” As Powell said: “Be very careful. I got around it all by not saying much and not using systems that captured the data.”

It was all advice Clinton was probably predisposed to take—in part because she eschewed technology herself. According to Cooper, “Clinton [as senator] usually carried a flip phone along with her BlackBerry because it was more comfortable for communications and Clinton was able to use her BlackBerry while talking on the flip phone.” But at State, she gave up the flip phone, conducting most discussions in person, reading most documents in printed hard copy, or using one of the three phones in her Mahogany Row office: A black phone, capable of making secure and unsecure calls, a yellow one used only for secure conversations, and a dedicated white phone for direct calls to certain government officials. She never had a computer or fax machine in her office.

Meanwhile, State Department IT and security teams were busy installing secure rooms in her two homes for reading and receiving material and conducting telephone conversations. Each house had its own SCIF. At Whitehaven—her brick Georgian-style house in northwest Washington—a State Department worker removed one of the regular doors on a third-floor room of the house, replaced it with a metal door secured by a key code lock, and outfitted the room inside with secure communications. A similar room was created at Chappaqua; while she rarely used the secure room at Whitehaven —preferring to just go into the office if she had work to do—she relied heavily on the one in Chappaqua when she was in New York, in part because cellphone coverage in the area was so poor that she needed the use of the SCIF’s phone. (The FBI interview reports differ on who precisely had access to Clinton’s home SCIFs—whether it was just Clinton herself or also top aides like Abedin.)

Each secure room was also equipped with a secure fax, but while Clinton was supposed to pick up the faxes herself at home, she often struggled to use the technology and had to rely on staff for help operate the machines. As one aide described it, Clinton “wasn’t very tech savvy and would get frustrated with the process.”

***

2. The setup

By March of 2009, Bryan Pagliano—who ultimately joined the State Department himself, working on IT programs related to mobile computing, teleworking and Bluetooth security vulnerabilities—had assembled all the components for the Clinton email server. He rented a minivan in Washington, loaded it full, and drove north on I-95, meeting Cooper at the Clinton residence in Chappaqua. The two men then lugged load after load of computers into the basement.

When he got into the basement for the first time, Pagliano laid eyes on the system that had been running the Clinton email until then: a basic Apple computer, connected to an HP printer used by Bill Clinton’s staff to print documents remotely for him from his primary post-presidential office in Harlem, which sat about 35 miles south.

Pagliano didn’t like the idea of housing the email server in a residential basement, because there was only a single unreliable Internet connection, but Cooper wanted physical access to the server. Pagliano told the FBI that he always thought it should be in a data center for “security and reliability.” But Cooper saw a series of advantages to housing it at Chappaqua rather than with the rest of the IT infrastructure at the Clinton Foundation’s office in Harlem: It helped segregate the family’s personal and political work from the foundation, and it minimized the number of people who could physically access the server. He also worried that an outside vendor might not report hacking attempts on the server, so preferred to rely on an in-house team of just him and Pagliano.

The two men loaded in a standard 12-unit server rack and the other hardware: a Kiwi Syslog Server, Cisco Private Internet eXchange firewall, a 3-terabyte hard drive and a power supply. Pagliano set up a Windows Small Business Server, as well as a BlackBerry Enterprise Server to run the Clinton devices. Cooper, meanwhile, registered a so-called SSL certificate to secure the server at Pagliano’s direction. Later, Pagliano assessed the setup as a “standard” and “B+” email server.

Pagliano began the migration of the email accounts from the old server to the new one at the house, then finished it up later in his hotel room. When he finished, Pagliano believed he’d “popped out” all of the Clinton staff email from the Apple server; he recalled that he didn’t transfer an email account for Hillary Clinton. The leftover Apple computer, meanwhile, was repurposed as a desktop for the household staff in Chappaqua. On the new system, Pagliano and Cooper both had administrative privileges. A backup system ran once a week.

On March 18, 2009, Hillary Clinton stopped using her longstanding email, hr15@att.blackberry.net, and switched to a new account: hrod17@clintonemail.com. When she switched accounts, all of her old email disappeared—including all of the email from her first seven weeks as secretary of state. To date, neither Clinton nor the FBI have located any of her email from that period.

That spring, Pagliano, while he was working on the email server, noticed that a new account had been created, labeled simply “H.” He asked Cooper who it was for; Cooper told him that it was Hillary Clinton’s new email.

***

3. The technophobe

Hillary Clinton, for her part, proved remarkably uninterested and unfamiliar with new technology. By time she moved into Foggy Bottom, much of the world had jumped aboard the iPhone bandwagon, but Clinton would cling stubbornly to her BlackBerry, even as the once-ubiquitous Washington icon slid toward tech oblivion.

According to Abedin, “It was not uncommon for Clinton to use a new BlackBerry for a few days and then immediately switch it out for an older version with which she was more familiar.” She deemed one upgraded BlackBerry “too heavy.” That personal preference proved challenging because she churned through devices at a steady clip—all told, the FBI figured that she’d used around a dozen BlackBerrys during her tenure at the State Department. While she never reported losing a BlackBerry, Clinton replaced one after she spilled coffee on it, another because its trackball started to fail slowly over time, and another when its screen cracked.

Aides would help set up the new devices and sync them with the email server; Cooper recalled disposing of old devices by breaking them in half or hitting them with a hammer. Clinton didn’t know her own email login information, so Hanley would generally enter the information as necessary, change the password, and tell Abedin, Cooper and Pagliano about a new password.

Clinton at one time requested a secure BlackBerry “after hearing President Obama had one,” but ultimately the State Department decided that it wasn’t feasible to give her one. Instead, her device of preference ended up being the BlackBerry Curve 8310, because the trackball feature was easier than the track pad on newer models like the BlackBerry 8700G, which she tried and rejected. Thus, as the BlackBerrys were upgraded, it became harder to find the style Clinton liked. Hanley said she generally purchased the devices from the AT&T store in Dupont Circle, though one came from Pentagon City Mall, and, later, she started to purchase them proactively to help ensure Clinton’s preference would be available. (She even recalled at one point turning to eBay or Amazon to purchase one.) After buying a device, she’d file for a reimbursement from the Clintons’ personal funds, managed by Justin Cooper in New York.

Why didn’t she have a State Department email address? That remains, to a certain extent, a mystery in the FBI files. At the beginning of the administration, the Executive Secretariat’s Office of Information Resource Management (S/ES-IRM)—the unit in the State Department that oversees information technology for the department’s top leadership—did offer the new incoming secretary a State.gov email address. But someone—exactly who is lost to history—on Clinton’s team declined. (Over the course of her tenure, the unit did create two email addresses for her, but neither was used personally. One address, SMSGS@State.gov, was used to send all-employee emails, while another, SSHRC@state.gov, was used to run the Outlook calendar and schedule meetings.)

Her preference for a personal email account was not technically against the rules. At State, FBI agents later found, there was “no restriction on use of personal email accounts for official business,” but employees were cautioned about security and records retention concerns. The State Department told employees that they should forward such emails to their official accounts for recordkeeping purposes. “There were no rules in place that specifically denied Secretary Clinton the use of her private network,” but, according to the State Department IG Steve Linick, private email was “highly discouraged.”

Officially “discouraged,” sure, but according to many that the FBI interviewed, the State Department’s culture uniquely embraced—and its poor information systems actively seemed to encourage—employees turning to private emails to conduct business. As FBI Director James Comey said in July when he reported the bureau’s findings: “We also developed evidence that the security culture of the State Department in general, and with respect to use of unclassified e-mail systems in particular, was generally lacking in the kind of care for classified information found elsewhere in the government.”

4. The state of State

Colin Powell had originally been shocked when he arrived at Foggy Bottom in 2001—he immediately realized that one of the largest problems he faced was the State Department’s outdated computer systems. At the time, the CIA and the State Department swapped responsibility for embassy communications every 12 months, an inefficient system that had caused the department to lose ground technologically. After Powell reviewed the situation, he worked out a deal with CIA Director George Tenet and “fired” his own State Department IT team, handing sole responsibility over to the CIA. More broadly, though, few State Department employees had their own computers—and Powell himself found himself facing a laptop in his office with a 56k modem, sluggish even then.

Powell invested in 44,000 new computers, giving every employee a computer on the desk, and monitored the adoption of the new systems as he traveled by conducting unofficial audits, sitting down at embassies overseas to check his own email and attempting to log into his account. As he told FBI agents, “This action allowed Powell to gauge if the embassy staff was maintaining and using their computers.” He also regularly checked the department’s internal “Country Notes” on the intranet to see if missions overseas were keeping their details up to date.

While during Powell’s tenure, the State Department rolled out a new unclassified email system called OpenNet, Powell personally preferred to use his own AOL address for email, treating it, he told the FBI, “like a home telephone line,” meaning that he felt he could use it for business or personal reasons. He regularly corresponded with foreign leaders by email, switching to secure calls if the conversations turned sensitive. (When he finished his term as secretary of state, Powell told the FBI, “he took no emails with him when he left State and knew of no official record requirements at the time.”)

All of Powell’s investment had taken the State Department only so far. By the time Clinton arrived, the State Department’s technology infrastructure was still outdated and balky. The “fob” system that was supposed to allow access to email outside the building—whereby employees would enter a special key or token to confirm their identity—was slow and prone to shutting down inconveniently. For employees who did use their official accounts, workarounds were common—particularly because many State Department officials and senior leadership, many of whom worked from the field or traveled regularly on missions overseas, didn’t have easy, regular access to the systems designed to transmit classified information securely.

One State employee told the FBI he regularly used nonsecure email and personal email simply because there was no other way to quickly transmit information. The FBI found “many DoS employees used personal email accounts because they were more easily accessible.” Clinton aide Monica Hanley told the FBI that “her state.gov email account was not as easily accessible as her Gmail account and on some occasions she used Gmail when she could not access her State.gov account.” There were particularly problems connecting to State.gov accounts on board the Air Force planes that Clinton used to travel, so staff often would use Gmail or other personal accounts while traveling.

As the FBI report concluded, “DoS does not have a restriction on the use of personal email accounts for official business. Personal email accounts are often used by individuals in the field who were not issued an official DoS mobile device, or who do not have the time or means to remotely log into the DoS system. Employees are not required to notify DoS that they are using a personal account for official business and there is no mechanism to track who is using a personal email.”

Even though the State Department’s unclassified network had been penetrated by at least one foreign adversary—exactly who isn’t revealed in the FBI notes—employees had actually come to rely on email even more as time passed, which meant playing fast and loose with information that other parts of the government treated much more carefully. As the officer explained to the FBI, “DOS has shown an increased tendency to communicate via email. He believed that they did this for simplicity, to avoid unauthorized disclosures such as [WikiLeaks] and to prevent other USG partners from seeing their ‘back channel’ discussions. [He] continued to say that the personnel at DOS were experienced and knew that this information was classified. However, they did it anyways and their actions hurt the CIA and other agencies whose [information was] conveyed in the emails.”

The department’s IT problems— both the culture of personal email and poor information security that it encouraged—were well known among those who worked with the State Department. One CIA official who reviewed a questionable email in the Clinton investigation told the FBI that the email in question technically “should be classified, but that he was not surprised that DOS had sent it on an unclassified channel.”

A potentially unlikely CIA executive echoed those same impressions: that the government’s classification system wasn’t necessarily a bright line; sometimes information was technically classified that a reasonable person could argue wasn’t necessary. Mike Morell—the former deputy director of the CIA who started working with former Clinton aide Philippe Reines’ firm Beacon Global Strategies after he retired in 2015—told the FBI after reviewing one email that “he understood why the email would be considered classified, but he did not believe that the email would jeopardize any sources, methods, or otherwise compromise national security.”

While “classified information” seems like it should be straightforward and binary—it either is or it isn’t—in practice government classification is a tricky and complicated issue. For one thing, different departments can treat the same information differently, as Under Secretary for Management Patrick F. Kennedy—a career Foreign Service officer who had started in the top position two years before Hillary Clinton came to the department—explained to the FBI. Whereas the intelligence community often “steals” information, leading it to be classified, the State Department may end up gathering that same information from nonsensitive sources and so never consider it classified; conversations with foreign diplomats may be classified or not—or later upgraded to classified if it’s determined that “the disclosure of such information might damage national security or diplomatic relationships.” (This was particularly true as governments and leaders shifted around the world.) Plus, the lines around documents and information could shift—many internal or even interagency drafts would be considered unclassified while they were being written, but would then be routinely classified when they were transmitted formally to the National Security Council.

Then, of course, there was the problem of the State Department’s unique mission to engage other countries. As one employee related to the FBI: “Generally the only way to discuss topics with Foreign Partners is via unclassified channels, or in very sensitive cases, by making arrangements to meet in person at Embassies or at DoS. Since there isn’t a classified system that allows DoS to communicate with its foreign counterparts, conversations that are held with foreign partners in unclassified channels are later ‘up-classified’ to Secret to protect the information.”

One State Department official, no fan of Hillary Clinton, said it was “business as usual” for her and others to have to communicate sensitive matters via the unclassified email system. “If you are a professional, you know how to do it and how much to do,” he said. The Department had only three real choices for passing along information: an official cable, a classified email and an unclassified email. “The process for sending a cable was not quick, nor were executives as likely to get a classified email in a timely manor [sic],” the official said, adding that he “tried to use his best judgment.” The classified emails were generally used primarily for passing “lateral information” to other ambassadors, the National Security Council or other parts of the intelligence community. The unclassified email was really the only functional choice “for day to day interaction,” and while the email system did let users mark a message using a lower-level warning—“Sensitive But Unclassified”—it didn’t grant any special protections to such messages.

Many employees, the FBI found, carefully worded emails to “talk around” classified subjects in unclassified email.

Clinton’s close aide Jake Sullivan saw the department’s leaders drowning in information. His own portfolio included managing dozens of employees and simultaneously keeping watch over the globe’s hot spots. He told the FBI his experience was that State Department employees “did the best they could to make a sound judgment when handling classified information” and they “worked hard while under pressure.” It wasn’t a perfect system—one email with potentially classified information about a foreign military’s activities arrived on his Gmail because he was in Idaho for a bachelor party and didn’t have access to his regular, classified email system—but he said he “could not recall an instance where anyone expressed a concern with the type of information coming over the unclassified email system.”

***

5. Emailing the president

While Hillary Clinton wasn’t much of an emailer, she did have one coveted Washington email address—President Barack Obama’s. The president’s email system allowed only select addresses to reach him, so when her email address changed, her staff had to notify the White House to add her new email to his approved list of contacts. Clinton said she never received any guidance on how or when to email the president.

The president, though, was more the exception than the rule in Clinton’s world. She had few correspondents. Just over a dozen individuals—mostly senior advisers and the department’s executive administrative staff—regularly emailed Clinton directly. It was a rare privilege reserved for senior advisers who needed regular contact. Excluding personal correspondence with family and close friends, Abedin, Mills and Sullivan together accounted for 68 percent of Hillary Clinton’s total email traffic as secretary of state. (Clinton also used her device to text staff and send BlackBerry messages.) While “at least a hundred, if not several hundred” State employees had her clintonemail.com address—emails from Hillary often arrived with just an “H” in the “from” field—and many of those employees, like Kennedy, were aware she used a personal email account, most didn’t understand she had a private server. Nor was Kennedy aware that the personal email account was her sole one.

In part, her email flew below the State Department radar both because of her tight circle of correspondents but also because, simply, as one aide said: “Clinton was not an email person.” And those who wanted to reach her knew it was better to email her top aides directly, anyway. As the FBI reported, “Multiple State employees advised they considered emailing Abedin, Mills and Sullivan the equivalent of emailing Clinton.” As for what arrived via unclassified email, Sullivan and Abedin both said, repeatedly, that they didn’t question the judgment of people sending that information and relied upon senders to properly mark sensitive information. Sullivan said he would regularly review “situation reports from around the world in an unclassified email.”

Rather than do business electronically, Clinton preferred to conduct meetings face-to-face and, as one close aide—a self-described “Clintonista”—said, she was a “paper person,” preferring to read documents in hard copy. While the Presidential Daily Brief—the government’s most valuable document—was often briefed in-person to her at the office, she read voluminously at the office and at home. Since Clinton did not have a classified email account herself, all classified material went to her in hard copy—a process overseen by her executive assistants, Joe McManus and, later, Alice Wells. Clinton, Sullivan recalled, had an “enormous” amount of information, including classified reports, briefed to her in-person or through the paper flow.

For reading at home, the State Department would also regularly deliver diplomatic pouches full of briefings and reports—the Diplomatic Security agent on duty at each residence would then deliver the pouch to a designated bench to await Clinton’s pickup. (At Whitehaven, the pouches went on a bench outside her bedroom; in Chappaqua, the bench was located near the home’s main entrance.)

Abedin, for her part, found that it was difficult to print from the State Department email system, so she’d often forward emails to her Yahoo email, Clintonmail.com accounts, or even another account that she’d previously used to support the campaign activities of her husband, Anthony Weiner. And there was a lot to print: Clinton didn’t like reading long emails—the BlackBerry font was too small—so she’d often forward such staff to staff to print. Deluged by tasks and information, Abedin reported that she’d often print and pass along documents to Clinton “without reading them.” The FBI also uncovered hundreds of emails sent to one of the Clinton family’s staff on the presidentclinton.com domain requesting that he print emails for her to read. Printing problems dogged Clinton’s team as they traveled the world, too. While special Mobile Communications Teams would outfit hotel rooms overseas with computers hooked up to the State Department network for Abedin or Hanley to use, the FBI found, “it was not uncommon for [aide Monica] Hanley to use her personal Gmail account to print from the mobile DoS unclassified terminal because even though she was using a DoS computer, the DoS connection was unreliable.”

In June 2010, Clinton received a new device: Just weeks after the first iPad was released, Philippe Reines purchased one for Clinton to use. The iPad, her staff hoped, would give her a way to read news articles on her own. She didn’t like reading news on her BlackBerry, but their hope was short-lived. She initially responded enthusiastically to the idea, responding to Reines’ email that her iPad had arrived by writing, “That is exciting news—do you think you can teach me to use it on the flight to Kyev next week?” But when the traveling party embarked on the Air Force plane for the meeting with President Viktor Yanukovych, Clinton instead fell asleep with the unopened iPad package in her lap. Reines told the FBI that this struck him as funny because, “in contrast, he would not be able to sleep if he had just received a new iPad.” Then he added a dour note: “This episode was a foreshadowing for how little she would use the iPad.” Over time, she warmed slightly to the device, using it in the evenings and while traveling for reading news, but once she got comfortable with it, she resisted attempts to upgrade it.

The next year, when her staff tried to upgrade her to an iPad 2, they had even less success. Abedin emailed Cooper on August 18, 2011, saying simply, “She doesn’t like ipad 2.” Clinton instead gifted the brand new device to Monica Hanley. There was no mistake that Hanley got a hand-me-down: When she logged on for the first time, the device still said “H’s iPad,” so Hanley wiped it clean before using it. As Hanley told the FBI, “It was not uncommon for Clinton to gift Abedin and Hanley with some of her personal items she no longer wanted.”

***

6. The State Department starts to worry

Pagliano’s role in helping run Clinton’s email was well known within the department, at least among its seventh-floor IT staff; he regularly interacted with them to keep Clinton’s email system running smoothly. His knowledge impressed those he interacted with: As one State IT employee said, Pagliano was “very sharp and technologically savvy individual who likely took action based on the security information and briefings provided.”

But not everyone at the State Department was pleased with the setup. At some point in the summer of 2009, two State IT specialists summoned Pagliano and asked whether he was aware of the clintonemail.com domain. He said yes. When Pagliano relayed this to one of Clinton’s aides, that person, Pagliano told the FBI, had a “ ‘visceral’ reaction and didn’t want to know any more.” Later in 2009 or early 2010, one of the same State Department employees asked Pagliano again about server, saying it might be a federal records-retention issue and asked him to relay that concern to Clinton’s “inner circle.” Pagliano approached Cheryl Mills in her office and passed along the information. Mills dismissed the worries, saying other former secretaries of state had done the same thing.

Just how far such concerns about Clinton’s email practices went inside the State Department is still a matter of some debate. An inspector general report on Hillary Clinton’s email reported that two IT staff approached the director of S/ES-IRM, John Bentel, and raised concerns about her email use, only to have him tell them it was approved and they should not discuss the server further. (However, in an interview with the FBI, he denies that any such conversation took place. As he told agents, he doesn’t recall saying that, adding that the account “was inconsistent with his open and welcoming management style.”)

Yet rather than appearing to be actively covering up Clinton’s paper trail, Clinton’s staffers—harried as they were and pulled in multiple directions by seemingly daily world crises—seemed simply uninterested in the details of record-keeping, either for Freedom of Information Act purposes or for the Federal Records Act, which governs official papers. Nor did they appear particularly curious even about Clinton’s own email setup. Aides like Mills, Abedin and Sullivan all said that while they knew her email address, they didn’t understand the technology behind it and were “unaware of existence of private server until after Clinton’s tenure.” Mills said she “was not even sure she knew what a server was at the time” she was Clinton’s chief of staff. It’s not even clear Clinton herself understood her email was running off a homemade computer in her Chappaqua basement: Clinton told the FBI she “had no knowledge of the hardware, software, or security protocols used to construct and operate the servers.”

While federal law has strict guidelines about the preservation of public records—both for historical purposes and for FOIA purposes—Mills, who said she received 400 to 700 emails a day, told FBI investigators that she believed that maintaining records was the responsibility of the “front office,” but she couldn’t say who was responsible for FOIA. Abedin told investigators she “always assumed all of Clinton’s communications, regardless of account, would be subject to FOIA if they contained work-related material,” but the process for that seemed unclear. As Mills saw it, since Clinton was emailing other State Department staff at their official email addresses, her missives were already being tracked. (The State Department’s inspector general later said, this “was not an appropriate method of preserving record e-mails.”)

Jake Sullivan, who told investigators that he’d never been offered nor requested a @clintonemail.com address of his own, reported that his State.gov inbox often ran afoul of size restrictions and he regularly had to “archive” “big chunks,” but that he “could not recall any methodology or science he applied when archiving email.” He said he knew about records-retention rules, and so he didn’t delete anything from his State.gov email and he handed over his official papers when he left the State Department, but he also told the FBI he sometimes used Gmail on weekends or while traveling.

But even the “front office” at the State Department, to use Mills’ term, didn’t fully comprehend the system for saving electronic records. As the Clinton team arrived at the State Department in 2009, the department was in the process of rolling out a new preservation system that allowed employees to electronically tag emails to preserve a record copy. The system was supposed to be rolled out departmentwide, but S/ES-IRM didn’t actually deploy it to the secure zone of Mahogany Row amid concerns that it would “allow overly broad access to sensitive materials.” The Office of the Secretary instead was left with the traditional “print-and-file” system. So the State Department’s new preservation system never even reached into its top echelons on the seventh floor, a fact that Lewis Lukens—the official who supposedly ran the executive leadership team—told FBI he didn’t even realize.

***

7. Hackers start sniffing

On January 9, 2011, Justin Cooper—who shared administrative privileges on the Clinton email server with Pagliano—noticed what he believed was a “brute force attack” on the server, in which a hacker was overloading the server with attempts to guess a username and password. Cooper, unable in that moment to reach Pagliano, “panicked,” according to Pagliano, and shut down the server. Cooper told Abedin that someone was trying to “hack” the server. Another email later in the day reported he had to reboot the server again as he tried to restore the system and defend against the unsophisticated attack. His concerns lingered, even the next day, he emailed saying, “Don’t email hrc anything sensitive. I can explain more in person.” The attack, though, was ultimately unsuccessful—nor was it particularly difficult to defend against. Pagliano later trained Cooper in the basics of how to respond and block specific Internet addresses that were attacking the site, and he warily told Cooper that he couldn’t be on call to tend the server constantly.

Such brute force attacks were regular occurrences during the years the server was in operation—but they proved ultimately unsuccessful as far as anyone, including the FBI, could tell. Pagliano could see the attacks unfold as the usernames that intruders attempted to use weren’t anything close to the names of the small handful of real users on the server. He told the FBI that he thought about implementing what was known as “two-factor authentication,” which would have required users to enter a special changing code from a digital fob when they logged in, and that he even went so far as install such measures on his own workstation as a test—but ultimately he decided it wasn’t worth the effort. He also never installed what was known as Transport Layer Security, which would have encrypted messages as they passed between the Clinton server and the State Department’s servers, telling the FBI that he figured there wasn’t a need for encryption on a “personal” server.

Clinton’s email faced other routine security threats. While the CloudJacket monitoring software caught “multiple instances of potential malicious actors attempting to exploit vulnerabilities,” the “FBI determined none of the activity, however, was successful.” (Similarly, the two iPads used by Clinton that the FBI tested showed no sign of cyber intrusion.) Clinton herself, though, faced multiple phishing or “spear-phishing” attempts, where someone would send a fake email or link hoping to infect her computer with malware or gain access to her email account. Clinton replied to one suspicious email from a regular email correspondent dubiously, “Is this really from you? I was worried about opening it!” Another contained a link to pornographic material. The attempts, though, seemed random and undirected; as Clinton recalled later, she “occasionally received odd looking email, but never noticed an increase in these types of emails that would be a cause for concern.”

Yet across the department, email security concerns lingered through that spring of 2011. In February, several State Department employees had their personal Gmail and Yahoo accounts hacked after they responded to a “phishing” email asking them to change their passwords. The hackers, unbeknownst to the employees, then changed the email settings to auto-forward copies of incoming mail to other accounts controlled by the intruders.

Those incidents prompted the department’s security chief, Eric Boswell, to send Clinton a memo on March 11 specifically saying that State emails were being targeted by a hacking threat. He encouraged employees to limit the use of personal email. (Normally, Boswell said later, Clinton was “very responsive to security issues.”) The next week, there was another attack, ultimately unsuccessful, on the Clinton server, but some in her inner circle didn’t hear about it: Abedin’s email address was misspelled on the note warning of the fresh attack and she never saw the warning.

In June 2011, Pagliano traveled to Chappaqua to upgrade the server’s technology. He replaced the Seagate external drive, which was beginning to see its disc drive fail with age, with a Cisco device. He added additional memory to the Dell PowerEdge 1950 server, added a Gigabit switch, upgraded the firewall, and added two new security devices: a Cisco botnet filter and a Cisco intrusion prevention service. He also replaced the batteries on the backup power source, updated the BlackBerry server software and installed any necessary patches. Pagliano, who had negotiated to be paid an hourly rate with Cooper for his work on the Clinton servers—Cooper had initially instead offered to pay a regular retainer—was ultimately paid $8,350.83, including expenses for travel and the equipment, for the trip.

***

8. A wall is breached

In January 2013, the Clinton server saw what the FBI determined was its only known “successful compromise.” According to the FBI’s forensic investigation later, on January 5, the account of a member of Bill Clinton’s staff—who all shared the server with Hillary’s email—was broken into by someone using the anonymizing software Tor. Over the course of the day, three known Tor IP addresses accessed the site and the intruder browsed through the staffer’s email folders and attachments. The FBI said it was “unable to identify the actor(s) responsible,” but that the damage, as far as it could tell, was limited to that lone staffer’s email that one day. And, by the end of the month, Hillary Clinton’s tenure as secretary of state had ended; after serving out Barack Obama’s first term, she handed in her resignation on February 1, 2013, and returned to private life.

Six weeks after she left office, though, another more mundane hack threatened to expose Hillary Clinton’s email address to the world. On March 14, a hacker known as “Guccifer,” a 40-something Romanian taxi driver named Marcel Lazăr Lehel, found in another account he’d cracked an email address belonging to Sidney Blumenthal, a longtime Clinton aide and confidant. (The other cracked account was likely the AOL address belonging to Hillary’s predecessor as secretary of state, Colin Powell, whose account Guccifer is known to have hacked around that time.) Guccifer was not a technically sophisticated hacker, relying instead on patience and research to crack users’ passwords and security questions; he later told authorities that he spent six months to get inside the email of a Romanian politician, Corina Cretu. Blumenthal’s email was much easier: He told the FBI he spent only 20 minutes on research and guessing before he was able to successfully reset Blumenthal’s password by answering a security challenge question.

The account included about 30,000 emails, and Guccifer says he spent seven hours carefully sorting and reviewing the emails, as well as downloading more than two dozen attachments. He took screenshots of various emails, including one email about Benghazi, and noticed that Blumenthal regularly emailed Hillary Clinton. He made a quick attempt to figure out where her server was located but, stymied, gave up. Finally, the next morning, Blumenthal realized he’d been locked out of his account and was able to reset the password again, halting Guccifer’s access.

Guccifer’s modus operandi was to send his findings to the media—earlier that year, he’d cracked the email of members of the Bush family, leaking to the world photos of paintings that George W. Bush was doing in his presidential retirement—and he sent Blumenthal’s emails to dozens of media outlets around the world. The release of Blumenthal’s emails gave the world its first knowledge of Clinton’s clintonemail.com domain name, and by the next day, March 15, Russian and Ukrainian Internet addresses were scanning the Clinton server, unsuccessfully trying to gain access.

The exposure of the email account encouraged Clinton’s aides to change the secretary of state’s address. Abedin selected hrod17@clintonemail.com, but the staff feared that they’d lose her existing emails when they changed addresses, so Monica Hanley retrieved an old MacBook laptop from Bill Clinton’s Harlem office and spent several days at her apartment transferring years of Hillary’s emails from the server files into Apple’s Mail program on the laptop. (The hope at the time was that the emails would also be useful in writing her future memoir.)

And, by that point, as she considered her post-State Department life and future political plans, the Clinton team was already reconsidering its email setup “due to user limitations and reliability concerns.” Bryan Pagliano had also moved on professionally, meaning that they couldn’t continue to lean on a close associate for day-to-day help. Staff for both Hillary and Bill Clinton initiated a search to find a vendor to manage the server. Cheryl Mills helped pull together a Request for Proposal outlining the Clintons’ IT needs and three vendors submitted pitches. Pagliano, who was a few months into a new job with the tech research firm Gartner, recommended one of the vendors, a Denver-based company called Platte River Networks.

So the Clinton email server moved to New Jersey. Over the course of the last 10 days of June, PRN took possession of the server’s equipment and its management. A PRN employee traveled to Chappaqua, removed the existing server hardware and transported it to a secure data center in Secaucus run by a company called Equinix. Beginning on June 30, the Clinton emails were moved from the old server to the new one, migrating the 20 to 30 email accounts associated with presidentclinton.com, wjoffice.com and clintonemail.com one at a time, by right-clicking and dragging each account. The new DATTO backup system took multiple snapshots a day of the server, retaining them for 60 days. Later, at the end of the year, confident in its new setup, PRN of its own volition powered down and disconnected the old Dell servers that the Clintons had used, leaving them to sit unused in the Secaucus data center rack until the FBI took custody of them in the midst of its investigation.

The Clinton servers were overseen by two PRN employees, one working remotely from home who handled the day-to-day systems administration; another, working at the company’s headquarters in Colorado, who handled hardware installation and “hands-on” maintenance. The team continually saw examples of the Clinton team’s relative unsophistication when it came to technology; Mills even occasionally asked PRN for help with her personal account. Such tendencies led them to limit some of the security on the account: The Clinton team had “originally requested that e-mail on the PRN server be encrypted such that no one but the users could read the content,” but PRN did not do so “to allow systems administrators to troubleshoot problems occurring within user accounts.”

At the same time, though, PRN wasn’t always entirely on top of all the technical details itself. It realized in August 2015 that due to “a technical oversight” its Datto backup system, which was supposed to be storing only local copies of the server backup, had also been backing up to Datto’s secure cloud storage—a practice that was promptly discontinued.

Indeed, what comes through time and again in the interview notes of the FBI’s email investigation is—far from a sinister careful coverup to avoid transparency and hide Clinton’s communications—just how disorganized and uncoordinated the technical details of her system actually were. In February 2014, Monica Hanley decided to upload five years’ worth of old Clinton emails to the new PRN server that had been saved to a laptop after Guccifer exposed her address; PRN tried to help Hanley remotely, but when that process failed, Hanley simply Fedexed the laptop to one of the PRN employees at home so he could convert the files and upload them to the server under a new email address, hrcarchive@clintonemail.com. The PRN employee completed the task after Googling how to successfully convert the Apple Mail files to the required .pst format using Gmail. No one was able to determine what happened to the MacBook once PRN was finished with it. The same emails may or may not have been saved to an external thumb drive as well, but no one could find it or remembered what happened to it.

Meanwhile the State Department was beginning to ask questions, trying to fill in gaps in its official papers for secretaries of state from Madeline Albright to Colin Powell to Condoleezza Rice to Hillary Clinton, and Congress was asking for documents related to the attacks on Benghazi. When the State Department’s recordkeepers belatedly realized that they’d never saved Clinton’s emails—and realized, simultaneously, that she’d been using a personal outside email account rather than an official one—they asked Clinton’s team to produce her emails. That task fell to her personal lawyer, Heather Samuelson, working with Cheryl Mills.

Samuelson, who worked in the White House liaison office at the State Department, told the FBI that that she had never been part of Clinton’s inner circle; she’d received only two emails from Hillary—one on her birthday one year and another after the death of her grandmother—and didn’t know about the private email system until she became Clinton’s private attorney. Over the course of 2014, she and Mills went back and forth with PRN repeatedly on various exports of the email archives. Samuelson described herself as “technically deficient,” and, according to her own statements to the FBI, paid little attention to the technical details of the queries and export of Hillary’s emails when she was gathering files to turn over to the State Department and the House Benghazi Committee, relying on PRN to execute tasks correctly.

Samuelson, Mills, and PRN searched Clinton’s archives for emails used .mil and .gov, as well as the names of members of Congress, foreign leaders and other contacts, as well as keyword searches on terms like “Afghanistan,” “Libya” and “Benghazi.” To assemble the required emails, she used a Lenovo Yoga 2 laptop, but accidentally spilled water on it at one point and, worried that the laptop would fail, she purchased a second Lenovo laptop and copied the email files over to it.

Throughout 2014, Samuelson and Mills struggled to fill in missing gaps in the email queries—many of which seem to have occurred when .gov emails were cc’d—and they made decisions about what to turn over for posterity based only on reading header information, not the body of the emails. They also didn’t have any system for removing duplicates—doing so only if they happened to realize they had duplicate emails.

PRN, at Mills’ instruction, also double-checked that there were no more old emails or server backups lying around on the outdated servers that had been abandoned as part of the switch to PRN’s services. PRN, after traveling to the data center in New Jersey, returned empty-handed, confident that they’d rounded up every Hillary Clinton email they could find.

In December 2014—the same month that Hillary Clinton and Huma Abedin switched email addresses yet again to a new domain, hrcoffice.com—Clinton’s lawyers handed over 55,000 pages of email correspondence to the State Department, totaling about 30,490 separate emails. The State Department’s Office of Information Program and Services picked up 12 banker’s boxes of emails from Clinton’s team. A PRN corporate note from that month about “the Hilary coverup operation” [sic], the employee told the FBI, was simply a joke.

But that’s when the questions really began.

***

9. The investigation

As the State Department team began to review the emails they collected from Hillary Clinton, officials began to raise potentially troubling questions—it appeared that dozens, perhaps even hundreds, of the former secretary’s unclassified emails contained national security secrets.

The potential scandal burst into public view on March 2, 2015, as the New York Times published a story entitled, “Hillary Clinton Used Personal Email Account at State Dept., Possibly Breaking Rules.” That week, the House Select Committee on Benghazi subpoenaed her emails. And by July, the FBI was investigating, encouraged by a referral from the inspector general of the intelligence community who saw evidence that Clinton’s odd email setup might have led to the mishandling of classified material.

The FBI unintentionally demonstrated the confusing and disorganized process behind the Clinton team’s own treatment of the server by managing to recover some 17,448 emails that hadn’t been turned over previously by Clinton’s lawyers. The Pentagon also informed the State Department that it possessed “approximately 1,000 work-related emails” between General David Petraeus and Clinton, most of which were “not believed” to among those the State Department had in its possession.

All told, the FBI found 81 email chains, including 193 individual emails, that either were or should have been classified at the time they were sent because, in government parlance, they included “classified equities” from either the State Department itself or the CIA, FBI, NSA, NGA—the National Geospatial-Intelligence Agency—or the Defense Department.

While in three of the email chains at least one paragraph was marked with only a (c) for Confidential and contained no additional classification markings, others supposedly contained much more highly sensitive information. According to the FBI’s analysis, conducted in conjunction with other government agencies, eight of Clinton’s email chains should have been Top Secret and 37 were Secret. Seven of the emails, all of which were forwarded to Clinton by Jake Sullivan, were associated with what the government calls a Special Access Program, a highly sensitive project subject to even stricter security precautions. As the FBI investigated, there was no consistent pattern to the supposedly classified emails—some came from career State Department officials, some came from presidential appointees, some from Foreign Service officers and some from other elected officials.

The FBI provided Clinton with her classified emails, ranging from Confidential to Top Secret/SAP, and “Clinton said she did

Show more