(CNN) -- Unlike many stories about powerful Washington figures having secret affairs, the downfall of spy chief David Petraeus goes beyond sex.

The scandal surrounding the decorated four-star Army general who once ran the wars in Iraq and Afghanistan involves questions of national security, politics and even the September 11 attack on the U.S. consulate in Benghazi, Libya, that left four Americans dead.

Petraeus, 60, resigned Friday after acknowledging he had an affair with a woman later identified as his biographer, Paula Broadwell, 40, a fellow West Point graduate who spent months studying the general's leadership of U.S. forces in Afghanistan.

FBI agents were at Broadwell's Charlotte, North Carolina, home late Monday, said local FBI spokeswoman Shelley Lynch. She declined to say what the agents were doing there.

Video from CNN affiliate WCNC showed a handful of people getting out of vehicles, carrying boxes and bags into the house. None spoke to reporters, even when asked who they were.

Timeline of the Petraeus affair

Days after Petraeus' resignation stunned Washington, information continues to emerge. Among other things, a video has surfaced of a speech by Petraeus' paramour in which she suggested the Libya attack was targeting a secret prison at the Benghazi consulate annex, raising unverified concerns about possible security leaks.

The affair came to light during an FBI investigation of "jealous" e-mails reportedly sent by Broadwell to a woman named Jill Kelley, a government source familiar with the investigation told CNN on Monday.

Kelley, 37, and her husband Scott released a statement saying they have been friends with Petraeus and his family for more than five years and asked for privacy.

Although Kelley lives in Tampa, Florida, she's known as a member of Washington's social circuit, according to the government source. The source has not spoken to Kelley, but says friends describe her as feeling like she is an "innocent victim."

Petraeus has denied having an affair with anyone other than Broadwell, according to a friend of the former general who has spoken with him since news of the affair broke.

The scandal also is rumbling through the halls of Congress, where leaders in both parties are seeking answers about the FBI investigation and there's much speculation about the impact Petraeus' resignation will have into the inquiry into the Benghazi attack.

Petraeus was scheduled to testify on the attack and the government's reaction to it this week.

King: Petraeus' resignation doesn't preclude testimony

Here's a look at the major threads of this still-unfolding story:

Why does it matter? Security and Benghazi

While affairs may be commonplace in Washington, when they involve the director of the CIA, things can take on a different tone.

Analysts say there is no evidence that a security breach occurred as a result of the affair, but that hasn't stopped discussion that Broadwell could have gained access to classified information as a result of what she has routinely described as "unprecedented access" to Petraeus.

That discussion seemed to gain momentum Monday thanks to comments Broadwell made in a speech last month at the University of Denver.

Is Petraeus pillow talk a security threat?

"I don't know if a lot of you have heard this, but the CIA annex had actually taken a couple of Libyan militia members prisoner and they think that the attack on the consulate was an effort to get these prisoners back," Broadwell said.

A senior intelligence official told CNN on Monday, "These detention claims are categorically not true. Nobody was ever held at the annex before, during, or after the attacks."

Broadwell's source for that previously unpublished bit of information remains unclear, and there's no evidence so far that it came from Petraeus. Administration officials have said the Benghazi assault was a terrorist attack.

The New York Times also reported Sunday that investigators found classified documents on Broadwell's laptop computer. The newspaper cited investigators as saying Petraeus denied he had given them to her.

Retired Gen. James "Spider" Marks, for whom Broadwell once worked and who knows Petraeus, said he doubts security protocols were breached despite what seems an unlikely indiscretion on the part of Petraeus.

"There's almost zero percent chance that national security was compromised or at risk," he said Monday.

A senior U.S. intelligence official said an extramarital affair by a CIA officer is not automatically considered a security violation.

"It depends on the circumstances," the official said.

The official also said Broadwell did not have a security clearance from the CIA.

Another official said Broadwell, who is an officer in the Army reserve, did have some kind of security clearance and that there are no issues with Broadwell having unauthorized access to classified information.

Petraeus' resignation also presents challenges to the congressional inquiry into the Benghazi attack.

CNN has confirmed that Petraeus recently traveled to Libya to meet the CIA station chief to discuss the attack. He was scheduled to testify before a congressional committee this week on the assault and the U.S. government response to it.

Will national security scandal create national security risk?

That now will not happen, but it is possible that he could be summoned by Congress to testify later.

Some Republicans have criticized the administration's response to the Benghazi attack and have speculated that Petraeus' departure was linked to the congressional inquiry.

Rep. Peter King, R-New York, said elements of the story "don't add up." He called Petraeus "an absolutely essential witness, maybe more than anybody else."

However, a senior U.S. official said Petraeus' departure wasn't connected to the Benghazi hearing.

"Director Petraeus' frank and forthright letter of resignation stands on its own," said the official, who spoke on the condition of anonymity because of the sensitivity of the topic. "Any suggestion that his departure has anything to do with criticism about Benghazi is completely baseless."

Who's who in the Petraeus scandal

The investigation

Congressional leaders are calling for an explanation of why they weren't notified sooner of the FBI's inquiry when it became clear Petraeus was involved.

Leaders of the House Intelligence Committee are expected to meet Wednesday with acting CIA Director Mike Morell and FBI Deputy Director Sean Joyce to discuss the Petraeus investigation and congressional oversight.

Sen. Diane Feinstein, the Democratic chair of the Senate Intelligence Committee, said on "Fox News Sunday" that she would "absolutely" push for answers.

"I mean, this is something that could have had an effect on national security," she said. "I think we should have been told. There is a way to do it. And that is, just to inform the chair and the vice chairman of both committees, to -- this has happened before, not with precise, same things, but, none of the four of us have ever breached that confidentiality."

On Monday, Feinstein told NBC that her concern about the situation "has actually escalated the last few days."

"...A decision was made somewhere not to brief us, which is atypical," Feinstein said. "This is certainly an operationally sensitive matter. But we weren't briefed. I don't know who made that decision."

The FBI investigation began when Kelley went to FBI officials to complain that Broadwell was sending harassing e-mails to her, a U.S. official told CNN. Kelley received the worrisome e-mails in May, an official said, describing the messages as along the lines of "stay away from my guy," but not explicitly threatening.

Opinion: How Petraeus changed the U.S. military

According to a source with knowledge of the e-mails, the messages accused Kelley of untoward behavior with some generals at MacDill Air Force Base in Tampa, Florida where Kelley did volunteer work.

The e-mails detailed the "comings and goings of the generals and Ms. Kelley," said the source, who declined to speak on the record because of sensitivity of the investigation.

Among those believed to be referenced in the e-mails was Petreaus. Because parts of Petreaus' schedule were not public, the e-mails raised questions about whether the sender of the e-mails had access to his private schedule or other sensitive information.

The content of the e-mails was first reported by NBC News.

At one point, Petraeus told Broadwell to stop sending the e-mails, a U.S. official said. It was not clear whether his request was made during or after his affair with Broadwell.

During the investigation, other communications surfaced between Petraeus and Broadwell, a married mother of two, according to a U.S. official.

Petraeus used a personal account to e-mail Broadwell, not his CIA account, a U.S. official said.

The FBI interviewed Petraeus, said the official, who stressed that the CIA director was never the target of the investigation and his communications were never compromised.

Broadwell was interviewed twice by the FBI, a U.S. official said.

The official said the investigation is not officially closed, but it appeared there will be no charges.

According to a congressional aide familiar with the matter, the House and Senate intelligence committees weren't informed that there was an FBI investigation into the situation until Friday.

Opinion: 5 things we've learned from Petraeus scandal

"The committees are required to be kept informed of significant intelligence activities," the aide said Saturday. "If there was an official investigation that was looking, at least in part, at information that was compromising the CIA director, then I think there's a solid argument to say that the committee leadership should have been notified to at least some level of detail."

But former FBI assistant director Tom Fuentes told CNN on Monday that if, as the investigation progresses, the FBI is not "uncovering criminal activity" or a "breach of security" then "there really isn't a need" to notify members of Congress.

The FBI has "very strict protocols" about who should be notified in this type of investigation, Fuentes said.

House Majority Leader Eric Cantor, R-Virginia, knew in October about Petraeus' involvement in an extramarital affair, a spokesman for the congressman told CNN on Sunday.

Doug Heye said Cantor was tipped to the information by an FBI employee. The congressman had a conversation with the official, described as a whistle-blower, about the affair and national security concerns involved in the matter, he said.

The affair

Broadwell and Petraeus first met in 2006 at Harvard University's Kennedy School of Government, where she was a student, Broadwell wrote in the preface of the biography she co-authored on Petraeus.

She told him about her interest in studying military leadership, and he offered his help.

"I later discovered that he was famous for this type of mentoring and networking, especially with aspiring soldier-scholars," Broadwell wrote.

She traveled to Afghanistan, where she interviewed Petraeus repeatedly, sometimes on long runs that likely increased the general's respect for her.

"She probably kicked his butt," Marks said. "And it was probably the first time that had ever happened to him, so he let his guard down. He brought her in."

Such runs were a common way for Petraeus to conduct business, an adviser who worked on and off in Afghanistan with Petraeus and Broadwell told CNN. Still, some staffers were jealous of the access she had to him and the lengthy interactions they had, the adviser said.

Out of those conversations and months of other research came the best-selling book "All In: The Education of General David Petraeus."

In promoting the book, and defending it against critics who said it was too sympathetic, she frequently spoke of her unprecedented access to the general and glowingly of his character.

"I'm not a spokesperson for him, and if showing a role model to other people in the world or other readers is a repugnant thing, then I'm sorry, but I think the values that he upholds and tries to instill in his organizations are valuable and worth pointing out," she told CNN in February.

The actual affair began about two months after Petraeus took over at the CIA in September 2011, according to one Petraeus friend.

It ended about four months ago, and the two last talked about a month ago, the friend said.

The decision to end the relationship was mutual, the friend said.

Another of the former general's friends said Petraeus felt isolated after leaving the camaraderie of the military, and it made him more vulnerable.

"I think leaving the Army, the emotions, and the psychological effect impacted on him more than he thought it would," the friend said.

Broadwell, with her background in military and intelligence issues, was someone he could talk to, the friend said.

"He enjoyed her company," the friend said. "She was an attractive gal and they had things in common."

According to her biography at the University of Denver, Broadwell graduated with honors from the U.S. Military Academy and has had "assignments with the U.S. intelligence community, U.S. Special Operations Command and an FBI Joint Terrorism Task Force."

She is a now a doctoral student at King's College in London, where her webpage indicates she is interested in studying military leadership, organizational and management theories and U.S. foreign, defense and intelligence policies.

CNN has not been able to contact Broadwell for comment. Her house in Charlotte did not appear occupied Monday.

Former Petraeus spokesman Steve Boylan said the retired general is devastated by the incident, and focused on his family.

"Furious would be an understatement," to describe Petraeus' wife, he said.

"He's taking this hard," Boylan said. "He is aware of the impact this has had on his family, and he knows what a wonderful family he has. On a personal level, he sees this as a failure, and this is a man who has never failed at anything."

CNN's Joe Sutton, Gloria Borger, Barbara Starr, Suzanne Kelly, Carol Cratty, Candy Crowley, Dana Bash, Pam Benson and Ted Barrett contributed to this report.