The National Transportation Safety Board says there is no evidence so far that there was anything wrong with the plane.
The multi-engine, propeller-driven plane struck two Cape-style homes a few blocks from Tweed New Haven Airport.
The aircraft's left wing lodged in one house and its right wing in the other.
Anthony Moscato, deputy chief of the East Haven Fire Department, said the bodies of two people from the plane and two people in one of the homes.
He said authorities now believe those were the only victims; they had previously said as many as six people could have been killed.
Those presumed dead are the pilot, Bill Henningsgaard, a former Microsoft executive, his 17-year-old son Maxwell, and two children in a home struck by the plane, 13-year-old Sade Brantley and 1-year-old Madisyn Mitchell.
Henningsgaard was taking his son on an East Coast tour of colleges.
The family learned it was Bill Henningsgaard's plane through the tail number, said his brother, Blair Henninsgaard, the city attorney in Astoria, Ore.
It wasn't his first crash. Four years ago, Bill Henningsgaard crash-landed his plane on Washington's Columbia River, and he and his 84-year-old mother were rescued by a passing boat as the plane began to sink.
The 10-seater plane, a Rockwell International Turbo Commander 690B, flew out of Teterboro Airport in New Jersey and crashed at 11:25 a.m., the Federal Aviation Administration said.
Tweed's airport manager, Lori Hoffman-Soares, said the pilot had been in communication with air traffic control and hadn't issued any distress calls.
"All we know is that it missed the approach and continued on. There were no distress calls as far as we know," she said.
"The plane struck two houses. One of the houses was struck more severely and part of the plane is actually in the basement," said Bob Gretz, NTSB Senior Air Safety Investigator.
A neighbor, David Esposito, said he heard a loud noise and then a thump. "No engine noise, nothing," he said.
Esposito said he ran into the upstairs of the house, where a woman believed her children were, but they could not find them. They returned downstairs to search, but he dragged the woman out when the flames became too strong.
"As to where they were, she said upstairs, and I flipped over the beds and flipped over the crib and looked in the closet, but couldn't find anybody. There was nothing," Esposito said.
East Haven Mayor Joseph Maturo offered sympathy to the family.
"It's total devastation in the back of the home," he said.
Neighbors said the woman moved into the neighborhood recently.
The NTSB said that they are primarily in a support position. They plan to document the wreckage and meet with the FAA. They noted that the plane's engine maker Honeywell was headed to the scene along with the maker of the plane's frame, Twin Commander. They hope to have a preliminary report up on the NTSB's website in five to 10 days.
This plane, like most small prop planes, was not carrying a data recorder or a black box. But it was carry a special piece of equipment that is critical to the investigation.
"The aircraft was equipped with a ground proximity system. We found that out through FAA records. That's what we're going to begin doing, looking for that piece of equipment," said Patrick Murray, NTSB investigator.
A vigil was held for the victims Saturday evening in the center of town at Margaret Tucker Park. The mother of the two children killed in the crash attended the vigil, and cried throughout the service.
Henningsgaard was a Microsoft vice president for 14 years before leaving in 2002 to establish Eastside Pathways, a nonprofit that helps students with reading and career preparedness.
(The Associated Press contributed to this report)