Protests on that day would be "disrespectful to all who see 9/11 as a day outside of politics, when we desire to remain united in honoring the lives and the courage of our many friends and loved ones," the groups said in letters sent to developers of the Islamic center and to those planning to protest it.
The messages were signed by representatives of the Tribute WTC Visitor Center, the September 11th Families Association, MyGoodDeed and others.
Two rallies are planned for the day: one against the center and one against anti-Islamic bigotry. Organizers of both say they will be respectful.
Another family group, 9/11 Parents & Families of Firefighters & WTC Victims, responded with a letter in support of the anti-mosque rally and its planners' First Amendment rights of free speech and assembly.
"By attending and participating in this rally, families can endeavor to ensure that the sacred ground will continue to be respected for posterity," the letter said.
Park51, a proposed cultural center two blocks from ground zero that would include a mosque, auditorium and other facilities, is not sponsoring any protests. A spokesman for the project did not comment on the protests or the letters.
Protesters at the rally against the mosque will carry flags instead of signs and won't chant, said Pamela Geller, the executive director of the group Stop Islamization of America and organizer of the protest.
"It is a rally of remembrance," she said, adding that organizers had carefully weighed the concerns but believed most family members wished the group to go forward.
Gavrielle Gemma, an organizer for the anti-bigotry rally, said the ad-hoc group's protest was being held on Sept. 11 only because Geller's rally was scheduled for that date.
"We definitely plan to be respectful of the day. But are we going to be silent? No," she said. "We cannot let people in this country be scapegoated because of their religion, their national origin, or their race."
David Paine, president of MyGoodDeed, which helps organize the annual National Day of Service and Remembrance on Sept. 11, called on all the groups to simply stand down.
"One demonstration begets another, which begets another," he said. "Next thing you know it'll be the National Day of Service, Remembrance and Protest. We don't want that."