The incident happened on Friday morning shortly after 10:00.
The Buildings Department says an inspection showed that the construction fence was installed contrary to approved plans. A full Stop Work Order has been issued for the site.
West 40th Street remained closed to vehicular traffic in the area until early afternoon.
Authorities say the scaffolding landed on three people. Two good Samaritans helped pull the debris off of two women.
The victims were taken to Bellevue Hospital. All were conscious and alert
A firefighter also sustained minor injuries responding to the scene. He was taken to Roosevelt Hospital
The collapse happened as a line of showers and storms brought strong wind gusts to the area.
The Department of Buildings issued a warning to property owners Friday to secure construction sites and buildings because of the high winds.