Garments workers frightened, factory owners concerned