Gove told the city council's chief executive, Stuart Lowe, that he was "personally concerned about how your organisation continues to meet its responsibilities to residents".
In his letter, Gove said: 'I am writing this letter after the Housing Ombudsman found serious maladministration on your part. This is because you have failed to address damp and mould problems in a resident's home for over two years.
You have done so in the knowledge that the resident had a four-month-old child in the home. Despite the investigation report finding clear signs of rising damp, you did not offer to move the resident out until four months later, and even then you told the resident that you would only consider moving the resident out after further inspections.
Eight months later you again told the tenants that they could not be rented out and then it took another five months to carry out another survey.
You provided no evidence that you had considered the vulnerability of the residents or the impact on their health and wellbeing. Two years later, a number of issues remain outstanding.
It is unacceptable that your residents have had to wait so long for a resolution, especially when a young child is involved. Residents should expect that when they report a problem it will be dealt with quickly and effectively, not months later, let alone years.
The council has been hit by government funding cuts since 2010, which some have blamed for the failings.
A Westminster City Council spokesman said: 'Earlier this year we launched a council-wide housing improvement programme. This included reviewing the repairs process and working with our contractors to significantly improve the services residents receive.
"We would like to thank the Ombudsman for engaging with us and our residents, who recently organised an event for us to allow them to speak directly to residents about the valuable role they play."