| Baudouin V de Hainaut - Count of Hainaut | Destroyed the castle in 1181. |
| Gérard de Douai - Provost and Lord of Emerchicourt | Cause of conflict with Baudouin V. |
| Gilles Brousse (ou Broisse) - Lord of Denain and Hasencort | Sells land in the 13th century. |
| Comte de Moutiers - Owner and Ambassador | Finished the renovation in the 19th century. |