I am being blackmailed into an "affair" ?

I have been trying to stop it all together but the only problem is he is blackmailing me. I am defined as a very professional, hardworking business woman. However the colleague is demanding I break up with my husband and if not he will tell my work colleagues. I am really confused on what to do as it such a complex situation. I have tried to stop the affair but if this news gets out around work I will no longer be looked on as a role model and such a high achiever. Please any advice would be very beneficial ? My husband is such a loving man and I am filled with guilt.