If they only have your IP then you should be fine, if they contact you and the only thing they have linked to you is your IP then they would have to either
A.) Contact your ISP (happens in most case)
or
B.)"illegally" access your information
More then likely A.
As for are you safe? I can't tell you for certain, but for now just relax. This isn't even set in stone for one thing.