Nothing more irritating then answering the phone last minute - running late for work, or streaking from the shower - and being greeted by a bleeping facsimile. Modem induced heart attack isn't any better. The AfMdS (Anti Fax/Modulator Defence System) would intercept the phonecall before you hear it ring,
detect if it is a modem or fax calling, and proceed using the rule defined by you. There would be two settings, 1) discard the call 2) discard with notify.
#1 is self explanatory, you wouldn't hear a ring or anything on your phone, but the system would create a log entry.
#2 would proceed as follows:
Fax: using the fax protocol return a page with user defined message that gets printed out on the remote fax notifying them of the mistake (saves you from having the poor fellow trying desperately over 20 mins to deliver that critical fax to the wrong number).
Modem: The firmware would also support a cut down PoP dialup server, quickly hosting the connection, assigning IP, address etc, and then sending some sort of net broadcast to the PC with the same message defined in the fax section, thus also saving you from someone trying to connect for 20 mins to their "ISP".
I envisage the device to be about the size of a small MP3 player, have an input phone socket, and an output phone socket for the phone.
Or you could just hunt them down, if that's your style.