I would first start by connecting the fax directly to the lines. There is a lot of back end stuff that telco need to do to ensure faxes go through.
the phone system should not play a factor unless there is a defective port.

I just fought with Century Link for a month on this very issue. they delivered lines via IP to Analog. however they didn't adjust the QoS allow priority to VOIP. Also you may want to turn off ECM on the fax machine and lower the baud rate



[Linked Image from i26.servimg.com]
TouchPoint Networks.

Serving the Northwest Since 1991
NEC Shoretel Zultys T3 Tadiran