AT&T Fiber Keeps Dropping? How We Proved the ISP Was the Problem

LIMEHAWK - Managed IT

Home>Incident Reports>AT&T Fiber Keeps Dropping? How We Proved the ISP Was the Problem

Time to fix

~5 weeks, most of it spent proving the ISP wrong

Incident

The ticket opened with one of the staff at an East Tennessee insurance agency saying the remote desktop kept "reconnecting and disconnecting" all morning. Within an hour a second employee chimed in. Within a day a third. Their agency management system and the twenty or so carrier portals they work out of were either spinning forever, throwing server errors, or timing out mid-page. Not slow all the time. Intermittent. Fine for ten minutes, then nothing for two, then fine again.

Intermittent is the worst word in networking. It's the reason this ticket took five weeks to close instead of an afternoon.

First pass — eliminating the obvious

AT&T pushed a modem swap on day one. Old modem out, supposedly more reliable model in, two hours of calm, then the flapping came back. We worked through the usual suspects:

- rebooted the firewall while the user rebooted their workstation
- replaced the Ethernet cable between the new modem and the firewall
- ran a script on affected workstations to disable their wifi adapters — two had wifi enabled alongside the cable and we didn't want them falling onto a weaker path when the wired connection hiccuped
- confirmed no firewall config changes in the last thirty days

None of it stopped the disconnects. The firewall was still reporting packet loss to the gateway. The flaps were real and on the WAN side of our equipment.

How we proved it was the ISP

AT&T's first move when you tell them their fiber is unstable is to blame your equipment. The on-site tech — who eventually lost his composure on the phone and had to be escalated past — kept insisting the firewall was the problem. Fine. The way you win that argument is to take your equipment out of the picture entirely.

We set up two independent external ping monitors from cloud hosts outside the client's network, each targeting the modem's public IP directly. Not the firewall behind it — the modem itself. If AT&T's circuit was clean the pings would be flat. If it was flapping, both monitors would show identical timeouts at identical moments, regardless of anything inside the office.

```
# one monitor pinging the public modem IP from a cloud host
# the other pinging from a different provider's network, for independence
# sample timeout windows from the 24-hour capture:
  09:14:22  request timed out
  09:14:23  request timed out
  09:14:24  request timed out
  09:14:25  58 ms
  ...
  11:47:02  request timed out
  11:47:03  request timed out
  11:47:04  163 ms (jitter spike)
```

Both monitors showed the same pattern: random multi-second gaps and latency spikes throughout the day, correlated across both external sources. That's not possible if the problem is inside the client's firewall. The firewall isn't in the path from a cloud host to the modem's WAN interface. If both external monitors see loss, the circuit is losing packets before it reaches the firewall.

We sent the graphs to AT&T. That changed the conversation.

What we did while AT&T fixed their circuit

ISP repair timelines are what they are. The agency still had to sell insurance. The site had a secondary cable circuit sitting mostly idle as backup. We reconfigured the firewall to run the two circuits in load-balancing mode instead of primary/failover: both WAN links up simultaneously, outbound sessions distributed across both, TCP reconnects landing on whichever link was healthy at the moment of retry.

Load-balancing doesn't eliminate the pain of a flapping circuit — an existing TCP session doesn't get teleported to the other link when its path dies — but it smooths the experience. New connections route around the bad link immediately. Over a workday of clicking between carrier portals, that's the difference between "the internet is broken" and "that page loaded a little slow."

Monitoring what the users actually use

This ticket surfaced something else: staff couldn't tell the difference between "the agency system is down" and "our internet is down." A SaaS throwing a 500 error looks identical to a network dropout from the user's seat. We had them list the sites they live in every day — agency management system, a dozen carrier portals, premium finance, payments — and stood up external uptime monitors for each. Next time the office yells "the internet is broken," we can look at a dashboard and tell them in thirty seconds whether it's the internet, a carrier, or something in between.

Outcome

Once the ping-monitor evidence went up the AT&T chain, the case got reassigned away from the original on-site tech and handled at a higher tier. Over the next two weeks the circuit stabilized. We left the load-balanced WAN config in place — no downside to running both circuits hot — and the uptime monitors stayed up as standing infrastructure.

Follow-up from the client a few weeks later: "Everything has been running smoothly on my end, and there are no issues encountered so far. Looks like the network is much more stable now."

time to fix: ~5 weeks end-to-end
evidence that moved the ISP: two independent external ping monitors
uptime improvements kept in place: dual-WAN load balancing, carrier-portal status monitors

Key takeaways

Takeaway

"intermittent" network issues are the hardest to escalate because the ISP can always claim it was working when they checked. Continuous external ping monitors capture the evidence they can't dismiss.

Takeaway

ping the modem's public IP from outside the network, not the firewall — that eliminates your own equipment from the argument before they can raise it.

Takeaway

when one circuit is unstable, load-balancing a second circuit beats passive failover. Failover only helps during a full outage; load-balancing smooths over flaps.

Takeaway

users can't tell the difference between "our internet" and "their SaaS." Status monitors for the web apps your users actually live in are worth standing up before the next outage, not during it.

Takeaway

when an on-site tech starts blaming your equipment and getting hostile, escalate past them. The case moves faster once it's out of their hands.

Need help with something like this? Contact us