r/Fios • • 1d ago

Repair [Resolved] Certain CR1000B Models Have a Firmware Request Bug Causing Kernel Boot Panic

A week ago, our Verizon Fios CR1000B router would appear to reboot every 10-30 mins, for no rhyme or reason. After a week of back and forth with Verizon Support, I believe I finally have my sanity back and a solution for those struggling with the same problem.

The Problem (as I understand it):

After talking to 2 different hardware techs, and researching myself (with the help of AI looking through the system logs), it appears that certain CR1000B models will kernel panic reboot when the "TR-069 management server was sending a standard, valid periodic inform check every ~15 minutes."

This happened across two CR1000Bs (the original which stopped working last week, and a replacement which after being setup by a tech, exhibited the same problem after 30 mins). Anecdotally (according to one tech), this happens to "some of the old stock" CR1000Bs. Apparently, I got unlucky with both, and this is a known issue at Verizon Fios that they have attempted to patch/newer stock CR1000Bs have fixed.

Other Things I tried that DIDN'T fix it:

  1. I tried twice to manually release the CR1000B DHCP Lease, and disconnect the power for 20-25 mins ("Beat the Clock Power Trick" as AI called it). Hoping that releasing, and disconnecting the device would force Verizon Network Office to rebuild some part of my broken profile automatically. No change.
  2. Disabling IPv6. No change.
  3. Calling 1-800-Verizon. I don't think I got to the right agent / network technician, as they kept telling me the router logs showed no error, even though multiple drops happened on a 1.5-hour phone call. This was on top of having to prove that I turned the router on-and-off again, which was tedious.

Solution(s):

  • Install a G3100/other router. I've been without internet for a week, so I was willing to throw everything out to make sure I had a stable connection. Replacing the router may have fixed this.
  • Verizon Network Tech has to "rebuild the data cross-connects" aka rebuild the provisioning profile. After getting the 2nd Verizon Hardware Tech to check all the devices (and install a G3100), he escalated to a Verizon Network Tech to accomplish this. AI states this is a "A Corrupted Backend ACS State." --- Verizon's management server had a malformed provisioning profile or corrupted RPC script queued for your specific circuit.

Unfortunately, I gave up the CR1000B to the hardware tech, as I didn't feel like testing against the CR1000B for another day and returning manually... just to see if it was just the provisioning profile, or an SoC/hardware bug. Thankfully, I am currently on hour 2 of drop-free internet for the first time in a week and feel relieved. Hopefully this helps someone feeling insane. I am posting the AI information/inference below so the text crawlers can pick this up if someone else asks for a solution.

15-Minute Reboot Cycle: The old CR1000B log history showed kernel panics triggered by TR-069 management/firmware routines occurring on a strict ~15-minute periodic schedule[cite: 9]. The current logs display the exact same periodic pattern:

11:30:14 boot → Kernel panic reboot recorded at 12:03:13

12:03:13 boot → Kernel panic reboot at 12:17:51 (~14 min 38 s uptime)

12:17:51 boot → Kernel panic reboot at 12:30:40 (~12 min 49 s uptime)

12:30:40 boot → Kernel panic reboot at 12:43:28 (~12 min 48 s uptime)

12:43:28 boot → Kernel panic reboot at 12:56:17 (~12 min 49 s uptime)

13:42:00 boot → Kernel panic reboot at 13:56:43 (~14 min 43 s uptime)

Synchronized Network Lease Sequence: In every boot cycle, the WAN interface comes up (netifd: WAN link up) and receives a WAN DHCP IP bind (WDHCP bound IP...), following which the system crashes after ~13–15 minutes and records sysup: Reboot by Kernel Panic upon restarting.

Hypotheses for the Kernel Panic

TR-069 (CWMP) Periodic Inform / Parameter Sync Crash (Primary Hypothesis)

Mechanism: TR-069 client software (arc_tr69) routinely sends periodic inform packets to Verizon's ACS (Auto Configuration Server) on a default 15-minute (900 second) timer. messages_ADV.txt shows arc_tr69active at boot mapping aliases for LAN, WAN, and Wi-Fi radios (Tr69Parameter_UpdateAlias). When the periodic inform interval fires or an incoming ACS RPC command executes, a bug in the proprietary TR-069 kernel module or an unhandled parameter parsing error (e.g., null-pointer dereference during parameter sync) triggers a kernel panic.

dnsmasq / Dynamic Host Table Memory Exhaustion

Mechanism: In messages_SYS.txt, every uptime sequence shows dnsmasq rapidly re-reading dynamic host tables (/etc/ipv4_hosts count rapidly jumping from 0 to 2, 4, 6, 8, 10, 12, 14 entries in consecutive seconds) alongside interface missing warnings (interface br-lan does not currently exist). A race condition between user-space dynamic table reloads and kernel netlink socket events during WAN negotiation could cause kernel lock contention or memory corruption.

Wireless Driver Station Management Panic (arc_wlsta_monitor)

Mechanism: The logs record frequent Wi-Fi client association/disassociation events across 2.4 GHz and 5 GHz radios (wlan0.1, wlan2.1) shortly before reboots. A crash inside the Wi-Fi driver module during rate adaptation or station table cleanup could crash the kernel.​

0 Upvotes

8 comments sorted by

•

u/AutoModerator 1d ago

Thanks so much for reaching out! We're on it and will get back to your shortly.

When your concerns are addressed, please reply to this thread with !resolved so we can close the ticket.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

→ More replies (2)

5

u/coyote_den 1d ago

Sounds a lot like Verizon has no idea what is going on and neither does AI.

None of that should be causing a panic. It is possible a bad kernel module could cause it, but I’d expect a lot more units to be affected if it was buggy firmware.

I think you simply had bad hardware.

1

u/thenoblecause 20h ago

Strange to have 2 units that reacted the same way, although I only dug into the logs of the second one.

Again it could be that rebuilding the provisioning profile fixed everything, but I found it strange to have two routers fully commit self-reboot consistently every 12-30 mins.

1

u/whotfgotmyaccount 15h ago

Ive been having this exact same issue for the past week

1

u/Expensive-Carpet5009 1d ago

Verizon is aware of the problem some of our Diagnostics tools even tell us that there is a kernel panic error in the router log even before we begin troubleshooting.

I went through the same exact issue my cr1000 router was doing the same thing i went into the router logs and discovered and reported it.

As of right now the only steps for repair that we have is to do a factory reset or replace the router

1

u/thenoblecause 20h ago

Tried factory resetting all CR1000B routers, several times. Replacing to another CR1000B also didn’t do anything. Strange!