This all began with my curiosity surrounding AT&T’s requirements as a subscriber to their networks. Unlike many cable carriers, AT&T demands you use their piece-of-sh*t equipment just to hop onto their sub-par networks. For this particular model, NVG595, the most common limiting factor that degrades user experience is the whopping 8,000 state table limit which triggers the most infamous hideous-don’t-need-know-you’re-a-piece-of-sh*t-device page that forces everyone to open the said page explaining that Arris had a potty accident and had to reset itself.
Can we get rid of these? Yes, hopefully. You could complain about it to AT&T, but what do think they’re going to do about it? I’ll tell you. They won’t fix it, but they’ll sell you another under powered device. And the loop goes on and on. Throughout the past couple of years there has been great progress into the idea of completely getting rid of these devices altogether. Some methods use the built-in “IP Passthrough” feature (doesn’t bypass the state table) while others call for plugging the device into your main switch initially and after a power loss, others call for an 802.1x proxy. But what about no Arris at all? It’s possible but much more uglier than all the other methods. It requires grabbing some certificates from the NAND on board. How you obtain those is up to you, and obviously how poorly secured your device is. Mine wasn’t poorly secured. It shipped with a newer firmware version, so all those exploits everyone has been talking about are pretty much all patched up. So…… Let’s see how we’re going to dig up those certificates from the NAND.
So we know that this version of the firmware isn’t suspectable to any known vulnerabilities. What about its guts? Lets open’em up.
We immediately see some interesting things. We got what appears to be JTAG on J100, UART on J5C1, and a 1Gb (128MB) Spansion S34ML01G1 NAND on the bottom.
My first approach was to target the UART pins. As you can tell, there are some pins already soldered onto the board which weren’t originally. Since there wasn’t any available documentation for the pinout, I had no choice to but whip out a volt meter to find out which pin does what.
HELO CPUI L1CI HELO CPUI L1CI DRAM ---- PHYS STRF 400H PHYE DDR3 SIZ4 SIZ3 SIZ2 DINT USYN LSYN MFAS LMBE RACE PASS ---- ZBSS CODE DATA L12F MAIN FPS0 BT02 0001 BT03 0321 RFS2 NAN5 CFE version 3.0.36 for BCM63268 (32bit,SP,BE,MIPS) Copyright (C) 2000,2001,2002,2003,2004,2005 Broadcom Corporation. Initializing Arena. Initializing Devices. NAND flash device: name Spansion S34ML01G1, id 0x01f1 block 128KB size 131072KB Partition information: #00 00000000 -> 0001FFFF (131072) #01 00020000 -> 07FFFFFF (134086656) CPU type 0x2A080: 400MHz CPU running TP0 Total memory: 0x8000000 bytes (128MB) Total memory used by CFE: 0x87D00000 - 0x87F43AE4 (2374372) Initialized Data: 0x87D368B8 - 0x87D388AC (8180) BSS Area: 0x87D388CC - 0x87D41AE4 (37400) Local Heap: 0x87D41AE4 - 0x87F41AE4 (2097152) Stack Area: 0x87F41AE4 - 0x87F43AE4 (8192) Text (code) segment: 0x87D00000 - 0x87D314A8 (201896) Boot area (physical): 0x80100000 - 0x87D00000 Relocation Factor: I:00000000 - D:00000000 CFE booting from NAND flash Current RootFS 2 Loading post: Entry:82a00000 Running POST..... POST version 1.3.0 Total POST tests: 2 Test: Memory........Pass Test: MII...........Pass Updating the POST results......Done Completed POST..... Booting from latest image (0xba820000) ... Decompression OK! Entry at 0x80014490 Linux version 22.214.171.124-motopia (fwbuild@MA35BLD08) (gcc version 4.7.2 (crosstool-NG 1.17.0) ) #1 Sat Sep 9 04:55:15 EDT 2017 Broadcom BCM63xx prom init Boot loader is : CFE console [early0] enabled CPU revision is: 0002a080 (Broadcom BCM63268) Determined physical RAM map: memory: 07e00000 @ 00000000 (usable) Zone PFN ranges: DMA 0x00000000 -> 0x00001000 Normal 0x00001000 -> 0x00007e00 Movable zone start PFN for each node early_node_map active PFN ranges 0: 0x00000000 -> 0x00007e00 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 32004 Kernel command line: root=mtd:rootfs ro rootfstype=jffs2 console=ttyS0,57600 wait instruction: enabled Primary instruction cache 64kB, VIPT, 4-way, linesize 16 bytes. Primary data cache 32kB, 2-way, VIPT, cache aliases, linesize 16 bytes NR_IRQS:128 PID hash table entries: 512 (order: 9, 2048 bytes)m▒▒k▒▒▒▒handover: boot [early0] -> real [ttyS0] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes) Inode-cache hash table entries: 8192 (order: 3, 32768 bytes) Allocating memory for DSP module core and initialization code Allocated DSP module memory - CORE=0x81118d40 SIZE=1907200, INIT=0x812ea740 SIZE=44 Memory: 122664k/129024k available (2406k kernel code, 6180k reserved, 442k data, 124k init, 0k highmem) Calibrating delay loop... 399.36 BogoMIPS (lpj=199680) Mount-cache hash table entries: 512 net_namespace: 1088 bytes NET: Registered protocol family 16 registering PCI controller with io_map_base unset registering PCI controller with io_map_base unset bio: create slab at 0 SCSI subsystem initialized usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb pci 0000:00:00.0: PME# supported from D0 D3hot D3cold pci 0000:00:00.0: PME# disabled pci 0000:01:00.0: PME# supported from D0 D3hot pci 0000:01:00.0: PME# disabled pci 0000:01:00.0: PCI bridge, secondary bus 0000:02 pci 0000:01:00.0: IO window: disabled pci 0000:01:00.0: MEM window: disabled pci 0000:01:00.0: PREFETCH window: disabled BLOG v3.0 Initialized BLOG v3.0 Initialized - mcast_state=3 BLOG Rule v1.0 Initialized Broadcom IQoS v0.1 Sep 9 2017 04:51:50 initialized Broadcom GBPM v0.1 Sep 9 2017 04:51:51 initialized NET: Registered protocol family 2 IP route cache hash table entries: 1024 (order: 0, 4096 bytes) TCP established hash table entries: 4096 (order: 3, 32768 bytes) TCP bind hash table entries: 4096 (order: 2, 16384 bytes) TCP: Hash tables configured (established 4096 bind 4096) TCP reno registered NET: Registered protocol family 1 JFFS2 version 2.2. (NAND) © 2001-2006 Red Hat, Inc. msgmni has been set to 239 io scheduler noop registered io scheduler cfq registered (default) Serial: Motopia BCM63XX driver $Revision: 1.4 $ ttyS0 at MMIO 0xb0000180 (irq = 13) is a BCM63XX ttyS1 at MMIO 0xb00001a0 (irq = 42) is a BCM63XX Driver 'sd' needs updating - please use bus_type methods tun: Universal TUN/TAP device driver, 1.6 tun: (C) 1999-2004 Max Krasnyansky <firstname.lastname@example.org> Probing for flash BCM Flash API. Flash device is not found. Broadcom DSL NAND controller (BrcmNand Controller) -->brcmnand_scan: CS=0, numchips=1, csi=0 mtd->oobsize=0, mtd->eccOobSize=0 NAND_CS_NAND_XOR=00000000 Disabling XOR on CS#0 brcmnand_scan: Calling brcmnand_probe for CS=0 B4: NandSelect=40000001, nandConfig=15142200, chipSelect=0 brcmnand_read_id: CS0: dev_id=01f1001d After: NandSelect=40000001, nandConfig=15142200 Block size=00020000, erase shift=17 NAND Config: Reg=15142200, chipSize=128 MB, blockSize=128K, erase_shift=11 busWidth=1, pageSize=2048B, page_shift=11, page_mask=000007ff timing1 not adjusted: 6574845b timing2 not adjusted: 00001e96 brcmnand_adjust_acccontrol: gAccControl[CS=0]=00000000, ACC=f7ff1010 BrcmNAND mfg 1 f1 SPANSION_S30ML01GP_08 128MB on CS0 Found NAND on CS0: ACC=f7ff1010, cfg=15142200, flashId=01f1001d, tim1=6574845b, tim2=00001e96 BrcmNAND version = 0x0400 128MB @00000000 brcmnand_scan: Done brcmnand_probe brcmnand_scan: B4 nand_select = 40000001 brcmnand_scan: After nand_select = 40000001 100 CS=0, chip->ctrl->CS=0 ECC level 15, threshold at 1 bits reqEccLevel=0, eccLevel=15 190 eccLevel=15, chip->ecclevel=15, acc=f7ff1010 brcmnand_scan 10 200 CS=0, chip->ctrl->CS=0 200 chip->ecclevel=15, acc=f7ff1010 page_shift=11, bbt_erase_shift=17, chip_shift=27, phys_erase_shift=17 brcmnand_scan 220 Brcm NAND controller version = 4.0 NAND flash size 128MB @18000000 brcmnand_scan 230 brcmnand_scan 40, mtd->oobsize=64, chip->ecclayout=00000000 brcmnand_scan 42, mtd->oobsize=64, chip->ecclevel=15, isMLC=0, chip->cellinfo=0 ECC layout=brcmnand_oob_bch4_4k brcmnand_scan: mtd->oobsize=64 brcmnand_scan: oobavail=50, eccsize=512, writesize=2048 brcmnand_scan, eccsize=512, writesize=2048, eccsteps=4, ecclevel=15, eccbytes=3 300 CS=0, chip->ctrl->CS=0 500 chip=878b3180, CS=0, chip->ctrl->CS=0 -->brcmnand_default_bbt brcmnand_default_bbt: bbt_td = bbt_main_descr brcmnandCET: Status -> Deferred brcmnand_scan 99 Creating 5 MTD partitions on "brcmnand.0": 0x000002820000-0x000005020000 : "rootfs" 0x000000020000-0x000002820000 : "rootfs_update" 0x000005120000-0x000007f00000 : "data" 0x000000000000-0x000000020000 : "nvram" 0x000005020000-0x000005120000 : "mfg" ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver PCI: Enabling device 0000:00:0a.0 (0000 -> 0002) ehci_hcd 0000:00:0a.0: EHCI Host Controller ehci_hcd 0000:00:0a.0: new USB bus registered, assigned bus number 1 ehci_hcd 0000:00:0a.0: Enabling legacy PCI PM ehci_hcd 0000:00:0a.0: irq 18, io mem 0x10002500 ehci_hcd 0000:00:0a.0: USB f.f started, EHCI 1.00 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB hub found hub 1-0:1.0: 2 ports detected ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver PCI: Enabling device 0000:00:09.0 (0000 -> 0002) ohci_hcd 0000:00:09.0: OHCI Host Controller ohci_hcd 0000:00:09.0: new USB bus registered, assigned bus number 2 ohci_hcd 0000:00:09.0: irq 17, io mem 0x10002600 usb usb2: configuration #1 chosen from 1 choice hub 2-0:1.0: USB hub found hub 2-0:1.0: 2 ports detected GACT probability NOT on u32 classifier Actions configured Netfilter messages via NETLINK v0.30. nf_conntrack version 0.5.0 (2016 buckets, 8064 max) ip_tables: (C) 2000-2006 Netfilter Core Team TCP cubic registered NET: Registered protocol family 10 IPv6 over IPv4 tunneling driver NET: Registered protocol family 17 Bridge firewalling registered Ebtables v2.0 registered 802.1Q VLAN Support v1.8 Ben Greear <email@example.com> All bugs added by David S. Miller <firstname.lastname@example.org> VFS: Mounted root (jffs2 filesystem) readonly on device 31:0. Freeing unused kernel memory: 124k freed Initiating system startup... Algorithmics/MIPS FPU Emulator v1.5 Starting UDEV... cannot open /dev/null Complete! Empty flash at 0x00f01f88 ends at 0x00f02000 mkdir: can't create directory '/data/system': File exists Motopia driver loading console [MotCrashDump-1] enabled crashdump_init done, registered console driver successfully Motopia event ready Reset button ready Push button ready LEDs ready: Power Internet SimpleConfig LAN WIFI VOIP1 VOIP2 WAN ONT pfs: Continuing in background sdb: detaching to run in the background Hit the enter key to continue...bcmPktDma: module license 'unspecified' taints kernel. Disabling lock debugging due to kernel taint bcmPktDma_init: Broadcom Packet DMA Library initialized Total # RxBds=1448 bcmPktDmaBds_init: Broadcom Packet DMA BDs initialized BPM: tot_mem_size=134217728B (128MB), buf_mem_size=20132655B (19MB), num of buffers=9532, buf size=2112 Broadcom BPM Module Char Driver v0.1 Sep 9 2017 04:51:21 Registered<244> NET: Registered protocol family 8 NET: Registered protocol family 20 PPP generic driver version 2.4.2 bcmxtmrt: Broadcom BCM3169D0 ATM/PTM Network Device v0.4 Sep 9 2017 04:51:23 NBUFF v1.0 Initialized Initialized fcache state Broadcom Packet Flow Cache Char Driver v2.2 Sep 9 2017 04:51:38 Registered<242> Created Proc FS /procfs/fcache Broadcom Packet Flow Cache registered with netdev chain Broadcom Packet Flow Cache learning via BLOG enabled. Constructed Broadcom Packet Flow Cache v2.2 Sep 9 2017 04:51:38 chipId 0x631690D0 Broadcom Forwarding Assist Processor (FAP) Char Driver v0.1 Sep 9 2017 04:51:33 Registered <241> FAP Debug values at 0x00000010 0x00000010 Enabling SMISBUS PHYS_FAP_BASE is 0x10c01000 FAP Soft Reset Done 4ke Reset Done Enabling SMISBUS PHYS_FAP_BASE is 0x10c01000 FAP Soft Reset Done 4ke Reset Done FAP Debug values at 0xa599b680 0xa5a1b680 Allocated FAP0 TM SDRAM Queue Storage (a59a1950) : 341376 bytes @ a5a80000 Allocated FAP1 TM SDRAM Queue Storage (a5a21950) : 341376 bytes @ a5b00000 bcmPktDma_bind: FAP Driver binding successfull [FAP0] GSO Buffer set to 0x00000000 [FAP1] GSO Buffer set to 0x00000000 [FAP0] FAP BPM Initialized. [FAP1] FAP BPM Initialized. [FAP0] FAP TM: ON [FAP1] FAP TM: ON Broadcom BCM63169D0 Ethernet Network Device v0.1 Sep 9 2017 04:51:18 Broadcom GMAC Char Driver v0.1 Sep 9 2017 04:51:27 Registered<249> wan physical port num: 3 Broadcom GMAC Driver v0.1 Sep 9 2017 04:51:27 Initialized fapDrv_psmAlloc: fapIdx=1, size: 4000, offset=b0a208a0 bytes remaining 7008 fapDrv_psmAlloc: fapIdx=0, size: 4000, offset=b08208a0 bytes remaining 7008 fapDrv_psmAlloc: wastage 8 bytes fapDrv_psmAlloc: fapIdx=0, size: 4808, offset=b0821840 bytes remaining 2192 fapDrv_psmAlloc: wastage 8 bytes fapDrv_psmAlloc: fapIdx=1, size: 4808, offset=b0a21840 bytes remaining 2192 eth0: MAC Address: 1E:14:48:37:55:9F eth1: MAC Address: 1E:14:48:37:55:9F eth2: MAC Address: 1E:14:48:37:55:9F eth3: MAC Address: 1E:14:48:37:55:9F eth4: MAC Address: 1C:14:48:37:55:91 wl: dsl_tx_pkt_flush_len=338 wl: high_wmark_tot=6195 wl: passivemode=0 wl: napimode=1 wl0: allocskbmode=1 currallocskbsz=3200 otp_read_pci: bad crc Neither SPROM nor OTP has valid image wl:srom/otp not programmed, using main memory mapped srom info(wombo board) wl:loading /etc/wlan/bcm6362_vars.bin Failed to open srom image from '/etc/wlan/bcm6362_vars.bin'. Reading vars image from kernel Error reading vars image from kernel wl: Read SROM/WLCAL from mfg info, sig:0x5372, rev 8, clk_offset 2142034800 - SUCCESS wl[435f]:Core revision: 16,D11CONF: 65c51800 wl0: MAC Address: 1c:14:48:37:55:90 wl0: Beacon padding set to 512 DSP Driver: DSP init stub Endpoint: endpoint_init entry Endpoint: endpoint_init COMPLETED arp_tables: (C) 2002 David S. Miller ip6_tables: (C) 2000-2006 Netfilter Core Team ADDRCONF(NETDEV_UP): eth0: link is not ready ADDRCONF(NETDEV_UP): eth1: link is not ready ADDRCONF(NETDEV_UP): eth2: link is not ready ADDRCONF(NETDEV_UP): eth3: link is not ready ADDRCONF(NETDEV_UP): eth4: link is not ready Broadcom Packet Flow Cache learning via BLOG enabled. FAP: set multicast reservation to 32 FAP: set max entries to 215 ADDRCONF(NETDEV_UP): eth0: link is not ready ADDRCONF(NETDEV_UP): eth1: link is not ready ADDRCONF(NETDEV_UP): eth2: link is not ready ADDRCONF(NETDEV_UP): eth3: link is not ready device eth0.16 entered promiscuous mode device eth0 entered promiscuous mode br1: port 1(eth0.16) entering forwarding state device eth1.16 entered promiscuous mode device eth1 entered promiscuous mode br1: port 2(eth1.16) entering forwarding state device eth2.16 entered promiscuous mode device eth2 entered promiscuous mode br1: port 3(eth2.16) entering forwarding state device eth3.16 entered promiscuous mode device eth3 entered promiscuous mode br1: port 4(eth3.16) entering forwarding state device wl0 entered promiscuous mode br1: port 5(wl0) entering forwarding state device wl0.1 entered promiscuous mode device wl0.2 entered promiscuous mode device wl0.3 entered promiscuous mode BOS: Enter bosInit BOS: Exit bosInit endpoint_open COMPLETED ***** bhiPrepareChannelTimeslotMap 1020 chanTsMap.chan = 0 ***** bhiPrepareChannelTimeslotMap 1021 chanTsMap.txTimeslot = 2 ***** bhiPrepareChannelTimeslotMap 1022 chanTsMap.rxTimeslot = 2 ***** bhiPrepareChannelTimeslotMap 1020 chanTsMap.chan = 1 ***** bhiPrepareChannelTimeslotMap 1021 chanTsMap.txTimeslot = 3 ***** bhiPrepareChannelTimeslotMap 1022 chanTsMap.rxTimeslot = 3 regStatus (reg 12 sel 0) = 0x10008401 regStatus (reg 12 sel 7) = 0x00000101 regCause (reg 13 sel 0) = 0x00000000 regCause (reg 13 sel 7) = 0x00808000 regCMT (reg 22 sel 0) = 0xe31e1406 regCMT (reg 22 sel 1) = 0x10008001 Secondary thread processor entry point at 0x81118d40after sendipc SLIC: Pass boardHalInit completed ******* DSP: Found BCM963268 ******* gDectRxOutOfSyncCounter = 0x811da8b0 gDectTxOutOfSyncCounter = 0x811da8b4 gApmAssertEnabled = 0x811ED7D0 gPcmAssertEnabled = 0x811ED7CC gInterruptCounter = 0x811DA8E0 gInterruptErrors = 0x811DA8E4 gPcmRxDescRing.next = 0x811ED80C gPcmTxDescRing.next = 0x811ED804 gApmRxDescRing.ext = 0x811ED7EC gApmTxDescRing.next = 0x811ED7DC gApmRxDescRing.ext = 0x811ED7F4 gApmTxDescRing.next = 0x811ED7E4 gDectTestMode = 0x811da884 32 ms ECAN tail-length halPcmInitDescriptors 144 nextTxDesc = 0xA0C01000 halPcmInitDescriptors 144 nextTxDesc = 0xA0C01008 halPcmInitDescriptors 148 Ownership for TX desc not set. Use this buffer. DSP: Interrupt Masks --------------- IrqMask = 0x00083000 IrqMask1 = 0x00000400 DSP: Interrupt Status ----------------- IrqStatus = 0x00000000 IrqStatus1 = 0x00000000 EndpointInit completed login:
Yaaaaaaaaay, we got console access! But wait, what’s the username and password? I’ve tried admin for the username and “/=%/?*&” (access code) for the password but no luck. Other default combinations were tried with no success. So what now? JTAG? NAND? I did try JTAG but also had no success. Reverse engineering a JTAG pinout is much more complicated. I was able to identify the grounds and VCC. Then I just assumed that the pads where organized in the standard EJTAG for the MIPS platform. I hooked these up onto a Bus Blaster with no luck.
The last method available was accessing the NAND. This requires desoldering the NAND from the board and plopping it into a programmer to read the contents.
The most logical thing to do once we have a raw dump is to binwalk it right? Yes, in most cases this will yield extracted OS files and kernels, especially when dealing with SPI EEPROMS. However this was not the case. Binwalk wasn’t able to extract JFFS2 filesystem automatically. This was because we took a raw dump of the NAND which included OOB data.
Usually the chip also has some extra storage per page to store ECC (error-correction code) and/or arbitrary “spare” data (also called OOB: out-of-band) which is not considered to be part of the useful storage of the chip but can still be read and written using low-level functions.
With this information, we’re basically going to strip the OOB area from the dump by passing it through Nand-dump-tool.py. We could have skipped this since I went back into the Xgpro programmer software and found an option to not include the “Spare Area In File”, but there wasn’t a way to prove this because at the moment the NAND had been resoldered back onto the board.
At this point I hit another brick wall which I couldn’t figure out why. It took me a while but binwalk still wasn’t extracting anything despite the good progress. I stumbled upon someone who converted their jffs2 dump from big endian to little endian. And guess what, It worked!!! Although binwalk still couldn’t extract the files, emulating it as a mtdblock worked just fine.
sudo jffs2dump -r -e 20000_converted.jffs2 -b 20000.jffs2 sudo mknod /dev/mtdblock0 b 31 0 sudo modprobe jffs2 sudo modprobe mtdram total_size=131072 erase_size=128 sudo modprobe mtdblock sudo dd if=20000_converted.jffs2 of=/dev/mtdblock0 sudo mkdir /mnt/jffs2 sudo mount -t jffs2 /dev/mtdblock0 /mnt/jffs2
One small problem. Where is the mfg data!?!? Isn’t 20000.jffs2 supposed to contain everything from rootfs, root_update, data, nvram, and the mfg partitions? Not exactly, thus I had to get crafty.
What basically did to get the mfg certificate was grab ARRIS_NVG595.bin (raw dump) > ARRIS_NVG595_split.bin (OOB data stripped) > ARRIS_NVG595_split_little_endian.bin (converted into little endian format) > Selected 0x5020000 – 0x5120000 (as mfg.bin) > binwalked it > mfg.dat
We’ve got our certificates! That is, those located in /etc/rootcert /etc/.ssl and /mfg/mfg.dat
I’m not going to dive into how one could use these certificates to mimic the device as many resources on that topic are readily available1,2. I’m not quite exactly done yet. I’ll continue to write about getting into a busybox shell with just console access since the only way others that don’t have the technical skills or tools will be left stranded in misery.
Upon playing with the bootloader, I discovered that they decided to lock us out. That means the possibility of dumping the NAND from console access is pretty much gone from the picture. I attempted to disrupt the normal boot process by depressing Ctrl-C, Ctrl-D, or Esc. Interestingly enough, the bootloader (cferam.00*, both 2 and 3 are the same file according to their sums) has references to Ctrl-C as an interrupter. This should have been enough to put anyone into a CFE command line. This really blows since CFE has many neat features, such as a tftp client, a micro_http server, and even a reverse tcp console!
Our next option is the console. We know that it’s password locked, we also know that the username is still admin but the password is no longer the device’s access code. We also know that by looking at /etc/inittab, we are automatically thrown into /bin/nsh. We have the binary, we could debug it but since critical daemons, such as sdb aren’t running, nsh just throws me into a shell. It would be nice if it did that on the device itself! But what about the password. What is nsh referring to as the password needed to get into nsh, not into an actual shell, but just nsh. i was left a bit confused. It seems as if the password was some sort of session number. However, I may be wrong. Take a look for yourself.